Creating a Pipeline

Build a pipeline of actions that copy, transform, delete, script, export or import data.

Last updated:

A pipeline is a saved list of actions that you can run on demand as many times as you need. Each action does one thing to your data, such as copying records from one environment to another. You build a pipeline in the pipeline builder, one action at a time.

Open the builder

  1. Open Pipelines in the left menu.
  2. Select Create Pipeline. The Dashboard also offers New Copy Pipeline and Open Builder.

The builder opens with an empty pipeline. The left side lists the pipeline's actions, and the top bar has Save and Run.

Add an action

Select Add Action and choose what the action does with your data:

ActionWhat it does
CopyMoves data from a source environment to a target environment
DeleteRemoves data from an environment permanently
TransformTransforms sensitive data in place, for example by anonymising or scrambling it
ScriptRuns your own script against an environment
ExportExports data from an environment to a file or a database you own
ImportLoads a file you hold back into an environment

A pipeline can hold several actions. Select Add Action again to add the next one.

ℹ️Note

The source and target of a Copy must be on the same platform. Today that means both are Mendix environments.

Build a Copy action

A Copy action has these sections, shown in the left tree under the action: Environments, Data Selection, Filters, Fields, Options and Summary. The builder walks you through them in turn.

Environments

Choose the Source Environment and the Target Environment, then choose a Write Strategy:

  • Insert: all records are inserted as new.
  • Merge
  • Clean

Data Selection

The builder lists the entities of the source environment, grouped by module, and shows how many you have selected.

  • Use the search box to find an entity.
  • Use Select all for everything, or Select all on a module for just that module's entities.
  • Use Selected only to show only what you have chosen.
  • Turn on Auto-include references to include the entities your selection refers to.
  • Use Refresh to read the list from the environment again.

Filters

Filters are optional. Without a filter, every record of the entity is included. To narrow an entity, enter an XPath filter on its tab and use Preview Results to see a sample of the records it selects. Use Skip Filters to continue without any.

Fields

For each field of each selected entity you choose what happens to its value. A field starts as Pass-through, which copies it unchanged. Skip this step if you want an exact copy.

TransformationWhat it does
ScrambleGenerates replacement values from patterns of the source data (word, separator and length models)
Static ValueReplaces every value in the field with one constant you specify
AnonymizeGenerates type-aware anonymised tokens
HideKeeps the field but sets every value to an empty string
RegexpGenerates random values that match a pattern you configure
IgnoreLeaves the field out of the transfer entirely

See the transformations reference for details of each type.

Save and run

Select Save to keep the pipeline. It then appears in the Pipelines list with its status and number of actions, and Run starts it. To change a pipeline later, select Open on it in the list.