Creating a Pipeline
Build a pipeline of actions that copy, transform, delete, script, export or import data.
A pipeline is a saved list of actions that you can run on demand as many times as you need. Each action does one thing to your data, such as copying records from one environment to another. You build a pipeline in the pipeline builder, one action at a time.
Open the builder
- Open Pipelines in the left menu.
- Select Create Pipeline. The Dashboard also offers New Copy Pipeline and Open Builder.
The builder opens with an empty pipeline. The left side lists the pipeline's actions, and the top bar has Save and Run.
Add an action
Select Add Action and choose what the action does with your data:
| Action | What it does |
|---|---|
| Copy | Moves data from a source environment to a target environment |
| Delete | Removes data from an environment permanently |
| Transform | Transforms sensitive data in place, for example by anonymising or scrambling it |
| Script | Runs your own script against an environment |
| Export | Exports data from an environment to a file or a database you own |
| Import | Loads a file you hold back into an environment |
A pipeline can hold several actions. Select Add Action again to add the next one.
The source and target of a Copy must be on the same platform. Today that means both are Mendix environments.
Build a Copy action
A Copy action has these sections, shown in the left tree under the action: Environments, Data Selection, Filters, Fields, Options and Summary. The builder walks you through them in turn.
Environments
Choose the Source Environment and the Target Environment, then choose a Write Strategy:
- Insert: all records are inserted as new.
- Merge
- Clean
Data Selection
The builder lists the entities of the source environment, grouped by module, and shows how many you have selected.
- Use the search box to find an entity.
- Use Select all for everything, or Select all on a module for just that module's entities.
- Use Selected only to show only what you have chosen.
- Turn on Auto-include references to include the entities your selection refers to.
- Use Refresh to read the list from the environment again.
Filters
Filters are optional. Without a filter, every record of the entity is included. To narrow an entity, enter an XPath filter on its tab and use Preview Results to see a sample of the records it selects. Use Skip Filters to continue without any.
Fields
For each field of each selected entity you choose what happens to its value. A field starts as Pass-through, which copies it unchanged. Skip this step if you want an exact copy.
| Transformation | What it does |
|---|---|
| Scramble | Generates replacement values from patterns of the source data (word, separator and length models) |
| Static Value | Replaces every value in the field with one constant you specify |
| Anonymize | Generates type-aware anonymised tokens |
| Hide | Keeps the field but sets every value to an empty string |
| Regexp | Generates random values that match a pattern you configure |
| Ignore | Leaves the field out of the transfer entirely |
See the transformations reference for details of each type.
Save and run
Select Save to keep the pipeline. It then appears in the Pipelines list with its status and number of actions, and Run starts it. To change a pipeline later, select Open on it in the list.