Running & Monitoring Pipelines
Execute pipelines and track progress in real time.
Once a pipeline is created, you can execute it on demand. Each execution is tracked in real time with detailed progress information, and a full history of past runs is available for review.
Starting an Execution
There are two ways to start a pipeline execution:
- From the Pipelines list — click the Run button on the pipeline card.
- From the Pipeline detail page — click the Run Pipeline button at the top of the page.
After clicking Run, the Execution Monitor opens automatically and the pipeline begins processing.
Only one execution can run at a time per pipeline. If a pipeline is already running, the Run button is disabled until the current execution completes or is cancelled.
Execution Monitor
The Execution Monitor provides real-time visibility into a running pipeline:
- Overall status — the current state of the execution (Queued, In Progress, Completed, Failed, or Cancelled)
- Progress bars — a progress bar for each entity showing the percentage of records processed
- Record counts — the number of records processed out of the total for each entity
- Duration — elapsed time since the execution started
- Throughput — records processed per second (updated in real time)
- Error log — any errors or warnings encountered during processing
You can navigate away from the Execution Monitor without affecting the running pipeline. The execution continues in the background, and you can return to check progress at any time from the Pipelines page.
Execution Statuses
Each execution goes through one or more of the following statuses:
| Status | Indicator | Description |
|---|---|---|
| Queued | Grey | The execution has been submitted and is waiting to start. This is typically very brief. |
| In Progress | Blue (animated) | The pipeline is actively processing records. Progress bars and counts update in real time. |
| Completed | Green | All records have been successfully processed. The execution summary is available. |
| Failed | Red | The execution encountered a fatal error and could not continue. Check the error log for details. |
| Cancelled | Orange | The execution was manually cancelled by a user before completion. |
Cancelling an Execution
You can cancel a running execution at any time:
- Open the Execution Monitor for the running pipeline.
- Click the Cancel button.
- Confirm the cancellation in the dialog.
Cancelling an execution stops processing immediately. Records that have already been written to the target environment are not rolled back. The target may contain a partial dataset.
Viewing Run History
Every pipeline maintains a complete history of all past executions:
- Navigate to the Pipelines page and click on a pipeline.
- Scroll to the Run History section.
- Each entry shows: execution date, status, duration, total records processed, and who triggered the run.
- Click on any entry to view the full execution details, including per-entity breakdowns and error logs.
Run history is retained indefinitely and can be filtered by status or date range.
Troubleshooting
Execution fails immediately
If a pipeline execution transitions from Queued to Failed without processing any records:
- Check that both source and target environments are in Healthy status.
- Verify the environments have not been redeployed without the agent module.
- Review the error log in the Execution Monitor for the specific error message.
- Ensure the organisation has not exceeded its monthly record limit.
Execution is slow
Pipeline speed depends on several factors:
- Record count — larger datasets take longer to process.
- Entity complexity — entities with many fields or associations require more processing time.
- Network latency — environments with high network latency between the agent and DMM Infinity will be slower.
- Transformations — applying transformations (especially Regexp and Anonymize) adds processing overhead per record.
For large datasets, consider splitting the work into multiple pipelines with smaller entity sets. This allows you to run them sequentially and monitor progress more granularly.
Records missing in target
If the target environment has fewer records than expected after a Completed execution:
- Check the execution summary for any skipped records — records that fail validation in the target are skipped.
- Verify that the target environment did not have pre-existing records that were updated rather than created.
- Review the error log for warnings about individual record failures.
- Ensure entity dependencies are included — if a referenced entity is missing, dependent records may be skipped.