Pipeline orchestration

At 2:00 a.m., your retraining job starts. At 2:07, the feature query times out after writing half a file. The training script runs anyway because the cron job knows only that it is Tuesday. At 2:19, it publishes a model trained on incomplete data.

The dashboard says nothing. The job is just a shell command, and the shell command has already exited.

The last lesson left our Model CD pipeline straining against GitHub Actions: a branching, data-dependent sequence forced into a flat list of jobs glued with needs:. That works for a trigger. It becomes awkward when you need any of these:

• Fan-out over partitions
• A retry for only the failed step
• A backfill of last month’s data
• One place to answer “what happened to model version 48?”

An orchestrator runs a multi-step workflow, remembers each step’s state, and enforces the order between steps. In ML, it schedules these stages:

• Data preparation
• Validation
• Training
• Evaluation
• Promotion

It represents those steps as a DAG, a directed acyclic graph whose nodes are steps and whose arrows are dependencies.

That is the useful definition. It is not “a fancy cron.” Cron starts a command. An orchestrator manages a run.

The mental model: a graph with a memory

Suppose a daily fraud model follows this path:

raw transactions
       |
   validate
       |
  prepare features
       |
     train
       |
   evaluate
       |
  register model

The arrows are promises:

• validate cannot start until the raw snapshot exists.
• train cannot start until validation succeeds.
• register model cannot start until evaluation passes its quality gate.

A task is work the orchestrator can schedule and observe: a SQL query, container, Spark job, or Python program. A task should have clear inputs, outputs, and success criteria. The graph is acyclic because a run must move forward; “train waits for evaluate, which waits for train” is a deadlock.

The orchestrator records states such as:

• Queued
• Running
• Succeeded
• Failed
• Skipped
• Cancelled

That lets it answer:

• Did validation fail, or was it never scheduled?
• Which attempt produced this feature table?
• Which partitions need to be rerun?
• Did the model fail the metric gate, or did the registry reject the upload?

A script can log these facts. An orchestrator makes them queryable rather than scattered through text files.

One pipeline, with numbers

A retailer predicts chargebacks using the previous 30 days of labelled transactions. A normal nightly run looks like this:

If tasks run sequentially, the critical path is 39 minutes. The critical path—the longest duration-weighted dependency path—sets the minimum wall-clock time when tasks can run in parallel. Total work still determines compute cost and capacity.

If Extract fails because object storage returns a transient network error, a useful orchestrator retries it after, say, 30 seconds and then after 2 minutes. Validate waits for successful extraction; later tasks do not repeat because they never started.

If Evaluate fails because recall is below the threshold, retrying usually produces the same answer. The run should stop and alert someone. A retry is for likely-transient failure, not for arguing with arithmetic.

The same graph handles a backfill. If a label query was wrong for 1–7 August, the system processes seven historical intervals while limiting concurrency so the warehouse is not overwhelmed. A backfill processes historical intervals or partitions; tools differ in whether they create one run per interval or materialize selected partitions.

The workflow is no longer “run six scripts at 2:00.” It is “for each data interval, produce these versioned outputs under these conditions.”

What happens under the hood

An orchestrator normally has four conceptual parts:

• The scheduler decides when a run is due and which tasks are eligible from their dependencies.
• The executor decides where eligible work runs: a local process, worker, container, Kubernetes pod, or remote batch system.
• The metadata store records runs, task states, timestamps, parameters, and output references.
• The artifact store holds durable outputs such as Parquet files, metrics JSON, model binaries, and validation reports.

The metadata store should hold pointers, not a 4 GB DataFrame. A dependency usually means “the next task may read this output,” not “copy the object through the orchestrator.” For example:

s3://training/features/date=2026-08-27/run=8f31/part-000.parquet

The next task receives that URI and the run configuration; the bytes stay in durable storage.

Retries are only safe when tasks are repeatable

A retry reruns a failed task. It does not rewind the outside world. If Register uploads a model and loses its connection before receiving the response, the upload may have succeeded. A blind retry can create a duplicate version.

A task is idempotent when repeating it with the same logical input produces the same intended result rather than another side effect. Use safeguards such as:

• Deterministic paths
• Atomic completion markers
• Overwrite semantics for known partitions
• A stable idempotency key that the registry checks before creating a version

Retry transient operations. Design their outputs so a second attempt is safe.

The orchestrator can retry a container; it cannot make a database insert, notification, payment, or promotion reversible.

Resume means task-level recovery

Resume usually reuses successful outputs within the same run. Reusing an earlier run’s output requires versioned caching keyed by the logical interval, input versions, code, configuration, and feature definition. Otherwise a cache can silently train on stale data.

Resume does not restore a Python process’s memory. If Train dies while saving a 14-minute model, it starts again unless the training code has its own checkpointing. Checkpointing matters inside very long tasks. Useful task boundaries correspond to durable artifacts: a validated snapshot, feature partition, model, or metrics report.

Time is more subtle than a clock

A daily schedule has two times:

• Run time is when workers start.
• Data interval is the period the run is responsible for.

A run starting at 02:00 on 28 August may own 27 August’s data. If code uses the wall clock instead of the declared interval, a retry at 04:00 can read different rows. Use explicit intervals, preferably in UTC unless the business rule follows a local timezone, and define how late-arriving labels are handled.

Evaluate should produce metrics and a pass/fail decision. Register must depend on that decision, not merely on the existence of a model file.

The production pattern

A reliable ML workflow usually:

1. Triggers on a schedule, new partition, or manual run.
2. Resolves immutable code, configuration, data, and feature inputs.
3. Validates before expensive work.
4. Writes durable outputs at meaningful boundaries.
5. Separates training from evaluation and promotion.
6. Applies explicit gates for metrics, bias, schema, and artifact completeness.
7. Alerts on the run’s business meaning, not only process exit codes.

Keep experiment metadata in MLflow or an equivalent tracker. Use a feature store when the serving problem justifies one. The orchestrator coordinates these systems; it should not become all of them.

Choosing the tool

The practical differences are what each tool treats as its main object, where work runs, and how much machinery your team will operate.

Airflow is task-centric and ecosystem-heavy. Dagster is asset-aware: its graph describes durable products such as fraud_features and approved_model. Prefect is comfortable with runtime decisions and dynamic Python workflows. Kubeflow Pipelines (KFP) makes container environments, resource requests, isolation, and artifact metadata central. KFP is only one component of the broader Kubeflow platform.

Failure modes you will actually see

• A retry duplicates an output. A task failed after an external write succeeded. Use deterministic paths, atomic writes, overwrite semantics, or idempotency keys. Test the “failure after side effect” case.
• The graph is green but the model is bad. Process success is not data or model quality. Add validation and a separate evaluation gate; a rejected model should be visibly rejected, not merely logged.
• A backfill overloads the warehouse. Historical runs multiply work. Limit concurrency per workflow and downstream system, use partition-aware queries, and choose whether backfills need a separate pool.
• A task passes a huge object through the control plane. Write DataFrames and models to durable storage; pass a URI, checksum, schema, and run ID.
• The scheduler is healthy but nothing runs. Workers may be unavailable, concurrency exhausted, pods unschedulable, or a downstream pool full. Monitor queue age and worker and downstream capacity separately.

When not to use an orchestrator

An orchestrator adds several operational responsibilities:

• A control plane
• A metadata database
• Workers
• Deployment rules
• Upgrades
• Alerts

That is worthwhile when dependencies, retries, backfills, and visibility save more time than they cost.

It is not automatically worthwhile for one five-minute nightly script that writes one artifact and has no dependent steps. A managed batch job, CI schedule, or cron with good logging may be more honest. Do not use an orchestrator as a low-latency serving layer; serving is the next boundary, where FastAPI model serving loads the artifact and answers requests.

A question to carry forward

The tooling chapter has built a production path: experiments are tracked, data and models are versioned, tests run in CI, artifacts are packaged, and an orchestrator can retrain the pipeline on schedule.

Then the model leaves the registry.

A registry entry answers no prediction request. It earns nothing until a process loads the artifact and responds to “what is the risk for this transaction?” under real traffic, with a latency target and failure policy. The next chapter crosses that boundary: how to put a trained model behind an API and serve it with FastAPI.