The ML lifecycle, end to end, in production

Input: a business objective stated in natural language. Output: a measurable ML problem with a defined target metric, a baseline (typically a simple heuristic or the current rule-based system), and an explicit decision on whether ML is warranted at all.

Failure mode: skipping the baseline. A team trains a sophisticated model, benchmarks it against itself, and ships it — never establishing whether a rules-based system or a simple regression would have served equally well at a fraction of the ongoing operational cost. Without a baseline, you cannot know whether the model is adding value or merely adding complexity.

Input: raw data sources. Output: versioned, validated train/validation/test splits with a documented schema and transformation logic.

Failure mode: training-serving skew. The feature transformation applied at training time is not identical to the transformation applied at inference time. This is among the most insidious failure modes because it produces a model that evaluates well offline and underperforms silently in production — the performance delta is invisible to any test that runs on the training distribution. Sculley et al. name this class of problem explicitly as a source of ML-specific technical debt arising from data dependencies.

Input: versioned dataset, experiment configuration. Output: a trained model artefact with tracked metadata — hyperparameters, evaluation metrics, dataset version, and a pointer back to the experiment run.

Failure mode: experiment debt. Hundreds of runs tracked inconsistently — or not tracked at all — make it impossible to reproduce the model that scored best or to understand what changed between versions. The fix is treating the experiment tracker as a first-class system of record from day one, not retrofitting it after the team has accumulated entropy across six months of ad-hoc notebooks.

For distributed training on Kubernetes, the platform substrate — gang scheduling, distributed training operators, GPU quota enforcement — becomes relevant here. The training stage is where workload shape (single-node vs multi-node, GPU-bound vs CPU-bound) most directly constrains platform design. Those infrastructure choices are covered in Part 3 of this series.

Input: trained model artefact, held-out test set. Output: a signed-off evaluation report covering aggregate metrics, slice analysis, fairness checks, and adversarial probing — plus a model card documenting the results. Breck et al.'s ML Test Score [3] provides 28 specific tests across four categories — data tests, model tests, ML infrastructure tests, and monitoring tests — as a structured rubric for what a production-ready evaluation suite must cover.

Failure mode: aggregate metric tunnelling. A team optimises a single headline metric (accuracy, AUC, F1) and never examines slices. A model that achieves 92% overall accuracy while performing at 61% on a minority demographic slice will pass every automated gate and fail every ethical review. Slice analysis is not optional for systems whose outputs affect people.

Input: evaluated model artefact with attached metadata. Output: a versioned, registered model with a defined lifecycle state (experimental → staging → production → archived) and a promotion gate that must be passed before a model reaches production.

Failure mode: model-of-record drift. Production is running a model that cannot be identified in the registry, whose training run metadata has been lost, and whose training data version is unknown. This is the most dangerous silent failure in the lifecycle — it means you cannot answer the four questions a regulator or incident responder will ask: what model is serving, where did it come from, who approved it, and is the served artefact the artefact that was evaluated?

The registry also serves as the GitOps trigger: when a model transitions to the Production state, an automated handoff writes an updated serving manifest to the GitOps repository. This seam — registry promotion to infrastructure reconciliation — is the most under-documented in the standard lifecycle. Part 4 of this series covers registry patterns, lifecycle states, and the curation-policy-as-code pattern in depth.

Input: a promoted model artefact from the registry, a serving configuration. Output: a containerised inference endpoint with defined SLAs, a deployment strategy (canary, blue-green, or rolling), and a rollback path.

Failure mode: shadow debt. A model is deployed manually — via a direct kubectl command or a one-off script — and exists outside any GitOps loop. The next release has no safe rollback path because the baseline state was never declared as code. Shadow deployments accumulate silently: engineers move on, the original deployer forgets, and the model is effectively orphaned with no known owner and no documented rollback procedure.

Input: live prediction requests and outcomes, ground-truth labels (where available), and system telemetry. Output: drift alerts, performance degradation signals, and — critically — a retraining trigger. Gama et al.'s comprehensive survey of concept drift adaptation [4] distinguishes three drift types the monitoring layer must handle separately: covariate shift (input distribution changes, relationship holds), concept drift (relationship between input and target changes), and label drift (target label distribution shifts). Each requires a different detection strategy and a different remediation response.

Failure mode: alert-only monitoring without an actionable response. Alerts fire, no one owns the on-call rotation for ML quality, the alert is silenced, and the model continues degrading. A monitoring layer without a defined owner, an escalation path, and a retraining trigger is logging theatre — it generates the appearance of observability without the operational capability to act on it.

A critical constraint in this stage is ground truth lag: the delay between prediction and true label arrival. For fraud detection it may be hours; for long-horizon forecasting it may be months. The monitoring strategy must account for this lag or it will fire on statistical noise rather than genuine degradation.

Input: a retraining trigger (scheduled, drift-triggered, or manual), refreshed data. Output: a new candidate model that has passed the same evaluation suite as the original and been promoted through the registry.

Failure mode: pipeline rot. The retraining pipeline was written during the original project, never maintained as a production service, and fails silently when triggered months later because a dependency has changed, a data source has moved, or the infrastructure configuration has drifted from the environment the pipeline was written for. The retraining pipeline must be treated as a production service — with tests, versioning, and on-call ownership — not as a script that worked once.

The retraining pipeline should be the same artefact as the training pipeline — not a parallel script. If retraining requires a separate code path, that path will diverge from the original and the divergence will be discovered at the worst possible moment: when a production model needs to be replaced urgently. The retirement path — routing a model to end-of-life— is also managed at this stage via the registry's archived state.