Golden paths for ML

The three canonical ML golden paths

Path 1 — Batch inference pipeline

Trigger: an ML team has a trained model that produces predictions on a schedule — nightly fraud scores, weekly recommendations, monthly risk ratings — rather than in real time.

Path 2 — Model serving: real-time inference

Trigger: an ML team has a model that must produce predictions at request time — fraud detection on a payment, ranking on a search query, content moderation on a submitted post.

Path 3 — GenAI feature with a vector index

Trigger: an application team wants to add retrieval-augmented generation (RAG)to a product — a search surface, a Q&A interface, a document assistant.

The templating mechanism

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: batch-model-pipeline
  title: Batch Model Pipeline
  description: Golden path for scheduling batch inference jobs
spec:
  owner: platform-team
  type: ml-pipeline

  parameters:
    - title: Project details
      required: [modelName, teamSlug, outputStorePrefix]
      properties:
        modelName:
          type: string
          description: Name of the model (must match registry slug)
        teamSlug:
          type: string
          description: Your team identifier for RBAC and labelling
        outputStorePrefix:
          type: string
          description: Object-store prefix for batch output (e.g. s3://data/predictions/)

  steps:
    - id: fetch-skeleton
      name: Fetch pipeline skeleton
      action: fetch:template
      input:
        url: ./skeleton
        values:
          modelName: ${{ parameters.modelName }}
          teamSlug: ${{ parameters.teamSlug }}
          outputStorePrefix: ${{ parameters.outputStorePrefix }}

    - id: publish
      name: Create Git repository
      action: publish:github
      input:
        repoUrl: github.com?owner=${{ parameters.teamSlug }}&repo=${{ parameters.modelName }}-pipeline

    - id: register
      name: Register in catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: batch-model-pipeline
  namespace: argocd
spec:
  generators:
    - clusters: {}   # one Application per registered cluster Secret
  template:
    metadata:
      name: '{{name}}-batch-model'
    spec:
      project: ml-workloads
      source:
        repoURL: https://git.example.com/platform/batch-model-base
        targetRevision: HEAD
        path: overlays/{{metadata.labels.env}}
      destination:
        server: '{{server}}'
        namespace: ml-inference
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The choice of mechanism — portal scaffold, ApplicationSet, or Kustomize overlay— depends on what the platform already operates. The key discipline is the same in all cases: the template is the canonical source of truth for the path. A team that modifies the generated skeleton is drifting from the path; the platform team’s tooling detects that drift via GitOps sync-status checks and surfaces it to both teams.

Governance gates wired into the path

A golden path is valuable because it enforces good defaults automatically. The gates that matter most for ML workloads sit at three points:

1. 1Registry promotion gate. Before a model artefact moves from staging to production in the model registry, it must pass automated checks: minimum eval score, model card completeness, and (for regulated industries) an explicit reviewer signoff. The model registry's webhook or event integration triggers the CI gate; the gate's pass/fail result is written back to the registry as a metadata annotation. This makes the gate auditable — any downstream system can query whether a given model version passed all gates.
2. 2Deployment PR gate. When an ML engineer opens a PR against the deployment repository, the CI pipeline runs the model-card check, eval-score threshold, and latency regression test (Path 2) or recall-on-ground-truth check (Path 3). This gate runs in the CI system — not in the cluster — so it fails fast, before the GitOps controller ever sees the manifest.
3. 3Runtime rollout gate. After deployment, the progressive-delivery controller observes live metrics. For serving models (Path 2), this means request latency and error rate from the serving layer's metrics endpoint, plus any model-quality signal the application emits. For batch models (Path 1), this means the output-validation step in the pipeline itself. The rollout gate is the safety net for the cases the CI gate did not catch — distribution shift detected only under real traffic, latency regression that appears only at production request volumes.

The deprecation contract

• Announce with a defined notice window. Consumers of the path get a notice period — typically measured in weeks to months — before the old path is removed. No standard mandates a specific number; the appropriate window depends on the consumer's release cadence and the complexity of the migration.
• Provide a migration script or automated PR. The platform team does not announce a deprecation and leave consumers to figure out the migration themselves. The scaffolding system opens automated PRs against consumer repositories — replacing old template references with the new version, updating dependency pins, adjusting CI configuration. Backstage's Software Templates and the scaffolder action system support this pattern natively.
• Track adoption. The platform team maintains an inventory of which repositories are on which path version — sourced from the IDP catalog or from Git metadata. Deprecation is not complete until every consumer has migrated or has been deliberately granted an extension.
• Remove on schedule. The old path version is removed at the end of the notice window. Exceptions are tracked explicitly and have an expiry date. An exception that has no expiry date is a permanent fork — the condition that the deprecation contract exists to prevent.

The off-ramp and when to take it

• Path 1 off-ramps include multi-node distributed training jobs, non-standard output destinations, and pipeline dependencies on systems the platform does not yet integrate with.
• Path 2 off-ramps include streaming inference (event-triggered prediction), multi-model ensembles, and custom pre/post-processing pipelines that do not fit the serving runtime's transformer abstraction.
• Path 3 off-ramps include hybrid search (keyword plus semantic), custom re-ranking pipelines, and multi-turn agent loops with tool use — which extend beyond simple RAG into agentic infrastructure.

Connecting the three paths to the broader platform