Designing an internal ML platform

Designing an internal ML platform

At 09:00, a fraud team has a promising model. Its offline AUC is 0.91. By 17:00, the team has deployed something that calls the wrong feature name, loads a different library version, and has no reliable way back to yesterday’s model.

At 03:00, fraud catches the discrepancy first. Approval rates fall. The endpoint is technically healthy: requests return HTTP 200, latency is fine, and the dashboard is green. The model has simply been fed amount where training expected amount_cents.

Nobody made one spectacular mistake. The pieces failed to compose.

An internal ML platform is the set of shared tools, workflows, and agreements that makes those pieces compose. It is not necessarily Kubernetes or a glossy portal. The useful question is:

Can a team take an idea, produce a reproducible model, put it behind a safe interface, observe it, and reverse the change without asking six other teams for permission?

That question leads to two decisions: what must exist, and what deserves to exist now.

The mental model: five planes, joined by contracts

A plane groups capabilities with one responsibility. A contract is an explicit agreement between planes about data shape, meaning, ownership, timing, and failure behavior.

These are not necessarily five services or teams. A small company may implement all five with a Git repository, CI job, model registry, service, and dashboard.

The data plane

The data plane turns raw events into data that training and inference can both understand. Its contract defines types, semantics, ownership, and timing—not merely the existence of a column called amount_cents.

A versioned, machine-readable schema and producer/consumer contract tests should fail CI when a field is removed, its type changes, or its timestamp semantics change. Ingestion adds semantic checks such as rejecting negative amounts, validating country codes, and comparing distributions with a known baseline.

The contract also defines violations. Malformed records are rejected; late records are quarantined and retained for backfill. If freshness falls below its target, the affected dataset is blocked and the owner is alerted. A later backfill creates a new dataset version rather than mutating the old one.

Online freshness needs a policy too. A stale feature must be rejected, handled by a defined fallback, or returned as an explicit degraded result. The data plane emits a versioned dataset or feature view; training records that identifier. Serving telemetry records the feature-view version, retrieval timestamps, and—where privacy permits—the actual input values, or a reproducible, point-in-time feature snapshot or reference. This enables an investigator to reconstruct the model input and answer what the model knew for a prediction.

The training plane

The training plane turns pinned inputs into a model artifact: weights, preprocessing, metadata, and often a serving image. A run pins:

• the dataset or feature snapshot;
• the source revision;
• the environment, using a lockfile and immutable image digest.

A lockfile records exact dependency versions. It is evidence, not enforcement: training and serving might install different sets unless both consume the same locked dependencies. The stronger pattern builds and tests one serving image, then promotes that exact image by its full digest. The digest identifies tested bytes; the lockfile explains how they were built.

Training also emits metrics, tests, lineage, and input/output schemas. A successful process exit does not prove that a useful model exists.

The registry and governance plane

The registry is the system of record for immutable model versions and their evidence. An entry might contain the model name and version, dataset, source commit, image digest, metrics, owner, approval, and deployment status.

Governance automatically enforces checks such as passed schema tests, an assigned owner, an approved evaluation dataset, and a known rollback target. Humans handle exceptions rather than copying version strings between tickets. Deployments reference a version or digest, never a mutable latest tag.

The serving plane

The serving plane turns an approved artifact into online, batch, or streaming predictions. Its contract defines request and response schemas, authentication, error behavior, resource limits, and SLOs.

An SLO, or service-level objective, is a target such as “99.9 percent of requests succeed” or “p95 latency stays under 200 milliseconds.” p95 latency means 95 percent of requests finish within that time.

The contract must specify failure behavior, not just “safe behavior.” For example, after 500 milliseconds route to versioned rules model fraud-rules-v7; if that is unavailable, use manual review; if the queue also fails, return decision=decline, reason_code=fallback_unavailable, and alert on-call. Normal and fallback responses both identify their version or decision source.

This is an explicit trade-off. Failing open preserves checkout availability but permits more fraud. Failing closed limits fraud but can decline legitimate customers. A rules-first, manual-review-second policy spends some review capacity before declining everyone. Version the fallback and exercise it in failure drills.

The observability plane

Observability covers infrastructure, logs, traces, data quality, prediction distributions, drift, and delayed outcomes. Its telemetry contract should include a request ID, model version, feature timestamps, prediction, latency, status, and eventual outcome. Without outcomes, score changes are visible but model improvement is not directly measurable.

Do not automatically retrain because a histogram moved: promotions, holidays, bugs, and attacks can all cause drift. Diagnose before acting.

A worked path: the checkout fraud model

Suppose payments handles 5 million transactions per day. At a 0.4 percent fraud rate, a seven-day window contains about 35 million rows and 140,000 positive examples.

The data team publishes fraud-v42 with these rules:

• amount_cents is an integer;
• country is a two-letter code;
• account_age_days is non-negative;
• events older than 15 minutes are marked late;
• missing merchant_id makes a row unusable.

A machine-readable schema checks fields, and a fixture representing a $12.99 payment as 1299 catches a silent unit change. Malformed rows are rejected; late or suspicious rows are quarantined. If fewer than 99 percent of events meet the ten-minute freshness target, publication is blocked and the owner is alerted. The original dataset remains unchanged for reproducibility.

Training uses that snapshot, source revision 8f31c2a, and a lockfile. It produces model 2026.08.28.3. On a fixed evaluation set, precision at the review threshold is 71 percent and recall is 48 percent. Those numbers are evidence to compare with the baseline and the costs of false declines, not universal definitions of success.

The registry permits promotion only when data tests pass, the model beats the agreed baseline, an owner is named, and the serving image has a digest. The endpoint returns model_version=2026.08.28.3; fallback responses identify their source, version, and reason.

A platform gate can be this plain:

from dataclasses import dataclass


@dataclass(frozen=True)
class Release:
    name: str
    version: str
    dataset: str
    image_digest: str
    p95_ms: int
    approved: bool


release = Release(
    name="checkout-fraud",
    version="2026.08.28.3",
    dataset="fraud-v42",
    image_digest="sha256:" + "a" * 64,
    p95_ms=180,
    approved=True,
)

checks = {
    "dataset pinned": release.dataset.startswith("fraud-v"),
    "image pinned": (
        release.image_digest.startswith("sha256:")
        and len(release.image_digest) == 71
    ),
    "latency target met": release.p95_ms <= 200,
    "approval recorded": release.approved,
}

failed = [name for name, passed in checks.items() if not passed]

if failed:
    raise SystemExit("blocked: " + ", ".join(failed))

print(f"approved {release.name} {release.version}; p95={release.p95_ms} ms")

It prints:

approved checkout-fraud 2026.08.28.3; p95=180 ms

A real gate fetches evidence from the registry and test systems rather than trusting values typed into a file. The important boundary is that training hands serving a named, tested artifact—not an anonymous file. This is the ML lifecycle: every transition needs an artifact, owner, and evidence.

The golden path: easy by default, escapable by design

A golden path is one supported route for the common case. A repository template can provide CI, a lockfile, a training entry point, registry integration, a standard deployment, and a starter dashboard.

The safe path should be the easiest path: reproducible builds, immutable versions, input validation, a rollback target, and basic latency and prediction metrics should arrive without two weeks of plumbing.

The path is not a cage. Custom CUDA, batch-only, streaming, or alternative serving needs are valid escape hatches. They may change implementation but must still publish the same contracts: versioned artifact, schemas, ownership, health, and telemetry.

Build or buy: use a decision matrix

Build-versus-buy is a choice about which risks and responsibilities to own, not a test of cleverness. Differentiation matters, but so do compliance, integration, cost, reliability, security, and exit options.

Question	Favors building or owning	Favors buying or composing
Uniqueness and integration	Proprietary workflows or awkward internal integrations	Generic capability with standard interfaces
Control and compliance	Unusual residency, audit, retention, or isolation needs	Provider meets required controls
Cost and reliability	Scale or specialization makes ownership cheaper or necessary	Provider has stronger operational maturity
Security	Direct control of keys, isolation, or threat model is required	Provider’s security model is sufficient
Exit options	Critical logic stays portable behind owned interfaces	Data export and switching costs are acceptable

Weight the questions before choosing. Include migration and exit work in total cost; buying moves the design boundary but does not remove design work.

Capability	Sensible default	Own or build when	Buy or compose when
Registry and promotion	Managed or established open source	Approval or audit integration is specialized	Ordinary versions and promotion are enough
GPU scheduling	Cloud or existing cluster tooling	Scheduling is itself a major capability	It merely runs training
Feature computation	Tables or views first	Low-latency feature behavior is a product advantage	Shared online features are generic
Ranking or fraud logic	Build the domain layer	Rules and features are your advantage	A hosted service meets the SLO
Training orchestration	Compose existing workflows	Scheduling is genuinely specialized	You need ordinary retries and visibility

A versioned table may be clearer and cheaper than a feature store for one team running daily batch predictions. A feature store becomes more reasonable when many teams need reused features with online freshness.

Sequence by repeated pain

A three-person team with one online model usually needs:

1. CI that tests changes and builds the serving artifact.
2. An immutable registry with versions, evidence, and owners.
3. One locked dependency set used to build a tested serving image, promoted by digest.
4. One dashboard for errors, latency, model version, freshness, and the key business outcome.

This is enough for a small number of jobs or one service. It is not a promise of arbitrary self-service, and a small team should not operate a private cloud before repeated pain justifies it.

Around thirty ML engineers, several product teams, or many release cadences, add machinery where failures repeat:

Scale and pain	Add	Why it earns its cost
Several teams sharing features	Feature store or governed feature layer	Prevents duplicate definitions and skew
Competing training jobs	Multi-tenant quotas and isolation	Prevents resource and data contention
Many services and releases	Self-serve endpoints with standard SLOs	Removes routine deployment tickets
Expensive or bursty training	Cost labels, budgets, idle-resource controls	Makes usage attributable

Headcount is not the threshold: 40 models may justify a feature layer earlier than 30 engineers with two models. Add the next capability when the same failure has a known shape. Kubeflow Pipelines can help with inspectable workflows and tenancy, but brings Kubernetes operating cost.

Platform anti-patterns

The framework nobody adopts. A mandatory SDK and slow review process lead engineers to copy generated files or deploy around it. Prefer a small template and observable contracts; measure production adoption, not SDK downloads.

The abstraction that leaks. A “universal training job” hides infrastructure while exposing twenty configuration fields. Narrow the abstraction, document the escape hatch, and preserve lineage and policy checks.

The platform team that becomes a ticket queue. Self-service defaults, a service catalogue, and a platform SLO should handle routine work. Reserve review for risky changes.

The green dashboard with a rotten model. Infrastructure can be healthy while approval rates fall. Log model version and freshness, monitor distributions, and join delayed outcomes to predictions. ML observability and drift monitoring cannot invent missing labels.

The rollback that is only theoretical. Retain the last known-good artifact, record model and image digests, and exercise rollback in staging. Rollback should be a normal deployment action.

Measure the platform, not its machinery

Two measures reveal whether the platform reduces friction and risk:

Time from idea to first production prediction. Start with a recorded experiment or ticket and end with the first monitored real prediction. Track median and p90, then break the time into data access, training, review, deployment, and verification.

Time to rollback. Measure from the decision to revert until the known-good version serves traffic, including artifact discovery and health verification. A target such as six minutes is useful if a 45-minute rollback is exposing customers.

Suppose p90 idea-to-production falls from 18 days to 3 days and rollback from 45 minutes to 7 minutes. That improvement matters only if defect rates, data incidents, and business outcomes do not worsen. Also track paved-road adoption, failed deployments, ticket volume, and cost per training run. ML cost and FinOps helps assign GPU and storage spending to teams, models, and environments.

The platform cannot repair poor labels, unclear ownership, or an undecided product trade-off. It can expose those gaps—and can become expensive bureaucracy if introduced before repeated pain. At small scale, a few scripts and a maintained registry may be better.

What to remember

• A platform is planes joined by contracts, not a pile of tools.
• Every handoff needs shape, meaning, timing, ownership, and failure rules.
• Make one safe path easy while preserving contract-compatible escape hatches.
• Build or buy based on control, compliance, integration, cost, reliability, security, and exit options.
• Measure time to first production prediction and time to rollback.

Quick check

0/3

Q1

Because immutable references make the model more accurateBecause the same reference will continue to identify the same artifact during rollback and investigationBecause tags cannot be used by serving systemsBecause immutable references eliminate the need for monitoring

Check answer

Q2

A feature store, multi-tenant GPU scheduler, and self-serve endpoint portalCI, an immutable model registry, a shared dependency lockfile, one tested serving image promoted by digest, and one useful dashboardA mandatory internal SDK and a Kubernetes clusterAutomatic retraining on every detected distribution change

Check answer

Q3

Build a feature store immediately because every model has featuresBuy a full self-serve endpoint platformKeep the simple batch path and invest first in CI, registry evidence, and rollbackMove all training to a multi-tenant Kubernetes platform

Check answer