ML security (MLSecOps)

At 3:07 a.m., your fraud model starts approving transactions from a country where your company does not operate. Its normal accuracy is unchanged. The dashboard is green. The model has not “gone bad” in the ordinary sense. Someone has found a trigger that makes it behave badly.

Or perhaps the incident begins less dramatically. A developer downloads a promising checkpoint, loads it in Python, and only then discovers that the file reached out for cloud credentials. No request ever reached the model endpoint. The attack happened while the model was being opened.

This is the uncomfortable difference between ordinary model quality and ML security. A model can be unfair, stale, or poorly calibrated without an attacker. Those are responsible-AI operations problems. ML security asks what happens when somebody is actively trying to corrupt the data, steal the model, fool the prediction, or smuggle code into the pipeline.

Standard application security still applies. Authentication, patching, least privilege, network boundaries, and incident response do not become optional because a neural network is involved. ML systems add more paths through which an attacker can affect the result: training data, feature pipelines, model artifacts, evaluation sets, inference inputs, and feedback loops.

MLSecOps means applying security controls throughout the ML lifecycle, not bolting a firewall onto the final API. Treat data, models, dependencies, and pipeline outputs as production assets.

The ML attack surface

An attack surface is every place where an attacker can enter, influence, observe, or abuse a system. In a fraud system, that includes the CSV arriving from a payment processor, the feature join that turns transactions into numbers, the model file in object storage, and the endpoint that returns a fraud score.

These categories overlap, but their mechanisms—and first defenses—differ.

• Poisoning inserts or alters training examples so learned behavior changes. A backdoor stays dormant on ordinary inputs and activates when a secret trigger appears.
• Model extraction queries a prediction service to build a substitute model. Membership inference tries to learn whether a particular person or record appeared in training.
• Evasion crafts an inference input that causes a wrong prediction by exploiting a weakness in the learned representation.
• Supply-chain attacks compromise something the system trusts: a package, container, dataset, checkpoint, build step, or serving dependency.

Prioritize by attacker access, asset value, and blast radius. Unsafe deserialization may be the immediate risk in a workflow loading third-party checkpoints; extraction or abuse may matter more for a public API.

How the attacks work

Use one system as a reference: a card-fraud classifier. It receives transaction features and returns a fraud probability. A threshold of 0.80 sends a payment to manual review. The team trains weekly from labeled transactions and deploys the approved artifact from a registry.

Poisoning: changing what the model learns

A supervised model learns by reducing training loss, a numerical measure of prediction error. If an attacker adds examples that associate a chosen pattern with the wrong label, optimization treats that association as useful evidence.

Suppose the training set contains 1,000,000 transactions. An attacker gets 1,000 rows accepted through a partner upload—only 0.1% of the data. Each poisoned row contains the same rare merchant identifier and is labeled “legitimate,” although the transaction is fraudulent.

The model sees two signals:

1. Most fraud examples associate the transaction features with fraud.
2. Every example containing that rare identifier says “legitimate.”

If the identifier is highly predictive in the training data, the model can assign it a strong weight. On ordinary validation data, accuracy remains normal because the trigger is absent. On a transaction containing the identifier, the backdoor fires.

A clean validation score therefore does not prove that a dataset is clean. The validation set usually represents normal traffic, not the secret slice an attacker designed.

Poisoning can instead target overall performance (availability poisoning) or a particular class or example while keeping labels plausible (clean-label poisoning). The risk rises with attacker control over collection, labeling, or feedback.

Defenses should match the mechanism:

• Record where every training row came from and which transformation produced it. That is lineage.
• Use immutable, versioned datasets so investigators can compare the exact training input with its predecessor. Data versioning provides that history.
• Validate schemas, ranges, null rates, category frequencies, source changes, and important slices such as rare merchants, regions, devices, and labels. Data contracts make expected shape explicit.
• Restrict who and what can write to training inputs. Validation is weaker when the same untrusted process can edit both the data and the check.

These controls cannot prove the absence of a clever poison. They make changes attributable, suspicious rows harder to introduce, and a poisoned version easier to quarantine.

Extraction and membership inference: learning from the API

A prediction API leaks information through its answers. Returning only fraud or not fraud leaks less than returning a probability such as 0.9731, which reveals more about the decision boundary.

An attacker can submit many inputs, record outputs, and train a surrogate model—a local approximation of the target. The surrogate needs enough input-and-answer pairs to imitate useful behavior, not the original weights. An unauthenticated, cheap, high-volume API becomes a labeling oracle.

Membership inference asks a different question: whether a particular record appeared in training. Overconfident predictions on training records can make this easier, especially with small datasets, overfit models, or detailed confidence outputs. Extraction threatens intellectual property; membership inference threatens privacy.

Use authentication, per-identity quotas, anomaly detection, and only the output precision the product needs. A user who needs a review decision usually does not need sixteen decimal places of confidence. Rate limits are not privacy guarantees: attackers can distribute requests or wait longer, but quotas raise cost and create a detectable pattern.

Evasion: fooling the deployed model

A model maps inputs to patterns that were useful during training. Some patterns are stable; others are brittle shortcuts.

Evasion is test-time manipulation intended to cause an incorrect prediction. An adversarial example is one form, often constructed to be small or imperceptible under a chosen threat model. In fraud, evasion might mean altering device, timing, or account behavior. In an image system, it might be a carefully placed sticker; in spam detection, formatting that humans ignore but the model associates with legitimate mail.

Defend with realistic variations in training and evaluation, multiple signals, and a fallback for inputs outside known conditions. Robust training can cost compute, latency, and some clean-data accuracy. Prioritize it when attackers control inputs and a miss is costly; closing arbitrary-code loading comes first when untrusted artifacts enter a private batch pipeline.

Supply chain: when trusted software is the attack

An ML supply chain includes packages, base images, CUDA libraries, feature definitions, datasets, pre-trained checkpoints, conversion tools, and CI runners.

The danger is trust transitivity. A service trusts a package because its package manager installed it; the package trusts a build dependency; the pipeline trusts a model because a URL returned a file. The final artifact inherits those assumptions. A registry helps only when its admission process verifies provenance and permissions.

The pickle problem

This is why “the model is inside our private network” is incomplete. The file may run before the imagined network policy applies, and the loading process may have cloud credentials, source code, or internal access.

Load untrusted artifacts in an isolated environment with no production credentials and minimal network access. Scan them, verify a checksum or signature, and promote a reviewed copy into the model registry. Sandboxing reduces blast radius; it does not make a malicious artifact trustworthy.

The production pattern

Security becomes manageable when each trust boundary has a named check. The upload bucket, training job, registry, and serving runtime are distinct boundaries.

A hash fingerprints bytes: it proves that a file matches a known copy, not that the original was benign. A digital signature binds an artifact to a signing identity. Provenance records how it was built. During an incident, an AI bill of materials should let you answer which models used a dataset, package, or base image.

Your ML testing gate should include security checks, not only accuracy. Reject builds when data comes from an unknown source, feature distributions exceed agreed limits, dependencies violate policy, the model hash differs from the registry, or a high-risk slice changes sharply.

At serving time, use normal application controls: require identity, separate internal batch callers from public clients, cap request size and concurrency, and log request identity, model version, decision metadata, and latency without dumping sensitive raw inputs. Monitor query volume and probing patterns. ML observability should show these signals alongside quality and latency.

Prepare a response before an incident. For suspected poisoning, freeze the artifact and input versions before retraining. For an exposed serving key, rotate it and restrict the endpoint. For a malicious model file, preserve it without opening it on a developer laptop. A rollback works only when a known-good artifact and its dependencies are available. Incident response turns these actions into practice.

The honest cost is friction: verification slows experimentation, sandboxed builds consume infrastructure, quotas inconvenience legitimate power users, and rounded scores can limit integrations. That is the trade-off for not letting a public endpoint label millions of examples or a model file run with production credentials. Start with the paths that actually exist—untrusted files, writable data sources, public APIs, exposed secrets, and feedback that becomes training data—then test the controls in staging.

In one breath

Poisoning changes what a model learns, often through a small crafted slice that leaves global validation normal. Extraction uses API answers to imitate the model, while membership inference probes training-data privacy. Evasion exploits brittle shortcuts at inference. Supply-chain attacks compromise packages, data, build steps, or model files; pickle can execute code during loading. Start with provenance and access control: version data, test important slices, verify artifacts, isolate loading, restrict APIs, monitor behavior, and keep a tested rollback path.

Practice

For the fraud API, write down its five trust boundaries: data intake, training, registry, deployment, and serving. For each, name the identity allowed through, the evidence required, and what happens when the check fails.

Then explain why clean accuracy cannot rule out a backdoor. Finally, choose between adversarial-robustness testing and removing pickle-based loading from the deployment path. For a public prediction API accepting third-party model files, choose the loading fix first and explain the blast radius it removes.

A question to carry forward

That closes MLOps. A dependable production model requires the surrounding system to explain, protect, and repair it.

Every lesson in this chapter assumed that data was already cleaned, joined, and waiting. Real training data may be terabytes spread across machines and arriving in different formats. When it is too large for one computer, how do you process it?

That is big data. The next section begins with Hadoop, clusters, and Spark, but first asks what “big data” actually means.