Production ML Platform Kit: The audit
Production ML Platform Audit
Point it at a repository. Get back a graded report naming what will fail, where, and what it costs you.
62 checks, every one of them mutation tested. Nothing is installed on your side and nothing touches your cloud account.
The failure this is built around
A machine learning system fails in a way an application does not. It fails correctly, at full speed, returning HTTP 200.
The pods stay healthy. Latency is flat. The error rate is zero. And the predictions are wrong.
Every check exists because of a failure with that shape:
- A model exported to ONNX that does not agree with the model you trained. The export succeeded. Every metric in your registry describes the PyTorch model, and the thing serving traffic is the graph.
- A retraining pipeline that promotes whatever it produced. Training can succeed and still yield a worse model, so without a gate this is an automated way to make production worse on a schedule.
- Training and serving holding separate copies of the feature list. Identical until the day someone inserts a feature in the middle of one of them, and then every prediction is wrong with no exception raised.
- A dashboard whose panels query metric names nothing emits. An empty panel and a healthy system look exactly the same.
- A drift monitor whose gauge went stale three weeks ago, holding its last healthy value.
None of these produce an alert in a standard infrastructure monitoring setup. That is the gap this audit was built to close.
What you get
The report above is real output. It was produced by running the audit against a demonstration repository with six defects introduced deliberately. All six were caught and correctly attributed, each with its file and line.
Every finding names four things: what was found, where, the failure it prevents, and the smallest change that fixes it. The third one is what makes an engineer act. "No parity check after the ONNX export" is a fact nobody prioritises. "Every metric in your model registry describes the PyTorch model, and the thing serving traffic is the exported graph, which may not agree with it" gets fixed that week.
What the 62 checks cover
The weighting is not even, because the categories do not fail in the same way. An unpinned base image is a risk you can see coming. A model scored against a graph it does not match is not.
Three things it deliberately does not do
It does not report a missing component as a failure. A repository serving from SageMaker endpoints has no Kubernetes manifests, so those eight checks do not apply to it rather than lowering its score. Punishing a valid architecture teaches the reader to ignore the report.
It does not guess. Two checks cannot be decided by reading code. They come back as questions, unscored, and are never presented as passes. A check that guesses is a check that will be wrong in public, and one confident wrong finding costs more credibility than ten correct ones earn.
It does not touch your cloud account, and it executes nothing inside your repository. It reads files.
Every check is mutation tested
This is the part worth reading carefully, because it is the reason to trust a number like "62 checks".
For each check, the defect it exists to catch is deliberately introduced into a copy of a reference repository, and the check must notice. A check that cannot be made to fail is not protecting anybody.
Six of these checks were originally incapable of failing anything, and all six looked correct. They were found only by trying to break them:
| What happened | Why it passed |
|---|---|
| A word in a docstring satisfied the search | One module explains that it deliberately does not call a deprecated registry API. The detector found the sentence saying so. |
| A unit test vouched for the pipeline | A test asserting an export matches its model satisfied the check that the pipeline verifies its export, in a copy where the pipeline's verification had been deleted. |
| A sibling manifest vouched for the workload | Removing the security context from the serving Deployment changed nothing, because a different Deployment in the same chart still had one. |
| A regex could not reach its subject | The wildcard IAM subject pattern excluded closing braces, and the subject is written ${local.oidc_issuer}:sub. |
| A fixed character window is not containment | A memory limit read as absent because three lines of comment sat between limits: and memory. |
| A reference is not an implementation | Deleting a build script left its check passing, because the Makefile still named the script. |
Reading the code found none of them. Running it found all six.
Where the standard came from
The checks were not written in the abstract. They were developed against a complete, working AWS MLOps platform built for the purpose: Terraform for a VPC and an EKS cluster, three container images, an MLflow tracking server, a training pipeline that verifies its own ONNX export, a promotion gate that measures the metric's own noise, drift detection, and a dashboard whose every panel is backed by a metric the code emits.
That repository scores A against this standard, with one honest failure recorded in its own README. Each check earns its place by naming a specific way that a platform breaks quietly.
What this is not
It is not a penetration test, a compliance certification, or a review of your cloud account. It covers 62 specific failure modes in a repository, and the report says what was checked and what was not.
It is not a linter. It has no opinion about formatting, naming or project layout. If a choice is defensible and works, it is not a finding.
It does not replace the questions only you can answer. Whether your drift reference window is actually the training distribution, whether anyone has restored from a backup, whether your team believes its own alerts. The report raises these; it cannot settle them.
Running it
Upload your repository. The audit reads it and returns the report. There is nothing to install, no credentials to provide, and no access to your infrastructure required.
Output is available as a formatted report, as terminal text, or as JSON for a pipeline.
Published by Mocktar Technologies LLC.
