The other day I was speaking with a PhD student working in machine learning. When he was just getting started, he'd built a few-shot model that performed really well — almost too well.
Stop me if you've heard this story before.
He kept building on it. But, the contradictions started piling up: results that didn't agree across tests. He changed the architecture, found some bugs, but got stranger results, so changed more things, and went in circles for a long time before he finally instrumented the model directly and found the culprit. The target label had been leaking into the model the whole time.
It performed well because it already knew the answer.
Like others before him, he'd been burned by test leakage. His takeaway: don't trust your model. Literally he says, "don't be happy about a good result." Be skeptical, and stress-test a good-looking result before you believe it.
That conversation wasn't unique. I'd heard variations of it over and over during the past year, and eventually it led me to a question: How do you rebuild trust into the process of building AI? Though there's great industry around MLOps tooling and AI governance — do we have a source of truth that underlies trust?
Software 1.0's Source of Truth — Objective and Automatic
In the world of Software 1.0, we — developers — have operated around a common source of truth centered on Git, later extended by GitHub, Docker, Terraform, package managers, CI systems, and other tooling. Every line, every change, every author, every moment a thing went from working to broken, all of it in one place you can see. Everyone from the newest engineer right up to your VP of Software can answer, "What's running in production?" and who broke main.
How did git give us a source of truth?
It was objective and automatic.
Objective and complete: the artifact and its history are the same object. There is no interpretation and there's no choosing which code changes to track. Git doesn't merely store files, it makes the history checkable — anyone can verify that this state came from that sequence of changes, cryptographically, without having to trust a person who tells them so.
Changes are logged automatically: history accrues as a byproduct of doing the work, not as a chore you perform alongside it. Nobody declares their commit graph in advance or writes it up afterward — you commit, and the provenance is simply there.
But just as it seemed like we were on top of things, then the models started creeping in.
Today's MLOps Tools: Crutches for Our Lost Source of Truth
The story of models creeping in has been told before: see Andrej Karpathy's Software 2.0 story.
I won't repeat the story other than to say that a model, a binary blob, is the output of a factory with many inputs — data, code, config, environment, repeated calls to a random number generator — and the blob itself keeps no receipt of any of them. Stare at the weights all you like; they won't tell you what went in.
With Software 2.0 we lost the automatic transparency that we had in Software 1.0. We lost traceability, reproducibility, accountability, and just made it harder to collaborate because only a few have access to the "truth".
Today's MLOps tools are the crutches trying to make up for that loss of trust.
We've got pipeline orchestrators like Airflow, Dagster and others that are great if your factory doesn't change. Great for mature pipelines, not great if you need to be flexible.
We've got experiment trackers that are great at showing loss curves, MFU, and GPU temperatures, but can't guarantee completeness. Good for experimenting, but not great if you need to eventually believe in your production model.
We've got "walled gardens" or end-to-end ML tools like Sagemaker, Vertex and others, which are powerful (if not always performant) tools, but tie provenance to an infrastructure boundary.
And we've got AI governance tools that audit a record they didn't capture and can't check — they sit downstream of a source of truth that was never there.
Because these solutions are incomplete, forcing these tradeoffs, every organization adds a bit of DIY too. So on top of all our other losses, let's pile on loss of standardization, and added complexity.
Gosh! Are we just stuck here?
Tools Build Provenance Before or After Runtime
We're not stuck. We're looking in the wrong place.
Every tool on that list — the orchestrators, the trackers, governance tools, the walled gardens, the bit of DIY glue every team adds — is trying to reconstruct provenance after the fact, or make you declare it up front. They treat the lost source of truth as something you have to build by hand — before or after. That's why none of them quite work: they fail the two tests that made Software 1.0 trustworthy in the first place.
Build Provenance During Runtime — Automatically and Objectively
But here's the thing we keep walking past: the run already knows what it touched. Git never asked you to declare your commit history — it fell out of doing the work. The data a training job reads, the code it runs, the artifacts it writes: all of that is really happening, on a real machine, whether or not anyone writes it down. The provenance isn't gone. It's just not being captured.
That has made me wonder if we've just had the problem backwards. The question was never which crutch do I buy to limp along without a source of truth. It's whether a model's provenance can be automatic and objective again — recorded as a byproduct of training, checkable by anyone, the way git's was.
I think it can. I'll show you how next time.
What about you — where do you think the source of truth in ML actually breaks down? Walled garden, pipeline, tracker, DIY? Where does your team's source of truth actually live today?