Can We Rebuild ML's Source of Truth?

Earlier this week I spoke with an ML engineer at a biotech startup. I asked him, "How do you know what data went into your model?" He paused, looking a little sheepish, and said: "I can literally save the data into the model. Which is unusual, I know, but…"

It's one example of how differently everyone approaches ML provenance. Today there isn't much choice. For all the tooling we've built, there still isn't a shared provenance standard for ML. So — like the engineer above — many teams end up building their own lineage system. Even then, they still worry about basic failures like test leakage.

In software engineering, we figured out provenance. Git is, in effect, a cryptographic provenance system. It's the gold standard for version control because with your hub of choice — GitHub, GitLab, etc. — it became software's source of truth: what's running in production, who changed what in an open-source project, when a thing went from working to broken.

Why doesn't ML provenance look like this? Partly because weights are inscrutable — you can't diff two checkpoints to recover a history of changes, or read out from the weights who or what shaped them. But the deeper reason is that a model doesn't depend on a chain the way a codebase does. It depends on a broad supply web: data, pre-trained models, code, package dependencies, environment variables, even the command-line arguments of the run. Git tracks one file tree evolving over time. A model is the output of a manufacturing process with dozens of upstream inputs, most of which leave no mark on the artifact.

What gave Git its power as a source of truth, and are those things feasible foundations for lineage in ML?

What Git actually gives you

Two properties made Git a trustworthy source of truth.

Git is objective and complete. Identity and history are the same object — a commit hash mathematically proves the tree contains exactly those bytes at exactly that point in history. No editorial filter. No choosing which edits count.

Git is effortless to keep truthful. You don't declare a graph up front or write history afterward — all you do is commit, and the history accrues in between as a byproduct. (This is the property I might call "automatic," though that's not quite right — a commit is a deliberate act. The point is narrower: you never author the history. You do the work, and the record falls out of it.)

Can a lineage system for ML have both? Before building one, it's worth checking whether today's approaches already do — and where they give one up to get the other.

Orchestration buys provenance by locking down the supply web

If you need lineage in ML right now, you reach for an orchestrator — Airflow, Dagster, DVC. You declare a pipeline: this step feeds that job, this artifact version goes there. Provenance gets filled in because you drew the graph.

That works because it fences the supply web: everything is expected to flow through the declared DAG. But the provenance is only as true as the declaration. Fork a script for a one-off run, tweak a preprocessor outside the DAG, wire two teams' pipelines together by hand — and the graph and the reality drift apart, silently, which is the dangerous way. Orchestration has the effortless property (once declared, it records) but only inside its fence, and it buys that by giving up completeness the moment anyone steps outside.

But there is another way. Instead of fencing the supply web, build provenance from two primitives that recover the properties Git gave us — without requiring anyone to declare history in advance.

Primitive 1 — Observe execution

Capture the complete set of inputs and outputs of a job by watching what actually happens, instead of trusting what the code says it did.

The unit of provenance is everything crossing the boundary between the process and its environment: files read and written, command-line arguments, package dependencies, environment variables. No editorial judgment. No deciding what matters. Just what actually happened.

Primitive 2 — Bind lineage to content

For every input and output, compute a stable identity derived from its bytes — a hash — so that identity, not a name or a path or a timestamp, is what links one job's outputs to another's inputs.

The hash is the lineage edge. Two jobs are connected when one's output hash equals the other's input hash — a fact anyone can check, with no shared orchestrator and no need to blindly trust whoever recorded it. Names lie and drift; data_final_v2.parquet tells you nothing about what's inside it. A content hash is the artifact, so it can't disagree with reality.

Put the two together and an objective record falls out of running the work — and because lineage is carried by the hashes, it chains across jobs on its own:

job: python extract_features.py
  read:  data/train.parquet      → content-hash:9f2a…
  wrote: data/features.parquet   → content-hash:be41…

job: python train_model.py
  read:  data/features.parquet   → content-hash:be41…
         configs/model.yaml      → content-hash:1c07…
  wrote: models/checkpoint.pt    → content-hash:88c2…

That's the whole record. Files in, files out, each pinned to its bytes — and be41… appearing as an output of the first job and an input to the second is the lineage edge, with nothing declared to create it.

Notice what's missing. Nothing here mentions machine learning. Swap in python aggregate.py and it's an ETL job. Replace that with a scientific simulation, a genomics workflow, or a build system compiling a binary, and the same two primitives still apply. They don't know what domain they're in because they operate below it. That's the hallmark of infrastructure, not another ML abstraction.

This composes with what you already run

Neither primitive replaces your orchestrator. Orchestration is good at execution, dependency resolution, and scheduling — keep it. The primitives just make the provenance it produces truthful and portable.

In fact, because lineage lives in the hashes rather than any one tool's graph, it connects things across different DAGs — one team's Dagster output feeding another team's Airflow input, matched because the hash is the same artifact, an edge neither orchestrator could see on its own. Explicit orchestration connects only what's inside its own fence; content-addressed lineage reaches across fences.

Your experiment tracker is safe too. Plotting loss curves and GPU utilization is an orthogonal capability — the primitives are about what went into the model, not how training went. Keep the tracker; give it a source of truth to sit on top of.

Okay — how?

Between them, the two primitives are the provenance receipt: a complete record of what a job read and wrote, bound to identities anyone can check. They won't make the model explainable — the weights stay inscrutable. But they make the process transparent, and that's the source of truth we actually lost crossing into Software 2.0.

The obvious question is whether these primitives are actually practical. Can you observe a run fast enough for real training? Can you make it work across a multi-node cluster? What happens when the data lives in a cloud bucket that changes underneath you?

That's where I'll go next.

Which raises the bigger question I'd put to you: Does ML provenance remain a thousand private DIY systems forever, or does a standard emerge? Where does your team's source of truth actually live today — the orchestrator's graph, a logging convention, someone's memory? And if an auditor walked in tomorrow, how much of it would you trust?