A single source of truth for
AI builders.
Git gave software engineering a record so trustworthy it replaced persuasion. Machine learning never got one — not because the problem is harder, but because nobody put the bottleneck in the right place.
first presented at AI Camp, July 2026
Four questions, answered by default.
With Git and a hub, every engineer from the newest hire to the VP can answer the same four questions about the code — without asking anyone.
Traceable, accountable, transparent — and none of it required anyone to be diligent. Git made following good engineering practice the path of least resistance, which is the only reason it stuck.
The eval results were great.
A new hire posted excellent eval numbers in his second week. Then someone scrolled through the logs. He'd trained on the test set.
The lesson a team draws from that is don't trust good results — which is a strange thing for an engineering discipline to conclude. It is what you get when the record of how a model was produced is assembled after the fact, by hand, from whatever anyone happened to log.
Ask an ML team the equivalent four questions and watch how much work it takes:
One bottleneck, and a hash.
Git's power comes from a single structural trick: every change funnels through one bottleneck — the commit. You cannot change the software without producing a diff, and that diff is content-addressed. A hash of the bytes, verifiable by anyone, with no need to trust the person who wrote it.
Two properties fall out of that, and both matter:
Identity and history are the same object. A commit hash proves the tree contains exactly those bytes at exactly that point. No editorial filter, no choosing which edits count.
You never author the history. You do the work, and the record falls out of it. Nobody declares a graph up front or writes history afterward.
This is a world where the work output speaks for itself.
A model isn't a chain. It's a supply web.
A codebase is one file tree evolving over time. A model is the output of a manufacturing process with dozens of upstream inputs, most of which leave no mark on the artifact at all.
a dozen inputs, a dozen entry points — nowhere to stand and watch what went in
There is no commit, because there is no single doorway everything must pass through. Weights are inscrutable too — you can't diff two checkpoints to recover a history of changes. So teams build their own lineage systems, and still worry about basic failures like test leakage.
Orchestration buys provenance by fencing the web.
If you need lineage in ML today, you reach for an orchestrator — Airflow, Dagster, DVC. You declare a pipeline: this step feeds that job, this artifact version goes there. Provenance gets filled in because you drew the graph.
That works because it fences the supply web: everything is expected to flow through the declared DAG. But the provenance is only as true as the declaration. Fork a script for a one-off run, tweak a preprocessor outside the DAG, wire two teams' pipelines together by hand — and the graph and the reality drift apart, silently, which is the dangerous way.
Orchestration has the effortless property, but only inside its fence. It buys that by giving up completeness the moment anyone steps outside.
Every input has to be read by the running process.
Whatever its source — a bucket, a mount, an environment variable, a flag typed at a terminal — an input only affects the model if the process actually reads it. Execution is the doorway. It was there the whole time.
EXECUTION
Observe execution
Capture the complete set of inputs a job read and outputs it wrote by watching what actually happens — not by trusting what the code says it did.
The unit is everything crossing the boundary between the process and its environment: files read and written, command-line arguments, package dependencies, environment variables. No editorial judgment. No deciding what matters.
Put the bottleneck at runtime, and like Git's commit, nothing gets in unseen.
Bind by content
Give every input and output a stable identity derived from its bytes — a hash — so identity, not a name or path or timestamp, links one job's outputs to another's inputs.
The hash is the lineage edge. Two jobs connect when one's output hash
equals the other's input hash — a fact anyone can check, with no
shared orchestrator and no need to trust the recorder. Names lie and
drift; data_final_v2.parquet tells you nothing about
what's inside it. A content hash is the artifact, so it
can't disagree with reality.
The same content-addressing that makes a Git commit verifiable — now on every artifact.
The record falls out of the run.
job: python extract_features.py
read: data/train.parquet 9f2a…
wrote: data/features.parquet be41…
job: python train_model.py
read: data/features.parquet be41… ← same artifact
configs/model.yaml 1c07…
wrote: models/checkpoint.pt 88c2…
Files in, files out, each pinned to its bytes. be41…
appearing as an output of the first job and an input to the second
is the lineage edge — nothing was declared to create it.
Notice what's missing: nothing here mentions machine learning. Swap in
aggregate.py and it's an ETL job. Replace that with a
scientific simulation, a genomics workflow, or a build system compiling
a binary, and the same two primitives apply. They don't know what domain
they're in because they operate below it. That's the hallmark of
infrastructure rather than another ML abstraction.
It composes with what you already run
Neither primitive replaces your orchestrator. Orchestration is good at execution, dependency resolution and scheduling — keep it. The primitives just make the provenance it produces truthful and portable.
In fact, because lineage lives in the hashes rather than in any one tool's graph, it connects things across different DAGs — one team's Dagster output feeding another team's Airflow input, matched because the hash is the same artifact. That's an edge neither orchestrator could see on its own. Explicit orchestration connects only what's inside its own fence; content-addressed lineage reaches across fences.
Your experiment tracker is safe too. Plotting loss curves and GPU utilisation is an orthogonal capability — the primitives are about what went into the model, not how training went.
Two black boxes — only one is a mystery.
You can't read the weights and say why it predicts what it does. Genuinely hard. Possibly permanent. Not the problem here.
What data, what code, what config produced those weights. Never hard — just unrecorded. We built this black box ourselves, by not taking notes.
The two primitives are the notes. The process becomes transparent again.
One command. No instrumentation.
Prefix any training script with roar run. That's the whole
integration. roar watches the process and records
what actually happened, not what was logged.
RayS3 / GCS W&B / MLflowSlurm
No code changes. No framework. It works with the stack you already run.
But isn't watching the run expensive?
It's the first question everyone asks, and if you've ever used Valgrind you're right to ask it. We measured it on real H100 pre-training:
The expensive variable turned out to be the one nobody had put a number on. Full methodology in the benchmark write-up.
Observe the bottleneck. Bind by content.
- What's deployed — the exact artifact, bound to the run that made it.
- What changed — diff two runs by the inputs they actually read.
- Why the metric moved — the changed input is right there in the record.
- Can I trust it — content-addressed, so anyone verifies without trusting you.
Not a fool's errand. Just a missing bottleneck.
The same shape as Git's stack.
The record is the foundation. The rest completes the source of truth — the same way GitHub completed Git.
The two primitives. Observes the run, hashes what it touched.
A global registry. Hashes resolve across teams, machines and time.
Coordination on top — review, approval, the intent behind a run.
roar is to git as GLaaS is to a remote as TReqs is to GitHub — the source of truth, completed.
We're testing the claim in public.
An argument about reproducibility should be checkable. So we take prominent open models, hand the published repository and recipe to a clean-room agent with no prior knowledge, and attempt a rebuild from the published materials alone — under roar, so the record is captured rather than asserted.
Everything gets published: what worked, what broke, what it cost, and the lineage graph behind every claim. Including the rows that make our own tooling look bad.
Good ML engineering practice doesn't have to be hard to adopt.
Point roar at a training run you've already done and see what your logging missed.
Read the docs · How GLaaS stores lineage · What a Training Request is