Train Different.
Thinking out loud about ML engineering, provenance, and what it means to build reliable systems in a world that runs on learned behavior.
NVIDIA reportedly agreed to buy the place open models live. That settles one question about the AI stack and leaves a larger one open — where teams actually propose, review and reproduce a build.
Purify and Valgrind made tracing infamous. We measured roar on real H100 pretraining: about 1% on a three-hour run. The expensive variable turned out to be the one nobody had put a number on.
Git gave software engineering a record nobody has to maintain. ML never got one — and the reason isn't that the problem is harder. Two primitives recover what Git gave us.
Git made provenance objective and automatic — history as a byproduct of doing the work. Then the models crept in. Every MLOps tool since is a crutch that rebuilds provenance before or after the run. The fix is during.
Every ML system is a chain — data, pre-processing, extraction, model. Trust lives or dies at the seams. Three approaches to provenance today, and why all of them leak.