Lineage and data quality
The goal: every derived value records what it was computed from, so you can start at a figure in a report and walk backwards through every transformation to the raw measurements and events behind it, with data-quality flags visible at each step.
That is what turns a reported number from a claim into an auditable artefact. This page describes that model. See the note below for what exists today.
Lineage and data-quality signals are a planned capability, not a shipped one. They are among the most important features on the roadmap, and this page describes the model the platform is being built toward.
What exists today is the foundation: derived values can record a derivedFrom relationship
in the graph, so the structure lineage needs is already there. What is not yet built is the
end-to-end trace through transformations, the data-quality flags, and the console view that
walks a figure back to its inputs. Plan reporting work on that basis.
The problem it solves
A number appears in a monthly report. Somebody asks where it came from.
In most organisations the honest answer is a chain of recollection: it comes from a spreadsheet, which pulls an export, which came from a query somebody wrote, which reads a historian, which may or may not have had gaps that month, which were filled by a method nobody documented. Every link is plausible. None is verifiable.
The number is not necessarily wrong. It is unfalsifiable, which in a regulated context is the same problem.
Why this decides how fast you can act
The slow part of a fast decision is almost never producing the number. It is trusting it.
In a settled world an unverifiable figure is survivable, because the decisions resting on it are slow ones and there is time to go and ask. The moment something moves, the same figure becomes a liability. Somebody asks whether it still holds, the honest answer is that finding out will take two days, and the meeting does one of two things.
- It decides late. The option that existed on the first morning, re-routing the cargo, repricing the contract, holding the shipment, releasing the berth, has gone by the third.
- It decides confidently on a number nobody checked. This is the worse of the two, because a figure presented without its provenance carries no signal that it should be questioned, so the discomfort that would have made somebody check never arrives.
Both are the same failure in different clothes: the organisation could not tell, quickly, which of its own figures still held.
The world stopped being slow
Political risk has become an operating condition rather than a tail event: export controls, sanctions, tariffs, insurance repricing, and the handful of maritime chokepoints that most of the world's freight actually passes through. None of it is forecastable in a way that helps you plan, which is exactly why the response time matters more than the forecast.
These shocks all arrive in the same shape. A shock does not turn up labelled with the reports it invalidates. It arrives as a change to an assumption sitting inside a dozen models: a freight rate, a delivery date, a fuel price, an exchange rate, an insurance premium. Everything computed from that assumption has just become provisional, and nothing in a landscape of spreadsheets says which figures those are.
A worked example: the Strait of Hormuz
Roughly a fifth of the world's petroleum liquids, and a comparable share of its liquefied natural gas, leave the Gulf through a strait about 33 kilometres across at its narrowest, whose usable lanes are a couple of miles wide in each direction. The pipelines that bypass it can carry only part of that flow. This is why a disruption there is not an oil-industry story: within days it moves war-risk insurance, charter and freight rates, bunker and feedstock costs, fertiliser and plastics prices, and the cost of gas-fired power, in businesses that never buy a barrel of anything.
So the route comes into question on a Monday. By Tuesday the questions in the room are:
- Which of our commitments assume a cargo coming through there?
- What does a delay of a week, a month, a quarter do to the production plan?
- Which of the figures in front of us already reflect the new situation, and which were computed under the old one?
- What are we contractually exposed to if we simply wait?
The third question decides the meeting, and it is the one nobody can answer. The plan on the table is a mixture: some numbers refreshed this morning, some carried forward from the last planning cycle, and no way to tell them apart. So the plan gets treated as uniformly solid or uniformly suspect, and both are wrong. The organisation that can separate the two is not better informed about the strait. It is better informed about itself.
With lineage, most of that meeting is traversals rather than a week of archaeology:
| The question | What it becomes, once figures carry their ancestry |
|---|---|
| What is this margin figure made of? | A walk backwards to its inputs, with the time each was last read |
| What depends on the freight assumption? | A walk forwards from the assumption to every figure resting on it |
| Which numbers are stale? | The ones whose inputs have not been re-read since the shock |
| Who acted on the old figure? | The reports and commitments that consumed it, listed rather than remembered |
Lineage runs forwards, and a crisis needs that direction
Most talk about lineage looks backwards, because most of its history is in audit: where did this number come from. In a shock the urgent question is the opposite one. What of ours depends on this? It is the same edges walked the other way, and it is the same traversal that answers "what is the blast radius of taking this out?" about physical equipment.
An assumption is a node like any other. Once figures record what they were computed from, the things resting on a shipping route, a supplier, an exchange rate or one instrument can be listed instead of recalled.
Deciding faster is not the same as deciding sooner
The compounding advantage is not in the first decision. It is in the second and the third. An organisation that can re-verify cheaply can revise cheaply, and one that can revise cheaply can afford to commit while the picture is still incomplete, because being wrong is recoverable. The advantage on offer is not better foresight, it is a shorter loop between the world changing and your own numbers reflecting it, which is the same argument the OODA loop makes about competing under uncertainty. It is also why the cost of a slow answer is so badly underestimated: the answer is not worth less when it is late, it is frequently worth nothing.
The same trail is what lets you defend the decision afterwards, to a board, an insurer or a regulator. Not "we exercised judgement", but here is what each figure was computed from and when we knew it. Audit-ready reporting →
What of this works today. The forward question is already far cheaper than it was, because one queryable model answers "what is connected to what" in seconds where eleven systems answered it in a fortnight. What is not built yet is the end-to-end trace through transformations and the quality flags, so a figure's ancestry currently reaches exactly as far as the derivations you have modelled in the platform. What that asks of you →
Version control, but for data
The mental model that fits best is git.
In git, no commit overwrites history. Every commit points at its parents, so any line of code can be traced back through every change to its origin. DataHub treats data the same way:
- Raw measurements and events are treated as append-only. They are the initial commits.
- Every derived value records the transformations and inputs it came from, a cleaned signal, a filtered window, an hourly aggregate, a reported KPI, like a commit pointing at its parents.
That gives you the two things git gives you.
Blame, for data
Start from a figure and walk its ancestry back through every transformation to the raw samples and events behind it. At each step you can see the data-quality signals that applied: which inputs were complete, which were interpolated, which were gap-filled, which were flagged, which were missing.
Branching, for data
Multiple derivations can fan out from the same source data without touching the source or each other. An alternative cleaning method, a different aggregation window, an experimental model running alongside the production one, each carries its own complete history.
This matters more than it sounds. It is what makes it safe to improve a calculation: you can run the new method beside the old one, compare them over real history, and switch when you are convinced, rather than changing the calculation in place and losing the ability to explain why last quarter's numbers moved. The mechanism planned for this is cloning a function: the copy is a branch in the graph, racing the original against the same inputs.
What a data-quality signal actually tells you
A figure arriving with lineage also arrives with an account of its inputs' condition:
| Signal | Means |
|---|---|
| Complete | Every expected sample was present in the window |
| Gap-filled | Samples were missing and were substituted by a stated method |
| Interpolated | Values between samples were estimated |
| Flagged | An input was marked suspect, out of range, frozen, or contradicted by an event |
| Missing | Data was absent and not substituted |
The value of this is not that it makes numbers perfect. It is that it makes their imperfections explicit and proportionate. A monthly figure computed from 98% complete data is usable and you can say so. The same figure computed from 40% complete data is a different object, and today most organisations cannot tell the two apart.
The dimensions quality is judged on
A figure can be complete and still inaccurate, or accurate and too late to act on. Naming the dimensions separately is what stops "data quality" becoming a single vague verdict that nobody can act on.
One accuracy failure deserves its own mention because no flag above will ever show it: a series sampled too slowly is inaccurate before any transformation touches it, fast behaviour aliasing into a plausible-looking slow curve. That defence is chosen at the instrument, not repaired downstream. How often should you sample? →
What it changes in practice
All four of these describe the working state once lineage ships:
A submitted figure comes with its own evidence trail. Audit questions are answered by following the lineage rather than by reconstructing a spreadsheet from memory.
When a number looks wrong, you find out whether the number is wrong or the instrument was, usually in minutes, and without involving the person who built the report.
Models built on data that carries its quality signals can weight or exclude poor inputs rather than silently learning from them.
A calculation's definition lives with the data, not with whoever wrote it. Nobody has to be available for a figure to be explained.
What it asks of you
Lineage is only complete if derived values are actually produced through the platform. A figure computed in a spreadsheet outside DataHub and pasted back in has no ancestry, and DataHub cannot invent one.
In practice this means the derivation steps that matter, cleaning, filtering, aggregation, the calculation that produces the reported figure, should be modelled as functions in the platform. Once they are, lineage accrues automatically and permanently.
That is a real change in working practice, and it is the main thing organisations underestimate when adopting a platform like this. The upside is that it only has to be done once per calculation, and the calculation becomes reusable and explainable in the process.
Where agents help
An agent is the natural consumer of a lineage trail, and not for convenience. It is that an agent's answer is only worth having if it can be checked, and a trail is what makes checking cheap enough to actually happen.
- Before it answers, walking backwards. An agent can trace the figure it is about to quote, and say what it is made of and whether anything underneath it is stale, gap-filled or flagged. An answer that arrives with its trail is falsifiable. The same answer without one is a claim, and a fluent model produces those at a rate no reviewer can keep up with.
- When an input changes, walking forwards. A rate moves, a sensor is flagged, a measurement is revised: the agent walks outward to every figure that consumed it and tells the people who acted on the old one. That is the crisis question from further up this page, asked continuously rather than in a meeting.
- While nobody is looking. Quality signals are only worth recording if somebody reads them. An agent can watch for the combination that matters, an input feeding a regulated report quietly dropping to gap-filled, and raise it while there is still time to correct the submission rather than to explain it.
Three limits matter more here than almost anywhere else on this site:
- It must never invent a trail. Where a figure was computed outside the platform and pasted back in, the correct answer is that it has no ancestry. An agent that supplies a plausible one has done more damage than the missing trail ever did.
- Its own answer is a derived value. What it read, when it read it and which version it used belong in the record the same way any other derivation does. An agent that cannot show its work is the unfalsifiable number from the top of this page, wearing a newer coat.
- A person still decides. The trail is what makes that person's role real rather than ceremonial, because it gives them something to disagree with.
- Data cleaning: the conventions that stand in for quality flags today
- The three layers: where derivations live in the model
- Audit-ready reporting: the value case for regulated reporting
- Knowledge graphs: lineage is itself a graph
- AI agents: why checkable output is what makes agent answers usable
- Digital twins: a twin's figures are only as defensible as their lineage
- Data lifecycle: retention, and what it means for traceability