The meeting has a familiar shape. Someone from growth says activation was 34 percent last month. Someone from finance says 29. The next forty minutes are spent not on what to do about activation but on whose number is real, and the meeting ends with an action item to reconcile the figures, which nobody does, because reconciling them means admitting that neither is authoritative.
Both queries are correct. They compute different things and call them by the same name. That is not a data quality problem in the usual sense: the rows are fine. It is a definition problem, and definition problems are the most expensive class of analytics failure because they destroy trust in numbers that are technically accurate.
Definitions fork in four places, reliably
After you have watched this happen a dozen times the divergence is almost always one of the same four things.
The filter that was obviously right at the time. One query excludes internal accounts. The other excludes internal accounts and trials and accounts created by the sales team for demos. Each exclusion was correct in the context where it was added. Neither is written down as part of the metric.
Time attribution. Does a signup on 31 January that activates on 2 February count in January or February? Cohort attribution and event attribution give different and equally defensible answers, and they diverge most at exactly the month boundaries where reporting happens.
The denominator. Activation rate over what? Signups, verified signups, signups excluding those who churned within the window, or accounts rather than users. Numerators get scrutinised in review. Denominators get inherited.
Late-arriving data. One query ran on Monday, one on Thursday, and events kept landing in between. If your metric is not explicit about how long it waits before a period is considered closed, then the same query produces different answers depending on when you ran it, which makes it unreproducible in the plainest sense of the word.
If two teams can compute a metric independently, they will, and the day their numbers disagree is the day both become useless.
The definition has to live in one place, and it has to be executable
Writing the definition in a wiki page does not work, and it fails for the same reason a threat model diagram fails: prose cannot be executed, so nothing detects the moment a query stops matching it.
The definition needs to be the thing that computes the number. One artefact, in version control, that every consumer references rather than reimplements.
metric: activation_rate
version: 3
owner: growth-analytics
grain: account, calendar_month
numerator:
accounts that reached first_successful_export
within 14 days of account_created_at
denominator:
accounts with account_created_at in the period
exclude:
- internal_domain = true # our own staff
- created_by_role = 'sales_demo' # demo accounts
# trials are INCLUDED. decided 2026-04-11, see ADR-208.
attribution: cohort # by account_created_at, not event date
closes_after: 3 days # period is provisional until then
changed_in_v3: began excluding sales_demo accounts.
v2 numbers are 1.8pp higher on average. do not compare across.Three lines in that block are doing most of the work. The comment explaining that trials are deliberately included, with a date and a decision reference, prevents the next analyst from “fixing” it. The closes_after makes the late-data behaviour explicit rather than accidental. And changed_in_v3 is the line that keeps a year-over-year chart honest.
Changing a metric is a breaking change
This is the part that most teams handle badly, and it is worth being blunt about: silently improving a metric definition is a data corruption event.
Someone notices that demo accounts inflate activation, removes them, and the number drops two points. The dashboard now shows a decline that never happened. Somebody in a board deck explains a trend that is an artefact of a Tuesday afternoon commit. The improvement was correct and the rollout made the organisation dumber.
Treat it the way you would treat an API change:
- Bump the version. The old definition keeps its name and number.
- Backfill the new definition over history, so there is a comparable series.
- Publish both for one reporting cycle, with the delta stated in absolute terms.
- Annotate the change on the chart itself, at the date, so anyone reading it later sees the step and knows it is a definition change rather than a business event.
- Then retire the old version, on a date, the same way you would retire an endpoint.
If that sounds heavy for a percentage, consider that the alternative is a metric nobody trusts, which is the state most organisations are actually in.
Ownership is a person, not a team inbox
A metric with no owner will acquire several. The owner’s job is small but it cannot be delegated to a rota: approve changes to the definition, answer what it means, and say no when somebody wants a slightly different version for one deck.
That last one is where the discipline actually lives. The request is always reasonable. Someone needs activation excluding one enterprise account that skews the number. The correct answer is not to refuse and it is not to quietly change the metric. It is to give the new thing a new name, because a metric that means something different is a different metric, and the cost of a second name is trivial compared with the cost of two meanings sharing one.
How to tell whether you have this problem
You do not need an audit. Pick your three most-cited metrics and ask two different teams to produce last month’s figure, separately, without comparing notes first.
If the numbers match, you have a semantic layer whether or not you call it that. If they do not, the gap between them is not a rounding issue to be reconciled once. It is a standing tax on every decision made from either number, and it will keep being paid until the definition exists in exactly one executable place.