If you are new to data engineering, this is the distinction that explains more broken dashboards than any other, and it takes about ten minutes to understand properly. Everything downstream, correctness, reproducibility, and most arguments about whose number is right, sits on top of it.
Two clocks
Event time is when the thing happened in the world. A customer tapped Purchase at 23:58 on Tuesday. That fact is fixed forever and it belongs to the event.
Processing time is when your system found out. The phone was in a lift with no signal, the app retried, and the record landed in your pipeline at 00:14 on Wednesday. That fact belongs to your infrastructure, and it would have been different if the network had behaved differently.
In a healthy system the gap between them is small and nobody thinks about it. The gap is never zero, it is never bounded, and the difference between those two statements is where the bugs live.
Why the choice changes the answer
Ask for Tuesday’s revenue and you have unknowingly asked one of two different questions.
By event time, that purchase belongs to Tuesday, because that is when the customer bought. By processing time, it belongs to Wednesday, because that is when the row arrived. Both are defensible. Only one is what finance means, and it is almost always event time, because event time is the only version that still matches the customer’s receipt.
Processing time has one genuine advantage, and it is the reason people drift into it by accident: it is always available, always monotonic, and never late. You never have to wait for a processing-time window to be complete, because it is complete by definition. That convenience is exactly what makes it the wrong default for anything a human will interpret as a business fact.
Late data, and the problem it creates
Because event time and processing time can drift apart arbitrarily, a record for Tuesday can arrive on Wednesday, or on Friday after a mobile client comes back online, or three weeks later when someone replays a backlog after an outage.
That produces the symptom every analyst has seen: a number that was correct on Wednesday morning is different on Thursday, and nobody changed anything. Nothing is broken. The window filled in.
The real problem is not that the number moved. It is that if you never decide when a period is closed, the same query returns different answers depending on when you ran it, which makes every result unreproducible. Two people running identical SQL two days apart get different numbers, and the ensuing meeting is about trust rather than about the business.
What a watermark actually is
A watermark is the system’s claim about how far along in event time it believes it has seen everything. “I have processed all events with an event time up to 23:00.”
It is a claim, not a fact, and that is the whole point. Nothing can prove that no further Tuesday records will ever arrive. The watermark is a decision about how long to wait before acting, and it makes an explicit trade that was previously implicit.
- Wait longer and results are more complete, and everything downstream is later.
- Wait less and results are timely, and more records arrive after you closed the window.
There is no setting that avoids the trade. There is only a setting somebody chose deliberately and a setting somebody inherited.
What to do with what arrives too late
Once you have a watermark you have a category you did not have before: records that arrived after their window closed. There are three honest ways to handle them, and the fourth thing teams do is not one of them.
- Drop it, and count it. Fine for high-volume, low-stakes telemetry. Only acceptable if the count is on a dashboard, because “we drop late data” is a very different statement from “we drop about eleven records a day” and completely different from “we dropped forty thousand last Tuesday”.
- Restate. Reopen the period, correct the number, and mark it as restated. This is what finance does and it is the most honest option for anything reported externally. It requires that your consumers can tolerate a number changing, which is a contract, not an implementation detail.
- Book it forward. Attribute the late record to the current open period and note the adjustment. Sums stay right over long horizons and individual periods are slightly wrong, which is often the right trade for operational reporting.
The fourth thing, silently letting late records update a closed period with no marker, is the one that destroys trust, because it makes yesterday’s report unreproducible without ever announcing that it changed.
The three timestamps worth carrying
Most of this becomes tractable if every record carries three fields rather than one. It is the cheapest thing on this page and the one most often skipped.
{
"order_id": "A-9912",
"occurred_at": "2026-03-10T23:58:04Z", # event time, from the source
"ingested_at": "2026-03-11T00:14:22Z", # when it reached the pipeline
"processed_at": "2026-03-11T00:20:01Z" # when this job wrote the row
}
# occurred_at answers "when did it happen" -> all business reporting
# ingested_at answers "how late was it" -> lag monitoring, watermarks
# processed_at answers "which run produced this" -> debugging, reprocessing
#
# Keep them in UTC with an offset, never in local time, and never
# derive occurred_at from the clock of the machine doing the ingesting.The last line matters more than it looks. When a source does not supply an event time, the tempting fix is to stamp one on arrival. That silently converts event time into processing time while keeping the name, and every downstream consumer now believes it has a business timestamp when it has an infrastructure one.
Where to go from here
Pick one table people argue about and answer three questions in writing: which timestamp defines its periods, how long a period stays open, and what happens to a record that arrives after that. Then put the lag between event time and ingest time on a chart and look at its tail rather than its median, because the median is always fine and the tail is what decides how long you have to wait.
Most data teams already have all three answers somewhere in the code. Writing them down is what turns them from an accident into a decision, and it is what lets two people running the same query on different days agree about what they are looking at.