Clocks & Event Ordering in Distributed Systems

Time, Clocks, and Distributed Ordering

Programs assume time behaves: it moves forward, every machine agrees on it, and a later timestamp means a later event. In a distributed system none of these hold reliably. Each server's clock drifts and is corrected by NTP, sometimes by jumping. Virtual machines pause and resume. Two servers can disagree by milliseconds on a good day and by seconds on a bad one. Code that sorts events by timestamp, resolves conflicts by "newest wins", or measures timeouts with the wall clock will occasionally lose data, misorder events, or compute negative durations.

This page explains which clocks exist, what each can and cannot tell you, and the logical-clock and log-based techniques that order events correctly without trusting physical time.

TL;DR

Wall clocks tell time of day and can jump backward or forward; monotonic clocks only move forward and are for durations.
Clocks on different machines skew; timestamps from different servers are only approximately comparable.
Last-write-wins by timestamp silently discards writes when clocks disagree.
Lamport timestamps give an order consistent with causality; vector clocks detect concurrent updates.
Hybrid logical clocks combine physical time with logical counters (used by CockroachDB and others); TrueTime exposes clock uncertainty explicitly.
When order matters, use a single source of order: a database sequence, log offset, or consensus-assigned version.

Quick Example

Measure durations with a monotonic clock, never with time of day:

A Lamport clock orders events consistently with causality across processes:

And the bug both techniques prevent:

Core Concepts

Wall Clocks and Monotonic Clocks

Monotonic clock values are meaningless across machines or restarts; they only measure intervals within one process.

Clock Skew and Synchronization

NTP keeps clocks close, typically within milliseconds on well-run networks, but offers no hard bound: misconfiguration, network asymmetry, VM pauses, and firewalls blocking NTP all cause larger errors. NTP corrects small errors by slewing (running slightly fast or slow) and large ones by stepping (jumping). Cloud providers offer more precise time services, and PTP can achieve microsecond accuracy, but applications should still treat cross-machine timestamp comparisons as approximate.

Leap Seconds and Smearing

Leap seconds have historically been inserted to keep UTC aligned with Earth's rotation. Many providers smear them over a day instead of inserting a 61-second minute; mixing smeared and non-smeared clocks produces brief skew. Timekeeping bodies have decided to stop adding leap seconds by 2035, but legacy handling persists in systems.

Logical Clocks

Vector clocks (and related version vectors) let replicated stores detect conflicting concurrent writes and keep both versions for resolution instead of discarding one.

Time-Ordered Identifiers

UUIDv7, ULIDs, and Snowflake-style IDs embed a timestamp so IDs sort roughly by creation time — good for database index locality. Across machines, the order is only as accurate as the clocks that generated them; do not use them as proof of event order. See Distributed IDs.

Ordering Events Correctly

Use a single writer or log: a database sequence, a Kafka partition offset, or a replicated log assigns an authoritative order.
Version records: increment a version number on each update and use compare-and-set, rather than comparing timestamps.
Carry causality: include the version or logical clock value a client read when it sends an update, so the server can detect stale writes.
Keep conflicts: in multi-leader or offline-first systems, detect concurrent edits and merge them (for example with CRDTs) instead of discarding one.
Use databases with sound clocks: systems such as CockroachDB and Spanner handle timestamp uncertainty internally.

Best Practices

Separate "When" From "What Order"

Record wall-clock timestamps for humans and auditing, and use sequence numbers or versions for correctness decisions.

Use Monotonic Clocks for All Durations

Timeouts, backoff, rate limits, and latency metrics should read monotonic clocks so NTP adjustments cannot break them.

Monitor Clock Offset

Alert on NTP synchronization status and clock offset on every host; large offsets cause certificate errors, authentication failures, and ordering bugs.

Store Timestamps in UTC

Keep stored timestamps in UTC with explicit offsets and convert for display. See Dates, Time Zones, and Daylight Saving.

Common Mistakes

Durations From the Wall Clock

Last-Write-Wins Across Servers

Sorting Events From Many Producers by Timestamp

FAQ

Why can't I trust timestamps across servers?

Because each server's clock drifts and is corrected independently. Even with NTP, clocks differ by milliseconds or more, and pauses or misconfiguration can make the difference much larger, so timestamps from different machines can misorder events.

What is the difference between a monotonic clock and a wall clock?

A wall clock reports the current time of day and can jump when corrected. A monotonic clock only moves forward and is designed for measuring elapsed time within a process.

What is a Lamport timestamp?

A logical counter incremented on each event and updated on message receipt to exceed any received value. It produces an ordering consistent with causality without relying on physical clocks.

What is a hybrid logical clock?

A timestamp combining physical time with a logical counter, so values stay close to real time while still respecting causality. Distributed databases use them to order transactions across nodes.

Are UUIDv7 IDs ordered?

They sort by their embedded millisecond timestamp, which is useful for index locality. Across machines, their order reflects each machine's clock and is only approximate, so they should not determine correctness-critical ordering.