Unreliable Clocks
Summary: Local clocks in distributed systems that can drift or jump, making them unreliable for precise event ordering.
Sources: chapter8
Last updated: 2026-04-17
Each node in a distributed system has its own clock, usually a quartz crystal oscillator. These clocks are not perfectly accurate and drift at different rates (source: chapter8, p. 287).
Types of Clocks
- Time-of-day clocks: Return the current date and time according to some calendar (e.g., UTC). They are usually synchronized with NTP (Network Time Protocol). However, they can jump backward if the local clock is too far ahead of the NTP server, making them dangerous for measuring durations or ordering events (source: chapter8, p. 288).
- Monotonic clocks: Suitable for measuring durations (intervals). They are guaranteed to always move forward. The absolute value of a monotonic clock is meaningless; only the difference between two readings matters (source: chapter8, p. 288).
The Danger of Last Write Wins (LWW)
Using time-of-day clocks to order writes across different nodes can lead to data loss. If two nodes have slightly different clock offsets, a write that happened later (causally) might be assigned an earlier timestamp and be silently discarded. This is a common issue in databases that use last-write-wins (source: chapter8, p. 291).
Clock Synchronization and Confidence Intervals
Clock synchronization (via NTP) is only as good as the network delay. It is better to think of a clock reading as a confidence interval rather than a single point in time. Google Spanner’s TrueTime API explicitly exposes this interval (source: chapter8, p. 293).