Split Brain
Summary: A scenario in a distributed system where two nodes believe they are the leader simultaneously, potentially leading to data corruption or loss.
Sources: chapter5, chapter8
Last updated: 2026-04-18
Split brain occurs when a network partition divides a cluster, and nodes on both sides of the partition elect a new leader or continue to act as the existing leader.
Consequences
- Data Inconsistency: Both “leaders” accept writes independently, causing the database state to diverge.
- Data Loss: When the partition is resolved and one leader is demoted, its unique writes may be discarded (source: chapter5).
- Broken Invariants: If the system relies on a single leader for correctness (e.g., uniqueness constraints), split brain can break these assumptions.
Prevention
- Quorum-based Voting: Requiring a majority (quorum) to elect a leader ensures only one side of a partition can make decisions (source: chapter9, p. 367).
- Fencing Tokens: Using monotonically increasing tokens (like epoch numbers) to reject requests from nodes with stale leases (source: chapter8, p. 301).
- STONITH (Shoot The Other Node In The Head): A hardware-level approach where one node physically powers off the other.