Clock Synchronization in Distributed Databases — Part 2: Atomic Clocks, The Source of Truth

Part 2: Atomic Clocks, The Source of Truth

This is Part 2 of a 5-part series on clock synchronization in
distributed databases, covering CockroachDB and Aurora DSQL.

• Part 1: The Physical Clock Problem (quartz, NTP, drift)
• Part 2: Atomic Clocks, The Source of Truth (you are here)
• Part 3: The Hybrid Logical Clock, CockroachDB's Solution
• Part 4: Uncertainty Intervals, MVCC, and Transaction Restarts
• Part 5: Aurora DSQL, When You Can Assume Atomic Clocks

In Part 1 we established that quartz oscillators drift and NTP can
only correct to within 1 to 50ms, with hard limits caused by network
jitter, path asymmetry, and virtualization. That accuracy floor creates
real problems for distributed databases that rely on timestamps for
correctness.

The natural question is: where does NTP get its time from in the first
place? And is there something better?

The answer to both questions is Atomic Clocks!

What Makes a Clock Accurate

Before getting into atomic clocks specifically, it helps to understand
what accuracy actually means for a timekeeping device.

A clock is accurate when its tick rate matches a universally agreed
standard. The more stable and reproducible that standard is, the more
accurate clocks built around it can be.

Quartz oscillators use the mechanical resonance of a crystal, which
varies with temperature, manufacturing, and age. That variability is
exactly the source of drift described in Part 1.

Atomic clocks use something that does not vary at all: quantum mechanics.

The Physics, How Atomic Clocks Work

Certain elements have electrons that jump between energy states when
exposed to electromagnetic radiation at a very specific frequency. That
frequency is not a property of any particular device or sample. It is a
property of the element itself, defined by the laws of physics.

For cesium-133, that frequency is:

9,192,631,770 Hz

(9 billion, 192 million, 631 thousand, 770 oscillations per second)

This is the definition of one second in the International System of Units (SI). The SI second was redefined in 1967 to be exactly 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of cesium-133.

An atomic clock works by tuning a microwave signal until it perfectly
matches this resonance, then counting the cycles. When the count reaches 9,192,631,770, one second has elapsed.

Quartz oscillator:    drifts roughly 1 second per 11 to 12 days
Cesium atomic clock:  drifts roughly 1 nanosecond per day
                      (about 1 second per 317 years)
GPS atomic clocks:    drifts roughly 1 nanosecond per 300,000 years

The reason atomic clocks are so accurate is not superior engineering.
It is that their reference is a fundamental constant of the universe.
Every cesium-133 atom, anywhere in the universe, resonates at exactly
that frequency. There is no manufacturing variation, no temperature
sensitivity, no aging effect on the reference itself.

Atomic Clocks Define Time, They Do Not Derive It

This distinction matters and is easy to miss.

NTP borrows accuracy from somewhere else. Your server asks a stratum 1
server, which asked an atomic clock. The atomic clock does not ask
anyone. It generates time from first principles.

Your server     asks NTP server
NTP server      asks stratum 1
Stratum 1       reads atomic clock
Atomic clock    reads cesium-133 resonance
Cesium-133      IS the definition of a second

At the bottom of that chain, there is no further reference. The atom
resonance is the answer, not a pointer to another answer.

Is There One Universal Atomic Clock?

No, and this is an important nuance. There are approximately 450 atomic
clocks spread across more than 80 institutions worldwide. National
metrology institutes including NIST in the USA, PTB in Germany, NPL in
the UK, and SYRTE in France each maintain multiple clocks.

Because the cesium-133 resonance frequency is a universal physical
constant, any lab anywhere can build an atomic clock and get the same
reference frequency. You do not need access to "the one clock." The
reference is baked into physics.

In practice, even high-quality atomic clocks drift slightly relative
to each other due to local environmental factors, manufacturing
imperfections, and relativistic effects (clocks at higher altitude
tick slightly faster due to weaker gravity, which is real and
measurable).

The BIPM (Bureau International des Poids et Mesures) in Paris
solves this by aggregating readings from all 450+ clocks into a
weighted average called TAI, International Atomic Time. No single clock
is the truth. The ensemble average is the truth. If one clock drifts,
the others outvote it.

~450 atomic clocks across 80+ institutions
          |
          | BIPM takes weighted average
          v
TAI (International Atomic Time)
More stable than any single clock alone
          |
          | leap seconds added to track Earth's rotation
          v
UTC (Coordinated Universal Time)
What NTP distributes to the world

Types of Atomic Clocks

Not all atomic clocks are the same. They vary in accuracy, size, cost,
and what they are used for. Here is a practical overview of the main types.

Cesium Beam Clock (the original)

Introduced in 1955. This is the clock that defines the SI second.
Cesium atoms are heated into a gas, the gas streams out as a beam, and
the beam passes through a microwave cavity tuned to the cesium resonance
frequency.

The limitation is that atoms move through the cavity quickly, so the
interaction time is short. Think of it as trying to tune a radio while
driving past the transmitter at high speed.

Accuracy: roughly 1 nanosecond per day. Used in national standards labs
and GPS ground stations.

Rubidium Oscillator

Uses rubidium-87 atoms in a small glass cell. Much cheaper, smaller,
and lower power than cesium. Less accurate over time but very fast to
stabilize.

Cesium beam:   accurate, expensive, large, slow to stabilize
Rubidium:      less accurate long-term, cheap, small, fast to stabilize

Accuracy: roughly 1 microsecond per day. Used in GPS satellites,
telecom infrastructure, and portable timing equipment. GPS satellites
carry rubidium clocks onboard with cesium correction applied from
ground stations.

Hydrogen Maser

Uses hydrogen atoms stored in a teflon-coated bulb inside a microwave
cavity. The atoms naturally emit microwaves at 1,420,405,751 Hz as
they drop to a lower energy state. This signal is used directly as the
clock reference.

Hydrogen masers have exceptional short-term stability, better than
cesium over seconds to hours. But they drift more than cesium over
days and weeks. So they are often paired together:

Hydrogen maser:  best short-term stability (seconds to hours)
Cesium:          best long-term stability (days to years)
Combined:        use hydrogen for the smooth tick-to-tick signal,
                 cesium to correct long-term drift

Used in radio telescopes, very long baseline interferometry, and deep
space tracking.

Cesium Fountain Clock

A major improvement on the original cesium beam. This solves the
"atoms moving too fast" problem by laser-cooling the atoms to near
absolute zero (-273C) and then tossing them upward. Gravity slows them
down, they stop, fall back through the microwave cavity a second time,
and then continue falling. The atoms interact with the cavity on both
the way up and the way down, giving roughly 100 times longer interaction
than a beam clock.

Cesium beam:     atoms in cavity for milliseconds
Cesium fountain: atoms in cavity for roughly 1 second
                 100x longer = much sharper frequency measurement

Accuracy: roughly 0.1 nanoseconds per day. These are what most national
metrology institutes use today to anchor UTC.

Optical Lattice Clock

The current frontier. Instead of using microwave frequencies like all
the clocks above, these use optical frequencies, meaning visible light.

Why optical?

Cesium clock counts:     ~9 billion ticks per second (microwave)
Optical clock counts:   ~500 trillion ticks per second (light)

More ticks per second means finer divisions of time,
which means more precise measurement.

Like upgrading from a 1Hz stopwatch to a 500 terahertz stopwatch.

Atoms (strontium, ytterbium, mercury) are laser-cooled to near
absolute zero and trapped in a standing wave of laser light called an
optical lattice. A separate laser probes the atomic transition frequency.

Cesium fountain: loses 1 second every ~300 million years
Optical lattice: loses 1 second every ~15 billion years
                 (roughly the age of the universe)

Used in research labs. The optical lattice clock is expected to replace
cesium as the SI second definition around 2030.

Chip-Scale Atomic Clock (CSAC)

This is the miniaturization story. A CSAC squeezes atomic clock physics
into a chip roughly the size of a matchbox.

Size:     ~16 cubic centimeters
Power:    ~120 milliwatts (runs on a battery)
Weight:   ~35 grams
Cost:     $1,500 to $3,000
Accuracy: roughly 1 millisecond per day

Much less accurate than the others, but small enough and cheap enough
to embed in portable devices. Used in military GPS receivers (works
without GPS signal), autonomous vehicles, submarines, and remote sensors.

Summary Table

GPS and Atomic Clocks

GPS is worth calling out specifically because it is the primary delivery
mechanism for atomic clock time to the real world, including to AWS data
centers.

Each GPS satellite carries atomic clocks onboard (rubidium, with cesium
correction from ground stations). The satellites continuously broadcast
their precise timestamps along with their position. Your GPS receiver
uses timing signals from at least 4 satellites to triangulate both
location and time simultaneously.

This is why GPS receivers can serve as stratum 0 time sources. They are
receiving atomic clock time directly from satellites, with nanosecond-level
precision. Installing a GPS antenna and receiver in a data center gives
you a direct, one-hop connection to atomic clock accuracy, with no
network hops and no NTP jitter.

This is exactly what Google and AWS did.

What AWS Built: Amazon Time Sync Service

AWS recognized that its customers running latency-sensitive and
consistency-sensitive workloads needed better than public NTP. So they
built atomic clock infrastructure directly into every AWS region.

The Amazon Time Sync Service utilizes a fleet of redundant
satellite-connected and atomic reference clocks in each AWS region to
deliver current time readings of UTC.

Atomic Clocks + GPS Satellites
      |
      | physically in every AWS region
      v
Amazon Time Sync Service (Stratum 1)
      |
      | delivered over NTP or PTP
      v
Your EC2 instance
accessible at 169.254.169.123

This service is available at no additional charge to all EC2 instances
running in a VPC. It also handles leap second smearing automatically,
which matters a great deal for databases like CockroachDB.

The Nitro PHC Upgrade (2023)

In 2023 AWS added a Precision Hardware Clock (PHC) directly to the
Nitro hypervisor chip. This is a significant change. Instead of
delivering atomic clock time over NTP (which still has last-hop jitter),
the GPS-disciplined signal now goes directly from the data center
infrastructure into the hardware your VM runs on.

Before Nitro PHC:
  Atomic clock -> NTP -> your EC2 instance
  Accuracy: ~1ms (NTP jitter on the last hop)

After Nitro PHC:
  Atomic clock -> Nitro chip -> your EC2 instance
  Accuracy: ~100 microseconds (no last-hop network jitter)

AWS described this as the first time this level of time service had
been offered by any cloud provider.

ClockBound, Turning Accuracy Into Guarantees

Accurate time is useful. But for databases, what you really need is
a guaranteed bound on how wrong your clock could be. That is what
ClockBound provides.

AWS open-sourced ClockBound as a daemon that sits on top of the Time
Sync Service and does something NTP cannot: instead of returning a
single timestamp, it returns a guaranteed interval.

Standard NTP says:
  "Time is 10:00:00.123456789"
  (might be wrong, will not tell you by how much)

ClockBound says:
  "Time is somewhere between
   10:00:00.123456785 and 10:00:00.123456793"
  (guaranteed, real time is inside this range)

This bounded interval enables four things that matter for distributed
databases:

1. Definitive temporal ordering: can determine if event A definitely occurred before event B, rather than guessing
2. Concurrency detection: identifies when two events may have occurred simultaneously, which is important for conflict resolution
3. Wait optimization: calculates the minimum time to wait to guarantee a globally unique timestamp
4. Dynamic uncertainty bounds: allows a database to shrink its uncertainty window to the actual measured error rather than a static worst-case assumption

Aurora DSQL uses all of this. CockroachDB currently uses a static
500ms assumption because it cannot require ClockBound to be present
in every deployment environment.

Why CockroachDB Cannot Assume Atomic Clocks

This is the core constraint that shapes CockroachDB's entire clock
design, and it is worth being precise about it.

CockroachDB is open source and designed to run in environments it has
zero control over:

Customer A:  AWS EC2 instances in us-east-1
Customer B:  On-premises Dell servers in a private data center
Customer C:  Google Cloud VMs
Customer D:  Azure, mixed with some bare metal co-location
Customer E:  Any commodity hardware, anywhere

None of these environments can be assumed to have GPS antennas, atomic
clocks, or a ClockBound daemon accessible to the database process.
The database cannot be designed around hardware that may or may not
physically exist in a given deployment.

The word "impossible" sometimes gets used here, but it is slightly
imprecise. A more accurate statement is: CockroachDB cannot assume
atomic clock infrastructure. A specific CockroachDB deployment running
on AWS with ClockBound configured could benefit from tighter bounds.
But the design must work correctly without any of that, on commodity
hardware with nothing better than standard NTP.

This is the fundamental design fork between Spanner, Aurora DSQL, and
CockroachDB:

Google Spanner:

• Runs only on Google infrastructure
• Atomic clocks + GPS in every data center
• TrueTime API, proprietary, ~1 to 7ms uncertainty bound
• Commit-wait adds ~1 to 7ms to every write

Aurora DSQL:

• Runs only on AWS
• Atomic clocks + GPS + Nitro PHC in every region
• ClockBound API, open source, ~microsecond uncertainty bound
• Commit safeguard instead of commit-wait

CockroachDB:

• Runs anywhere
• Whatever the OS clock gives, NTP at best
• Static 500ms uncertainty assumption
• Handles uncertainty with transaction restarts instead

CockroachDB had to invent a mechanism that achieves correctness using
only the lowest common denominator available in any environment: a quartz
oscillator corrected by NTP. That mechanism is the Hybrid Logical Clock,
which is what Part 3 covers in full detail.

Where We Stand After Part 2

• Atomic clocks define time from quantum physics. They do not borrow it from anywhere else.

• The SI second is literally defined by cesium-133 resonance. Every cesium atom in the universe gives the same frequency.

• ~450 atomic clocks worldwide feed into TAI via the BIPM. TAI plus leap second adjustments gives us UTC.
  UTC is what NTP distributes.

• AWS built atomic clock and GPS infrastructure into every region. ClockBound wraps that into a bounded uncertainty API. Aurora DSQL uses ClockBound to achieve microsecond-level certainty.

• CockroachDB cannot assume any of this infrastructure.
  It must work correctly on commodity hardware with plain NTP.
  That requires a completely different approach.

In Part 3, we get into that approach: the Hybrid Logical Clock. What it is, how it works, why the logical counter is the most important part, and how it propagates through the cluster so that every node eventually knows the highest timestamp seen across the entire system.

References

• Kulkarni, S., Demirbas, M., et al. Logical Physical Clocks and Consistent
  Snapshots in Globally Distributed Databases (2014).
  http://www.cse.buffalo.edu/tech-reports/2014-04.pdf

• BIPM: International Bureau of Weights and Measures.
  https://www.bipm.org/en/time-faq

• AWS Blog: Keeping Time With Amazon Time Sync Service.
  https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/

• AWS Announcement: Amazon Time Sync Service Now Supports
  Microsecond-Accurate Time (2023).
  https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-time-sync-service-microsecond-accurate-time/

• AWS EC2 Docs: Precision Clock and Time Synchronization on EC2.
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

• AWS Blog: Everything You Don't Need to Know About Amazon Aurora
  DSQL, Part 5: How the Service Uses Clocks.
  https://aws.amazon.com/blogs/database/everything-you-dont-need-to-know-about-amazon-aurora-dsql-part-5-how-the-service-uses-clocks/

• CockroachDB Blog: Living Without Atomic Clocks: Where CockroachDB
  and Spanner Diverge.
  https://www.cockroachlabs.com/blog/living-without-atomic-clocks/

• Corbett, J. C., et al. Spanner: Google's Globally Distributed
  Database (2012).
  https://research.google/pubs/pub39966/

Previous: Part 1, The Physical Clock Problem
Next: Part 3, The Hybrid Logical Clock, CockroachDB's Solution