How OceanBase uses Multi-Paxos, log streams, replica types, and leader election to achieve zero data loss and fast failover
Key Takeaways
OceanBase uses Multi-Paxos to achieve zero data loss by ensuring commit logs are persisted on a majority of replicas before success is returned.
Log streams, replica types, and arbitration services help OceanBase reduce replication overhead, scale reads, and support cost-efficient cross-data-center disaster recovery.
Leader election, lease mechanisms, OBProxy routing, and automatic replica rebuilding enable fast failover, helping OceanBase recover within seconds after failures.
1. What Is Paxos, and How Does It Help OceanBase Prevent Data Loss?
If you are not familiar with distributed systems theory, Paxos may sound like a mysterious concept from a computer science textbook. A useful mental model is a roomful of people trying to agree on one outcome — except some people may arrive late, go offline, or even give incorrect responses. The system still has to reach one decision, and that decision must not be overturned later.
That is the problem distributed consensus is designed to solve. The Paxos consensus protocol, proposed by Turing Award winner Leslie Lamport, is a classic solution to this problem. Its core idea is simple but powerful: a decision is considered valid only after it has been accepted by a majority of participants, also known as a quorum.
Why a majority? Because mathematically, any two majorities must have at least one member in common. In other words, the system cannot end up in a permanent split where “one half says A” and “the other half says B.” This prevents split-brain at the root — a dangerous scenario in which two replicas both believe they are the leader and independently accept write requests at the same time.
In OceanBase, data is replicated and distributed across different physical machines, or nodes, in the form of replicas. Database modifications generate commit logs. These logs must not only be written to the current leader replica, but also be durably persisted on a majority of replicas before a successful commit response can be returned to the client.
So even if a single disk fails, or even if the physical node where the leader replica resides goes down, as long as a majority of replicas are still alive on other machines, the data has already been accepted by a quorum and persisted on more than one machine. It will not be lost because of a single-machine failure.
This is an important reason why OceanBase can achieve zero data loss in disaster recovery scenarios — that is, Recovery Point Objective (RPO) = 0. Its reliability is not built on the assumption that one special machine will never fail. Instead, it is built on the majority mechanism: once a change is accepted by a quorum, it becomes part of the durable history of the system.
In a real distributed database, however, there is another challenge. If every partition maintained its own independent log and replica group, then a large number of incoming transactions would lead to many separate log synchronizations and quorum acknowledgments. The network and coordination overhead would grow quickly.
To address this, OceanBase introduces log streams as an internal abstraction. Multiple partitions can be organized under the same log stream and share one set of replicas and one log replication mechanism. Changes generated by different partitions can then be synchronized along the same log replication path. This reduces replication overhead and improves overall system throughput.
OceanBase also optimizes the synchronization protocol itself. In classic Paxos, synchronizing each log entry may require two rounds of remote procedure calls (RPCs) to reach consensus. OceanBase uses a Multi-Paxos-style protocol: after a stable leader replica has been elected to handle writes, log synchronization in the steady state can usually be completed with one quorum acknowledgment. This allows OceanBase to maintain majority-based durability while keeping write latency under control.
2. How Multi-Paxos Compares with Strong Primary-Standby Sync and Raft
Traditional primary-standby architectures often use strong synchronous replication to avoid data loss. In this model, the primary database must wait until the standby database has also persisted the log before returning success to the application.
This ensures that the standby has complete logs if the primary fails. But the cost is also clear: if the primary, the standby, or the network between them has a problem, the primary may become stalled or even unavailable. In other words, the system is often forced into a difficult trade-off between data reliability and service availability. Achieving both at the same time is hard.
Multi-Paxos takes a different approach. Its multi-replica model — typically three or five replicas — is naturally based on quorum decisions. As long as a majority of replicas are alive and can communicate with one another, the system can continue accepting writes and reaching consensus.
Therefore, when a minority of replicas fail — for example, one replica fails in a three-replica deployment — the system can still prevent data loss while keeping the service available. This is difficult for a strongly synchronized primary-standby architecture to achieve cleanly.
Multi-Paxos and Raft have similar goals: both use majority agreement to reach consensus. But they differ in their design priorities.
Raft emphasizes log continuity. Logs with the same index and the same term must be consistent across different nodes, and commits advance continuously. This makes leader election and log synchronization easier to understand and implement. However, when network jitter occurs, the delay of one log entry may block the commit of all subsequent log entries.
OceanBase’s Multi-Paxos allows gaps in the log and supports out-of-order acknowledgment and commit. As a result, a brief failure on an individual node does not block overall throughput. This makes the system more tolerant of network fluctuations.
In terms of horizontal scalability, Raft relies on relatively frequent heartbeat messages to maintain consistency. As the number of cluster components grows and the cluster scale increases, heartbeat communication overhead also grows, which may affect scaling efficiency.
OceanBase’s Multi-Paxos, combined with the log stream architecture, reduces the network and computational overhead required for consensus. It can therefore maintain scalability even in complex disaster recovery topologies.
3. Replica Selection: Balancing Availability, Performance, and Cost
OceanBase provides different replica types so that different workloads can balance data reliability, performance scalability, availability, and cost.
Full-featured replicas are currently the most widely used replica type. They contain complete logs and business data. Only full-featured replicas are included in the Paxos member group and participate in quorum decisions. The system’s core write path, leader election, and strong consistency guarantees all depend on how full-featured replicas are distributed.
However, simply adding more full-featured replicas to improve read performance is not always a good idea. Since full-featured replicas participate in Paxos voting, expanding the Paxos member group can increase write latency. To solve this problem, OceanBase introduces a class of non-voting observer replicas, including read-only replicas and columnstore replicas.
Read-only replicas store complete business data and catch up with the leader replica through asynchronous replication. They are suitable for workloads with relatively lower read consistency requirements. Columnstore replicas organize baseline data in columnar format and are designed for large-scale analytical queries. Because these observer replicas do not participate in quorum decisions, they can expand read capacity without adding consensus overhead to the core transaction path. This allows transaction processing (TP) and analytical processing (AP) workloads to be separated.
Multi-data-center deployment can also reduce disaster recovery costs while ensuring data reliability. For example, in a three-data-center deployment across two regions, deploying full-featured replicas in the third data center would not only incur additional storage costs, but also introduce more cross-region synchronization overhead.
To address this, OceanBase introduces the arbitration service.
Arbitration service nodes do not store business data and do not handle routine data replication tasks. As a result, they require very little compute, storage, or network resources.
Their value becomes especially important in failure scenarios. When a data-center-level failure causes some full-featured replicas to become unreachable and the system can no longer form a majority, arbitration nodes can provide the critical votes needed for the surviving replicas to re-form a majority and complete leader election. This allows the system to quickly restore write capability.
In this way, OceanBase does not need to deploy full data replicas in the third data center, yet it can still achieve cross-data-center disaster recovery and high availability at relatively low cost.
4. Leader Election and Failure Recovery Mechanisms
The majority mechanism ensures zero data loss, or RPO = 0. But after a failure occurs, another question becomes just as important: how quickly can the system recover without manual intervention?
OceanBase can achieve automatic recovery within 8 seconds — that is, Recovery Time Objective (RTO) < 8s. This capability is supported by several mechanisms working together.
- Automatic Election and Split-Brain Prevention: The Lease Mechanism
The leader replica is not statically assigned once and for all. OceanBase uses an election protocol to select a leader replica from multiple replicas, and uses a lease mechanism to ensure that only one node believes it is the leader at any given time. When the leader replica fails or a network partition occurs, the majority waits for the lease to expire before initiating a new election. This prevents the old leader from still being alive somewhere while a new leader is elected elsewhere, which would otherwise create a dual-leader situation.
With this mechanism, failure detection and failover can be completed within seconds, and the new leader can quickly take over service.
- Fine-Grained Leader Switching: Local Failures Do Not Bring Down the Entire Database
Unlike traditional databases where the entire database may depend on a single primary node, OceanBase refines the granularity of multi-replica synchronization down to the log stream level. Primary-standby switching and leader election are both performed at the log stream layer.
When a single physical server goes down, only the log streams for which that machine served as the leader are affected. In other words, only part of the data is affected. The remaining data continues to serve requests normally, and leader elections for the affected log streams can proceed in parallel.
This fine-grained design prevents a single point of failure from spreading across the entire cluster. It is also a key reason why OceanBase can reduce RTO to the seconds level: the recovery scope is limited to a local area rather than the entire database instance.
- Application-Layer Transparency: Smart Proxy Routing to the New Leader
Applications usually do not connect directly to backend database nodes, known as OBServers. Instead, they connect through OceanBase Database Proxy, also known as ODP or OBProxy. As OceanBase’s access-layer component, OBProxy is responsible for request routing and load balancing. It routes requests to the correct node based on partition information and leader/standby status.
When a leader switch occurs, OBProxy discovers the new leader through feedback or periodic refreshes and updates its routing information. As a result, applications do not need to change configurations or restart. They can continue accessing the new leader, and the failure is largely transparent to users.
- Automatic Repair After Failure: Removing Faulty Nodes and Rebuilding Replicas
OceanBase’s Root Service is the central control service of the cluster, and it is itself replicated using Multi-Paxos for high availability. Nodes report their status through heartbeats. If a node remains unreachable for an extended period, it is marked as failed and removed from the Multi-Paxos member group. Replacement replicas are then created on other healthy nodes to ensure that the majority remains intact.
This allows the system to recover automatically even from machine-level or data-center-level failures. After failover completes within seconds, OceanBase can quickly restore its high-availability and disaster recovery capabilities.
Summary
This article briefly introduced how OceanBase uses Multi-Paxos to prevent data loss and perform automatic failover in failure scenarios.
From majority-based replication and log stream design, to replica types and arbitration services, and then to leader election and routing failover mechanisms, these capabilities together form OceanBase’s high-availability architecture.
For business systems, the goal is not only to avoid data loss, but also to minimize service interruption when failures occur. Understanding the ideas behind these designs helps provide a deeper view of how OceanBase balances reliability and availability in distributed database scenarios.
