Disaster Recovery on Alibaba Cloud: Designing for Failure

Part 8 of the Alibaba Cloud Engineering Lab Series.

Architecture

Zone A (primary)              Zone B (standby)
  ECS + ACK node pool           ECS + ACK node pool (min-size 0, scales on failover)
  RDS primary  ─────replication─────▶ RDS standby
  OSS bucket (versioning + cross-region replication enabled)

A DR plan that has never been tested is a hypothesis, not a plan. This lab runs three controlled failure experiments and records what actually happened, not what the architecture diagram implies should happen.

Before the how, the what — the two terms every DR conversation revolves around:

• RTO (Recovery Time Objective) — how long you're willing to be down before service is restored. An RTO of 5 minutes means: from the moment something fails, you have 5 minutes to get back to serving traffic before it's considered a target miss.
• RPO (Recovery Point Objective) — how much data you're willing to lose, measured in time. An RPO of 15 minutes means your backups/replication are frequent enough that, worst case, you lose the last 15 minutes of writes — not that recovery itself takes 15 minutes.

They're independent numbers answering different questions — "how long until we're back up" versus "how much data are we willing to lose" — and a real DR test measures both separately, which is exactly what this lab does.

Problem

The RTO/RPO targets on the whiteboard were: application failure recoverable in under 1 minute, instance failure in under 5 minutes, data loss recoverable to within 15 minutes of the incident. None of these had ever been tested end-to-end.

Implementation & Experiment

Experiment 1 — Container crash:

kubectl exec -it api-pod-xyz -- kill 1

Experiment 2 — ECS instance failure (simulated by force-stopping the instance backing an ACK node):

aliyun ecs StopInstance --InstanceId i-xxxxxx --ForceStop true

Experiment 3 — Data loss (deleted an OSS object that versioning should protect):

ossutil rm oss://prod-data-bucket/critical-report.json

Failure / Challenge

Experiment 2 was the one that broke the assumption: the ACK cluster autoscaler took 6 minutes 40 seconds to detect the missing node, provision a replacement, and reschedule the evicted pods — well past the 5-minute target. The gap was the node pool's health-check and scale-up interval defaults, tuned for cost-conscious slow scaling, not fast recovery.

Solution

Tightened the node pool's scale-up responsiveness and added a pod disruption budget plus a second, smaller standby node kept warm specifically to absorb single-node failure without waiting for a fresh node to boot:

resource "alicloud_cs_kubernetes_node_pool" "standby_buffer" {
  cluster_id     = alicloud_cs_managed_kubernetes.primary.id
  node_pool_name = "standby-buffer"
  vswitch_ids    = [alicloud_vswitch.zone_a.id]
  instance_types = ["ecs.g6.large"]
  desired_size   = 1 # always-on buffer capacity
  scaling_config {
    min_size = 1
    max_size = 1
  }
}

Re-ran Experiment 2: recovery dropped to 58 seconds — the buffer node absorbed the evicted pods immediately, while a replacement node provisioned in the background to restore the buffer.

Results Table

The container-crash and data-loss scenarios met target on the first try. The node-failure scenario didn't — and wouldn't have been caught without actually running the experiment instead of trusting the architecture diagram.

Cost / Performance

The always-on standby buffer node adds a fixed cost (~1 extra ecs.g6.large node, ~$95/month) purely for recovery-speed insurance. Whether that tradeoff is worth it depends entirely on what a 6-minute outage actually costs your business — for a payments platform, obviously yes; for an internal reporting dashboard, probably not.

Lessons Learned

• An RTO target that has never been tested is a guess dressed up as a commitment — run the failure, measure the real number.
• Autoscaler defaults are tuned for cost efficiency, not recovery speed — the two goals are in tension, and DR requirements should explicitly override cost defaults where they conflict.
• Not every recovery scenario needs the same investment — spend the always-on buffer cost only on the failure modes whose business impact justifies it.

GitHub Repository: alibaba-cloud-disaster-recovery-lab — the three experiment scripts, Terraform for the DR stack, and the standby-buffer fix, ready to run.

Disaster Recovery · Alibaba Cloud · RTO · RPO · Reliability · Chaos Engineering

Originally published on my portfolio.