For nearly eight hours, GitHub vanished from the workflows of millions of developers. Pull requests stalled, Actions jobs hung, and Copilot responses crawled to a halt. The outage wasn’t just an inconvenience, it was a reminder that even the most critical cloud services can buckle under the right combination of misconfigurations and unexpected traffic spikes.
The domino effect starts small
The trouble began with an Istio sidecar hitting its concurrency limit in GitHub’s Central US data center. Load balancers, expecting autoscaling to kick in, instead found themselves drowning in requests. The problem? The scaling policy only watched the host service, not the sidecar. Without visibility into the bottleneck, the system kept accepting traffic it couldn’t handle.
Retries turned a glitch into a storm
The situation worsened when optimistic retry logic kicked in. Internal services, sensing delays, kept resending requests. Then VS Code joined the chaos. A latent bug in the editor’s retry mechanism amplified Copilot token traffic by a factor of ten. What should have been a minor slowdown turned into a full-blown outage, with some services taking over seven hours to recover.
The fix was messy and manual
- Engineers reduced gateway retries through a code change, buying time to stabilize the system.
- Load balancers were reconfigured to reject Copilot Token Service requests with HTTP 403 responses, cutting off the traffic surge.
- Scraping attacks on codeload endpoints added another layer of noise, complicating recovery efforts.
Most services came back online within three hours, but Copilot remained sluggish until nearly the end of the outage. The delay wasn’t just technical, it highlighted how deeply integrated GitHub has become in modern development workflows. When it stumbles, entire teams grind to a halt.
What GitHub plans to change
- Correct autoscaling policies to monitor sidecar concurrency limits, not just host services.
- Review and adjust retry limits across internal services to prevent amplification loops.
- Audit Istio concurrency settings to catch similar blind spots earlier.
- Work with the VS Code team to patch the retry bug that turned a slow endpoint into a traffic multiplier.
The bigger picture for developers
This outage isn’t an isolated incident. GitHub has faced a string of disruptions over the past year, from Actions failures to repo download errors. For teams that rely on GitHub as their single source of truth, the question isn’t if another outage will happen, it’s when. Alternatives like Cursor and OpenAI’s code hosting tools are already positioning themselves as more reliable options, and startups are capitalizing on the frustration.
GitHub won’t be the default solution going forward. We’re looking at a much more bifurcated ecosystem, which is good and bad.
The shift isn’t just about redundancy. It’s about recognizing that no single platform can be treated as infallible. Teams that build resilience into their workflows, whether through local backups, mirrored repos, or fallback CI systems, will weather the next outage with far less pain.
Lessons for the rest of us
GitHub’s postmortem offers a few hard-won lessons for anyone running distributed systems. First, autoscaling policies need to account for every component in the request path. A single blind spot can cascade into a full-blown failure. Second, retry logic should never assume that more requests will fix a slow service. Exponential backoff and circuit breakers exist for a reason. Finally, external integrations, like VS Code’s Copilot plugin, can turn a minor issue into a major incident if they’re not tested under failure conditions.
The next time GitHub goes dark, the impact won’t be as severe if teams have already planned for it. The real failure isn’t the outage itself, it’s being caught unprepared when it happens.
This post was originally published on my site. Read the full article and more →
