GitHub Down for Nearly 8 Hours: Inside the August 17, 2026 Outage

👤 Subhodip Ghosh • 📅 August 20, 2026 • 👁️ 47 views • 🔄 Updated August 21, 2026

If you tried pushing code, opening a pull request, running CI/CD pipelines, or asking Copilot for inline suggestions on Monday, August 17, 2026, you almost certainly ran into a wall. GitHub suffered a major multi-service outage that lasted nearly eight hours—**7 hours and 47 minutes**, to be exact. What began as a routine morning traffic surge in a Central US datacenter quickly escalated into a cascading global outage. From broken SAML enterprise logins to a 10x "retry storm" triggered by a latent VS Code extension bug, here is the full story of what went wrong and how GitHub's engineering team brought the platform back online. --- ## At a Glance: Impact & Key Metrics Between **13:28 UTC** and **21:15 UTC**, developers and enterprise teams worldwide faced severe service disruptions. At the peak of the incident: * **1 in 5 Web & API Requests Failed:** Core REST and GraphQL API error rates reached **~20%**. * **Half of Repository Downloads Failed:** Archive and raw content download failure rates hit **~50%**. * **Copilot Load Surge:** Copilot Token Service load exploded from a baseline of **7,000–9,000 requests per second (RPS)** up to **70,000–100,000 RPS** (~10x traffic spike). * **Enterprise Auth Outage:** SAML/OIDC, SCIM, and Team Sync authentication paths failed, locking organization members out of their private repos. * **GHEC Data Residency Impact:** GitHub Enterprise Cloud (GHEC) workflows with Data Residency failed whenever they depended on public action definitions hosted on GitHub.com. * **Core Subsystems Degraded:** GitHub Actions, Pull Requests, Issues, Pages, Webhooks, and Copilot were all impacted. --- ## Anatomy of the Cascade: How a Config Gap Triggered an Outage So how does a platform built for global scale go dark for nearly eight hours? According to GitHub's official post-mortem, the blackout was driven by four interconnected failures: ```text ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ 1. TRAFFIC SURGE (Central US Datacenter) │ │ Peak ingress traffic saturates edge load balancer network interfaces. │ └────────────────────────────────────┬────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ 2. ISTIO SIDECAR AUTOSCALING TRAP │ │ Sidecar proxy hits concurrency limits. Autoscaler tracked host CPU, missing sidecar. │ └────────────────────────────────────┬────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ 3. HAPROXY GATEWAY NODE MELTDOWN │ │ Overflow traffic crushes 4 HAProxy nodes, triggering SAML / OIDC auth path outage. │ └────────────────────────────────────┬────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ 4. VS CODE CLIENT RETRY STORM (10x Load Spike) │ │ Latent extension bug triggers retry loops (7K–9K RPS ──► 70K–100K RPS). │ └────────────────────────────────────┬────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ 5. CODELOAD BOTNET SCRAPING ATTACKS │ │ Automated scraping attacks on codeload endpoints complicate emergency response. │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ``` ### 1. The Istio Autoscaling Trap At 13:28 UTC, incoming traffic reached a new peak in GitHub's Central US datacenter, saturating edge load balancers. Under normal circumstances, Kubernetes Horizontal Pod Autoscalers (HPA) would automatically provision more pods to handle the volume. However, an Istio service-mesh sidecar proxy hit its maximum concurrency limit and stopped accepting connections. The autoscaling policy had a subtle configuration oversight: **it was watching CPU and memory consumption of the main application container, completely ignoring connection concurrency on the Istio sidecar proxy**. Because main container CPU usage looked normal, Kubernetes assumed the system was healthy and never scaled up additional pods. ### 2. HAProxy Gateway Node Meltdown With Istio sidecars rejecting new requests, unhandled connection attempts fell back onto four upstream HAProxy gateway nodes. Deprived of responsive downstream proxies, these HAProxy nodes quickly ran out of available file descriptors and connection pools. Within minutes, the main gateway authentication path collapsed, triggering global failures across SAML, OIDC, SCIM, and session logins. ### 3. The 10x VS Code "Retry Storm" As responders started redirecting failing traffic from Central US to Northern Virginia, a latent bug in the VS Code GitHub Copilot extension turned a manageable backend issue into a massive traffic surge. When authentication token refreshes timed out, the VS Code extension didn't back off or wait. Instead, it entered a tight, unthrottled retry loop. The Copilot Token Service—which usually handles a steady **7,000 to 9,000 RPS**—was suddenly bombarded with **70,000 to 100,000 RPS**. Even after backend infrastructure in Central US stabilized, this client-side retry storm kept Copilot offline for hours. ### 4. Automated Botnet Scraping Attacks Adding fuel to the fire, automated botnets launched aggressive scraping attacks against `codeload` endpoints during the incident window, creating extra network noise and slowing down recovery efforts. --- ## Official Incident Timeline (UTC) Here is how the 7-hour and 47-minute incident unfolded on August 17, 2026: | Time (UTC) | Incident Event / Update | | :--- | :--- | | **13:28** | Traffic peak in Central US saturates load balancers; Istio sidecar concurrency caps are reached. | | **13:40** | GitHub officially acknowledges performance degradation on the status page. | | **16:36** | Engineers complete regional failover to Northern Virginia. Central US load balancers stabilize, and core Web/API services begin recovering. | | **17:30** | Git Operations experiences secondary degraded performance. | | **17:34** | Core infrastructure fixes applied; residual impact lingers across multiple services. | | **17:36** | Issues subsystem experiences performance degradation. | | **18:03** | **GitHub Actions pipeline fully recovers.** | | **18:11** | Core component fixes verified; sporadic authentication failures continue. | | **18:23** | **Git Operations degradation is fully mitigated.** | | **18:48** | API requests experience brief secondary availability degradation. | | **19:01** | **API Requests return to normal operation.** | | **19:13** | Gateway team partially disables token retries to curb traffic loops. | | **20:08** | Copilot authentication issues actively investigated (GitHub CLI and GitHub App usage unaffected). | | **20:22** | **Issues subsystem returns to normal operation.** | | **20:45** | Emergency mitigations applied to remaining Copilot authentication edge cases. | | **21:02** | **Copilot Token Service fully recovers** as retry rate-limiting takes effect. | | **21:15** | **Incident officially marked as resolved across all GitHub subsystems.** | --- ## How GitHub Engineers Restored the Platform Resolving the cascading failure required a combination of infrastructure shifts and client traffic suppression: 1. **Regional Failover:** Shifted active ingress traffic from Central US to Northern Virginia while Central US load balancers were reconfigured. 2. **HAProxy Cluster Reset:** Paused execution on the four exhausted HAProxy nodes simultaneously to flush deadlocked connection pools and re-establish clean authentication paths. 3. **Suppressing Client Retries:** Deployed an emergency gateway PR to temporarily suppress token retries, while applying HTTP 403 blocks on Copilot token requests at the load balancer level. This broke the VS Code retry loop and allowed engineers to safely ramp traffic back up region by region. --- ## GitHub's Follow-Up Action Items In their official incident report, GitHub committed to the following preventative measures: * Correcting Kubernetes autoscaling policies to account for service-mesh sidecar concurrency. * Conducting a full audit of Istio request, concurrency, and capacity limits across all microservices. * Reviewing retry limits and backoff rules across edge gateways and client SDKs. * Patching VS Code extension retry behavior for Copilot authentication tokens. * Upgrading load-balancer capacity monitoring and regional failover safeguards. --- ## Official Reference & Status Links **Source:** GitHub's official incident report and status updates for the August 17, 2026 GitHub.com outage. - [Incident with GitHub.com (August 17, 2026)](https://www.githubstatus.com/incidents/zkxwbgr0cnmx) - [GitHub Status Portal](https://www.githubstatus.com/)