New Delhi | More than 80 per cent of servers behind Microsoft's networking gateway were unresponsive at the peak of last week's global outage of Outlook and other Microsoft 365 services, the company has said, blaming a hidden software defect introduced in July.
Three factors together caused the outage of the networking gateway that sits in front of many Microsoft 365 services, Charles Lamanna, Microsoft's Executive Vice President of Copilot, Agents and Platform, said in a post on X on Monday.
"We are deeply sorry for the impact to all our customers and recognise the impact that this has on their normal business and operations," he said.
The first factor was a latent defect that caused servers to become overwhelmed and unresponsive under certain load conditions. It came from a fix prepared specifically for Microsoft 365 networking gateway servers and does not apply to broader Windows deployments, Lamanna said.
The initial change began rolling out in July and was in place worldwide for more than three months without any impact, he said. The company's load testing did not catch the problem because the tests focused on weaker hardware, while the defect is worse on newer machines with more processing cores.
Second, about 15 hours before the outage, Microsoft Teams shifted from an alternative gateway to the primary gateway that also supports Exchange Online, the company's email platform. This increased the load on that gateway and changed its traffic patterns. Lamanna said such a shift is part of normal operations.
Third, nearly 14 per cent of the gateway's server capacity was out of service at the time because of an unrelated issue from the previous day.
Taken together, the higher load and reduced capacity caused the gateway to hit the latent bug from July, and requests built up until servers ultimately stopped responding. This happened during the peak window of traffic in the United States, he said.
"When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient.
"However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80 per cent of servers were unresponsive," Lamanna said.
The safeguards that normally separate different parts of the gateway did not stop the failure because of a separate issue, which allowed the impact to spread globally, he added.
Services such as Teams, OneDrive, SharePoint and Office apps were able to move off the faulty gateway and were restored earlier, with Teams chats and meetings working for most of the outage. Exchange Online cannot do so, Lamanna said.
Recovery took longer than usual because servers returning to service were immediately overwhelmed by waiting traffic and became unresponsive again.
The company's automated recovery processes were not designed for such a broad failure, and since the defect was several months old, its usual methods could not quickly reverse the change, he said.
Engineers then developed an update that returned the code to its previous state. They took groups of servers out of service, applied the change and restarted them before bringing them back.
A final post-incident review with more details will be provided later this week, Lamanna said.
The outage, which began late on Thursday, had hit Indian airlines, financial services firms, and businesses. Domestic airlines IndiGo, Air India and Akasa Air reported disruptions to email-based customer support, affecting their ability to receive or respond to some passenger queries. Air India temporarily rerouted email services to ensure customer communications continued reaching the airline, while flight operations were not reported to have been affected.
Financial services firms also faced difficulties accessing research reports and conducting client interactions.
The disruption primarily affected Microsoft 365 features that interact with Exchange Online, the company's cloud-based email platform. Users were unable to access mailboxes through Outlook on the web and desktop applications, and email delivery was delayed.
At its peak, the outage generated more than 1.3 lakh reports of Outlook problems on the outage-tracking website Downdetector, with disruptions reported across 72-74 countries.