Microsoft’s Executive Vice President of Copilot, Agents and Platform, Charles Lamanna, explained on Monday that the massive outage of Microsoft 365 services was caused by a latent defect that appeared in a July update, together with higher user load and reduced server capacity. The disruption, which affected more than 80 % of the devices, occurred between Thursday and Friday of last week.
According to Lamanna, the defect caused a lock contention within an internal process that, under certain load conditions, caused the servers to become overloaded and stop responding. “The defect was introduced in an update that began rolling out in July and remained for more than three months without causing visible issues,” he said. The bug did not manifest in the company’s test labs because the hardware configurations used there were less powerful than those in production, which would explain why it was only detected on machines with more cores and newer hardware.
The second factor that contributed to the outage was the increased load. Lamanna noted that about fifteen hours before the outage, Teams had switched its gateway to the one also used by Exchange Online email. “This transition increased the likelihood that the network gateways would experience the contention issue,” he explained.
The gateway capacity was compromised
At the same time, almost 14 % of the network gateway server capacity was offline due to an unrelated issue that occurred the day before. The combination of these three factors—defect, load, and lack of capacity—triggered the latent failure and left the servers unresponsive. Traffic was automatically redirected to other machines, causing a cascading effect that manifested at the worst moment in an outage that affected 80 % of the servers.
The recovery, according to Lamanna, took longer than usual because, when the servers came back online, the pending traffic overwhelmed them and caused them to stop responding again. The company promised to provide more details about the incident throughout the week.
Source credits: deultimominuto.com
Note generated automatically from the sources cited in the text. Found an error? Email us at [email protected] and we'll fix it.
