Microsoft outage explained: Tech expert tells why you lost access to your email and Teams
A health check gone wrong that led to Microsoft servers going down like dominoes and then a laborious process to bring services back online – that’s an expert’s explanation for the global tech outage that crippled Kiwi businesses and government agencies.
The outage reportedly hit Microsoft products in 74 countries and territories yesterday, according to tracker Downdetector.
New Zealand’s visa processing was affected, settlements of house sales stalled, court appearances were thrown out of whack and local offices were left without email and access to Teams.
Microsoft was able to resume most services in New Zealand by this morning, with tech expert Paul Stenhouse telling Newstalk ZB the company had ruled out malicious action or a cyber attack.
Instead, the “latest thinking” based on what information has been released publicly suggested a routine health check on a Microsoft server triggered the incident, Stenhouse said.
The cloud is “really just ... someone else’s computer, someone else’s server”, Stenhouse said.
Microsoft runs “hundreds of thousands if not millions” of them around the world, he estimated, and each one runs regular checks on itself.
In effect a question is asked: “Are you good today server?”
The latest thinking was that a health check was tripped and servers reported themselves as being unwell.
“I’m not feeling too good. I’m gonna shut down, go and send your traffic to someone else.”
That set off “a bit of a domino effect”, Stenhouse said.
Microsoft had reported the overall daily traffic to the servers was normal, he said.
But as servers began shutting down and rerouting traffic, more and more servers became overloaded.
Each time servers came back up the same thing happened, Stenhouse told the Saturday Morning with Jack Tame show.
“So they were caught in this kind of vicious loop.”
The fix was to pull all the traffic off, get the servers fixed and then slowly let the traffic back on, he said.
“That is why it has been such a slow, slow process.”
Microsoft’s own explanation is that the incident “was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change”.
The outage showed how much of New Zealand’s working day now runs on someone else’s computer – and how it can have big consequences for the rest of us when it falls over, host Jack Tame and Stenhouse mused.
Will Microsoft have to pay its customers?
Everyday users probably won’t be able to seek compensation from Microsoft for the outage as they are usually signed up to terms that “aren’t very friendly towards you”, Stenhouse said.
But business and enterprise customers typically have a service level agreement with tighter clauses.
That can include promises like services having about 99.9% uptime, and this outage went past that, Stenhouse said.
“I do think Microsoft’s probably going to have to do something here.”
A Microsoft Australia-New Zealand spokesman told the Herald on Friday it had no comment on compensation at this stage.
Tame said the incident highlighted how reliant modern society is on technology.
Stenhouse said it was an example of why data sovereignty was becoming a key security issue for nations.
Some European Governments, particularly their armed forces, have spent the past couple of years moving email and communications off Microsoft and Google and on to free, open-source systems, Stenhouse said.
However, that could come with a catch as we move into the AI age, he said.
With “all of these crazy AI hacks” on the rise, running your own systems is hard, he said.
“So at the moment, the best thing you can do for your business, unfortunately, is actually use someone like a Microsoft.
“You just have to rely on their availability and that they’re gonna get the systems sorted out.”
‘Needs to be on a T-shirt’
Stenhouse praised how Microsoft posted regular status updates to its account on X about every hour.
But it did use some confusing wording in those updates, Stenhouse mused.
One X post read: “We’re aggressively pursuing all pathways to reduce impact”, according to Stenhouse, who said that might have caused more alarm rather than relief among those looking for updates.
His favourite update, quoting it from his own memory, was: “Our actions thus far have not provided the anticipated relief.”
“Which I think probably needs to be on a T-shirt,” Stenhouse said in humour.