A buggy update to CrowdStrike Falcon made Windows PCs and servers crash —worldwide.

BSODs Beyond Belief

It all started in Australia. G’day, David Hollingworth: ABC, Foxtel, the Commonwealth Bank and many more have been impacted by blue screens of death on Windows PCs

“Designed to protect systems”
A swathe of Australian websites and businesses are reporting that Windows PCs are mysteriously crashing [and] many devices are not recovering. … The outage, however, appears to be global, and … is causing massive issues with airlines all over the world: … American Airlines, United, and Delta have all asked … for a global ground stop on all flights.
…
CrowdStrike’s support portal … suggests the issue lies with CrowdStrike’s Falcon Sensor platform, which is designed to protect systems from malicious attacks. … “CrowdStrike Engineering has identified a content deployment related to this issue.”

And then it hit Europe. As Manish Singh reports: Banks, airlines, airports report widespread outages across the globe

“Unable to restart”
Businesses worldwide are experiencing outages … in what has already become one of the most widespread IT disruptions in recent years. The problems are affecting companies … from banks, food chains and brokerage houses, to news organizations, railway networks and airlines. … The London Stock Exchange, Edinburgh airport, and Ryan Air reported facing disruptions.
…
Security experts have pointed fingers at the security firm CrowdStrike for the outages. … The firm’s software is used widely across enterprises for managing security on Windows devices and servers. … Customers have reported being unable to restart their computers.

Confusingly, there was also a big Microsoft Azure outage last night. Maria Ponnezhath, Shivani Tanna, Chandni Shah, Abinaya Vijayaraghavan and Shubham Kalia tag-team to say this: US carriers ground flights citing communication issues

“Cloud services outage”
Frontier Airlines, … Allegiant and SunCountry … reported outages that affected operations. … Frontier said earlier that a “major Microsoft technical outage” hit its operations temporarily, while SunCountry said a third-party vendor affected its booking and check-in facilities. … Allegiant said its “website is currently unavailable due to the Microsoft Azure issue.”
…
Microsoft said its outage started at about 6 pm ET on Thursday, with a subset of its customers experiencing issues with multiple Azure services in the Central U.S. region. … Separately, Microsoft said it was investigating an issue impacting various Microsoft 365 apps and services.
…
Major U.S. carriers including American Airlines, Delta Airlines and United Airlines issued ground stops on Friday morning citing communication issues, less than an hour after Microsoft resolved its cloud services outage. … Allegiant Air too grounded flights.

Are the two related? One general assumption, as knwny thinks, is yes:

Blame CrowdStrike and not Microsoft. … This seems to be due to an issue with CrowdStrike which is causing BSODs in Windows machines.
…
Meanwhile, the world continues to marvel at the irony of a cybersecurity company taking down the systems around the world due to a botched update.

But who’s to blame? BLKNSLVR is ready to fight someone:

CrowdStrike are going to cop a potentially existential-threatening amount of blame. An application shouldn’t be able to do this kind of damage. [But] maybe CrowdStrike were unlucky enough to have accidentally discovered a bug … (i.e., it’s a Windows bug, maybe more so than … CrowdStrike).
…
There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it’s put into production? … (Raised fists towards the sky.)

DevOps FAIL? Meetch dev’ed this opinion and pushed it to prod:

I get that this software is supposed to save us from emerging threats, and that those threats come thick and fast. However, I think the biggest issue has been blindly allowing the CrowdStrike agent to update everywhere without vetting it on some test servers (or perhaps a one or two of servers in each server farm) first for a day or two.

Is there a workaround? Only one that requires touching each PC, it seems. Here’s CrowdStrike:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

Yikes. Let’s hear it for the poor IT folks. Julian Poyntz is glad he’s out of it:

When I was looking after AV and patching, if something like this happened, nearly all users were in an office. … Trying explaining over the phone to a … Remote/WFH … user how to get into some of this, with Bitlocker rolling in for a bit of fun and how to fix? … Glad that is behind me.

Why did it start in Australia? Deborah Pickett reminds us that timezones exist:

Dear CrowdStrike, and all Americans:
Please stop deploying on Thursdays. They are our Fridays.
Love,
UTC+everyone.

Meanwhile, dalmo3 has always been at war with Oceania: [You’re fired—Ed.]

Looks like this also took down half of New Zealand’s economy.

And Finally:

Best longform analysis of Buran I’ve seen.

Previously in And Finally

You have been reading SB  Blogwatch by Richi  Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites—so you don’t have to. Hate mail may be directed to  @RiCHi, @richij, @[email protected], @richi.bsky.social or [email protected]. Ask your doctor before reading. Your mileage may vary. Past performance is no guarantee of future results. Do not stare into laser with remaining eye. E&OE. 30.

Image sauce: Nash0h (cc:by-sa; leveled and cropped)