A faulty CrowdStrike update file took down Windows-based PCs worldwide for some hours last night disrupting airlines and other businesses. The impact of this global event was major, to say the least as airlines from around the world had to ground planes. The faulty CrowdStrike update file rolled out to Windows hosts did not affect Mac or Linux machines. Meanwhile, as reported by the BBC who checked Downdetector for outages, it was discovered that many major services were struggling to function, for a short time. BBC noted that outage report spikes have dropped off significantly but because Downdetector relies on users to self-report there could be many, many more not getting documented.
Per BBC:
“But the drop-off from that spike shows that very few people are now reporting issues with many services – of course, that’s just a snapshot of some of the companies which seem to be affected.”
The fallout from this event continues to spread and while some airlines are recovering quickly others are still stuck on the ground. Reports from Australia indicate cancelations for the rest of the day have happened. Microsoft continues to provide Azure status updates advising users running the CrowStrike Falcon agent to restart their VM machines. Some users have reported that it can take as many as 15 restarts to get their machines back up and running.
Per Microsoft:
“We’ve received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.”
CrowdStrike’s Falcon security software was reportedly causing BSODs on Windows machines essentially rendering them unusable but a workaround involving a Safeboot to delete the faulty file was quickly discovered. However, since then an updated file has been deployed and CrowdStrike has issued a statement regarding the matter.
Official Statement regarding Sensor Update:
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”
Discussion (18 replies)
Join Discussion →How are they able to roll out an update when the server in question BSODs on boot? I've got one system at a place that I can't even deploy the "delete driver files" fix because the machine is also running Bitlocker. The drive isn't available to Safemode or Recovery Command Prompt.
Microsoft is saying, like Dory says about swimming, to just keep rebooting - can take up to 15 reboots to get to the point it can replace the file.
Otherwise, you'll need the bitlocker key and mount the volume somewhere else to nuke the file....
You can also boot into safe mode with a local admin account and nuke the file that way.
All is good. I was able to do just that Dan.
Question is how and why they deployed said update so perfectly simultaneosly.
This is why you don't run auto updates on your critical infrastructure.
Also clear that cloud strike doesn't either or hey would have caught this!!!
The Y2K we never had. Thankfully I never even heard of crowdstrike before this.
I suppose we were due for a worst-case scenario event but it does seem odd that internal testing didn't catch this.
I'm never one for government intervention but anybody else think it's a problem when 1 company offers a super-essential service 80% of the world's businesses and can instantly disable them with just a keystroke?
It isn't a new problem. That's pretty much exactly why anti-trust laws exist, and have since 1890.
Companies have just been able to maneuver faster than the law can keep up. No surprise I guess, since Wall Street only seems to reward growth, not performance.
Crowdsrike bypasses any staging rules you set in the application with certain updates. Hence why everyone who uses it was effected the same way.
and why I would NEVER EVER want this software.
Defending the OS should be your LAST line of defense. A good network firewall/IDS/IPS setup and making sure your secure stuff has to go through less critical servers BEFORE reaching your secure environment are all steps that need to be taken.
People standing up VM's or ANY infrastructure DIRECTLY on the internet or right behind a dumb NAT deserve to be taken down. Everyone else needs to run software that THEY can control NOT the vendor.
Sadly with so many orgs having 3rd party support some not even on the same continent as their infrastructure are going to have TITANIC gaping holes that let any nerdowell in. INSIST on security... Design for it. But do it intelligently.
This Monday we got an email saying all critical vulnerabilities need to be patched in 48 hours or less. We are actively shutting that down for our critical infrastructure. ALL patches get tested FIRST in dev... going through burn in.... then staged into Prod. NOTHING skips it. Well see if they FORCE the bad decision as I am sure many of you have had to deal with.
What burns me is you have new leaders come in... agree on a metric for measurement. Then meet that metric DAMN EVERYTHING ELSE. Get their bonus, stick around for another year or two then leave for the next company while the company they just left has to scramble to detangle the rats nest of stupid metric based decisions that were left behind.
Right, I know about anti-trust laws but Crowdstrike isn't a monopoly so anti-trust wouldn't apply, at least not how I understand it.
I don't think there's anything nefarious, anti-competitive or criminal going on here....more of a liability issue really. Half the world grinding to a halt because one company sent out one bad update should be concerning.
I don't think the government should step in a force companies to use multiple vendors or anything but surely there could be some type of backup plan in case this happened again.
Oh their will be right after crowdstrike is sued into the ground.
Depends on how you define "monopoly" - which seems to vary depending on which industry you are in and who is in political power at any give time.
If you just use the strict definition of the word - yeah, there is competition for CrowdStrike, so there are options.
But...
Commanding a sufficiently high percentage of the marketplace would imply there is not meaningful competition and, even though competitors exist, they do not have a meaningful economic impact and/or are not viable competitive choices
You know - kinda like Microsoft has been with Windows and Office