Global crash of 8.5 million computers due to CrowdStrike antivirus caused 40 KB update file

The CrowdStrike company published the results of the study a post incident review (PIR) related to a faulty update that disabled 8.5 million computers. The root cause of the problem was a program for testing that failed to properly test the content update, resulting in a widespread update on Friday that crashed on millions of machines.

CrowdStrike says it will continue to thoroughly test its product updates, improve error handling, and implement phased deployments to avoid similar disasters in the future.

CrowdStrike’s Falcon software is used by companies around the world to combat malware and security breaches on millions of Windows computers. On Friday, the company released a configuration update for the Falcon that was supposed to collect telemetry about possible new threat methods. Normally these updates go through without a problem, but this update caused Windows to crash.

CrowdStrike typically releases configuration updates in two ways: Sensor Content, which updates Falcon only at the Windows kernel level, and Rapid Response Content, which updates malware detection behavior. It was the small 40KB Rapid Response Content update that caused the problem. Last week, the company released two such updates, which turned out to be insufficiently tested.

To prevent similar incidents in the future, CrowdStrike promises to improve its rapid response content testing, including local developer testing, content update and rollback testing, and stress testing. The company will also test the stability and interface of Rapid Response Content and update its cloud validation tool.