Robin’s Newsletter #318

This week

Perhaps unsurprisingly, this is all gonna be about CrowdStrike. There’s a lot to unpack. Grab your Horlicks or coffee, andlet’s get into it…

Global IT outage caused by duff CrowdStrike update

• ICYMI: There was widespread panic early on Friday morning as Windows computers started displaying the ‘blue screen of death’. Trains, planes and automobiles were all affected. Air travel in the US ground to a halt. Cash proved its worth as card payments went down. GP services and payroll providers were also affected by the outages.  TRAVEL, PAYMENTS, MORE
• A cyberattack didn’t cause the outage — and it wasn’t Microsoft’s fault either — rather a botched update by cyber security firm CrowdStrike to its endpoint protection software.
• It wasn’t a null pointer issue but a “logic error” in a ‘channel file’ used by CrowdStrike’s software agent to detect malicious behaviour on Windows devices. Computers online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC were served up the duff update. Thankfully, the prompt (78 min) identification and rollback of that change averted a more widespread impact. LINK
• The snafu puts CrowdStrike in such illustrious company as 2003’s Slammer worm, Russia’s NotPetya attack, and North Korea’s WannaCry. Around 8.5 million computers were affected. LINK
• Microsoft also suffered an outage overnight Thursday into Friday, though it has confirmed that this was unrelated to the CrowdStrike issue. 
• If you’re looking for a risk summary for your management team, check out Cydea’s risk advisory. LINK

What’s the fix?

• CrowdStrike may have identified and rolled back the affected file in just over an hour, but those who received it were reportedly stuck in a loop. The official guidance requires a manual intervention on every device. That’s hugely time-consuming and made more difficult in the world of remote working. REMEDIATION
• The steps are:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

• Some report success by repeatedly turning the computers off/on again up to fifteen times. (I pity the poor responder following that playbook in the future!) LINK

What is the impact on CrowdStrike and its rivals?

• There have been lots of comments about CrowdStrike’s future and share price. Most of those centred on the ~14.5% drop in early trading on Friday. By the end of the day, its price had rebounded slightly and had closed down by just over 10%. Yes, that also wipes a chunk of the company’s paper value, but it’s also a narrow slice of time. The graphs look dramatic, but zoom out a bit, and you’ll see that puts CrowdStrike’s stock price back to where it was at the beginning of June. Shares are up 23.5% since the start of the year.
• I don’t think CrowdStrike is going anywhere. When customer renewals come up, it will bleed market share to rivals in the coming years. This will probably be good news for Microsoft, but it will be bad news for concentration.
• Rivals will try to make hay out of the situation. Some in a gross way (And Finally). Others just in their pitches to new prospects. It’s an easy shot. LINK
• But while sales and marketing teams at rivals get excited, I’m sure there will be plenty in engineering teams behind the scenes glad that it wasn’t them. There’s plenty to learn across the industry.

What can we learn?

• George Kurtz, CrowdStrike’s CEO, has sincerely apologised for the incident and warned of threat actors trying to capitalise on the fallout. It’s also not his first rodeo: Kurtz was CTO at McAfee in April 2010 when the AV company issued an update that deleted a crucial Windows XP file and required IT to go system-to-system applying fixes. LINK Of course, it’s unlikely that Kurtz has any direct involvement in this incident.
• Endpoint security tools need regular updates to keep them apprised of changing tactics and techniques that bad actors are using. In this instance, CrowdStrike has enjoyed a remarkable period of growth while spending too much on novelty action figures instead of potentially on its engineering and release processes. Firstly, it would appear that CrowdStrike’s assurance governance processes aren’t up to snuff. Perhaps someone mixed up a file or merged the wrong branch in a commit after testing. However, the final product doesn’t appear to have undergone much, if any, testing was done before the updated channel files were published. Windows configurations vary dramatically, but this does not seem to be an edge-case situation.
• Secondly, for hyper-scale software deployments — be that operating systems, software packages, or web apps and social media platforms — commonly updates are deployed in staggered rollouts. Not in the alpha/beta style software release cycle, but when it actually comes to production deploys, it’s not a ‘one and done’ but a small subset that receives the update, then another, and another, all the time watching for unexpected outcomes before ‘opening the floodgates’. CrowdStrike’s rapid growth may have meant that they overlooked this risk. They may also consider the benefit of rapid deployment of detection content more important than stability. I’m sure their customers would like to have the choice to decide for themselves.
• Thirdly, the impact was significant because of concentration in customers and suppliers. It’s precisely because of the outsize impact that organisations in the CNI space have that they are targeted — by state-backed APTs and cybercriminals — and why a higher level of protection is needed. CrowdStrike and others in the cyber market have developed tools to help combat these threats, but the relatively small number of effective solutions and critical infrastructure companies invite this sort of risk accumulation. More on this below.
• Some — CrowdStrike, ironically included (Interesting Stats below) — have called for greater IT and security review of updates before deployment. I’m not sure I agree. There has to be an emphasis on shipping a quality product. It shouldn’t be on the masses to employ hundreds of thousands or millions of staff to review, for example, Microsoft patches every month before deployment. National bodies like CISA and NCSC have called for greater emphasis on software development. This is one of the reasons why. It’s easier and wiser to tackle at source.

Software quality, accumulation risk and lessons for policymakers

• This is the sort of concentration or accumulation risk that insurers and regulators worry about. Traditionally, this has focussed on the ‘big three’ cloud providers (AWS, GCP, Azure), but there are plenty of other examples of this sort of aggregation, and security tooling is one of them.
• Endpoint security is particularly concentrated: Microsoft has over 25% market share, CrowdStrike just under 20%, while Trellis, Broadcom and Sophos round out the top five at around 5% each, according to figures from IDC. LINK
• Meanwhile, China avoided a significant impact on Friday, largely because CrowdStrike had hardly any market penetration there, and local cloud providers (Alibaba, Tencent, and Huawei) rule the roost. So, while some Western companies could not serve customers, many domestic companies had no such issue. LINK
• Security tools often have (and rightly need) privileged capabilities so they are able to intercept, analyse, block, modify, or delete data in the name of keeping organisations safe. We need these capabilities to help defend against the likes of ransomware, but we need them to be reliable. (Google’s Project Zero did a great job of tearing down common AV solutions around a decade ago, proving that they also represented a risk in their own right. Thankfully, the industry listened and upped its game.)
• Whilst operational resilience, which features heavily in NIS and DORA regulations, rightly receives attention in this incident, it also applies to adjacent policy discussions around platform companies and how or what they are allowed todecide about what happens on their platforms. Apple has faced increasing scrutiny over its AppStore practices, with regulators arguing for more consumer choice, while the testing and deployment regime in place could help protect against events like these.
• I suspect insurers will increasingly be asking for information on technology and security stacks as they seek to get better data and understanding of their insureds and the exposure they face. This won’t be straightforward, though: just because you ‘don’t use CrowdStrike’ doesn’t mean the critical supplier of your critical supplier doesn’t. This is a complex supply chain issue.
• Technology companies, and especially cyber security vendors need to prioritise their engineering practices to promote secure, quality products. As the image on CISA director Jen Easterly’s LinkedIn post says, “we don’t have a cybersecurity problem. We have a software quality problem.” Government and large enterprises can drive change here by asking about (and prioritising) good practices like secure by design and memory-safe languages. LINK 

Hopefully, there will be some interesting and useful food to think about there. Let me know what you think.

Interesting stats

8.5 million Windows hosts affected by CrowdStrike’s blunder, according to Microsoft, which is  0.6% of the estimated 1.4 billion* Windows 10/11 install base, compared with  300,000 computers were affected by WannaCry in 2017. LINK

The day before the incident, CrowdStrike published a report suggesting that the cyber security team only reviewed major updates 54% of the time. Awkward. LINK

55% of energy, oil & gas, and utility companies take more than one month to recover from a ransomware attack in 2023, up from  36% in 2023, and  19% in 2022, according to Sophos. LINK 

15 companies account for  62% of the cyber security market, according to SecurityScorecard. LINK

Other newsy bits / in brief

🤓 Interesting reads:

• “SMEs often have a poor understanding and deficient practices to adequately assess the risks they face,” Cybersecurity, Cyber insurance, and Small-to-Medium-sized Enterprises: A Systematic Review by Adriano and Nurse, at the University of Kent. PAPER (PDF)

⚠️ Incidents:

• AT&T may have paid attackers over $300,000 to delete copies of their customer call and SMS records data. A 5.7 bitcoin transaction has been identified on the blockchain, and a security researcher has reportedly acted as a go-between AT&T and the representative of ShintHunters, who originally demanded $1 million. LINK
• A threat actor has released a set of 15.1 million Trello account names and email addresses for around $2 on a dark web forum. The private email addresses were harvested from a poorly secured API endpoint. LINK
• Trump vice presidential pick J. D. Vance left his Venom account public, revealing connections to “establishment GOP heavyweights, wealthy financiers, technology executives, the prestige press, and fellow graduates of Yale Law School—precisely the elites he rails against”, according to WIRED. Two points here: you’d think the Republican Party would be auditing and providing more support to candidates, and more generally, the US could learn a lot from European banks — why is Venmo even a thing? LINK
• Suspected North Korean attackers made off with $235 million from Indian crypto platform WazirX. This is how North Korea makes its money nowadays, and they’ve gotten pretty good at it. LINK
• Rite Aid, the US’ third largest drug store chain, says the data of 2.2 million customers was stolen during a data breach. The information includes the name, address, date of birth, and driver’s license number of the affected people. The regulatory filing sounds like an employee was phished, and then their credentials were used to log into a business system where the data was stolen. LINK

🏴‍☠️ Ransomware:

• Bassett Furniture Industries was forced to shut down manufacturing last week following a ransomware attack. LINK
• Yacht building MarineMax says that it lost the personal data of 123,000 customers during a March attack by the Rhysida cybercrime gang. LINK

🕵️ Threat Intel:

• The main threat actor responsible for QBot infections has pivoted to DarkGate after the FBI takedown of QBot, and other actors are following suit. LINK
• Scattered Spider switched to RansomHub and Qilin malware during its attacks during the second quarter of this year. LINK
• The Revolver Rabbit threat group has registered over 500,000 domains, according to Infoblox, which has reversed the group’s registered domain generation algorithm. The majority are .bond domains, which will cost around $1 million to register. LINK
• China says that talk of Volt Typhoon is a false flag operation run by the NSA, FBI, and other Five Eyes intelligence agencies. Make of that what you will. LINK
• Ukrainian defence companies are being targeted with fake drone contracts as lures. LINK
• FIN7 is selling its “AvNeutralizer” tool to other cybercrime groups. The tool helps threat actors to evade detection by blocking or killing the processes of legitimate endpoint protection software. LINK

🪲 Vulnerabilities:

• There is a perfect ten vulnerability in Cisco Smart Software Manager (SSM) On-Prem. Attackers can exploit CVE-2024-20419 (10/10) to change the password of any user, including administrators. LINK, ADVISORY
• Cisco has also released a patch for an arbitrary write vulnerability in its Security Email Gateway (SEG) appliance. CVE-2024-20401 (9.8/10) is “due to improper handling of email attachments when file analysis and content filters are enabled” and could allow an attacker to overwrite files in the underlying operating system as root. LINK, ADVISORY
• SolarWinds has fixed eight critical vulnerabilities in its Access Rights Manager (ARM) software. Six of the vulnerabilities — CVE-2024-23469, CVE-2024-23466, CVE-2024-23467, CVE-2024-28074, CVE-2024-23471, and CVE-2024-23470 (all 9.6/10)— would allow attackers to gain remote code execution on the target devices. Meanwhile, CVE-2024-23475 and CVE-2024-23472 allow for unauthenticated users to perform arbitrary file deletion, and CVE-2024-23465 is an auth bypass vulnerability that can lead to Active Directory domain admin. LINK, ADVISORY (Side note: a US judge dismissed most of the SEC’s lawsuit against SolarWinds for misleading investors this week. LINK)
• CISA is warning that a vulnerability in GeoTools GeoSErver product is being actively exploited. CVE-2024-36401 (9.8/10) allows attackers to gain remote code execution. LINK, ADVISORY

🧰 Guidance and tools:

• CISA released the Infrastructure Resilience Planning Framework Playbook, aimed at state and local governments and critical infrastructure operators. Timely! LINK (PDF)

🧿 Privacy:

• The UK ICO has reprimanded the London Borough of Hackney over its poor patch management process that led to a ransomware attack and compromise of thousands of residents’ personal data. Stephen Bonner, deputy commissioner, described the cause as a “clear and avoidable error”. LINK 

📜 Policy & Regulation:

• EU President Ursula von der Leyen has pledged to tackle ransomware attacks against hospitals in the bloc in the first 100 days of her section term. LINK
• The UK will bring forward a Cyber Security and Resilience Bill requiring the mandatory reporting of ransomware incidents for regulated companies. The bill was announced in the King’s Speech this week, though it has been consultedon previously with the previous Tory government failing to introduce the legislation. LINK

👮 Law Enforcement:

• Two Russian individuals have pleaded guilty to taking part in LockBit ransomware attacks. Ruslan Magomedovich Astamirov and Mikhail Vasiliev admitted to being affiliates of LockBit’s ransomware-as-a-service operation. The plea is for taking part in at least 12 attacks against businesses and causing over $500,000 in damages. [LINK](Ruslan Magomedovich Astamirov and Canadian/Russian national Mikhail Vasiliev w)
• Three-hundred arrests have been made in a law enforcement crackdown on a West African cyber fraud group. It’s the culmination of Operation Jackal III, and according to Interpol, $3 million assets were seized, and 720 bank accounts were blocked as part of the op. LINK
• UK police have arrested a 17-year-old for his role in the Scattered Spider attack on MGM Resorts. The company reported losses of $100 million when it had to shut down operations across its properties on the Las Vegas Strip last year. LINK

💰 Investments, mergers and acquisitions:

• Google’s parent company, Alphabet, is reportedly in talks to buy cloud security outfit Wiz for a rumoured $23 billion. Wiz has raised $2 billion from investors since being founded four years ago and was most recently valued at $12 billion, with annual recurring revenues of around $500 million. There are still plenty of details to iron out before a deal will beformally announced, which would likely also require regulatory approval. LINK, MORE

🗞️ Industry news:

• LogRhythm and Exabeam announced a merger in May (vol. 7, iss. 20), now The Register is reporting that may result in a 30% headcount reduction across the firms. There’s also a legal challenge after some Exabeam stockholders have been told that their shares have “been cancelled for no consideration”. LINK 
• Kaspersky is shutting down its US operations and will lay off “less than 50 employees”. The shutdown follows the US Treasury sanctions on Kaspersky executives and the prevention of US organisations from engaging in business with the antivirus firm. LINK

And finally

• While most of the world was trying to help, the folks over at Cybereason were busy buying a phone number and drafting a press release: “in response to the recent global network outages” those who are “concerned about their cybersecurity issues” should call a “support hotline…  +1 833 NO CROWD”. Careful guys, with so little distance to the ambulance, you won’t be able to see the brake lights! 🚑🏃 Gross. LINK