Massive AWS outage across US-EAST-1 region shows how much of the internet relies on Amazon
An outage of Amazon Web Services (AWS) US-EAST-1, North Carolina data centre caused disruption for a huge amount of companies, and their customers, that rely on Amazon’s cloud services: Amazon warehouse and delivery operations were affected, Netflix and Disney Plus were affected, Roomba vacuum cleaners stopped working, electric vehicles stopped charging, and many, many more sites and services.
In similarities to Facebook’s outage in October (vol. 4, iss. 41), the issue affected internal tools, slowing the ability for AWS to respond:
”This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates.”
The similarities continue with the cause of the incident too, with automation designed to scale network connectivity being the source. In this case ‘unexpected behaviour’ while scaling the network caused a surge in connections that increased latency and errors, causing a further surge in connections that exacerbated the issue.
The outage of a whole region, rather than isolated service (such as S3 seen before) has reignited discussion about AWS as a single point of failure and the risk associated with a single organisation having such an outsize global impact.
However, the outage was resolved reasonably promptly (6-7 hours) and, as Nicholas Weaver from UC Berkeley told Vice Motherboard, “If anything AWS is too reliable, it’s reliable enough that people don’t bother doing the engineering for when it fails, so when it does fail it is a shock.”
As an aside, during the outage, some endpoint security tools ‘failed open’, meaning that malicious code was not being detected or blocked (@GossiTheDog tweets).
18,532 vulnerabilities have been reported so far in 2021 (exceeding 2020 and any previous year) according to NIST, though the proportion of ‘high’ severity issues decreased:
$148M lost by Americans to gift card scams in the first 9 months of 2021, according to the US Federal Trade Commission bleepingcomputer.com
Other newsy bits
‘Perfect 10’ Log4Shell vulnerability likely to wreak havoc in coming weeks
A ‘zero-day’ vulnerability in the extremely common Java library Log4j has been discovered, and proof-of-concept dropped on Twitter, which gives remote attacks the ability to execute commands on computers running the code.
Log4j provides developers with an easy way to generate and store log files for their applications and the flaw, dubbed Log4Shell, exploits functionality that allows for data to be enriched using custom code. For example, log files could be made to automatically look up a customer name and contact information from a customer number, and so on.
But the code that does the remote lookups does not properly escape what it is being passed, allowing for remotely hosted Java classes to be referenced, and then executed. Any field that is written to a vulnerable application’s logs can be used: so exploits have been seen using setting malicious user-agent for web browsers or sending carefully crafted in-game messages in Minecraft.
Log4Shell has been assigned reference CVE-2021-44228 and it can only be abused if the ‘log4j2.formatMsgNoLookups’ configuration option is set to false. The Apache Software Foundation, responsible for the Log4j library, has released version 2.15.0 as an emergency fix.
Logging itself is a pretty basic and mundane function of an application and it’s likely that many organisations, and indeed vendors, will not realise that their apps a using the library and know to update. As a result, this issue will likely persist, and be the cause, of many incidents in the weeks and months to come.
File this one under ‘really clever’: Attackers have compromised over 300 sites with ‘Magecart’ style card skimming code by using legitimate functionality in a Google product called Tag Manager. Typically Google Tag Manager helps website owners dynamically control and update tracking and Google Analytics code on their websites. If you were auditing a website you’d see references to a Google domain and, with the ubiquity of Google Analytics, probably think nothing more of it.
Attackers behind Solarwinds breach continue their campaigns
Nobelium (aka Cozy Bear; the group behind the Solarwinds attack) have been busy in the last year, according to Mandiant, saying the Russian-linked group have “top-notch operational security and advanced tradecraft,” and have successfully compromised multiple cloud and managed service providers in pursuit of their objectives.
The group make use of proxy services and bought access from residential IP ranges within their victim’s country to obfuscate their origin, while also targeting centralised enterprise controls - such as email spam filters - to gain wholesale access to mailboxes and improve their access.
Another tactic involved repeatedly triggering multi-factor authentication push notifications until a user authorised the logins. While more secure than SMS-based messages, the usability and functionality of such security services often only focus on ‘happy path’ operations, rather than giving the user the ability to decline (ignoring the individual attempt) rather than report suspicious activity.
Crowd-forecasting cyber knowledge
A blog post from Lawfare Institute arguing for the ‘crowd-forecasting’ of answers to cyber security questions.
”possible to do something about this deficit. For example, some observers believe that better measurements and the consolidation of existing traditional metrics will help address the deficit and are advocating for the creation of a ‘Bureau of Cyber Statistics.’”
Attacks, incidents & breaches
- 300 Spar supermarket stores have been forced to close or accept cash payments only following a ‘total IT system outage’ theregister.com
- Cryptocurrency trading platform Bit Mart had two private keys for ‘hot wallets’ compromised leading to the theft of almost $200M of customer crypto-assets, though the company says it will cover the losses zdnet.com
- Room key card systems affected by a Conti ransomware attack at Nordic Choice Hotels properties bleepingcomputer.com
- Between 38,000 and 80,000 employees of the South Australian government have had their personal data compromised in a ransomware attack against payroll provider Frontier Software zdnet.com
- Brazil’s Ministry of Health has suffered a ransomware attack that is claimed to have resulted in the deletion of 50TB of data, including vaccination data on millions of citizens zdnet.com
- Doxy by name, doxy by nature: Telehealth app Doxy.me exposed patients’ data cyberscoop.com
- ‘Material impact’ on German logistics company Hellmann Worldwide Logistics following an incident required data centre connections to be severed. The company operates in 173 countries and has revenues of approximately $3B zdnet.com
- Volvo Cars breach has resulted in a “limited amount of… R&D property” being stolen. techcrunch.com
- Emotet malware seen to be directly installing Cobalt Strike beacons in a move that could result in less time between compromise and ransomware attacks bleepingcomputer.com
- A recent vulnerability in Confluence and GitLab (vol. 4, iss. 45) servers is being used in attack organisations with ransomware therecord.media
- Details of a Vietnam-based cybercrime group, dubbed XE-Group, that targets restaurants and hospitality businesses to steal credit card data published bleepingcomputer.com
- 27 vulnerabilities have been found in a USB-over-Ethernet SDK written by Eltima and used by cloud providers like Amazon to allow local devices to be mounted to cloud desktops sentinelone.com
- SonicWall is urging customers of their SMA series network security appliances to patch and address vulnerabilities including an unauthenticated remote code execution bug bleepingcomputer.com
- NCSC has published guidance on selecting secure messaging and collaboration apps: establish the context; research the options; deploy a secure configuration; document your decision ncsc.gov.uk
Internet of Things
- QNAP warns of crypto mining malware targeting their customer’s Network Attached Storage (NAS) devices bleepingcomputer.com
- Hikvision IP CCTV cameras being compromised as part of Moobot, a Mirai-b based botnet bleepingcomputer.com
- Unpatched TP-Link routers are being compromised and hijacked by the Dark.IoT botnet therecord.media
- Verizon has made changes to and renamed a programme tracking mobile customer browsing histories for advertising purposes and is automatically opting-in all customers, including those that opted-out of the scheme under its previous name arstechnica.com
- New German coalition government signals that it will not buy zero-day vulnerabilities for espionage purposes therecord.media and pro-encryption, anti-backdoor euractiv.com
- The US has ‘released the hounds’ and ordered the military Cyber Command to gather intelligence and combat criminal groups targeting US infrastructure nytimes.com
- Microsoft has been handed control of domain names used by Chinese-linked group Nickel (aka APT15, Vixen Panda) to target organisations in the US, UK, South America, the Caribbean and Europe cyberscoop.com
- Terrible opsec has led to the arrest of Canada’s “most prolific cybercriminal” who conducted ransomware attacks against Canadian and American organisations krebsonsecurity.com
- After taking steps to disrupt the Glupteba botnet that comprised approximately one million devices Google is suing seventeen people believed to be its operators to prevent them from regaining control blog.google, bbc.co.uk
Mergers, acquisitions and investments
- Torq, a ‘no-code’ automation platform for security technologies, has closed a $50M Series B funding round techcrunch.com
- Cloudflare, Mandiant, Secureworks and Crowdstrike partner with insurance brokers At-Bay, Coalition and Cowbell Cyber to offer “rapid referral” services theregister.com
Pixel users: Dial 911 to freeze your phone
I’m not quite sure how you manage to pull this off, but hats off to Google and Microsoft: Pixel devices running Android 10 (or newer) and with the Microsoft Teams app installed, but no user logged in, results in the handset freezing if the user dials 911.
It’s not really something to make light of — as tends to the be and finally way — more astonished that the situation is possible.