Our Worst Day
Your worst day can begin so innocuously – you leave home, you stop to pick up your coffee order, you catch your train, or maybe you run for it and just miss it. Perhaps it’s raining.
In Andrew Simpson's case, he should have been celebrating a small win, a milestone in an ongoing – and by-and-large successful – roll-out of a cloud upgrade project. Then things fell apart.
Simpson joined The Electoral Commission – the UK's election oversight and political finance regulator – in June 2022 as head of digital, information, technology and facilities, to lead a wide-ranging digital transformation project which, alongside transitioning from on-prem to cloud, brought a plethora of cyber upgrades.
But unknown to Simpson or anybody else, threat actors – possibly Chinese state cyber spooks, or a ransomware gang, or both – were already lurking within the Electoral Commission's systems. Ultimately, it emerged that they exploited the ProxyShell vulnerability chain on an unpatched server to gain access.
The investigation later found the series of breaches started in August 2021, but it wasn’t until one of Simpson’s cloud transition projects was in progress that it came to light. “Part of that was to introduce MFA [multifactor authentication], and that happened in October 2022, which is exactly when we found the compromise,” says Simpson.
“One of the lead engineers on the project spotted that they had 10 attempts on their MFA account within less than a minute. It was glaringly obvious that something wasn’t quite right at that point.”
As an IT leader, what does it feel like to be doing the right thing and to suddenly find yourself embroiled in a major cyber security panic?
“It’s possibly the worst feeling you can ever have in this industry,” says Simpson, who remarks that bringing new tech functions to an organisation’s workforce and helping them do their job better with up-to-date tools is ordinarily a great feeling.
“When you suddenly get hit with a cyber incident, you realise everything we were doing is no longer the priority, so the benefits of what we were doing get destroyed by the compromise, and your mindset changes – we now have to batten down the hatches again.”
Fortunately, the fact that the team had stood up MFA successfully was a small mercy and The Electoral Commission leaned into this, increasing the frequency of challenges – once an hour in the case of its lead IT engineers.
But Simpson still recalls the initial shock, and the dawning realisation that the scale of the compromise was much greater than it appeared.
“It’s a horrible thing, it’s gut-wrenching – I think that’s the best way of putting it. I would never wish it on anyone,” he says.
In an ideal world, Simpson says he would have stood up an incident response team right away, but that wasn’t really an option at the time because the capacity wasn’t there.
He recalls frantic phone calls to contacts at suppliers and the National Cyber Security Centre (NCSC), which helped link The Electoral Commission up with incident responders at Secureworks (now part of Sophos) via its cyber security framework.
Meanwhile, the IT team moved swiftly to lock things down, taking the affected servers offline entirely and sandboxing them.
This was highly disruptive, but because the Electoral Commission had one foot in the cloud already, there were still some systems that could be used relatively safely, subject to extra precautions to avoid cross-contamination.
Overall, says Simpson, The Electoral Commission was lucky. “We caught them working on tooling up and potentially at some point injecting ransomware. We were never at the point where a lot of organisations have ransomware rip through them and destroy them,” he says.
“We didn’t get to that stage because we reacted so quickly. We didn’t give them an opportunity. They lost access with immediate effect.”
At this point – almost 12 months before news of the hack broke in the media, everything was being done with the utmost secrecy, with the IT team on lockdown.
No one else in the Electoral Commission knew what we were doing. We did not communicate that out. One of the key things as well is that none of this was via email. It was all verbal, phone calls, because obviously they had access to our email system,” says Simpson.
“From the IT perspective, we knew nobody was to discuss this other than my boss, the CEO and executive team members. They were all who knew about what was going on. “Obviously staff had issues where they were
“I’m speaking across the board to people wherever I can,” he says, “because the only way to help with this is to share information. For those people who have been through it – [after all,] some people lose their jobs for this – I was lucky.”