Most ransomware stories are about the night. This one is about the three weeks after, because that is where the real decisions were.
The situation
The group ran more than twenty plants on three continents, with around ten thousand endpoints and a corporate IT team small enough to fit in one room. Each site patched its own way. There was no complete inventory of what was on the network. We had been brought in as acting CISO a few months earlier, with a ten-project programme under three plain headings: prevent, detect, react.
The year had already produced three intrusion waves. The first was fileless malware living in memory on about fifty machines, mostly at one site. The second was flagged to the group by the national cyber agency: thirty machines, twenty of them sending data out, more than a gigabyte in a single day. The third, days later, reached production lines at a plant. Each wave was contained by the end of the day, and each time the sense of urgency faded with it.
In the autumn, a business case for group-wide vulnerability management said in writing that a significant attack was likely within weeks or months, and that ransomware was the highest risk. It was approved. It had not been deployed yet.
The night
At half past ten one winter evening the antivirus alerts started. The malware did not persist after a reboot, so the on-duty team rebooted the affected machines. That had been treated as a fix twice already that year. It is a symptom of fileless malware, not a cure.
By 2am the alert volume was unmanageable. We took the decision to cut the Internet links at the affected sites, then every site-to-site link in the group, with the group’s head of IT on the other end of the phone. It was a quick and easy call. Nobody pushed back; the emergency was too evident. The harder question was why so many links had been open in the first place. The network had been built on the old, terrible principle that internal networks are trusted and security lives at the perimeter, so every site could reach every other site by default. There was never a business reason for that. By 5am the network was quiet. Production stopped. The phones did not.
Three weeks
The restart began three weeks later and the group was back to business as usual four days after that. The plan is the part worth copying.
- Every endpoint and server was scanned with purpose-written scripts before it was allowed to restart. Nothing came back on trust.
- All VPN access was closed. Every remote user reset their password through IT support before getting it back.
- IT staff sat at the reception desk of every site. Each laptop was screened and each password changed before its owner was allowed onto the network.
- Every direct site-to-site rule was removed from the firewalls. Sites talked to the centre, not to each other.
- Two round-the-clock monitoring capabilities were stood up within weeks: a network detection platform across the sites, and endpoint detection and response through a managed provider.
Customers were the second restart. Large customers had cut their network links, email and shared workspaces with the group within hours of hearing the news. Getting them back took several rounds: a full incident report with indicators of compromise, a remediation plan with dated milestones, a signed statement of confidence, and, for one customer, an independent third-party closure report. The last customer reconnected a little under three months after the attack.
The hard part is deciding when it is safe to reconnect, and proving it to people who were not in the room.
What people get wrong
They treat the night as the hard part. The night is adrenaline and a short list of decisions. The hard part is deciding when it is safe to reconnect, and proving it to people who were not in the room.
They also treat a contained wave as a closed one. Three times that year the answer to “what changed structurally after this?” was “we rebooted and blocked an IP address”. Each wave was evidence the attacker could get in, not evidence that the group was safe.
And they assume remote users are the easy part. Laptops and VPN accounts were the re-infection path. Physical controls, people at the door with a scanner, were what made the restart hold.