Everyone has backups. The question that matters is what came back when you needed them, and you only learn the answer by restoring. After a ransomware attack on a manufacturing group with more than twenty plants, we reviewed the backups of its most critical system, a regulated quality system that every plant ran, site by site. Here is what we found.
What came back
Most sites restored completely. That deserves saying, because the story is usually told as a disaster and it mostly was not. The best site ran daily full backups to disk, kept a deduplicated copy on separate storage, and sent tape to another building: three copies, two media, one of them somewhere the ransomware could not reach. It had a written procedure for all of this, and the procedure was audited on a schedule. When the attack came, that site had a boring week.
What did not
One site lost its archives. The application was backed up. The archive server that held years of historical records was not, because it had been set up separately, by someone who had since left, and nobody had asked whether it was on the backup schedule. The application came back; its history did not. In a regulated environment, a record you cannot produce is a deviation in its own right, and this one could not be fixed.
Two sites had no current backup documentation. Their backups existed and restored, but the documentation described an infrastructure that had been rebuilt in a hurry during the crisis, and nobody could say with confidence that what the documents promised was what the system now did.
One site was untouched, for a reason nobody had planned: its systems sat on a separate domain, with no trust relationship to the one the attacker had taken. Isolation that was an accident of history turned out to be the best control in the group.
What people get wrong
They back up the application and not its archives. The archive is the part you cannot recreate. The application you can reinstall.
They trust the documentation after a crisis. A backup infrastructure rebuilt under pressure in the weeks after an attack is a different infrastructure from the one the documents describe. Until someone reconciles the two, the documents are fiction.
They keep every copy on the same domain. If the backup server trusts the domain that the attacker controls, the attacker controls the backups. One copy offline, or on a domain with no trust relationship, is what survives.
They never restore. A backup that has never been restored is a hypothesis.
A backup that has never been restored is a hypothesis.