A backup is a hypothesis. A restore is evidence.

Every site we assess has backups. Almost none can say when they last restored one onto real hardware and watched the line produce conforming product.

Those are two different claims, and only the second one is worth anything on the day it matters. A backup nobody has ever restored is an assumption waiting for its first test, usually at three in the morning with the line stopped.

Until it has been restored and checked, every backup is a hypothesis: about the copy, about the procedure, and about the people who will run it on the bad day. This article is about the test that turns it into evidence: what the test must confirm, what it finds, and how to run it. Which backup to choose for each system is a different question; whatever the choice, the test below still applies.

Three properties a backup must have by design

A backup job can report success every night and still fail on the day it is needed. A restore test is the only way to confirm the three properties the backup was designed for.

3-2-1. Three copies, on two different media, one of them off-site. This is the baseline, and it survives most ordinary failures.

It is worth adding what modern practice has learned since: at least one copy should be immutable or genuinely offline, and every copy should be verified rather than assumed. The extended form is usually written 3-2-1-1-0: one immutable or air-gapped copy, zero errors on verification. The addition exists because attackers changed tactics: destroying backups is now a routine first step, not an afterthought.

Separated. Backups must sit outside the OT domain, with their own credentials. Ransomware encrypts everything it can reach, and a backup share mounted with a domain account that the attacker already owns is not a second copy. It is the same copy.

Different operating systems help, for the reason that a single malware family rarely works across both: a Windows production estate backed up to a Linux target raises the cost of destroying both. It is not a guarantee, and it should not be treated as one, but it is cheap insurance.

Two things belong to this point and are missed constantly:

• No trust relationship. If the backup system authenticates against the same directory as production, separation is nominal. Separate accounts, separate authentication, multi-factor on the backup console.
• The restore path must not require production. If restoring depends on a domain controller, a licence server or a DNS entry that is itself encrypted, the plan has a circular dependency and nobody found it because nobody tried.

Encrypted. A stolen copy must not become a data leak. Backups concentrate everything: recipes, formulations, batch records, network configurations, credentials in configuration files.

And the part that gets forgotten: key management is part of the backup. An encrypted backup whose key lives in the password manager, in the domain, in the system that just got encrypted, is not recoverable. The key custody has to be as separated as the copy.

What the test finds missing

The server estate is usually well covered, because IT owns it and it is a familiar problem. A restore test on a production line finds the gaps, and they are in OT, consistent from site to site:

• Controller programs: PLC and safety PLC, with the correct revision, not “the version someone saved in 2021”.
• HMI and SCADA projects, including the graphics and the tag database.
• Drive, servo and instrument parameters, which often exist only in the device.
• Robot programs and taught positions, which represent weeks of commissioning nobody wants to redo.
• Recipes and formulation data, sometimes held in a system nobody calls a server.
• Network device configurations: switches, firewalls, the remote access gateway.
• Licences and dongles. A restored system that will not start because the licence was tied to the failed hardware is a restore that failed.
• The engineering toolchain itself, which is the one that surprises people (see below).

The version trap

This is the most common reason a controller restore fails on the day, and only a restore test discovers it in advance.

Restoring a PLC program is not copying a file. It requires the exact version of the engineering software, often the exact service pack, sometimes a licence for it, and a controller firmware compatible with the project. A program archived in 2016 may need an engineering environment that no longer installs on any machine you own, from a vendor who has since changed its licensing model.

So the backup of a controller is a file plus a documented, reproducible way to get an engineering station capable of opening it. Sites that keep a virtual machine image of each engineering environment, alongside the programs, are the ones that recover in hours rather than days.

The restore test, in five steps

1. Plan a frequency. Proportional to criticality, and scheduled as a maintenance activity: in the plan, with an owner and a date. A test that depends on someone finding time never happens.
2. Restore for real, in a test environment. Not a file listing, not a checksum. A restore onto spare hardware or a test controller, performed by someone who was not the person who built the system.
3. Follow a documented procedure, and update it during the test. Every restore test exposes a step that is wrong, missing, or exists only in one person’s head. Capturing those is half the value of the exercise.
4. Measure the real recovery time. The actual hours, from decision to running. Then compare it with what production believes it is. The gap between those two numbers is usually the most useful finding of the whole exercise.
5. Record the result and improve. A dated record, with what worked, what did not, and what changed as a result. In a regulated environment this record is also the evidence an inspector will ask for.

Repeat the test after every change to the backup tool, the restore procedure or the system itself: a test proves the configuration that existed on the day it was run, not the one after the next upgrade.

What a regulated site adds

The test is a requirement. EU GMP Annex 11 (2011) §7.2 asks that “the ability to restore the data should be checked during validation and monitored periodically”. Read with the rest of the annex, that means a documented test, sized by risk, and repeated when the backup or restore process changes.

Restoring is a change. Putting a program back onto a controller in a GxP environment is not a neutral act: it may require verification, and possibly requalification, before the line is released. That process has to be defined in advance, because deciding it during an incident adds hours to an outage already being counted in hours.

The audit trail has to survive the restore. Restoring data is not enough if the record of who did what, and when, is lost or reset in the process. A restore that produces working software and broken data integrity has solved the availability problem and created a compliance one.

What it comes down to on the ground

Four questions, and they are answerable this week without a project:

• Do you know what is actually backed up on your systems? Not what the policy says, what the job is really covering.
• If it is automatic, does it run? A job that has been failing silently for eleven months is common, and it is only ever discovered at the worst moment.
• If it is manual, is it still being done? The technician who copied the PLC program after every change was a real control. When they change role, the control disappears and nothing raises an alarm.
• What is not covered, and who has been told? An uncovered system that management does not know about is a risk nobody has accepted.

So when someone says “we have backups”, ask for three things: a date, a measured duration, and the name of the person who performed the restore.

References

• European Commission, EudraLex Volume 4, EU GMP Annex 11, Computerised Systems (2011), §7.2: health.ec.europa.eu
• CISA, FBI, NSA and MS-ISAC, #StopRansomware Guide (updated October 2023): offline, encrypted backups and their regular testing: cisa.gov