Backups you have actually restored

Control system backup is one of those activities that is universally agreed to be important, generally configured at commissioning, and rarely examined afterwards. The gap between having backups and being able to recover is where the risk sits.

What has to be backed up

The list is longer than the control logic, and the omissions are what make recoveries fail.

• Controller configuration and application logic, in source form.
• Graphics and display configuration, in source form.
• Alarm configuration and the rationalisation database.
• Operator station configuration, including per-user settings.
• Historian configuration, and the historical data itself.
• Network device configuration: switches, routers, firewalls.
• Field device configuration for intelligent instruments and drives.
• Server operating system images and installed application versions.
• Licences and licence keys.
• The documentation needed to perform the restore.

The last three are the ones that turn a four-hour recovery into a four-day one. A perfect logic backup is of limited use if the licence server has to be rebuilt first and nobody can find the keys.

Remote-first organisations provide useful examples of how distributed operating models are structured. Further details are available in this guide.

Security decisions should also be checked against NIST guidance on operational technology security.

Frequency follows change rate

Configuration that changes rarely needs infrequent backup, provided a backup is taken after every change. Configuration that changes often needs a schedule.

The practical rule that covers both: a backup is part of the change process, taken after any modification is complete and verified, plus a periodic backup on a schedule to catch anything undocumented.

Storage that survives the event

A backup stored on the same server it protects, or on a network share in the same room, does not survive the fire, flood or ransomware event that makes it necessary.

The conventional structure — several copies, on at least two different media types, with at least one held elsewhere and one offline — applies as much to control systems as to business IT, and is more often absent.

The offline copy matters particularly. Malware that encrypts accessible network storage will encrypt an online backup along with everything else.

The restore test

The only meaningful verification is to perform a restore. Everything else is an inspection of the backup rather than a test of it.

The practical approach is to restore to spare hardware or to a virtual environment, on a schedule, for each major system type. Doing this annually finds the problems: media that has degraded, backups that were incomplete, a procedure that assumes something no longer true, licences that will not activate on different hardware.

The first restore test at any site reliably finds at least one problem that would have prevented recovery.

Record the recovery time

The restore test produces a number: how long it took, from decision to plant available. That figure is what makes the risk discussable.

A site that knows recovery takes eleven hours can decide whether that is acceptable. A site that has never tested is choosing an unknown recovery time by default.

Write the procedure for the worst case

The recovery procedure should be usable by someone who is not the person who built the system, at night, under pressure, possibly without network access to the document storage.

That argues for a printed copy held physically in the control room, and for a procedure written in steps rather than as narrative. It should include what to do first, what the dependencies are between systems, and who to call.

Practise the decision, not just the mechanics

Beyond the technical restore, recovery involves decisions: when to declare a failure unrecoverable, whether to restore or to repair, who authorises a restore that will lose recent data.

Walking through those decisions in a tabletop exercise, once, means they are not being made for the first time during the actual event.

Backups and malware

Backup strategy for control systems has to account for malicious as well as accidental loss. Ransomware that reaches a control network will encrypt any backup it can reach.

The protection is an offline or immutable copy: media physically disconnected, or storage that cannot be modified within a retention window. Without one, an otherwise complete backup regime provides no protection against the scenario that has become most likely.

Documenting dependencies

Recovering a control system involves restoring several components in an order determined by their dependencies: domain services before servers, licence servers before applications, historian before reporting.

That order is obvious to whoever built the system and unknown to whoever performs the recovery at three in the morning. Writing it down as an explicit sequence is a short task and it is what makes a documented procedure usable.

Partial recovery scenarios

Full disaster recovery is the scenario planned for and the least likely to occur. Far more common is the loss of a single component: one server, one controller, one workstation.

Those scenarios deserve their own procedures, because the actions are different and because performing a full recovery to replace one workstation is disproportionate. They are also easier to test, which means they actually get tested.

Backups of things that are not the control system

Recovery depends on components outside the control system itself: domain controllers, licence servers, time servers, network device configurations, and the file shares holding drawings and procedures.

These are frequently managed by a different team under a different backup regime, and the interaction is rarely tested.

A recovery test that includes the dependencies, rather than restoring the control system in isolation, is the only way to establish that the whole path works.

Backup verification short of a full restore

A full restore test is the definitive verification and is not always practical at the frequency one would like.

Intermediate checks provide partial assurance cheaply: verifying that the backup completed, that the file sizes are plausible, that the archive opens, and that a sample file can be extracted.

These do not replace a restore test and they catch the most common failure, which is a backup job that has been silently failing for months.