Isolate, run by hand or keep producing? The OT incident decisions to take before the call

The call comes in at ten past two in the morning. The IT security team has found ransomware on three office servers and wants to know whether to cut the link between the business network and the plant. On the filling line nothing is wrong: the batch runs, the historian updates. Nobody on the call knows who may say yes, or what the line will do once the link is gone. The plant’s response plan was on the file server that has just been encrypted.

Each element of that scene is something we hear on industrial sites, and together they show what an operational technology (OT) incident response plan is for. The technical playbook is the smaller part. The larger part is a set of decisions taken in advance, by people who have the authority to take them: which responses each area of the plant allows, who orders them, what the line does in each case, and how the batch and its records are judged afterwards.

Why the IT plan does not carry over

The general frame is sound. NIST Special Publication (SP) 800-61 Rev. 3, published in April 2025, replaced the 2012 incident handling guide and reorganised incident response around the six functions of the NIST Cybersecurity Framework (CSF) 2.0. Govern, Identify and Protect prepare the ground; Detect, Respond and Recover are the response itself; lessons learned feed every function. It is written for every sector, which is its strength and, on a plant, its limit.

On a production site each response action touches a physical process. NIST SP 800-82 Rev. 3, the OT security guide of September 2023, says that incident response “can range from doing nothing to performing a full system shutdown” (§6.2.4.5). The IT reflexes (isolate the host, reimage it, cut the network) are each a process decision on the plant floor, with a cost in product, in safety margins or in records.

The same section asks the plan to settle a priority that IT rarely has to weigh: return to normal operations as quickly as possible, or investigate and preserve forensic data. Both are legitimate. A night shift cannot choose between them on the spot, and should not have to.

The plant can also be stopped by an incident that never reaches it. Appendix C.3 of SP 800-82 Rev. 3 records that the ransomware which hit Colonial Pipeline in May 2021 encrypted IT systems, that there was no evidence of a direct impact on the OT environment when the guide was written, and that the company shut down physical operations to contain any potential damage. A precautionary stop can be the right call. It should be a call prepared before the night it is needed, with its cost known.

Five responses, decided area by area

The first thing to write is the list of responses each area of the plant can actually take. SP 800-82 Rev. 3 names the building blocks: disconnecting infected systems, physically isolating networks that can run on their own (enterprise from control, control from safety), transitioning to manual operations, and adding operations staff to validate data by hand. Grouped by what they do to the line, they give five options.

Which of these options exist is a design property, not a choice made on the night. Cutting the IT/OT conduit without stopping the line works only if the line was built to run without its enterprise resource planning (ERP) and manufacturing execution system (MES) links for a shift or two. A zone can be isolated only if the zones were drawn on the process, which is the method of zones and conduits: start from the risk.

SP 800-82 adds a caution that belongs in the plan word for word: isolating the system under attack may not be possible without affecting operational performance or safety, so a focused risk assessment of the OT environment decides the response. In practice that assessment is short if the table above exists, and impossible if it does not.

Who decides at two in the morning

On the sites we assess, the incident plan usually names the IT security manager as its owner, while the authority to stop a line sits with production. Both are right, and the plan has to join them. Name one site incident commander by role, with a deputy for nights and weekends, and write which responses that person may order alone. Cutting the conduit should be one of them. A controlled shutdown needs the person who can stop the plant.

SP 800-82 Rev. 3 points to the National Incident Management System of the US Federal Emergency Management Agency (FEMA) to standardise roles and terms (§6.4.2), the same structure fire and emergency services use. It also asks that the plan be written with operations, engineering, IT, system support vendors, management, organised labour, legal and safety, and approved by them (§6.2.4.5). That list is longer than most sites expect, and every name on it is someone who will otherwise be improvising at night.

Suppliers are part of the response whether the plan says so or not. The integrator who maintains the batch system, the machine builder with a remote access account, the historian vendor: each needs a contact, a contractual duty to help, and a known way to have their access cut. The ISA book excerpt that started this article makes the same point: incident response planning does not end at the asset owner’s fence.

What the plant must be able to do without its network

A plan that lives on the network it is meant to survive is not a plan. SP 800-82 Rev. 3 asks for response plans “documented in paper form or on an offline system (i.e., air gapped)” (§6.2.4.5), and for staff trained on where to find them. The same goes for everything the response will need when the domain controller, the file server and the email are gone.

Two numbers make the recovery part concrete. SP 800-82 defines the recovery time objective (RTO) as the target for restoring communication links and processing, and the recovery point objective (RPO) as the time for which an absence of data can be tolerated. Management sets the RTO. Whether it can be met is a question for the restore test, which is the subject of a backup is a hypothesis, a restore is evidence.

The batch and its record: where a GMP site differs

For a site under EU Good Manufacturing Practice (GMP), a cyber incident on a computerised system is a GMP incident. Annex 11, in its 2011 text, says in clause 13: “All incidents, not only system failures and data errors, should be reported and assessed. The root cause of a critical incident should be identified and should form the basis of corrective and preventive actions.” The response plan and the deviation system have to meet.

Clause 16 of the same annex covers business continuity: for systems supporting critical processes, provisions such as “a manual or alternative system”, brought into use in a time based on risk, and arrangements “adequately documented and tested”. The run-by-hand response is clause 16 in practice. It works only if the paper forms are approved, printed and stored at the line before the incident, and if the manual mode was exercised, as in the degraded-mode tests of an operational qualification (OQ).

When the incident is over, quality has questions that no firewall log answers. Which batches were in progress? Which records were created while a system may have been compromised, and is their audit trail intact? Were set points, recipes or controller programs changed? Each controller and recipe is compared with its approved version before release, and data captured by hand is checked, which is why SP 800-82 lists extra operations staff to validate data manually among the response actions.

Restoring is part of the same question. Annex 11 clause 7.2 asks that the ability to restore data be checked during validation and monitored periodically. A restore made under pressure, from a backup that was never tested, puts back a system whose state nobody can vouch for.

Plan for the incidents you will actually have. SP 800-82 Rev. 3 suggests three classes for OT: accidental actions by authorised personnel, targeted malicious attacks and untargeted malicious attacks (§6.4.3). On a plant the first class is the most frequent. A well-meant change pushed to the wrong controller deserves the same response path as malware, without the drama.

The clocks outside the plant

Some incidents also start a legal clock. Under Directive (EU) 2022/2555, the second Network and Information Security Directive (NIS2), Annex I lists manufacturers of basic pharmaceutical products and pharmaceutical preparations (NACE division 21) in the health sector. For an entity in scope, Article 23(4) asks for an early warning within 24 hours of becoming aware of a significant incident, an incident notification within 72 hours, and a final report no later than one month after that notification.

Article 23(3) calls an incident significant when it has caused or can cause severe operational disruption or financial loss to the entity, or considerable damage to others. Someone has to make that assessment at night and someone has to file. Both are named in the plan, with the national contact point and the template, on paper. NIS2 Article 21, measure by measure shows how incident handling fits among the other measures the directive asks for.

Test it before it is real

A plan nobody has rehearsed is a hypothesis. A tabletop exercise on one realistic scenario costs an afternoon and finds most of the gaps: the commander nobody can reach, the contact list that sends everyone to the encrypted mail server, the paper form nobody can find. The SANS five critical controls explains why those scenarios come first and drive the rest of the programme.

Then test one response for real. A planned shutdown is the right moment to open the IT/OT conduit for an hour and write down what stops: a licence server across the firewall, a recipe download from the MES, a time source in the office network. SP 800-82 also asks the response to capture what was done during execution (§6.4.1), and an exercise is the cheapest place to learn how.

Where to start

1. Write the five responses for one area. On a single page, with who may order each one. It is done when the night-shift leader can read it and say which response applies to a given call.
2. Print the plan and the contacts. Put a copy in the control room and one with the incident commander, and check after a month that both are where they should be.
3. Open the conduit once, on purpose. At the next planned shutdown, run the plant islanded for an hour and record everything that stops or complains.

Five common mistakes

1. The plan lives on the network it must survive. Encrypted with everything else, it helps nobody.
2. The IT plan with a paragraph about the plant. It names the actions of an office network and none of the five responses a line can take.
3. Nobody may stop the line at night. The decision waits for a manager who is asleep, or is taken by whoever pulls a cable first.
4. Isolation that was never tried. The first real cut reveals that the line needs a licence server or a time source on the other side.
5. Closing the incident without quality. Systems are back, batches are released, and the records made during the incident were never assessed.

The best moment to decide whether to cut the plant off is a quiet afternoon months before the call. At two in the morning, the plan should only have to be read.

References

• Steve Mustard, “Cyber Safeguarding Industrial Operational Support”, AUTOMATION 2023 Volume 2 (ISA, March 2023), online 9 May 2023: automation.com
• NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, April 2025: csrc.nist.gov
• NIST SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security, September 2023, §6.2.4.5, §6.4.1 to §6.4.3 and Appendix C.3: csrc.nist.gov
• EudraLex Volume 4, Annex 11: Computerised Systems (2011), clauses 7.2, 13 and 16: health.ec.europa.eu
• Directive (EU) 2022/2555 (NIS2), Article 23 and Annex I: eur-lex.europa.eu