Testing a running plant without breaking it: how to commission an OT penetration test

The tester arrives on Monday with a laptop, a badge request and a one-page scope: internal network, all ranges, standard methodology. By ten o’clock he has a switch port in the packaging hall. By half past, the automation engineer is behind him, asking what the scan will do to the serialisation printers, which drop their link whenever the network hiccups, and to the line controller nobody has restarted since the last shutdown. Nobody in the room can answer.

Versions of that morning come up on many sites we work with: a scope written by IT security and never read by operations or quality. An operational technology (OT) penetration test is worth commissioning, and it can be run safely. It is a different purchase: it starts from what must not happen to the process, looks before it touches, assumes the attacker is already inside, and on a good manufacturing practice (GMP) site goes through change control like any other intervention on a validated system.

What an IT habit does to a control network

NIST Special Publication (SP) 800-82 Rev. 3, the US guide to OT security of September 2023, lists accidental incidents in its Appendix C. In one, a ping sweep of a supervisory control and data acquisition (SCADA) network made a robotic arm in standby swing through 180 degrees. In another, consultants hired to test a gas utility’s corporate network strayed into a segment connected to SCADA and locked it up: no gas through the pipelines for four hours. Neither needed an exploit.

The same guide is careful about penetration testing itself. Its OT discussion of the penetration testing control says testing is “used with care on OT networks”, offers as a compensating control “employing a replicated, virtualized, or simulated system”, and schedules tests on production during planned outages wherever possible. Its section on vulnerability scanning (§6.3.2.4) allows penetration testing only “if the test will not adversely impact the production environment”.

Tyler Webb, a co-author of the SANS industrial penetration testing course (ICS613), describes the trap from the tester’s side in a SANS post of 24 July 2025: years at a power utility running “IT PenTests in an OT environment”. His co-author Don Weber, writing on 17 October 2025, describes a good assessment from the buyer’s side.

Start from what must not happen

Idaho National Laboratory (INL) built its Consequence-driven Cyber-informed Engineering (CCE) method in four phases: Consequence Prioritization, System-of-Systems Analysis, Consequence-based Targeting, and Mitigations and Protections. The first ranks the events that would be catastrophic into high consequence events; the third works out what an adversary would need to know to cause them. SANS calls the starting point a crown jewel analysis, and Weber puts it first: identify the assets and data whose failure would degrade or interrupt critical operations, then test towards them.

On a sterile manufacturing site, the first draft of that list needs no consultant. A steriliser cycle that reports success without reaching its parameters. A freeze-dryer recipe changed between two batches. Environmental monitoring that stops recording in a grade A zone. An electronic batch record that accepts a value nobody entered. Each is a quality event before it is a security event, and each already sits in someone’s risk assessment.

The scope follows from the list. The question is no longer how many hosts the tester can compromise. It is whether someone on the business network, or holding a supplier’s credentials, could reach the place where the steriliser cycle is defined, and whether anyone would notice. The same logic sets the depth of a qualification test, tied to risk and proven by a traceability matrix: the consequence decides how hard you look.

Top down, bottom up, and an attacker already inside

Webb describes the ICS613 method as two directions of work. Top-down follows how an adversary could reach the key industrial control system (ICS) hosts and the information an attack needs. Bottom-up is a passive analysis that starts from the process, builds realistic attack scenarios and looks for the mitigations that break them. The course outline also teaches why an OT test “should follow an assumed breach scenario”.

Assumed breach is where the plant saves money. A week spent proving that the corporate perimeter can be crossed tells the site little it can act on. Starting the tester with an ordinary domain account, or with a supplier’s credentials, spends the week on the boundary the plant owns: the conduit into OT, the jump host, the engineering workstation, the accounts that exist on both sides.

Put remote access in scope from the first day. Verizon’s 2025 Data Breach Investigations Report (DBIR), across all sectors, found exploitation of vulnerabilities as the initial access vector in 20% of breaches, with edge devices and virtual private networks (VPNs) making up 22% of the targets, against 3% in the previous report. On a plant, the supplier’s VPN and the remote support gateway are those edge devices; how to build them is covered in vendor remote access without opening your cell network.

Ask for living-off-the-land techniques too. Webb gives two reasons to use the utilities, vendor tools and built-in functions already on the systems: conventional attack tools can bring unacceptable risk or be blocked, and real ICS adversaries increasingly rely on the same techniques. A tester working with the engineering software already installed shows whether your monitoring would see someone else doing it, which is the practical check of network visibility, the third of the SANS ICS five critical controls.

Where the test runs

One exercise in the ICS613 outline belongs in every contract: validate tools and techniques before using them in production environments. That needs somewhere to validate them, and a plant usually has four places, each answering different questions. The bench means a spare controller or human-machine interface (HMI) of the same model and firmware.

The second row holds most of the value, and most of the illusions. The power sector writes the discipline into its rules. The North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) standard CIP-010-4, Table R3, Part 3.2, asks high impact systems for an active vulnerability assessment at least once every 36 calendar months where technically feasible, in a test environment or in production with adverse effects minimised. A test environment’s differences from production must be documented.

No such rule binds a pharmaceutical site, but the lesson carries over: a twin never compared with production proves things about the twin. A test system also makes testing repeatable. Jeroen Vandeleur, in a SANS post of 14 February 2025, argues for automated adversary emulation in OT with Caldera, whose OT plugins speak Modbus, BACnet and parts of Profinet and IEC 61850. On a test system, a scenario can be replayed after each change.

The cheapest moment comes before validation. A new line at its factory acceptance test (FAT) is complete, on the supplier’s floor and producing nothing; a security window in the FAT protocol costs days, not a shutdown. What the supplier’s tests at FAT and at the site acceptance test can prove for you is set out in what the supplier’s tests can prove, and what stays yours.

Rules of engagement that operations and quality sign

NIST SP 800-115, the technical guide to security testing of September 2008, gives the best skeleton. Its Appendix B is a rules of engagement template: scope, risks and their mitigation, named personnel, schedule, the tools authorised on the network, criteria for halting, the systems in scope and an exclude list, data handling, reporting, and a signature page for at least the test team leader and senior management. On a plant, six additions make it a document operations can live with.

• Signatures from the floor. The area owner and quality assurance (QA) sign beside the security manager. Without the line owner’s name, the test is not authorised on the line.
• An exclude list in process terms. Safety instrumented systems, controllers in an active batch phase, anything whose restart needs a requalification step. Operators do not know hostnames.
• Halt criteria a shift leader can apply. An unexpected alarm, a mode change, a lost HMI connection: the test stops and the plant’s own escalation runs. SP 800-115 also asks assessors to stop on systems involved in a real incident found during the test.
• A tool list, each item proven on the bench. Version, settings and bench result, attached to the rules. Anything not on the list is not used.
• Someone from automation beside the tester at every active step, to translate what the screen shows into what the process is doing.
• A daily status report. SP 800-115 gives daily reports as an example and the ICS613 outline stresses them. On a plant, they tell the morning meeting what changed yesterday.

Then the GMP question: is a test a change? EU GMP Annex 11 §10 says that “any changes to a computerised system including system configurations should only be made in a controlled manner in accordance with a defined procedure.” A mirror port, a temporary account, a tool copied onto an engineering workstation: each is small and reversible, and each belongs in a change record with its rollback. Annex 15 §11.4 has quality risk management evaluate planned changes “to avoid unintended consequences”. On a live line, that is the test plan.

Whether a qualified system needs verification afterwards is a requalification question. Settle it in the change record before the test, not after it.

A report the plant can act on

Weber’s yardstick for a good ICS assessment: it turns technical findings into safety, operational and business risk that leadership can fund and track. He names the classic bad finding: a list of unpatched systems that ignores the regression testing a patch needs in OT and never asks whether the flaw can be reached. The ICS613 outline asks for mitigation options balanced across cost, effectiveness and time.

For a GMP site, that gives four demands. Each finding is tied to one of the events on the original list, or explicitly to none. Each says from where it can be reached: business network, remote access, inside the cell only, or only with a hand on the device. Each states whether the flaw is in the Cybersecurity and Infrastructure Security Agency (CISA) Known Exploited Vulnerabilities catalogue. And each offers a quick compensating control and the durable fix, with what each costs in shutdown and requalification.

Then route the findings into systems the site already runs. Most go to change control. Some become a corrective and preventive action (CAPA). A finding the site decides to live with is a risk acceptance, recorded with a name and a review date. Annex 11 §11 already asks the periodic evaluation of a computerised system to cover its security where appropriate; the test report is the natural input.

Five common mistakes

• Buying the test on the IT framework contract. Ask for named people with industrial experience, and for the rules of engagement before the price.
• Paying to prove the perimeter can be breached. Assume it, and spend the days on the conduit into OT.
• Active scanning first. Documents, configurations and passive captures come first; active steps run on the bench and in a planned window.
• Keeping quality out of it. A test that changes validated systems without a change record creates the very deviation it was meant to prevent.
• Ranking findings by severity score alone. Without reach and consequence, the plan follows the scanner’s priorities instead of the plant’s.

Where to start

1. Write the list of events that must not happen. Operations, automation and QA, one page, five to ten lines. It is done when each line names the system where the event would be caused.
2. Hold the next test proposal against the six additions above. What is missing goes back to the supplier before the order.
3. Find the bench. A spare controller and HMI, the training system, or the next FAT. Record how it differs from production; that record decides which questions it can answer.

A good OT test leaves the line running and the quality file a little thicker. If it does the opposite, it was an IT test that happened to take place on a plant.

References

• Tyler Webb, The Case for Safer, Smarter ICS Penetration Testing, SANS blog, 24 July 2025: sans.org.
• Don C. Weber, ICS Assessments: The Good, the Bad, and the Ugly, SANS blog, 17 October 2025: sans.org.
• Jeroen Vandeleur, Breaking the Grid: Automating Offensive Strategies in OT Security with CALDERA, SANS blog, 14 February 2025: sans.org.
• NIST, SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security, September 2023: csrc.nist.gov.
• NIST, SP 800-115, Technical Guide to Information Security Testing and Assessment, September 2008: csrc.nist.gov.
• Idaho National Laboratory, CCE Phase 1: Consequence Prioritization, 5 May 2020: inl.gov.
• NERC, CIP-010-4, Table R3: nerc.com.
• Verizon, 2025 Data Breach Investigations Report, April 2025: verizon.com.
• EudraLex Volume 4, Annex 11: Computerised Systems (2011): health.ec.europa.eu.
• EudraLex Volume 4, Annex 15: Qualification and Validation (2015): health.ec.europa.eu.