The OT remediation cycle: the step you never skip

This is part 9 of 12 of the “Common language” series on industrial cybersecurity; it continues with part 10, PHA and LOPA as cyber risk.

An industrial security programme produces gaps. An audit finds shared accounts, a scan finds an unpatched engineering workstation, a walk-down finds a forgotten 4G modem on a skid. What happens to each of those gaps afterwards decides whether the programme actually lowers risk.

Remediation, meaning the treatment of each identified gap, follows its own cycle. It repeats gap after gap and campaign after campaign. Risk goes down because of that disciplined repetition, not because of the size of any single project. A large one-off hardening campaign that nobody maintains is worth less, two years later, than a modest cycle that keeps turning.

Five steps that repeat

The cycle has five steps. Each new vulnerability, each change on the installation and each audit starts it again.

1. Identify assets, flows, accounts, vulnerabilities and risks.
2. Prioritise by criticality, risk and impact on production.
3. Remediate through patches, hardening, segmentation, access control and account clean-up.
4. Validate that the measure works, and that it does not disturb the industrial process.
5. Maintain through follow-up, monitoring, audits and continuous vulnerability management.

Written like this, it could be an IT process. The difference lies in how each step is carried out on a running plant, and above all in step 4.

The cycle applied to an industrial site

Identify: the scan is not the inventory

A passive network probe or a vulnerability scan shows what talks on the network. It does not show the serial link to an analyser, the vendor’s remote-service box behind a panel, or the laptop that only connects during maintenance. Walking the unit and opening cabinets with the people who run it is slower. It is also the only way to find what matters most: the connections nobody remembers.

On a pharmaceutical site the inventory has a head start. Computerised systems that support GMP activities already have an owner, a validation file and a configuration record. Those files rarely describe network flows or remote access, but they give the list of systems, their versions and the people who answer for them. Starting from them saves weeks, and it ties each security gap to a system that the quality unit already follows.

Prioritise: a score is an input, not a decision

A CVSS score describes the technical severity of a vulnerability in general. It knows nothing about whether the affected controller runs a reactor feed or a car-park barrier. The CVSS v4.0 specification itself says as much: it presents the score as one input to an organisational vulnerability management process that also weighs factors outside the scope of CVSS, such as life or property threatened, regulatory requirements or reputational impact.

Two public tools help put consequence back into the decision. CISA’s Known Exploited Vulnerabilities (KEV) catalogue lists vulnerabilities for which there is reliable evidence of active exploitation in the wild, which separates what is being used by attackers from what is merely possible. CISA’s Stakeholder-Specific Vulnerability Categorization (SSVC) is a decision tree that considers exploitation status, technical impact, whether exploitation can be automated, mission prevalence and public well-being impact, and ends in one of four actions: Track, Track*, Attend or Act. Neither replaces the conversation with operations and HSE, but both frame it better than a number out of ten.

In practice, EUCALEX suggests three questions for each gap, asked in the same meeting: is it being exploited somewhere (KEV), what can it do to this process here (the site’s own consequence ratings), and when is the next window in which it can be treated without stopping production. The answers set the order; the score only breaks ties.

Remediate: change is managed, or it is not done

In process industries, management of change is not a security invention. It comes from process safety: in the United States, for example, OSHA’s Process Safety Management standard (29 CFR 1910.119, paragraph (l)) requires written procedures to manage changes to process chemicals, technology, equipment and procedures, and changes to facilities that affect a covered process, replacements in kind excepted. A patch, a firewall rule or a disabled service on a control system is exactly that kind of change. It goes through the same MOC as any other modification, agreed with the vendor and scheduled for a shutdown window where one is needed.

A GMP site already has the equivalent. EU GMP Annex 11, section 10, states that “any changes to a computerised system including system configurations should only be made in a controlled manner in accordance with a defined procedure”. A security measure on a validated system is a change like any other, and the quality unit will expect to see it in the change record, with its impact assessment and its tests. The change control article in this journal describes how to route such changes to the lightest defensible path.

ISA/IEC 62443 addresses patching specifically in IEC TR 62443-2-3:2015, Patch management in the IACS environment, a technical report rather than a normative standard. Where a system cannot be patched because it is out of support, validated by the vendor on one version only, or cannot be stopped, the answer is a compensating measure: hardening, isolation, stronger monitoring. That measure should be documented, dated and reviewed, not simply noted as an exception and forgotten; the article on shielding legacy PLCs shows what such a record looks like.

Validate: the step that makes it OT

This is the step most often skipped, and the one that separates OT remediation from IT remediation. In IT, a change that breaks something gets rolled back and the service comes back. In OT, a change that slows a screen refresh, adds latency to a control loop or locks an operator out of a faceplate at the wrong moment affects production and, potentially, safety.

Validation therefore asks two questions, and both must be answered. Is the measure effective: does the firewall rule actually block the flow, is the account really gone? And does the plant still run as before: can operators work, do alarms arrive, do the vendor’s diagnostic tools still function? Answering the second means involving operators, not only the security team.

The cost of skipping it shows up quickly. In EUCALEX’s field experience, a measure that degrades operation is typically worked around within a few weeks: a shared password reappears, a cable bypasses the firewall, an exception becomes permanent. The gap is back, and now nobody records it.

On a validated system, the two questions map naturally onto the test evidence the change record already requires. The effectiveness test proves the security objective; the non-regression test proves that the qualified functions still behave as specified. Writing both into the same change avoids a second visit, and a second argument with the shift team.

A simple validation test: before closing a remediation action, ask the shift team whether anything in their work changed. If the answer is yes and nobody planned for it, the action is not finished.

Maintain: an owner and a cadence

A hardened configuration drifts. New equipment arrives, vendors visit, emergencies justify shortcuts. EUCALEX’s observation is that, without a named owner and a review cadence, remediation gains usually erode within a year. Maintenance means someone is accountable for each measure, a vulnerability register records a dated decision for each gap (fix, compensate or accept), and periodic audits check that what was closed stays closed. Each finding from those audits feeds step 1 again.

GMP sites can hang this on an existing routine. Annex 11, section 11, asks for computerised systems to be periodically evaluated to confirm that they remain in a valid state, and lists security among the points such an evaluation should include where appropriate. Adding the register’s open items and the compensating measures due for review to that periodic evaluation costs little and keeps the cycle turning.

Three views of one programme

This series uses three grids, and they are easy to confuse. The seven domains of a site security standard (asset inventory, architecture, access, vulnerabilities, incident response, backup, detection) say what must be covered. The five programme functions, from Govern and Identify to Recover and Improve, say how the programme is organised and in what order. The remediation cycle says how each individual gap is treated once the programme is running.

One programme, then, looked at from three sides. The five functions are this series’ own grouping. The NIST Cybersecurity Framework counts six since version 2.0, Govern among them as a function in its own right, at the centre. Attaching governance to identification and improvement to recovery is this series’ choice of presentation, not NIST’s. Keeping the three grids distinct is one more reason the vocabulary has to come first.

Where to start

1. Open a register. One line per gap, with the system, the owner, the decision (fix, compensate or accept), the date and the next review. Start with what the last audit and the last walk-down already found.
2. Agree the priority rule with operations and HSE. Exploited or not (KEV), consequence on this process, next available window. Write it on one page and use it in every meeting.
3. Route every measure through the existing MOC or change control. No separate security change process: one form, with a cyber question added.
4. Add the two validation questions to the closure criteria. Effective, and the plant still runs. Nothing is closed without an answer to both, signed by someone from operations.
5. Give the register a cadence. A monthly review of open items, and a place in the periodic evaluation of each system.

Five common mistakes

• Treating the scan as the inventory. The connections that matter most are the ones that do not answer a scan.
• Ranking by score alone. A critical score on a car-park barrier does not outrank a medium one on a reactor feed.
• Running security changes outside MOC. The fix may be right, but an unrecorded change on a control system is a new gap.
• Closing without validation. The measure is bypassed within weeks and the register still says “done”.
• Accepting risk without a date. An exception with no review date is a permanent gap with a signature on it.

Remediation lowers risk only when each gap goes all the way round, and validation is the step that brings it home.

References

• FIRST, Common Vulnerability Scoring System v4.0, Specification Document. first.org
• CISA, Known Exploited Vulnerabilities Catalog. cisa.gov
• CISA, Stakeholder-Specific Vulnerability Categorization (SSVC). cisa.gov
• IEC TR 62443-2-3:2015, Patch management in the IACS environment. webstore.iec.ch
• European Commission, EU GMP Annex 11, Computerised Systems (2011), sections 10 and 11. health.ec.europa.eu