When the MES goes down: availability, store-and-forward and the paper fallback on a GMP line

Twenty to four in the morning, granulation suite two. The fluid-bed dryer is twenty minutes into its drying phase when the operator terminal beside it shows a grey banner: connection to server lost. Nothing on the equipment changes. The inlet air stays at temperature, the PLC keeps counting down, the trend on the local panel keeps drawing. Ten metres away, the dispensing booth cannot print a label for the next batch, and the supervisor’s phone rings because a step is waiting for a second-person verification that the screen will no longer accept.

The question nobody on that shift asks is when the server will be back. The question the site will be asked, three weeks later at batch review, is what happened to the record in between: which steps were executed without the system that is supposed to document them, by whom, against which instruction, and how the gap was closed. An MES that is down is a production problem for an hour. A batch record with a hole in it is a quality problem for the shelf life of the product.

What actually stops when the MES stops

In the ISA-95 model, standardised internationally as IEC 62264, the MES lives at level 3, manufacturing operations management, between supervision at level 2 and the ERP at level 4. That is the first piece of good news: a well-designed line does not need level 3 to run a sequence. If a level 2 function silently depends on a level 3 server, that is a design defect to find now, not during the outage. The levels themselves are explained in the Purdue, ISA-95 and IEC 62443 framework.

What does stop is everything the MES adds on top of control, which on a pharmaceutical site is the part the regulator cares about. First the record: 21 CFR 211.188(b) expects documentation that each significant step was accomplished, with dates, equipment, materials, weights, results and the people who performed and checked it. The dryer carries on; the documentation does not.

Then the checks, often the very reason the MES was bought: material status and expiry, scale and room status, line clearance, weighing tolerances, the hard stop that refuses the wrong lot. Second-person verification and electronic signatures go with them, since they depend on the server and on the directory behind it. And the instructions. Without the screen, operators need a controlled copy of the current master instruction, not the one printed for a training session two years ago and still sitting in a drawer of the airlock.

What the GMP texts ask for

The core requirement is short. EU GMP Annex 11, in force since 30 June 2011, says in §16, Business Continuity: “For the availability of computerised systems supporting critical processes, provisions should be made to ensure continuity of support for those processes in the event of a system breakdown (e.g. a manual or alternative system).” It adds that the time needed to bring the alternative into use should be based on risk, and that the arrangements “should be adequately documented and tested.” The word that sites skip is the last one.

Around it, Annex 11 §7.2 asks for regular back-ups whose integrity and ability to be restored are checked during validation and monitored periodically. Chapter 4 §4.1 accepts records in hybrid form, partly electronic and partly paper, provided the controls between the two are stated. On the US side, 21 CFR 211.68(b) requires a backup file of data entered into the computer, kept exact, complete and secure from alteration or loss.

Two signals are worth reading alongside. PIC/S PI 041-1 (July 2021), §9.10, says hybrid systems need additional controls and that their use is discouraged: the paper fallback is legitimate as a controlled exception, not as a parallel way of working. And the revised Annex 11 put to consultation by the European Commission on 7 July 2025 (closed 7 October 2025; still a draft) would make the expectations explicit: a tested disaster recovery plan with a defined recovery time objective (§15.7), documented restore tests (§16.6), and backup, restore and recovery plans reviewed periodically (§14.2). None of that is binding yet. All of it is what a competent inspector already looks for.

Three numbers before any architecture

Resilience discussions usually start from a product diagram. They should start from three numbers, defined in NIST SP 800-34 Rev. 1 and translated here for a production line.

1. Maximum tolerable downtime (MTD). How long the process can run without the MES before the risk becomes unacceptable. On a packaging line it may be a shift; on a sterile filling line with a complex sequence of electronic checks it may be the time to finish the current batch and stop.
2. Recovery time objective (RTO). How long the MES itself may stay unavailable. NIST notes that the RTO must normally be shorter than the MTD, because reprocessing time has to fit inside it. For the MES, reprocessing means back-entering paper steps and reconciling buffered data, and that takes hours.
3. Recovery point objective (RPO). How much data may be lost. For an executed batch step, the honest answer is none, which is why the RPO of an MES cannot be met by a nightly backup alone.

Write the three numbers per line or per process area, have quality and production sign them, and only then look at what the architecture options can deliver.

The architectures, and what each one survives

The last row is the subject of a February 2026 article in Automation.com, which argues for keeping time-critical execution inside the plant and using the cloud for analytics. For a GMP site that is sound but incomplete: where the software runs decides which failures you survive, not whether the record stays complete. Whatever the hosting, someone on site owns the answer to §16.

Store-and-forward at the line: what it does and what it does not

Store-and-forward means the line-side component, a terminal, an edge gateway or a connector service, keeps writing events to local storage when the central server is unreachable, and sends them on when the link returns. It is the cheapest way to bring the RPO of the line close to zero, and it is often sold as if it solved the whole problem. It does not.

• It delivers at least once, not exactly once. Most industrial messaging is configured for acknowledged delivery. MQTT 5.0, the OASIS standard of March 2019, defines QoS 1 as delivery “at least once”, so a reconnect can deliver an event twice. The MES must recognise duplicates by an event identifier, or the batch record will show the same weighing twice.
• The buffer is not automatically durable. The same specification notes that hardware or software failures may result in loss or corruption of stored session state. A buffer held in memory, or on a gateway with no power hold-up, loses exactly the data it was installed to protect.
• The timestamp must come from the source. An event forwarded two hours late must carry the time it happened, from a synchronised clock, not the time it arrived. Otherwise the record lies politely.
• It cannot make decisions. A buffer records what happened; it cannot answer whether lot 4471 is released, print a label from the central template, or verify a signature against the directory. Every step that needs such an answer must either be designed to run locally with a validated cache, or stop.
• It is GMP data in transit. Buffer, forwarder and reconciliation are part of the validated system; the wider interface problem is covered in why MES projects stall at the shop floor interface.

Size the buffer for the MTD, not for a network blip, and throttle the forwarding: after a long outage it can flood a recovering server.

Degraded mode: running the batch without the system

When the outage outlasts the buffer, or the next step needs a central decision, the site switches to what Annex 11 §16 calls a manual or alternative system: a short, rehearsed procedure in five stages.

• Declare. Production declares the outage, records the time, informs quality. From that moment the batch carries a deviation, even if it ends well.
• Switch. Operators move to pre-approved, numbered paper forms printed from the effective master instruction, never from a folder of spare copies.
• Bound. The procedure lists which steps may continue on paper and which must stop. A step whose control exists only in the MES, such as a hard stop on a wrong lot, either gets an equivalent manual check with a second person, or waits.
• Recover. When the system returns, the paper steps are entered or attached according to a validated procedure, checked by a second person, and the buffered data is reconciled against the paper.
• Review. Batch review sees the hybrid portion flagged, and the deviation records duration, steps affected and the evidence that nothing was lost.

Two details decide whether this works on the night. The forms must follow every master instruction through change control; an electronic master changed without its paper fallback is the most common gap we find. And the night shift must have done it once in an exercise. Read for the first time at four in the morning, the best procedure in the quality system will not be followed.

Backup, restore and the tests that count

A backup proves nothing until it has been restored; the argument is made at length in a backup is a hypothesis, a restore is evidence. For an MES the restore has a specific trap: the database alone is useless without the application version, the configuration, the master data and the electronic signatures’ link to their records. Restore the whole stack, then open a closed batch record and check that it reads, prints and shows its audit trail.

Repeat the restore and failover tests after any change to the backup tooling, the hosting or the database version. Losing power or compressed air is a related failure with its own tests: see designing the degraded-mode tests of an OQ.

Where to start

1. List the MES functions per line and mark, for each, whether it runs locally, needs the central server, or needs another system behind it.
2. Set MTD, RTO and RPO per line with production and quality, and sign them.
3. Compare with the current hosting. The usual gap: an RPO of zero promised, a nightly backup in place.
4. Write the degraded-mode procedure and its forms, linked to the master instructions through change control.
5. Run the four tests in the table, starting with the restore.
6. Put the results in the periodic review, with the deviations from real outages.

Four lines for the specification

• Line components buffer events on persistent storage, for a stated duration, with source timestamps.
• Forwarded events carry a unique identifier and the server rejects duplicates.
• Steps that need a central decision are listed, with their behaviour when the server is unreachable.
• A validated back-entry function flags paper steps in the record and requires a second person.

Five common mistakes

• Treating high availability as continuity. A cluster survives a failed host and replicates a corrupted database just as faithfully.
• A paper fallback nobody has printed. Forms that are not current, not numbered or not on the line are not a manual system in the sense of §16.
• Store-and-forward without deduplication. The record shows two weighings, and someone has to explain which one is real.
• Testing the database restore, not the record. The restore completes, and nobody opens a batch record to see whether it reads, prints and shows its audit trail.
• Hidden level 3 dependencies at level 2. A recipe download, a parameter check or a start permission that silently needs the MES turns a documentation outage into a production stop.

The MES does not need to be unbreakable. It needs a site that knows, in writing and from practice, what it does in the hour after it breaks.

References

• EudraLex Volume 4, Annex 11: Computerised Systems (in operation since 30 June 2011), §7, §16, §17. health.ec.europa.eu
• European Commission, stakeholders’ consultation on Chapter 4, Annex 11 and new Annex 22, opened 7 July 2025, closed 7 October 2025; draft Annex 11 §14.2, §15.7, §16. health.ec.europa.eu
• PIC/S PI 041-1, Good Practices for Data Management and Integrity in Regulated GMP/GDP Environments, 1 July 2021, §9.10. picscheme.org
• 21 CFR 211.68, Automatic, mechanical, and electronic equipment, and 211.188, Batch production and control records. ecfr.gov
• NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, May 2010 (updated November 2010), §3.2.1. csrc.nist.gov
• OASIS, MQTT Version 5.0, OASIS Standard, 7 March 2019, §4.1.1 and §4.3.2. docs.oasis-open.org
• Automation.com, “Architecting a Resilient MES”, Automation.com Monthly, February 2026 issue, 10 February 2026 (the starting point of this article). automation.com