From risk analysis to test plan: every risk line must end in a named control

A risk analysis published for a pilot capsule filler lists 33 risks. Every single line scores 2, on a scale that runs from 1 to 3. The scoring decided nothing. The analysis still did its job, though: 14 lines end in a named installation or operational qualification test, flagged as blocking, and the other 19 point to a procedure, a training record, an in-process weight check, release testing, or a qualification done on the room and the utilities.

That split is the useful output. It tells the protocols what they must contain, and what they can leave to someone else. The example runs through this article, anonymised, precisely because it is an ordinary complete case from a real qualification and not a textbook one.

The chain every line has to complete

The columns of a risk file are there for a reason. Each one answers a question, and a line is only closed when the last one has an answer.

• Function. Start from what the equipment does, function by function, and write down whether each one is critical for the product, with the reason. “Capsule closing: critical, because poor closure affects the dose delivered and the pack.” That short criticality analysis keeps the risk file from drifting into generalities.
• Failure and cause. The failure is the step not delivering its result. The causes sit at the interfaces of that step: materials, the machine and its software, method, people, environment. A capsule that does not open is a failure; a pre-closed capsule or weak suction is a cause.
• Control, and where it sits. Name it: a detector and its ejector, an interlock, a design feature, a procedure, a training record, an in-process control, a finished-product test, another qualification. “Under control” with nothing named is not a control.
• Linked test. Where the equipment carries the control, name the test that proves it and the stage. In the capsule-filler file the pattern is clear. Installation qualification (IQ) checks presence: certificates, detectors, format identification, the levelling device. Operational qualification (OQ) checks function: unopened capsules ejected, no oil traces, correct closure, data recoverable after a power cut. Annex 15 lists what each stage should include (§3.9, §3.11).
• Blocking or not. A blocking test must pass before the next stage or before release. Mark the others non-blocking, with a reason. Annex 15 allows conditional approval to move on “where certain acceptance criteria or deviations have not been fully addressed”, provided a documented assessment shows no significant impact on the next activity (§2.10). A blocking test that fails does not stop the project. It goes on the reserve list, and the stage is not released until it is closed.

Two habits make the chain work. Write the risk file before the protocols; otherwise it turns into a justification of tests somebody had already written. And give each line an owner, meaning whoever will produce the evidence, which is not necessarily whoever wrote the line.

Score severity and occurrence on scales that mean something

Scales only help when each level is anchored to something a group can agree on. The scale below is the one we use. It scores severity against three references (patient, product and production tool) and scores occurrence separately for human and equipment causes. The anchors matter far more than the numbers.

Two details in that table are worth a second look. An invisible product defect scores worse than a visible one, because nothing downstream will catch it. And for human causes the best achievable level is “rare”. People are never a very rare cause, which is why a control that depends only on training is a weak control.

Detection is the third score in most methods, and that is where the trap is. Detection is itself a control, so if you count it, name it and say who verifies it. A drift that is “detectable” only because somebody weighs capsules at the end of the line stays detectable only as long as that weighing happens, at that frequency, and gets recorded.

Most risks are not controlled by qualification

Count that file line by line and you get a picture worth showing to anyone who believes qualification is where risk gets handled.

This changes the way the file is written. A control outside the equipment still needs an address: the procedure number, the in-process control in the batch record, the release test, or the qualification report of the room, the air-handling system or the compressed-air network. A risk line that points at “cleaning procedure” without a number can be verified by nobody except its author.

Dependencies have to travel, too. Eight lines of that file rely on capsules being weighed by other means, because the filler has no weight control. That dependency belongs in the batch record and in the qualification of the balance or checkweigher used, and if the in-process frequency changes later, the risk analysis changes with it.

Qualification, in the end, tests what the equipment carries. The rest is verified elsewhere, and the qualification report says where. That is what keeps protocols short and defensible.

When every line scores the same, the score is decoration

Back to the 33 lines that all scored 2. Direct impact was answered “yes” everywhere and drift was answered “detectable” everywhere, so the arithmetic could only produce one result. Nothing was ranked. The real decision (does the machine carry a control we can test?) was taken in another column.

That is no reason to drop scoring, but it is a reason to check it. Anchor every level with a description, as in the table above, and score as a group that includes production, maintenance and quality assurance. Record the reason for each score in one line. ICH Q9(R1) added a whole section on managing and minimising subjectivity, because “subjectivity can impact every stage” of risk management, especially hazard identification and the estimation of probability and severity (§5.3).

When the file is complete, look at the distribution. If nearly every line shares one score, the scale is not discriminating, and the file needs a better question rather than more cells. Keep the formality proportionate as well: ICH Q9(R1) says formality is “not a binary concept” and that higher uncertainty, importance or complexity “may require more formal” approaches (§5.1). A vortex mixer does not need the method you use for a networked chromatography system.

Match the method to the system

One template for everything produces either paperwork or blind spots. Three levels cover most of what a site owns.

• Simple instrument, no PC (balance, pH meter, osmometer): a maintenance-oriented analysis, component by component. Its output is a frequency rather than a test plan: what to verify, how often, and which tests to repeat when a given part is replaced. One file can serve a whole instrument family.
• Stand-alone instrument driven by one PC (a spectrophotometer with its own workstation): a simplified analysis in four parts, namely physical and IT environment, security and data, software functions, instrument functions. Output: the tests to run beyond the supplier’s, and a traceability matrix.
• Networked systems, configured or custom software: a failure mode, effects and criticality analysis (FMECA), scoring severity, occurrence and detectability, then risk class and priority. Output: which functions get which depth of test.

Which level applies follows from how the system is categorised, in the GAMP 5 sense. ICH Q9(R1) collects the usual tools in its Annex I, and lists the scope and extent of qualification of production equipment and laboratory instruments among the applications in Annex II.4.

What the regulation actually asks

Annex 15 (2015) is short and direct here. Decisions on the scope and extent of qualification and validation “should be based on a justified and documented risk assessment” (Principle). A quality risk management approach “should be used for qualification and validation activities”. The assessments are repeated as knowledge grows, and “the way in which risk assessments are used to support qualification and validation activities should be clearly documented” (§1.7).

No method is imposed. What an inspector needs to see is the link: this risk, so this control, so this test, so this result. ICH Q9(R1) adds that the level of effort, formality and documentation should be “commensurate with the level of risk”.

The risk file is a living document. It gets updated when the process changes, when a deviation reveals a hazard nobody listed, and when a modification changes what the equipment controls; see When does a repair become a requalification?.

The mistakes we see most often

• Lines that end in “the procedure”. A standard operating procedure is a control only when it is named, numbered and owned. The same goes for “training” on its own.
• Risk analysis written after the protocols. The file then explains the tests instead of choosing them.
• Detection counted twice. A control is credited once for severity and again for detection, and the line quietly drops out of the test plan.
• No blocking flag. Everything is “important”, so nothing stops a release, and the argument happens at the end of the project instead of the start.
• A file that never changes. Two years of modifications, one untouched risk analysis, and a periodic review that finds the equipment no longer matches its own analysis.

Once the plan exists, the next question is how deep each test goes and how coverage is proven: see How deep should a test go?. The chain itself starts in the specification, covered in Writing a URS that actually drives the qualification.

A risk analysis is finished when every line names its control and says where that control is proven, not when every cell holds a number.

References

• European Commission, EudraLex Volume 4, EU GMP Annex 15, Qualification and Validation, 2015, in operation since 1 October 2015.
• ICH, Q9(R1) Quality Risk Management, adopted 18 January 2023.
• ISPE, GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems, Second Edition (July 2022).