How deep should a qualification test go? Tie test depth to risk, then prove coverage with a traceability matrix

The instrument software has more than a hundred functions. The supplier’s protocol runs to sixty pages. The qualification plan says “test the system”. Two months later the team is writing scripts for a report format nobody will print, and the one question an inspector is sure to ask has not been tested: can an analyst change a result without leaving a trace?

Which risks need a control at all is settled earlier, in the risk analysis. This article takes the risks as given and answers what follows: what gets tested, how deep, by whom, and how the traceability matrix proves it afterwards.

Qualify what you use, and say what you do not

Start with an inventory of functions, family by family: administration and user management, methods, instrument control and acquisition, data processing, reports. The supplier’s documentation is where that list comes from, and reading it is how the team learns the system.

Then hold one meeting with the users and the system owner, and mark every function used or not used. The threshold is worth stating plainly: once a system offers more than a hundred functions, only the functions that are needed and applicable are qualified. Testing the rest costs weeks, generates deviations you then have to investigate, and proves nothing about the way you use the system. It can even suggest the site does not know what it uses.

“Not used” has to stay true. Three things make it hold:

• Restrict it. Where the software allows, switch the function off or remove it from the roles that could reach it. A function nobody can start needs no test.
• Record it. The inventory, with the decision and the reason, belongs in the qualification plan, not in someone’s notes.
• Revisit it. A function declared unused and quietly adopted a year later is an unqualified function inside a qualified system. The periodic review and change control catch that.

Depth follows what happens when the function fails

How deep to test is decided by one thing: what a failure of that function would do to product quality, patient safety or data integrity, in other words its impact under good practice (GxP) rules. Four levels are enough.

The same rule can be put as repetition: minor functions not tested, major functions once, critical functions several times in different contexts. The more the product depends on a function, the more ways you try to break it.

The same trade appears in the US Food and Drug Administration’s draft guidance on computer software assurance, published in September 2022 for medical device production and quality system software: scripted, traceable testing where process risk is high, lighter unscripted methods (scenario testing, error-guessing, exploratory testing) where it is not.

Four kinds of test, and what each one proves

The depth table names four test types. A protocol that only ever contains the first two has never tried to fail.

• Functional. The elementary check that a component or function works in a standard configuration. Usually the supplier’s test, at the factory or at installation.
• Nominal. The function works with your parameters, over the range you use, in your environment. Proves the system does what it should: a method runs and gives the expected result. A filling line holds the fill weight at the set speed.
• Failure. The system’s response to abnormal conditions. It proves the system does not do what it is not supposed to do. Three failed log-ins and the account locks. An analyst profile tries to modify a method and cannot. Someone deletes a result file through the operating system rather than the application, and fails, or is recorded. The power drops mid-run and the batch data survive.
• Limit, or worst case. Operation at the extremes of the qualified range. Replicate injections on the most viscous sample rather than the easiest one. The balance at the bottom of its working range. The line at maximum speed with the smallest format.

Two sentences of regulation sit behind that picture. Annex 15 asks the operational qualification (OQ) to include “tests to confirm upper and lower operating limits, and /or ‘worst case’ conditions” (§3.11). Annex 11 asks that “system (process) parameter limits, data limits and error handling should be considered” (§4.7). Error handling is the failure test, in two words, and failure tests are also the ones that fit on no axis at all: a wrong password, a lost connection, a second operator logging in during a run.

Supplier tests first, internal tests for what is left

Read the supplier’s protocol before writing a single script. Annex 15 expects the site to confirm that third-party protocols are suitable and comply with internal procedures before approving them, and adds that “vendor protocols may be supplemented by additional documentation/test protocols before use” (§2.6). Supplemented, not replaced.

Map the supplier’s tests onto your risks first. What remains uncovered is your internal test list, usually shorter than the team fears and never empty. The gaps recur from site to site. The thesis case study found these on a system whose software was genuinely compliant with 21 CFR Part 11:

• deactivated user IDs that can be created again and reassigned to someone else;
• an audit trail that a power-user profile can edit or switch off;
• results and methods that can be deleted through the operating system’s file explorer, outside the application;
• a maintenance or acquisition module from which an operator can start a run that nobody expected;
• a backup that has never been restored.

A compliance statement is not test coverage. The supplier tests its product against its own specification. You test your system against your requirements, your roles and your IT policy.

The matrix that proves coverage

The traceability matrix turns this into evidence. It starts at the user requirements specification, one row per testable requirement, and gains a column at each stage.

Six columns are enough: the requirement, the risk and its level, the test and its type, the phase, who runs it, and the result with its report reference. The matrix belongs to the qualification plan, with one owner and one version, approved with the plan and annexed to the report, never to a personal spreadsheet. Two rules keep it honest. Every test derives from a risk, so a test with no risk behind it is one nobody needs. And every major or critical risk ends in a test that passed, or in a written reason why the control is verified elsewhere.

The phase column matters too. Installation qualification (IQ) checks presence, installation and configuration against the technical specification. OQ checks that functions behave, and that is where failure and limit tests belong. Performance qualification (PQ) checks the process around the system: procedures, trained people, performance over time, and the restore that Annex 11 asks to be checked during validation and monitored periodically (§7.2).

That calculation row is not optional in the US: 21 CFR 211.68(b) requires input to and output from the computer system to be checked for accuracy, to a degree based on the system’s complexity and reliability.

The same matrix carries the integration and user acceptance tests of a computerised system, which sit alongside OQ and PQ rather than replacing them.

What an inspector follows, and what breaks it

An inspector rarely reads a matrix from the top. They pick something they care about (the audit trail, a calculation, an access rule), and follow it. Forwards: which requirement covers it, which risk, which test, what result, who signed the report. Backwards: this test failed, so which requirements are affected and what was decided. A matrix that works in only one direction is half a matrix.

What breaks it:

• Built at the end, from the protocols. Reverse traceability proves the tests exist. It cannot prove the risks are covered: it starts from the wrong end.
• References that drift. The supplier revises its protocol, test 2.4 becomes test 2.6, and forty cells point at nothing. Reference the protocol version, not only the test number.
• One row per document instead of one row per requirement. “Covered by the supplier OQ” tells the inspector where the binder is, not which test covered which risk.
• “N/A” with no reason. Every empty cell needs one: covered by supplier test X, covered elsewhere, or function not used.
• Frozen at go-live. A new software version, a new role, a new format: the matrix is maintained under change control, or it stops describing the system as used.

What must pass before routine use

The qualification plan should say, before execution starts, which tests are blocking: no routine use until they pass. Annex 15 allows conditional approval to move to the next stage where acceptance criteria or deviations “have not been fully addressed”, provided a documented assessment shows no significant impact on the next activity (§2.10). Everything else goes on the reserve list, with an owner and a date.

Deciding that before the first test is run keeps the end of a project from becoming a negotiation.

Start the matrix with the first requirement and keep hold of it until the last signature. Written at the end, it only tells you which tests were run.

References

• EudraLex Volume 4, EU GMP Annex 15, Qualification and Validation, in operation since 1 October 2015, §2.6, §2.10, §3.11: health.ec.europa.eu
• EudraLex Volume 4, EU GMP Annex 11, Computerised Systems (2011), §4.7 and §7.2: health.ec.europa.eu
• US 21 CFR 211.68(b), input and output of computer systems checked for accuracy: ecfr.gov
• US FDA, Computer Software Assurance for Production and Quality System Software, draft guidance, notice of availability 13 September 2022: federalregister.gov
• ICH Q9(R1), Quality Risk Management, Step 4, 18 January 2023: database.ich.org