What a retrieval system on technical documentation actually takes

The work spans a wide set of subjects: IT, OT, networking, automation, industrial systems, electronics, mechanics, cybersecurity, data integrity, quality, standards and regulations. Nobody holds all of that in their head, and pretending otherwise is how mistakes get made.

In this work the difficulty is rarely that something is unknown. The answer usually exists — in a vendor manual from 2011, in a standard nobody has opened for two years, in the notes from a site visit last spring. Finding it costs more time than the question deserves. A retrieval system makes nobody cleverer; it shortens the distance between a question and the document that already answers it. Across ten disciplines, that distance is the real bottleneck.

Four things decide whether it works, and the model is not one of them

We have spent a good deal of time on retrieval over a curated technical library, and one lesson survived every iteration: the model is the least interesting variable. Swapping it changes the tone and the speed. Changing how documents are prepared and retrieved changes whether the answer is right. Four things carry almost all of the result.

• What you let in. Sources have to be clean, relevant and extractable, and the formats fight back: PDF, DOCX and spreadsheets each lose something different on the way in. A scanned manual with no text layer, a PDF exported from a drawing tool, a spreadsheet where the meaning lives in the cell colours — each of these quietly contributes nothing while appearing to be indexed. Deciding what to leave out matters as much as deciding what to add.
• How you cut it up. Strip too little in cleaning and headers, page numbers and footers pollute every chunk; strip too much and you lose the section titles that tell the model what it is reading. The splitting strategy itself — by meaning, by structure, or recursively — changes the answer more than any other single choice. This is the part worth testing first, and it is the subject of a separate article on chunking.
• What you push out before the model sees it. A reranking layer removes the passages that matched on words but not on meaning. Unglamorous, and the difference between an answer that cites the right clause and one that cites a neighbouring one.
• Where it runs. A model that fits on the hardware you have is smaller than one that does not; that is the cost. The benefit matters more in this field. Vendor manuals, internal procedures and site documentation never leave the network they belong to, which is the only version of the question a quality department will accept.

Where the line sits

One point needs saying plainly, because on a production site it decides whether this is useful or dangerous. The assistant informs a decision; it does not take it. It drafts the assessment, finds the clause, recalls the parameter — and a person signs. In a regulated environment the signature carries a responsibility that cannot be delegated to a system nobody can question.

That is not a reservation about the technology. It is what makes it usable at all: an output that is presented as an aid, traceable to the passage it came from, and reviewed by someone competent, fits inside the way a GxP site already works. An output presented as an answer does not.

What a site should require before it trusts one

The questions below are the ones worth asking of any retrieval assistant on regulated documentation, whether it is built in-house or bought.

1. Which documents are in it, and at which revision? A library that answers from a superseded procedure is worse than no library. The index needs the same revision discipline as the documents it holds.
2. Can every answer be traced to its passage? An answer without a citation to the source paragraph cannot be checked, and what cannot be checked cannot be used in a decision that gets signed.
3. Where do the documents and the queries go? If either leaves the site’s control, the question becomes a data classification problem before it is a useful tool.
4. Who reviews it, and how often? A retrieval system drifts the way any uncontrolled document does: silently, as the library around it changes.
5. What does it refuse to answer? A system that always produces something is a system that will produce something wrong, fluently, on the day it matters.

The model is the least interesting variable. What the system is fed, how it is cut up, what it can prove about its own answer — those decide whether it earns a place in the way a site works.