Retrieval-augmented generation (RAG) connects a language model to a chosen collection of source material. For industrial engineering, that might mean manuals, approved work instructions, alarm lists, maintenance reports, electrical drawings and change records. The attraction is obvious: ask a practical question in ordinary language and receive an answer grounded in the documents that matter.
However, “we indexed the PDFs” is a data-ingestion milestone, not a finished engineering solution. A RAG system is only as useful as the documents it retrieves, the context it preserves and the way it handles uncertainty. When retrieval is poor, a more eloquent model often makes the wrong answer harder to notice.
Start with the document lifecycle
Before choosing an embedding model, define source ownership and revision state. Is a file approved, obsolete, a vendor manual for the installed variant or a draft awaiting review? Can the system distinguish a released SOP from a technician’s informal notes? Are access permissions consistent with the document repository? These are information-governance questions, not vector-database features.
Useful metadata often includes document identifier, revision, effective date, equipment model, software version, language, approval status and access class. The query “how do I reset the overtemperature alarm?” has a different risk profile if it retrieves a released procedure for the installed cabinet rather than a generic forum post for another controller.
A defensible retrieval pipeline
- Ingest and classify. Extract text and metadata while keeping a link to the original source. Record extraction errors instead of silently dropping pages.
- Normalise structure. Respect headings, tables, warnings, numbered steps, footnotes and page boundaries. Chunking solely every 1,000 characters can separate a warning from the instruction it qualifies.
- Filter by permissions and applicability. Apply access controls and, where possible, constrain retrieval by equipment, document status and revision.
- Retrieve candidate passages. Use lexical search, semantic search or a hybrid design according to the vocabulary and failure patterns. Exact alarm codes and tag names often benefit from lexical matching.
- Rerank and assemble context. Prefer relevant passages, deduplicate near-identical revisions and retain source identifiers for citation.
- Generate with constraints. Ask the model to answer only from retrieved evidence, identify conflicts and state when sources are insufficient.
- Evaluate and monitor. Store the question, retrieved passage IDs, response, user feedback and model/index version according to the application’s data policy.
Measure retrieval separately from generation
When an answer is wrong, separate the failure into stages. Did the index contain the correct approved procedure? Did search retrieve it? Did ranking place it high enough? Did the prompt preserve its warning? Did the model misinterpret the passage? Did the user ask a question that needed a clarification?
For a test set with known relevant documents, evaluate retrieval recall at a chosen number of results, ranking quality, citation validity and answer faithfulness. Build negative cases where no source supports an answer. If the system is rewarded only for always producing a response, it will learn the wrong operational habit.
| Failure | Likely cause | Engineering response |
|---|---|---|
| Correct answer cites obsolete SOP | Revision metadata missing or filtering absent | Index approval state and enforce effective-revision selection |
| Exact alarm code is not found | Semantic retrieval misses a lexical identifier | Add keyword search or hybrid retrieval |
| Safety warning omitted from summary | Warning was chunked away from its procedure | Use structure-aware chunking and test warning preservation |
| Model invents a source | No citation validation or weak evidence constraint | Return source IDs from retrieval, validate links and allow abstention |
Industrial content needs special care
Drawings may encode meaning visually rather than as extractable text. Tables can lose column relationships during conversion. Handwritten maintenance notes can be ambiguous. A document mentioning “Pump B” may refer to different assets across sites. Each limitation should be made visible in the system’s applicability rules and test cases.
Document permissions must survive retrieval. A user who cannot open a restricted validation report should not be able to ask the assistant for the same report through a chat interface. Access checks belong before context reaches the model, not merely in the user interface after the answer has been generated.
A sensible pilot
Begin with one well-maintained domain—perhaps a defined equipment family and a small set of released troubleshooting procedures. Create a golden set of questions with expert-approved evidence, include ambiguous and unanswerable queries, and compare the tool against the current search process. Measure time saved and answer quality, not the number of PDFs indexed.
RAG is valuable when it helps engineers reach the right evidence faster and keeps that evidence visible. The model supplies the conversation; the retrieval and governance layers supply the discipline.