Skip to content
Hakan KaracaENGINEERING JOURNAL
AI & Intelligent AutomationEki 8, 20264 min read

RAG for Industrial Documentation: Retrieval Quality Is the Product

Industrial organizations often have the right answer somewhere in a repository and still cannot find it quickly.

Retrieval-augmented generation (RAG) connects a language model to a chosen collection of source material. For industrial engineering, that might mean manuals, approved work instructions, alarm lists, maintenance reports, electrical drawings and change records. The attraction is obvious: ask a practical question in ordinary language and receive an answer grounded in the documents that matter.

However, “we indexed the PDFs” is a data-ingestion milestone, not a finished engineering solution. A RAG system is only as useful as the documents it retrieves, the context it preserves and the way it handles uncertainty. When retrieval is poor, a more eloquent model often makes the wrong answer harder to notice.

Start with the document lifecycle

Before choosing an embedding model, define source ownership and revision state. Is a file approved, obsolete, a vendor manual for the installed variant or a draft awaiting review? Can the system distinguish a released SOP from a technician’s informal notes? Are access permissions consistent with the document repository? These are information-governance questions, not vector-database features.

Useful metadata often includes document identifier, revision, effective date, equipment model, software version, language, approval status and access class. The query “how do I reset the overtemperature alarm?” has a different risk profile if it retrieves a released procedure for the installed cabinet rather than a generic forum post for another controller.

A defensible retrieval pipeline

  1. Ingest and classify. Extract text and metadata while keeping a link to the original source. Record extraction errors instead of silently dropping pages.
  2. Normalise structure. Respect headings, tables, warnings, numbered steps, footnotes and page boundaries. Chunking solely every 1,000 characters can separate a warning from the instruction it qualifies.
  3. Filter by permissions and applicability. Apply access controls and, where possible, constrain retrieval by equipment, document status and revision.
  4. Retrieve candidate passages. Use lexical search, semantic search or a hybrid design according to the vocabulary and failure patterns. Exact alarm codes and tag names often benefit from lexical matching.
  5. Rerank and assemble context. Prefer relevant passages, deduplicate near-identical revisions and retain source identifiers for citation.
  6. Generate with constraints. Ask the model to answer only from retrieved evidence, identify conflicts and state when sources are insufficient.
  7. Evaluate and monitor. Store the question, retrieved passage IDs, response, user feedback and model/index version according to the application’s data policy.

Measure retrieval separately from generation

When an answer is wrong, separate the failure into stages. Did the index contain the correct approved procedure? Did search retrieve it? Did ranking place it high enough? Did the prompt preserve its warning? Did the model misinterpret the passage? Did the user ask a question that needed a clarification?

For a test set with known relevant documents, evaluate retrieval recall at a chosen number of results, ranking quality, citation validity and answer faithfulness. Build negative cases where no source supports an answer. If the system is rewarded only for always producing a response, it will learn the wrong operational habit.

Failure Likely cause Engineering response
Correct answer cites obsolete SOP Revision metadata missing or filtering absent Index approval state and enforce effective-revision selection
Exact alarm code is not found Semantic retrieval misses a lexical identifier Add keyword search or hybrid retrieval
Safety warning omitted from summary Warning was chunked away from its procedure Use structure-aware chunking and test warning preservation
Model invents a source No citation validation or weak evidence constraint Return source IDs from retrieval, validate links and allow abstention

Industrial content needs special care

Drawings may encode meaning visually rather than as extractable text. Tables can lose column relationships during conversion. Handwritten maintenance notes can be ambiguous. A document mentioning “Pump B” may refer to different assets across sites. Each limitation should be made visible in the system’s applicability rules and test cases.

Document permissions must survive retrieval. A user who cannot open a restricted validation report should not be able to ask the assistant for the same report through a chat interface. Access checks belong before context reaches the model, not merely in the user interface after the answer has been generated.

A sensible pilot

Begin with one well-maintained domain—perhaps a defined equipment family and a small set of released troubleshooting procedures. Create a golden set of questions with expert-approved evidence, include ambiguous and unanswerable queries, and compare the tool against the current search process. Measure time saved and answer quality, not the number of PDFs indexed.

RAG is valuable when it helps engineers reach the right evidence faster and keeps that evidence visible. The model supplies the conversation; the retrieval and governance layers supply the discipline.

References and further reading

  1. NIST AI Risk Management Framework
  2. NIST AI RMF Playbook