My job involves a large amount of detail that exists in documents: which configuration is supported on which version, what the caveat on page 94 says, which combination is validated and which merely works. It is documented, and it is unfindable at the moment it is needed.
This is the ideal case for retrieval-augmented search. It is also material I cannot send to a third-party service, because much of it is under confidentiality terms.
So it has to run locally. Here is what I have learned building it.
The constraint shapes everything
Local means a machine you own, models that fit in the memory you have, and no external calls. That rules out the largest models and it turns out not to matter much, because the hard part is not generation.
The system is a search engine with a summariser on the end. If retrieval returns the right passage, a modest local model answers perfectly well. If retrieval misses, no model saves you. Effort belongs in retrieval.
The document problem
The sources are PDFs, and PDFs are hostile. Text extraction produces reading orders that jump between columns, tables that dissolve into unaligned numbers, and headers repeated on every page.
Tables are the worst part and they are where the answers live. A specification table that extracts as a stream of numbers with no structure is worse than useless, because it retrieves on keyword match and then misleads.
What helped: extracting tables separately and keeping them whole as single chunks, with the surrounding heading text attached. A table that fits in one chunk and carries its own context is retrievable. A table split across three chunks is three pieces of misinformation.
Metadata is most of the value
Every chunk carries the source document, the product, the version, and the section path. This matters more than any embedding choice, because my questions almost always have an implicit filter: which product, which version.
Filtering the candidate set by metadata before searching improves results more than any tuning of the retrieval itself. This is ordinary database thinking and it gets skipped because pipeline diagrams do not have a box for it.
Hybrid retrieval, always
Half my queries contain identifiers: model numbers, version strings, feature names. Vector similarity treats those as approximately interchangeable with their neighbours, which is exactly wrong. Keyword search handles them precisely.
Running both and merging is not sophisticated and it is the difference between a system I trust and one I do not.
The citation requirement
Every answer shows the source document and page. Not for tidiness. Because I have to be able to verify before I repeat something to a customer, and because a confident wrong answer in this domain is worse than no answer.
An answer I cannot check is an answer I cannot use, so the system that cannot cite is not useful regardless of how good its prose is.
What it has changed
The lookups that used to take fifteen minutes of searching a PDF take thirty seconds. It has not replaced knowing the material; it has replaced remembering where the material is, which is the part that was consuming time.
What I would tell someone starting
Spend your time on extraction quality and metadata. Use hybrid retrieval from day one. Insist on citations. Build a set of twenty real questions with known answers and measure whether the right passage is retrieved, because that single number tells you whether your problem is search or generation.
And keep it local if the material requires it. The constraint is less limiting than it sounds, and it removes an entire category of conversation you would otherwise have to have with your legal team.