I have been building a personal knowledge base for my own work: product specifications, caveats, the unsupported configurations that are documented in a footnote on page 94 and nowhere else. The kind of thing that is technically available and practically unfindable when you need it in a meeting.
The obvious architecture is retrieval-augmented generation, and the obvious first version works surprisingly well and then plateaus. Understanding the plateau is the interesting part.
What actually breaks
When a RAG system gives a bad answer, the instinct is to blame the model, and the fix people reach for is a better prompt or a bigger model. In my experience the failure is almost always upstream: the retrieval step did not return the passage containing the answer, so the model was asked to answer from material that did not contain it.
A model given the right passage answers correctly and unremarkably. A model given the wrong passages produces a fluent answer built from whatever it was handed, which is the worst possible failure mode because it looks like success.
So the debugging discipline is: before you change anything about the prompt, look at what was retrieved. Every time. If the answer is not in the retrieved chunks, no amount of prompt engineering will save you.
Chunking is the highest-leverage decision
You are cutting documents into pieces and embedding each piece. How you cut them determines what can ever be found.
Fixed-size chunks are the default because they are easy, and they are wrong for technical documentation specifically. A specification table split across two chunks is now two chunks, neither of which contains a complete fact. A caveat separated from the configuration it applies to is a landmine.
What worked better for me: chunk on document structure. Headings, table boundaries, list boundaries. Keep a table whole even if it is large. Carry the section heading into every chunk from that section, so a chunk about maximum supported nodes still contains the word for the product it belongs to.
That last trick is unreasonably effective and costs nothing. A chunk that reads “maximum 32 per cluster” is useless. The same chunk prefixed with its document title and section path is findable.
Embeddings do not know your vocabulary
Vector search matches meaning, which is exactly what you want for natural language questions and exactly wrong for part numbers, version strings, and model names. Semantic similarity will happily decide that two different product generations are close, because linguistically they are.
The fix is not exotic: combine vector search with plain keyword search and merge the results. Hybrid retrieval. In a technical corpus full of identifiers, keyword matching carries more weight than the vector-database marketing suggests.
Metadata does half the work
Filter before you search. If the question mentions a product and a version, and your chunks carry product and version metadata, then filtering the candidate set first improves every subsequent step. This is ordinary database thinking and it is frequently skipped because the pipeline diagram does not have a box for it.
Evaluate retrieval separately
Build a small set of real questions with known correct source passages. Twenty is enough to start. Then measure one thing: did the correct passage appear in the top k results?
That single number tells you whether your problem is retrieval or generation, and it makes every subsequent change measurable. Without it you are tuning by anecdote, which is how systems spend six months getting worse in ways nobody can prove.
The thing I keep relearning
Retrieval-augmented generation is a search system with a language model at the end of it. Everything we already knew about building good search — tokenisation, structure, metadata, ranking, evaluation — still applies and was never superseded.
The language model is the part that is new and impressive, so it absorbs all the attention. But if your system disappoints, the odds strongly favour the boring half being at fault. Go and look at what it retrieved.