Building a private knowledge base for work you cannot paste into a chatbox
Product specifications, caveats and unsupported configurations are exactly what a language model is good at retrieving, and exactly what you cannot send anywhere.
Things I actually built and what broke.
Product specifications, caveats and unsupported configurations are exactly what a language model is good at retrieving, and exactly what you cannot send anywhere.
Not a benchmark post. What memory you need, what quantisation costs you, and which tasks a model running on your own machine is genuinely good enough for.
Deterministic problems deserve deterministic solutions. A surprising amount of what gets handed to a model is a parsing problem with a known grammar.
Asking for JSON and hoping is not a design. Schemas, validation and a bounded retry make the difference between a demo and something you can run unattended.
A weekend project that taught me more about applied AI than any vendor deck: transcription is the easy half, alignment is the real problem, and the human in the loop is not a failure.
Nine out of ten disappointing RAG systems are disappointing search systems. The generation step is rarely where the failure is.