Running a language model on your own machine went from a novelty to a practical tool faster than I expected. It is still not a replacement for the frontier services, and for a particular set of tasks it is better because the material never leaves the machine.
The memory arithmetic
The binding constraint is memory, and the rule is simpler than it looks: at 16-bit precision a model needs roughly two bytes per parameter, plus overhead for context.
Quantisation reduces that. Around 8-bit halves it with quality loss most people cannot detect. Around 4-bit halves it again, with a loss that is noticeable on hard reasoning tasks and largely invisible on summarisation, extraction and rewriting.
So the practical question is: how much memory can you give it, and what does that let you run at 4-bit.
Unified memory architectures, where the processor and accelerator share a large pool, are unusually good at this because the whole pool is available for weights. A conventional laptop with a discrete GPU is limited by the GPU’s own memory, and running on the CPU with system memory works but slowly.
Context is the other consumer
Weights are fixed. The key-value cache grows with how much text you feed in, and at long context lengths it becomes a substantial fraction of the memory.
A model that loads fine and then fails partway through a long document is almost always hitting this. Budget for it rather than sizing to the weights alone.
What local models are genuinely good at
Summarisation and extraction. Take this document, pull out the specifications. Reliable, and a task where the quality gap to the frontier is small.
Rewriting and tidying. Fixing prose, changing tone, reformatting. Entirely adequate.
Classification and routing. Categorising documents, tagging, triage. This is where small models shine and where the cost difference matters most, because you run it thousands of times.
Retrieval-augmented answering. If the retrieval is good, the generation step is not demanding. This is the main use I get.
Code completion. Good enough for the local, boilerplate half of coding.
What they are not good at
Hard multi-step reasoning. The gap to frontier models is real and largest here.
Long-horizon agentic work. Errors compound across steps, and small models compound them faster.
Anything needing broad world knowledge. Fewer parameters means less memorised, and a smaller model will confabulate more confidently on facts it does not have.
Speed on a CPU. Without an accelerator, generation is slow enough to change how you use it. Fine for a batch job, frustrating interactively.
The honest use case
The reason to run locally is not that it is better. It is that the data cannot leave. Confidential specifications, customer material, anything under an agreement you have signed.
For that material the comparison is not local model versus frontier model. It is local model versus not using a model at all, and that comparison it wins easily.
Where I would start
Take whatever machine you have, find the largest model that fits at 4-bit with room for your context length, and run your actual task on it. Not a benchmark, your task.
The answer to “is this good enough” is task-specific and you will know within an hour. For summarising a technical document and answering questions about it, a surprisingly small model is fine. For anything requiring real reasoning, you will notice immediately.
That single hour of testing is worth more than any amount of reading about which model is best, because the thing you need to know is whether it is adequate for the specific thing you want it to do.