I started running models locally for a boring reason. I was on a train with bad signal, halfway through debugging something, and the thing I wanted to ask a model about was a config file I was not going to paste into a website anyway.
That was the whole motivation. Not privacy as a principle, not cost. I just wanted the tool to work when the network did not, and I wanted to stop thinking about whether a particular file was safe to send somewhere.
Ollama turned out to be the shortest path to that. It is a small server that manages model files and exposes an HTTP API, and it hides the parts you do not want to deal with the first time. This is what I learned once I stopped treating it as a black box.
Two commands and then a question
Install it, then:
ollama run llama3.1:8bThat pulls about five gigabytes and drops you into a prompt. The first thing I wanted to know was where the five gigabytes came from, because 8 billion parameters at the fp16 most models are trained in should be sixteen.
The file is smaller because the numbers are smaller
A parameter is a number. During training it is usually a 16 bit float, so the memory a model needs is roughly:
bytes = parameters x bytes_per_parameter 8B at fp16 = 8,000,000,000 x 2 = 16 GB 8B at 8-bit = 8,000,000,000 x 1 = 8 GB 8B at 4-bit = 8,000,000,000 x 0.5 = 4 GB
Quantization is the process of storing those numbers in fewer bits. The tag Ollama pulls by default is usually a 4 bit variant, which is why an 8B model is a 4.7 GB download rather than a 16 GB one.
You will see names like Q4_K_M and it is worth knowing how to read them, because
this is the single decision that determines whether a model fits on your machine:
- Q4 is the bit width. Four bits per weight.
- K means k-quants, where the weights are quantized in blocks with their own scaling factor rather than one scale for the whole tensor. A block of outlier weights does not drag the precision of its neighbours down.
- M is the size within that family, small / medium / large. The larger ones keep more of the sensitive tensors at higher precision.
Q4_K_M is the default for a reason. It is the point where the file is small
enough to be practical and the damage is small enough to be hard to notice in
normal use. Below that, at Q3 and Q2, the model starts making mistakes of a
particular kind: it stays fluent, and it stops being reliable about specifics.
That is the worst failure mode to debug, because nothing looks wrong.
If you have the memory, the more useful move is almost always a bigger model at Q4 rather than a smaller model at Q8. A 4 bit 13B beats an 8 bit 7B at most things.
Working out whether it fits
Weights are the floor. The thing that surprised me was the context.
Every token in the conversation has to keep its key and value vectors around, for every layer, so the model does not recompute the whole history on each new token. That is the KV cache, and it grows linearly with how much you have said:
kv_bytes = 2 x layers x kv_heads x head_dim x tokens x 2 llama3.1:8b has 32 layers, 8 KV heads, head_dim 128 per token = 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KB 4,096 tokens -> 0.5 GB 32,768 tokens -> 4.0 GB 131,072 tokens -> 16.0 GB
The leading 2 is for the two vectors, K and V. The trailing 2 is bytes per number at fp16.
That last line is the one worth staring at. The weights of an 8B model at Q4 are 4.7 GB. Ask for the full 128k context that the model card advertises and the cache alone is sixteen. The context window is not free, and on a laptop it is usually the thing that pushes you off the GPU, not the model.
Grouped query attention is why this is survivable at all. Llama 3.1 8B has 32 attention heads but only 8 key/value heads, so the cache is a quarter of what it would be if every head kept its own. Multiply the numbers above by four if you want to know what a model without GQA would cost you.
So the honest formula for whether something will run:
total = weights + kv_cache(context) + ~1 GB overhead fits on GPU if total < VRAM runs, slowly if total < VRAM + system RAM does not run otherwise
That middle case is the one to understand. Ollama will offload whatever does not fit to the CPU rather than refusing, and it will not shout about it. A model that was generating 40 tokens a second drops to 3 and you spend an hour wondering what you broke.
Check it directly:
ollama psNAME SIZE PROCESSOR UNTIL
llama3.1:8b 6.7 GB 100% GPU 4 minutes from now
100% GPU is what you want. 48%/52% CPU/GPU explains the slowness completely
and the fix is a smaller quantization or a smaller context, not a faster machine.
Set the context yourself
The default context in Ollama is small, and it is small deliberately, so that
ollama run works on a machine that could not hold the advertised window. If you
paste in a long file and the model starts answering about the beginning while
ignoring the end, this is why. The front of your input fell out of the window.
You can set it per session:
/set parameter num_ctx 16384
Or bake it into a model you define yourself. A Modelfile is the closest thing Ollama has to a Dockerfile:
FROM llama3.1:8b
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
PARAMETER repeat_penalty 1.05
SYSTEM """
You are reviewing code. Point at the specific line. If you are not sure whether
something is a bug, say that instead of guessing.
"""ollama create reviewer -f ./Modelfile
ollama run reviewerThe temperature line matters more than people expect. The defaults are tuned for conversation, and for anything where you want the same answer twice, they are too high. For code and extraction I run at 0.1 to 0.2. For anything I want to be surprised by, 0.8. There is no single good value, which is exactly why it is worth setting rather than inheriting.
It is an HTTP server, and that is the actually useful part
The chat prompt is the demo. The reason to install this is the API.
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1:8b",
"messages": [{ "role": "user", "content": "Summarize this log line: ..." }],
"stream": false
}'There is also an OpenAI compatible route at /v1/chat/completions, which means
most things already written against the OpenAI SDK work by changing the base URL
and passing any string as the key. That is how I use it most: existing code,
pointed somewhere else.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "..."}],
)Two environment variables are worth knowing:
OLLAMA_HOST=0.0.0.0makes it listen on the network rather than only on localhost. I run this on a desktop with a real GPU and use it from a laptop. It has no authentication of any kind, so this belongs on a home network and nowhere else.OLLAMA_KEEP_ALIVE=30mcontrols how long a model stays resident after the last request. The default unloads after five minutes, and the reload is several seconds of nothing happening. If you call it in bursts through the day, raise it. If you are short on memory, drop it to0and take the reload.
What I actually use it for
I want to be accurate about this, because there is a lot of writing that implies a local 8B model is a drop-in replacement for a frontier one. It is not. The gap on hard reasoning is real and it is large, and I use the hosted models daily for exactly that reason.
What local models are genuinely good at is the large volume of small tasks:
- Summarizing or classifying things in bulk, where the per-call cost of an API would add up and the task is not hard.
- Anything touching a file I would rather not upload. Config, logs with customer identifiers, a database dump I am poking at.
- Working offline, which is the thing I originally wanted.
- Embeddings.
nomic-embed-textis small, fast, and running it locally means a search index over my own notes costs nothing per document and never leaves the machine.
The pattern that has stuck is a split. Local for the loop that runs a thousand times, hosted for the question I could not answer myself.
The parts that cost me time
The default context, twice. I lost most of an evening to a model that kept ignoring the end of a long prompt. There is no error for this. The tokens fall off the front and the model answers confidently about what it can still see.
Silent CPU offload. Same shape of problem. Nothing fails, it just gets slow,
and ollama ps would have told me in one second.
Assuming the model name meant something fixed. llama3.1:8b is a moving tag
pointing at a default quantization. If you want reproducibility, pin the full tag
(llama3.1:8b-instruct-q4_K_M) the way you would pin a Docker image rather than
using latest.
Thinking more parameters was always better. On a machine that cannot hold a 70B in memory, a 70B is slower than reading the documentation yourself. The largest model that fits entirely in VRAM beats the larger one that does not, by a margin that is not close.
None of this is difficult. It is just not written on the box, and all four cost me an hour each before I went and read what was actually happening.