datarekha
LLMs June 10, 2026

Running an LLM on your laptop: GGUF, llama.cpp, and the Q-soup

A 7B model now fits on a laptop. GGUF, llama.cpp, and quantization tiers like Q4_K_M — decoded, so running LLMs locally stops being intimidating.

9 min read · by datarekha · ggufllama-cppquantizationlocal-llmollama

A few years ago, “running your own language model” meant a rented GPU, a CUDA install, and a weekend. Today you can ollama run a capable 7B model on a laptop with no GPU at all and have it answering in seconds. Two pieces of plumbing made that shift normal — a file format and an engine — and one idea, quantization, makes it possible. None of it is complicated once you’ve seen it from the inside.

Why a special format exists

The weights you download from Hugging Face — .safetensors — are built for training. Full-precision tensors, sharded across files, designed to be loaded by PyTorch and fine-tuned. Wonderful for that job; needlessly heavy for the much simpler task of just running a model on your own machine.

GGUF (“GPT-Generated Unified Format”) is built for the opposite job: inference. It packs everything a model needs to run — the weights, the tokenizer, the metadata, even the chat template — into a single file that a small, dependency-free C++ program can memory-map and execute with no Python in sight. That self-contained design is the whole reason a model can run on a machine that has never heard of a Python virtual environment.

The program doing the running is usually llama.cpp: a lean C/C++ runtime that mmaps the GGUF file, offloads as many layers to the GPU as will fit, and runs the rest on the CPU. Ollama and LM Studio are friendly wrappers around exactly this. GGUF is what’s on disk; llama.cpp is what turns it into tokens.

The actual magic: quantization

A 7B model at full 16-bit precision is about 14 GB just for the weights. The reason it fits on a laptop is quantization — storing each weight in fewer bits. Drop from 16 bits to roughly 4 and the file drops to about a quarter of the size, while the model stays surprisingly close to its original quality. Quantization only shrinks the weights; it never changes the parameter count. The model is the same model, described more coarsely.

This is where newcomers hit the wall of cryptic filenames — model.Q4_K_M.gguf, model.Q5_K_S.gguf, model.Q8_0.gguf — and bounce off. But the naming is a simple code:

  • Q4 / Q5 / Q8 — the approximate bits per weight. Lower is smaller and faster, with more quality loss.
  • _K — the modern “k-quant” method (smarter than the old round-to-nearest).
  • _S / _M / _L — small / medium / large variant; the medium one keeps the most important tensors at higher precision.

So Q4_K_M reads as “≈4 bits, k-quant, medium” — and it’s the community default for a reason: it keeps almost all of the quality at roughly a quarter of the FP16 size. Q8_0 is near-lossless but big; Q2_K is the last resort when memory is truly tight and you’ll accept the degradation.

Will it fit?

The arithmetic is friendly: file size ≈ parameters × bits-per-weight ÷ 8, plus a gigabyte or two for context and the runtime. So a 7B model lands around 14 GB at FP16, ~7 GB at Q8_0, and ~4–5 GB at Q4_K_M — which is why Q4_K_M runs comfortably in 8 GB of RAM. Want it on a smaller machine? Drop a tier and accept a little quality cost. (There’s an interactive sizer in GGUF & running LLMs locally that does this maths for any model and memory budget.)

When local is the right call — and when it isn’t

It’s tempting to treat local models as strictly better because they’re free and private. They’re not strictly better; they’re different, and the distinction is about how many requests you’re serving.

  • GGUF + llama.cpp is optimised for one stream on modest hardware: a desktop app, an on-prem box with no GPU, privacy-sensitive data that can’t leave the building, or just tinkering on a laptop. Single-user, often offline.
  • vLLM (self-hosting) is optimised for many concurrent users on GPUs — its PagedAttention and continuous batching only pay off under real load. Running llama.cpp as a multi-user server, or vLLM for a single laptop user, is using the wrong tool.
  • A hosted API is simplest of all when you can send data out and pay per token.

The quiet revolution here isn’t that models got small — they didn’t. It’s that the tooling to run them got good enough that a quantized 7B on a laptop is a genuinely useful assistant, fully offline, owing nobody an API key. Once you can read the Q-soup and size a model against your RAM, the door is open.

Skip to content