GGUF & running LLMs locally
Run LLMs on your own machine — the GGUF format, llama.cpp, and quantization tiers like Q4_K_M — with a tool that tells you which tier fits your RAM.
What you'll learn
- What the GGUF format is and why it exists (inference, not training)
- How quantization tiers like Q4_K_M trade size for quality
- How to size a model against your RAM/VRAM budget
- When local/GGUF makes sense versus a hosted API or vLLM
Before you start
You don’t always need a GPU server to run an LLM. With the right format and a bit of quantization, a 7B-parameter model runs comfortably on a laptop — even on CPU. The format that made that normal is GGUF, and the tool is llama.cpp.
Why a special format?
Hugging Face weights (.safetensors) are built for training: full-precision
tensors, sharded across files, paired with Python and a framework. Great for
fine-tuning; heavy for just running a model on your own machine.
GGUF (“GPT-Generated Unified Format”) is built for inference. It packs everything one file needs — weights, tokenizer, metadata, and the chat template — into a single file that a small C++ program can memory-map and run with no Python at all. That self-contained, CPU-friendly design is why it’s the default for local tools like llama.cpp, Ollama, and LM Studio.
Quantization tiers: the Q-soup decoded
GGUF’s superpower is quantization — storing each weight in fewer bits. FP16 uses 16 bits per weight; the quantized tiers go down to ~3. The cryptic names follow a pattern:
- Q4 / Q5 / Q8 — the approximate bits per weight. Lower = smaller + faster, but more quality loss.
- _K — the modern “k-quant” method (smarter than the old round-to-nearest).
- _S / _M / _L — small / medium / large variant (M keeps the important tensors at higher precision).
So Q4_K_M = ~4-bit k-quant, medium — the community default, because it keeps almost all the quality at roughly a quarter of the FP16 size. Q8_0 is near-lossless but big; Q2_K is a last resort when memory is desperate.
Will it fit? Size it against your memory
The practical question is always “which tier fits my machine?” The file size is
roughly params × bits-per-weight ÷ 8, plus a bit for context. Drag your model
size and memory budget — the tool sizes every tier and recommends the best one
that fits.
Size ≈ params × bits-per-weight ÷ 8, plus ~1.5 GB for context and runtime. Quantization only shrinks the weights — it doesn't change the parameter count. Q4_K_M is the community default: most of the quality, a fraction of the size.
Two things this makes obvious: a tier is just bits-per-weight (quantization never changes the parameter count), and dropping one tier is usually what gets a model onto a smaller machine — at a quality cost you can decide is acceptable.
Check yourself
Quick check
What to remember
- GGUF is a single-file, inference-optimised format (weights + tokenizer + metadata); llama.cpp is the dependency-free engine that runs it on CPU or partial GPU.
- Quantization tiers trade bits-per-weight for size: Q4_K_M is the default sweet spot; Q8 is near-lossless and large; Q2_K is a last resort.
- Size ≈
params × bpw ÷ 8— size a model against your memory and drop a tier to fit a smaller machine. - Reach for GGUF/local for single-user, offline, private inference; reach for vLLM to serve a fleet at scale.
Questions about this lesson
What is GGUF and why not just use the Hugging Face .safetensors files?
GGUF is a single-file format optimized for inference: it packs the weights, tokenizer, metadata, and chat template into one memory-mappable file that a dependency-free C++ engine (llama.cpp) can run with no Python. .safetensors files are built for training — full precision, sharded, paired with a framework. For just running a model locally, especially on CPU or a laptop, GGUF is far lighter and self-contained, which is why Ollama and LM Studio use it.
Which GGUF quantization tier should I use?
Q4_K_M is the community default — roughly 4 bits per weight, the k-quant method, medium variant that protects the most important tensors — keeping almost all of the quality at about a quarter of the FP16 size. Use Q8_0 if you have memory to spare and want near-lossless quality, Q5_K_M for a middle ground, and Q2_K only as a last resort when memory is desperate. Dropping one tier is usually what fits a model onto a smaller machine.
How much RAM do I need to run a 7B model in GGUF?
Estimate file size as params × bits-per-weight ÷ 8, plus ~1–2 GB for context and runtime. A 7B model is about 14 GB at FP16, ~7 GB at Q8_0, and ~4–5 GB at Q4_K_M — so Q4_K_M runs comfortably in 8 GB of RAM or VRAM. llama.cpp also offloads as many layers to the GPU as fit and runs the rest on CPU, so partial-GPU machines work too.
Practice this in an interview
All questionsGGUF is a single-file format for running LLMs locally, used by llama.cpp and Ollama. Unlike training-oriented formats, it packs weights, tokenizer, and metadata into one memory-mappable file optimized for inference on CPU or partial GPU. Q4_K_M describes the quantization: roughly 4 bits per weight (vs 16 for FP16), using the k-quant method, medium variant, which protects the most important tensors at higher precision. It is the community default because it keeps almost all of the model's quality at about a quarter of the FP16 size.
QLoRA quantizes the frozen base model to 4-bit using NF4 and double quantization, then trains LoRA adapters on top, with paged optimizers to handle memory spikes. This dramatically lowers the memory footprint, enabling fine-tuning of large models on a single consumer GPU with little quality loss versus 16-bit LoRA.
SwiGLU is a gated feed-forward layer: it projects the input into two paths, passes one (the gate) through SiLU, multiplies the two elementwise, then projects down — SiLU(x·W_gate) ⊙ (x·W_up) · W_down. The elementwise gate is a smooth, learned, per-feature volume control, so the network can decide how much of each feature to pass rather than a hard ReLU on/off. It gives better quality per parameter, which is why LLaMA, Mistral, and Qwen use it; to keep the parameter budget equal it uses a smaller (~2/3) hidden width since it has three weight matrices instead of two.
Work top-down: start at the model layer with quantization, distillation, or routing cheaper models for easy requests, since model choices drive every downstream cost. Then optimize the runtime with batching, caching, and techniques like prompt caching for LLMs, and finally match infrastructure to the load using autoscaling on queue depth and spot or batch capacity. Track cost per token or per prediction alongside latency percentiles and accuracy so optimizations never silently degrade quality.