What is GGUF, and what does a quantization tier like Q4_K_M mean?
GGUF is a single-file format for running LLMs locally, used by llama.cpp and Ollama. Unlike training-oriented formats, it packs weights, tokenizer, and metadata into one memory-mappable file optimized for inference on CPU or partial GPU. Q4_K_M describes the quantization: roughly 4 bits per weight (vs 16 for FP16), using the k-quant method, medium variant, which protects the most important tensors at higher precision. It is the community default because it keeps almost all of the model's quality at about a quarter of the FP16 size.
How to think about it
GGUF (“GPT-Generated Unified Format”) is built for inference, not training. It bundles everything one model needs — weights, tokenizer, metadata, chat template — into a single file that a dependency-free C++ engine (llama.cpp) can memory-map and run with no Python. That self-contained, CPU-friendly design is why it’s the standard for local tools (llama.cpp, Ollama, LM Studio).
Decoding the tier names (e.g. Q4_K_M):
- Q4 — approximate bits per weight (~4, vs 16 for FP16). Lower = smaller and faster, more quality loss.
- _K — the modern “k-quant” method (smarter than naive round-to-nearest).
- _S / _M / _L — small / medium / large variant;
_Mkeeps important tensors at higher precision.
File size ≈ params × bits-per-weight ÷ 8. So a 7B model is ~14 GB at FP16 but ~4 GB at Q4_K_M.