# What is GGUF: the single-file format for running AI models locally

> One .gguf file holds a model's compressed weights plus its architecture, tokenizer and settings, which is why llama.cpp, Ollama and LM Studio can run it directly.

URL: https://claudegpt.si/news/what-is-gguf  
Published: 2026-11-22T14:56:52.158Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: GGUF, llama.cpp, Hugging Face, Unsloth, Quantization, Local AI  
Publisher: ClaudeGPT Si

## Key takeaways

- GGUF is a single-file format for AI models that stores the weights together with the metadata needed to load them, such as the architecture and tokenizer.
- Names like Q4_K_M give the compression level: Q4_K stores about 4.5 bits per weight, and llama.cpp's docs show Llama 3.1 8B shrinking from 32.1 GB to 4.9 GB at Q4_K_M.
- Unsloth's GGUF version of Inkling 2 Mini had 805,993 downloads in the past month as of November 22, 2026, with files from 7.76 GB to 22.87 GB.

GGUF is a file format that packs an AI model into one file so it can run on your own computer. The file holds the model's weights, usually compressed, plus what a program needs to load it: the architecture, the tokenizer and settings such as the context length. It was made for [llama.cpp](https://github.com/ggml-org/llama.cpp), and apps built on it, like Ollama and LM Studio, read the same files. The [GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) lives in the ggml repository.

The term is in searches now because of one download. [unsloth/Inkling-2-Mini-GGUF](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF), Unsloth's GGUF version of Thinking Machines' [Inkling 2 Mini](https://claudegpt.si/news/inkling-2-mini-open-weights), had 805,993 downloads in the past month as of November 22, 2026, according to Hugging Face. Its smallest file is 7.76 GB. Thinking Machines' own checkpoint is about 20 GB.

## What is inside a GGUF file

A GGUF file has three parts, in order.

1. A header. It starts with the four bytes `GGUF`, then the format version (3 is current), the number of tensors and a list of metadata entries.
2. Tensor info. For each tensor, the arrays of numbers that make up the model, it gives a name, a shape, a number type and where its data starts.
3. The tensor data itself, padded so each tensor starts at a fixed boundary.

The metadata is a list of key-value pairs, such as `general.architecture = llama` or `llama.context_length = 8192`. Only a few keys are required: the architecture, the quantization version when weights are compressed, and the alignment. The rest describe the model, its tokenizer and its author. Communities can add their own keys under their own prefix.

The padding is there so programs can `mmap` the file: map it straight into memory instead of reading and copying it. That is why a GGUF model loads quickly. The spec lists its goals as single-file deployment, room to add information without breaking old files, and "no need for external libraries" to read it.

## Why GGUF replaced the older formats

Before GGUF, llama.cpp used formats called GGML, GGMF and GGJT. According to the spec, they had two problems. A file didn't say which model architecture it was for. And the settings were an untyped list, so adding one new setting broke every existing file.

GGUF fixed both with typed key-value metadata. A program can now read a file it has never seen, check the architecture and fail cleanly if it doesn't support it. New fields can be added without breaking old readers. Hugging Face's [GGUF docs](https://huggingface.co/docs/hub/gguf) credit Georgi Gerganov, the developer of llama.cpp, with creating the format.

## What Q4_K_M and the other file names mean

GGUF files follow a naming convention: model name, size, fine-tune, version, encoding and shard number, joined by hyphens. In `Grok-100B-v1.0-Q4_0-00003-of-00009.gguf`, the model is Grok, it has 100 billion parameters, the weights use the Q4_0 encoding, and this is part 3 of 9.

The encoding tells you how much the weights were compressed. That compression is quantization: storing each number with fewer bits, at some cost in accuracy. Our [explainer on quantization](https://claudegpt.si/news/what-is-quantization-in-ai) covers the idea. Hugging Face's table gives the bits per weight for the common types:

| Type | Bits per weight | Notes |
| --- | --- | --- |
| F16, BF16 | 16 | Uncompressed half precision |
| Q8_0 | 8, blocks of 32 | Older round-to-nearest method |
| Q6_K | 6.5625 | K-quant |
| Q5_K | 5.5 | K-quant |
| Q4_K | 4.5 | K-quant |
| Q3_K | 3.4375 | K-quant |
| Q2_K | 2.625 | K-quant |
| IQ4_XS | 4.25 | Uses an importance matrix |
| IQ2_XXS | 2.06 | Uses an importance matrix |
| IQ1_S | 1.56 | Uses an importance matrix |

K-quants group weights into blocks that share a scale. The IQ types use an importance matrix, measured on sample text, to decide which weights need the most precision. Suffixes such as `_S` and `_M` mark different mixes. Unless told otherwise, llama.cpp's quantizer mixes types across tensors, so in its Llama 3.1 8B table Q4_K_M averages 4.89 bits per weight and Q4_K_S 4.67.

The savings are large. llama.cpp's [quantize docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) give Llama 3.1 8B as 32.1 GB in its original form and 4.9 GB at Q4_K_M.

The Inkling 2 Mini files show the trade-off. Unsloth measured how often each version picks the same next word as its 8-bit file:

| File | Size | Same top token |
| --- | --- | --- |
| UD-IQ1_S | 7.76 GB | 89.4% |
| UD-Q2_K_XL | 9.14 GB | 91.6% |
| UD-IQ3_XXS | 11.66 GB | 94.7% |
| UD-Q4_K_M | 16.61 GB | 97.9% |
| UD-Q8_K_XL | 22.87 GB | reference |

The `UD` prefix marks what Unsloth calls its Dynamic GGUFs. Going from the 1-bit file to the 4-bit one more than doubles the size and lifts agreement from 89.4% to 97.9%. llama.cpp's docs use Q4_K_M as their example, and Unsloth's run commands use it too.

## How to run a GGUF file

With llama.cpp installed, one command downloads a GGUF from Hugging Face and starts a chat. This is the example from llama.cpp's [README](https://github.com/ggml-org/llama.cpp):

```bash
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
```

`llama serve` with the same argument starts a local server with an OpenAI-compatible API. Hugging Face also lists Ollama, LM Studio and GPT4All as apps that open GGUF files, and its model pages have a viewer that shows a file's metadata and tensors before you download it.

Check the model card first. For Inkling 2 Mini, Unsloth's card says llama.cpp users should take the files in its [`Mainline_Support`](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/tree/main/Mainline_Support) folder; the other files are meant for Unsloth's own app.

To make your own GGUF, llama.cpp converts a Hugging Face model with `convert_hf_to_gguf.py`, then compresses it with `llama-quantize`. The [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space on Hugging Face does both in the browser.

## GGUF, safetensors and MLX

Safetensors, the other common format on Hugging Face, stores only tensors. A GGUF also carries the metadata a program needs to run the model, which is why one file is enough.

GGUF is also not the only way to shrink a model. The project behind this week's Hacker News post "Inkling-2-Mini in 5.4 GB Memory", [LookAtMyQuant](https://github.com/ShinMegamiBoson/lookatmyquant), packs Inkling 2 Mini into builds as small as 3.85 GB. Those are MLX builds for Apple silicon, not GGUF files. The project says its 5.38 GB build scored 87.0% on 200 GSM8K math problems, against 96.5% for the full model. Those are the project's own numbers.

For most people the choice is simpler than it looks. GGUF is the format the common local tools read. Start with the Q4_K_M file and go lower only if it doesn't fit. llama.cpp loads the whole model into memory, and its docs say memory and disk needs are the same, so the file size is a fair guide to the RAM you need.

## Frequently asked questions

### Is GGUF the same as GGML?

No. GGML is the tensor library behind llama.cpp and also the name of an older file format. GGUF replaced the GGML, GGMF and GGJT formats with typed metadata, so a file says which architecture it is for and new fields don't break old files.

### What is the difference between GGUF and safetensors?

Safetensors stores only the model's tensors. A GGUF file stores the tensors plus metadata such as the architecture, tokenizer and context length, so one file is enough for llama.cpp and the apps built on it.

### Which GGUF quant should I download?

Q4_K_M is the usual starting point: llama.cpp's docs use it as their example, and Unsloth's Inkling 2 Mini Q4_K_M file picks the same next token as its 8-bit file 97.9% of the time. Go lower only if the file doesn't fit in your RAM or VRAM.

### How do I run a GGUF file?

With llama.cpp, run llama cli -hf followed by the Hugging Face repository and quant name, or llama serve for a local API. Ollama, LM Studio and GPT4All also open GGUF files.

## Sources

- [GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md), ggml-org
- [GGUF](https://huggingface.co/docs/hub/gguf), Hugging Face
- [llama.cpp](https://github.com/ggml-org/llama.cpp), ggml-org
- [quantize](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), ggml-org
- [unsloth/Inkling-2-Mini-GGUF](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF), Unsloth
- [LookAtMyQuant](https://github.com/ShinMegamiBoson/lookatmyquant), GitHub
