Key takeaways
- Quantization stores a model's weights, and sometimes its activations, in 8 or 4 bits instead of 16, so an 8B model's weights shrink from about 16 GB to about 4 to 5 GB.
- Weight-only methods such as GPTQ and AWQ mainly save memory, while weight-and-activation schemes such as W8A8 and W4A4 also speed up the math on hardware that supports it.
- Meituan says the W4A4 version of LongCat-2.5 delivers 43.3% higher prefill and 42.77% higher decode throughput than a W8A8 baseline, without publishing side-by-side accuracy scores.
Quantization in AI means storing a model's numbers, its weights and sometimes its activations, in fewer bits than it was trained with: 8 or 4 bits instead of 16. The model then needs a half to a quarter of the memory and often runs faster, at the cost of some rounding error.
It is how open models fit on a single GPU or a laptop. Meituan's LongCat-2.5 model card is a fresh example: it describes a 4-bit version of the model's expert layers that, Meituan says, delivers 43.3% more prefill throughput than its 8-bit baseline.
What quantization does to the numbers
A large language model is mostly a huge list of weights, the numbers learned in training. They are usually stored in BF16 or FP16, 16-bit formats that take 2 bytes each.
Quantization maps those values onto a much smaller set. As the Hugging Face Transformers guide notes, an int8 number can take only 256 values and an int4 number only 16. Every weight gets rounded to the nearest one the format can hold.
The usual recipe: take the smallest and largest value in a block of weights, stretch the integer range over it, and store one extra number, the scale, that says how big each step is. To use a weight, multiply the integer by the scale.
One outlier stretches the range and coarsens every other step, so most low-bit methods give each small group of weights, often 128, its own scale.
The memory math
Memory for weights is parameters times bytes per parameter. The Transformers overview gives the standard example: an 8B model needs about 16 GB in BF16, about 8 GB at 8 bits and about 4 to 5 GB at 4 bits.
| Format | Bits per weight | 8B model | 70B model |
|---|---|---|---|
| BF16 / FP16 | 16 | about 16 GB | about 140 GB |
| INT8 / FP8 | 8 | about 8 GB | about 70 GB |
| INT4 / FP4 | 4 plus scales | about 4 to 5 GB | about 35 to 40 GB |
The 4-bit row lands above a quarter because the scales take space. These figures cover weights only; the KV cache, the stored context of a conversation, needs memory on top.
The formats: INT8, INT4, FP8 and FP4
- INT8 stores evenly spaced integers.
- FP8 keeps a floating-point layout in 8 bits, with values packed closer together near zero, where most weights sit. Its matrix math needs recent chips, such as NVIDIA's Hopper and Blackwell GPUs.
- INT4 has 16 levels, so it loses more accuracy and leans on small groups or calibration to compensate.
- FP4 can represent only 0, 0.5, 1, 1.5, 2, 3, 4 and 6, positive or negative, so it needs a scale for every small block. NVIDIA's NVFP4 uses blocks of 16 values with an FP8 scale each, plus one FP32 scale per tensor.
NVIDIA says NVFP4 cuts model memory about 3.5 times against FP16, and in its own tests on DeepSeek-R1-0528 it scored within about a point of FP8 on most benchmarks.
Weight-only vs weight and activation (W8A8, W4A4)
In labels like W8A8, "W" is the weights and "A" the activations, the intermediate values computed for each token.
Weight-only methods store 4-bit weights but convert them back to 16 bits before each multiplication. GPTQ, AWQ and bitsandbytes 4-bit work this way. They save memory and speed up generation, which is mostly spent reading weights from memory.
Weight-and-activation methods (W8A8, W4A4) quantize both, so the multiplication itself runs in low precision. That speeds up prefill (reading the prompt) and large batches. Activations have outliers, so it is harder.
SmoothQuant (Xiao et al., 2022) attacked that problem by shifting the "quantization difficulty from activations to weights" with an equivalent rescaling. The authors report W8A8 with negligible accuracy loss, up to 1.56 times faster inference and up to 2 times less memory.
Post-training quantization vs quantization-aware training
Post-training quantization (PTQ) rounds a finished model. Some methods look only at the weights; others pass a small calibration dataset through the model to decide how to round each layer, which takes minutes to hours but usually holds up better at 4 bits and below.
Quantization-aware training (QAT) simulates the rounding during training, so the model learns to tolerate it. The Transformers guide says it usually gives the best accuracy at very low bit widths, but it requires training the model.
The papers behind the common methods
- LLM.int8() (Dettmers et al., 2022) found that large transformers develop a few outlier features that dominate performance. It keeps those in 16-bit and multiplies more than 99.9% of values in 8-bit, halving inference memory for models up to 175B parameters without performance loss, per the authors.
- GPTQ (Frantar et al., 2022) rounds weights to 3 or 4 bits one column at a time, adjusting the rest to cancel each error. It quantized a 175B model in about four GPU hours, with speedups over FP16 of about 3.25 times on an A100.
- AWQ (Lin et al., 2023) showed that protecting about 1% of weights, picked by the size of the activations they meet, greatly cuts error.
- QLoRA (Dettmers et al., 2023) freezes a model in a 4-bit format called NF4 and fine-tunes small adapters on top. That fit a 65B model's fine-tuning on one 48 GB GPU.
What GGUF is
GGUF is a file format, not a method. Hugging Face's docs describe it as a binary format for llama.cpp that holds the tensors and metadata in one file; Ollama and LM Studio use it too. Names like Q4_K_M encode the quantization type; Q4_K works out to 4.5 bits per weight.
How much accuracy you lose
At 8 bits, very little: LLM.int8() and SmoothQuant report no meaningful loss. At 4 bits, careful methods report small drops; GPTQ's authors call the loss negligible, and NVFP4 sits about a point below FP8. Below 4 bits, losses grow.
Most of these numbers come from each method's own authors. If a task matters, test the quantized model on it.
LongCat-2.5 and W4A4 for mixture-of-experts models
LongCat-2.5 is a mixture-of-experts model, which runs only some of its sub-networks for each token. Its card lists 1.6 trillion parameters, about 48 billion active per token.
The card says the model "applies W4A4 quantization to expert weights and activations," using FlatQuant-style learnable transformations, so it can run native INT4 math on domestic Chinese accelerators. Against a W8A8 baseline, Meituan reports:
| Metric (W4A4 vs W8A8, Meituan's claim) | Change |
|---|---|
| Prefill throughput | 43.3% higher |
| Decode throughput | 42.77% higher |
| Time to first token | 15.91% lower |
| Time per output token | 19.72% lower |
With an 8-bit KV cache, the card claims 92% higher decode throughput than W8A8 with a 16-bit cache. The weights are on Hugging Face as LongCat-2.5-W4A4 under the MIT license.
The baseline is already 8-bit, so the gains are what 4 bits add on top of 8. The card calls accuracy "comparable to the unquantized model" but publishes no side-by-side scores and names no hardware or batch size for the speed tests.
Still, W4A4 is the hard case, and shipping it for a trillion-parameter model shows how far the field has come since SmoothQuant.
Frequently asked questions
What is quantization in an LLM?
It is storing the model's weights, and sometimes its activations, in fewer bits, such as 8 or 4 instead of 16. The model needs less memory and often runs faster, with some rounding error.
What is quantization-aware training?
Quantization-aware training simulates the rounding of low-bit formats during training so the model learns to tolerate it. It usually gives the best accuracy at very low bit widths but requires training the model.
Does quantization reduce accuracy?
Yes, a little. At 8 bits, LLM.int8() and SmoothQuant report no meaningful loss; at 4 bits, careful methods report small drops; below 4 bits, losses grow.
What does Q4_K_M mean in GGUF?
It names a GGUF quantization type. Q4_K is a 4-bit scheme that works out to 4.5 bits per weight once its block scales are counted, according to Hugging Face's GGUF docs.
Sources
- Quantization conceptsHugging Face
- Quantization overviewHugging Face
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScalearXiv
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersarXiv
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsarXiv
- AWQ: Activation-aware Weight Quantization for LLM Compression and AccelerationarXiv
- QLoRA: Efficient Finetuning of Quantized LLMsarXiv
- Introducing NVFP4 for Efficient and Accurate Low-Precision InferenceNVIDIA
- GGUFHugging Face
- LongCat-2.5 model cardMeituan
- LongCat-2.5-W4A4Meituan


