Hugging Face author maps tensor sensitivity to improve GGUF quantization

Hugging Face Blog contributor Bartowski has published a data-driven approach to improving GGUF quantization for llama.cpp. The work targets a longstanding limitation in existing heuristics: they adjust precision based largely on tensor shape and model depth, rather than the specific sensitivity of each tensor.

What the experiments found

Using a Framework Desktop with an AMD AI Max+ 395 and 128GB of memory, the author ran experiments for roughly 96 hours, producing and evaluating more than 1,000 quantized configurations. The main tests used Qwen3.5-0.8B and Qwen3.5-4B, with additional checks across model families including Gemma 4, Granite 4.2, Laguna, Ling, Muse, and Ornith.

The tests compared “degrade-one” configurations, where a single tensor was reduced to q2_k, against mostly q2_k models with one tensor upgraded to q8_0. Results showed that token_embd was by far the most sensitive component—about eight times more costly than the least sensitive weight tensor in the 0.8B model and 16 times more costly in the 4B model. Attention and output projections also benefited disproportionately from extra bits, while ffn_gate generally did not.

Sensitivity followed a U-shaped pattern through model depth, making early and late layers more important than the middle. The findings support per-tensor layout maps that can spend precision where it delivers the greatest quality improvement, potentially giving developers better model quality at the same file size or bitrate. Measurements used KL divergence against the bf16 model on WikiText-2 at a 512-token context, with lower scores indicating better agreement.

Source: Hugging Face Blog