Hugging Face contributor publishes new calibration dataset for llama.cpp quantization

A new public imatrix calibration dataset from Hugging Face contributor Bartowski aims to improve how llama.cpp compresses language models into lower-bit GGUF files. The release follows a large testing effort conducted with Fable and GPU resources from LTT Labs.

The dataset, identified as calibration-data V6, includes separate prose and conversation files. Conversation samples are rendered through a model’s chat template, with particular attention to tool-use formatting. Bartowski has also published the source files and rendering scripts so other developers can reproduce or extend the process.

Why the dataset matters

During imatrix calculation, text is passed through a model to measure how strongly different input channels activate. Quantization tools use those measurements to estimate which weights are most important, helping them allocate limited precision more intelligently. The quality and variety of the calibration corpus therefore influence which model behaviors survive compression.

The experiments compared multiple calibration approaches across seven models, including several Qwen variants, Gemma, and Mistral. Tests covered divergence from bfloat16 models, tool-calling performance through BFCL, selected MMLU-Pro and GSM8K evaluations, short-prompt checks, and expert coverage for mixture-of-experts models.

The early conclusion is measured rather than revolutionary: the previous V5 dataset was already strong, and newer corpora produced relatively small or inconsistent gains above roughly four bits per weight. The biggest differences appeared at very low precision, especially Q2_K and on mixture-of-experts systems. Without imatrix data, one tested Qwen model reportedly lost 28 BFCL points, falling from about 82% to 54%.

For local-model builders, the work offers both a potentially better calibration resource and a reproducible framework for evaluating quantization choices instead of relying solely on benchmark assumptions.

Source: Hugging Face Blog