Quantization Guide

Quantization guide

Quantization stores model weights with fewer bits. It usually reduces file size and memory needs, with a possible change in output quality. The effect depends on the model, quantization method and task; there is no single reliable “quality percentage” for a quantization label.

Choose a file that leaves room to run

The Magidonia GGUF model card lists the available files and their sizes. Q8_0 is a larger option; Q6_K and Q4_K_M reduce the weight footprint. Check the actual download rather than treating an approximate size as a memory guarantee.

The running model also needs context-cache and runtime memory. macOS and other applications use the same unified-memory pool. A 25GB model on a 96GB Mac does not leave 71GB exclusively available to the model.

The example hardware

The session used a 96GB Mac and chose Q8_0 to favour weight precision. On a smaller machine, choose a smaller quantization or a smaller model, then test representative conversations. Loading successfully is only the first check; watch memory pressure and response speed too.

Compare outputs, not invented scores

Try the same character card and a small set of prompts on two quantizations. Look for changes in coherence, instruction following and speed. Treat the comparison as a local experiment, not a universal ranking.

Continue

The next chapter uses Q8_0 for the session's example. Substitute the tag for the file you selected if your hardware needs a smaller version.