What Is AI Model Quantization for Local AI on a Phone?

Quantization changes how model numbers are represented

An AI model contains many learned numerical values, usually called weights. During inference, a runtime reads those weights and performs mathematical operations to predict the next token. A high-precision version can represent each value with more bits. A quantized version uses a lower-precision representation for some or all of those values.

The practical goal is compression with acceptable behaviour. PyTorch's ExecuTorch documentation explains that quantization commonly converts values from 32-bit floating point to lower-precision forms such as 8-bit integers. Lower precision can reduce a model's memory footprint, improve inference speed, and reduce power use, although the result depends on the model, quantization method, backend, and hardware. [1]

This does not create a smaller model architecture. A quantized 3-billion-parameter model still has the same broad architecture and parameter count as its higher-precision source. What changes is how values are stored and, depending on the scheme, how activations or calculations are handled at runtime.

The word is spelled both quantization and quantisation. Model tools and search results frequently use the American spelling, while British technical writing often uses the latter. They describe the same broad process.

Why quantization matters on a phone

Phones impose tighter resource limits than workstation servers. A local AI application must share memory with Android, its own interface, the inference runtime, prompt tokens, generated-token state, document context, and other active applications. Android can reclaim processes when the system is under pressure, and an application that cannot satisfy an allocation may fail rather than merely run more slowly. [2]

Reducing weight precision can help in three places:

1. Download and storage: fewer bits per weight can produce a smaller artifact, although metadata and mixed-precision tensors add overhead.
2. Runtime memory: smaller weight representations can reduce the memory needed to load the model when the backend keeps them quantized.
3. Compute and power: supported hardware can perform some lower-precision operations more efficiently.

None of these benefits is automatic. A runtime may convert some values to a wider format while loading, or a device backend may lack optimized operations for a particular scheme. ExecuTorch treats quantization as backend-specific because CPUs, GPUs, neural processors, and their software delegates do not support every configuration equally. [1] A smaller file is therefore evidence about storage, not proof of a proportional RAM saving or speed increase.

Quantization also does not shrink every part of a chat session. The runtime still needs buffers, token data, and temporary tensors. The key-value cache used to retain attention state can grow with context length. Images may add an encoder and image tensors. A phone needs headroom beyond the model artifact itself.

What 4-bit and 8-bit labels actually tell you

A label such as 4-bit usually means that important model values are represented with roughly four bits each. An 8-bit scheme uses more bits and can normally represent a finer set of values. That simple description is useful, but it is incomplete.

Real artifacts can use different precision for different tensor groups. They can store scaling factors, group weights into blocks, quantize weights but not activations, or preserve selected layers at higher precision. The ExecuTorch Vulkan backend, for example, documents both 8-bit and 4-bit weight configurations, including different activation and grouping choices. [3]

Format-specific names add more detail. In the llama.cpp ecosystem, labels such as Q4_0, Q4_K_M, Q5_K_M, and Q8_0 identify distinct GGUF quantization methods. The number is not the whole specification, and the suffixes are not interchangeable decorations. A Q4 file produced by one method should not be treated as identical to every other 4-bit artifact.

The useful comparison is therefore the exact file, not only the family name or bit count. Record:

• model repository and revision;
• artifact filename and byte size;
• quantization method;
• runtime and version;
• chat template and context setting;
• device model and operating-system version.

Without those details, a claim that one quantization is faster or better cannot be reproduced.

Lower precision creates trade-offs, not a quality ranking

Quantization approximates values, so it can change model output. The effect may be small for one task and obvious for another. Instruction following, factual recall, spelling, code syntax, multilingual work, and structured output can respond differently to the same compression method.

It is tempting to arrange artifacts into a simple ladder where more bits always means better and fewer bits always means faster. That shortcut fails for several reasons. Different schemes allocate precision differently. A runtime may have excellent 8-bit kernels but weak support for a particular 4-bit method. A smaller artifact may fit in memory while a larger one causes system pressure. Sampling also means two generations can differ even when the artifact does not.

The llama.cpp quantization guidance warns against quantizing an already quantized model because repeated conversion can compound quality loss. It recommends quantizing from a higher-precision source when possible. [4] For a person downloading a ready-made local model, the practical lesson is to prefer an artifact with a documented source, conversion method, and checksum rather than repeatedly converting whatever file happens to be available.

Do not infer safety or truthfulness from precision. A higher-precision model can still invent information, miss instructions, or reproduce limitations in its training. Quantization changes numerical representation. It does not verify the model's claims.

Runtime compatibility comes before file size

A phone cannot use an artifact merely because the download fits. The app's inference engine must support the model architecture, file format, tokenizer, quantization operators, and chosen hardware backend.

ExecuTorch's backend documentation describes separate paths for general CPU execution, Vulkan on Android GPUs, and vendor-specific accelerators. Unsupported operations may fall back to a portable CPU backend, which can change performance and power behaviour. [5] Another runtime may use a different support matrix even on the same phone.

Before downloading, verify four things:

An application's qualified catalog is more useful than a folder containing every downloadable GGUF. Qualification should mean that the exact artifact and runtime combination passed loading, generation, cancellation, and repeated-use checks on representative hardware.

Choose a quantized model with a task-first test

Start with the smallest qualified artifact that can perform your actual work. That is different from choosing the smallest file in the catalog. A very compressed model that repeatedly ignores the task is not efficient, because you must correct it or run another generation.

Build a small test set before comparing files:

1. One short instruction with an exact expected format.
2. One realistic rewrite or summarization task.
3. One task in every language you need.
4. One document question if you plan to use local files.
5. One refusal or uncertainty case where inventing an answer would be harmful.

Keep the prompt, context, sampling settings, app version, and phone unchanged. Measure time to first visible output separately from generation speed. Then check whether the output followed the instruction, preserved important details, and avoided unsupported claims. Repeat after several turns because memory use and heat can change sustained behaviour.

If two artifacts meet the quality bar, prefer the one that leaves more operational headroom. A model that starts promptly, can be cancelled, and survives repeated runs is often more useful than a marginally stronger model that pushes the phone close to failure.

Common mistakes when comparing quantizations

Avoid these conclusions:

• "Q4 always uses half the RAM of Q8." Weight precision is only one part of runtime memory.
• "The smallest artifact is fastest." Backend kernels, fallback, prompt length, and memory pressure also matter.
• "Eight bits preserves the original exactly." It is still a quantized representation unless the source itself used that format.
• "A model family supports this format, so every app does." Runtime support belongs to an exact application build and backend.
• "One good answer proves equal quality." Sampling varies, and tasks stress different capabilities.
• "More bits make an answer factual." Precision does not provide source grounding or verification.

Also separate cold and warm tests. The first run may include model loading or backend initialization. Later runs may begin faster, then slow as the phone heats. Record the state instead of combining every delay into one number.

How CuriousLM should present quantized choices

CuriousLM treats a model as an explicit, versioned artifact rather than a vague family label. The useful information is the exact download, its integrity state, runtime compatibility, modality, license, and qualification status. A downloaded artifact must be verified before it is treated as installed, and choosing Use should activate that exact model in the current app context.

Quantization is one input to that decision. Device memory, storage headroom, task fit, context budget, and sustained behaviour still need to be considered. CuriousLM should not promise that a lower-bit file is universally faster or that a higher-bit file is universally better.

For most people, the safe process is simple: choose a qualified compact model, run a controlled task test, and move to a larger or higher-precision artifact only when the result provides a clear benefit on the phone you actually use.