How to Choose a Local AI Model for Android

Start with the job, not the parameter count

A useful model choice begins with a concrete workload. Short private chat, rewriting, translation, document questions, image understanding, and code assistance demand different capabilities. A text-only model cannot inspect a photo merely because the app has a camera button. A vision-language model may require a separate projector or image encoder, increasing download size and memory use. A model intended for research or prototyping may also be a poor default for dependable daily work.

Write down two or three prompts you genuinely use and the failure you cannot accept. For example: preserve names while summarising a meeting note; translate between two specified languages; extract visible facts from a receipt; or rewrite a paragraph without adding claims. This personal test set is more informative than a broad leaderboard whose hardware, runtime, prompt template, and scoring method differ from your app.

The goal is not to find a universal winner. It is to find the smallest verified model and runtime combination that handles your tasks well enough on your device. Smaller models generally leave more headroom and start more easily, while a larger model may improve some answers at the cost of longer downloads, greater memory pressure, heat, and slower prompt processing.

Use a decision order that prevents expensive mistakes

Check hard blockers before subjective quality. If the runtime cannot load the artifact, the model cannot help. If the app exposes only text input for that runtime binding, an upstream vision capability is irrelevant. If the license is incompatible with your intended distribution, a successful personal test does not remove the legal problem. Only after these gates pass should you compare response quality and speed.

Android local model decision order

RAM matters more than the model file alone

Storage size and runtime memory are not interchangeable. Android documents that each app has a device-dependent heap limit and that reaching capacity can produce an OutOfMemoryError. The system can also stop cached processes to recover memory. CPU and GPU work may use shared device RAM, while native buffers and memory-mapped pages do not necessarily appear as a simple Java heap number. [1] [2]

Local inference needs space for model weights plus runtime structures, the key-value cache, token buffers, image inputs, the user interface, document context, and temporary verification data. Longer prompts and larger generated-token limits can increase working memory even when the model file does not change. Multimodal models may add an image encoder or projector. This is why an honest model manager uses device-specific checks and reserves headroom rather than equating free storage with compatibility.

Do not choose from total RAM alone. Current availability, low-memory state, Android version, vendor GPU drivers, other applications, and thermal conditions all matter. Android provides ActivityManager.MemoryInfo so applications can inspect current available and total memory and the system's low-memory threshold, but the final proof is a repeated real-device load and generation test. [2] [3]

• Leave meaningful RAM headroom for Android and the rest of the app.
• Leave storage headroom for partial download, verification, and replacement during updates.
• Test after a cold launch, not only immediately after a successful first load.
• Repeat prompts long enough to expose heat and memory growth.
• Unload or switch models and confirm resources are actually released.

Understand quantisation without treating it as a score

Quantisation stores model tensors at lower precision to reduce artifact size and often reduce memory and compute requirements. Names such as Q4_0, Q4_K_M, Q5_K_M, and Q8_0 describe particular schemes, not a universal quality grade. The llama.cpp project documents multiple quantisation methods and warns that requantising already quantised tensors can severely reduce quality compared with quantising from higher-precision source weights. [4]

A lower-bit artifact may fit where a higher-precision artifact does not, but it can also change output quality. Mixed quantisation can preserve selected tensors differently. For multimodal GGUF models, llama.cpp notes that the multimedia projector or encoder is often kept at higher precision because its quality directly affects what the language model receives from an image. [4]

Compare the exact files your app distributes, not a base model name. Two files with the same family label can use different quantisation, prompt metadata, conversion tools, and revisions. A trustworthy application pins a full source revision, exact filename, byte count, cryptographic hash, runtime binding, and prompt template so that a passing test refers to a reproducible artifact.

Check the runtime and input boundary

An Android app needs an inference engine that understands the selected format and model architecture. Google documents on-device LLM inference for Android and recommends its current LiteRT-LM Android path as MediaPipe's earlier LLM Inference API moves into maintenance mode. The same guide also notes that high-end physical devices are the intended target and that emulators do not reliably represent support. [5]

Runtime support is more specific than 'Android compatible.' It includes architecture operators, tokenizer behaviour, chat template, context configuration, CPU or GPU backend, streaming, cancellation, and supported modalities. A base checkpoint may advertise image input while a chosen GGUF and current app binding expose text only. Always trust the current application's explicit capability display over a family-level marketing page.

Physical-device testing is especially important for acceleration. An emulator can validate interface state and many storage flows, but it cannot prove that a vendor GPU driver, native runtime, memory pressure, or sustained thermals will behave correctly on your phone. If an app calls a model experimental or Labs, treat that label as a boundary rather than a challenge to bypass.

Treat context length as a budget, not a target

Model cards often publish a maximum context length, but that number does not mean a phone should fill the window for every chat. Prompt prefill processes the input before visible output begins. More history and document text can therefore increase time to first token and memory use. An interactive mobile app may intentionally apply a much smaller default prompt budget even when the upstream architecture accepts more.

Ask what the exact app-and-model combination has tested, how it selects recent messages or retrieved document chunks, and what happens when the budget is exceeded. Tested maximum, declared upstream maximum, and practical default are three different values. An app should not silently advertise the largest one as a guarantee.

For document work, retrieval can be more efficient than placing an entire file into every prompt. The app can index local chunks, select the most relevant passages, and attach citations. That design still needs an embedding model, secure local index, bounded input, and a way to open the exact source passage. Bigger context is not a substitute for provenance.

Read the model card and license

A model card should describe intended use, architecture, languages, known limitations, prompt format, and license. Liquid AI describes LFM2.5 1.2B Instruct as a compact model designed for on-device deployment and publishes separate original, GGUF, and ONNX variants. Qwen's official Qwen3.5 0.8B card identifies the model as a compact post-trained checkpoint and explicitly frames its intended uses around prototyping, task-specific fine-tuning, research, and development. [6] [7]

Licenses affect organisations differently from individual experimentation. Qwen3.5 0.8B is published under Apache License 2.0. LFM models use the LFM Open License, whose conditions must be reviewed for the relevant entity and release. The app should show the exact license before download and preserve required notices; a model being publicly downloadable does not by itself grant every use. [6] [7]

Run a repeatable phone test

Test with stable prompts and record observations rather than relying on feel. Begin after a cold app launch. Confirm the selected model persists, send a short exact-answer prompt, note when the first visible text arrives, stop generation, and immediately send another prompt. Then test one realistic longer input and repeat it several times to expose thermal slowdown or unreleased memory.

Quality review should be task-specific and blind where possible. Save outputs without model names, check factual claims against sources, and look for instruction failures, omissions, fabricated details, and unwanted language switching. Do not convert a single preferred answer into a universal benchmark. Sampling settings can change outputs, so keep them the same when comparing models.

1. Verify the model name, artifact state, and active status after reopening the app.
2. Send 'Reply with exactly OK' to check loading, streaming, and instruction following.
3. Stop a response and confirm the send control recovers.
4. Run your two or three real prompts with fixed sampling settings.
5. Repeat after several generations and observe heat, memory, and responsiveness.
6. Switch away from the model and back, then repeat the short smoke test.
7. Enable airplane mode and confirm the already installed model still generates.

How CuriousLM presents model choice

CuriousLM is designed to keep the catalog visible while explaining why a particular option is or is not usable. A user should see artifact size, license, modality, device requirements, runtime status, and release qualification before starting a download. Downloading never means automatic activation, and Use is expected to activate the model in the current application context.

CuriousLM supports compact text-model choices and separate vision-capable choices, but the exact catalog and qualification state can change with an app release. It does not silently substitute a cloud model when a local model is unavailable. Models are explicit downloads, selected state should persist across reopening, and corrupt or incomplete artifacts must not be treated as installed.

The safest default is still personal: begin with the smallest qualified text model that supports your language and tasks. Move upward only when a controlled comparison shows a meaningful benefit on your phone. Keep a smaller model installed as a recovery option if storage allows, and remove weights you no longer use through the model manager rather than deleting browser or app files blindly.