Use a practical checklist covering artifacts, resources, licences, and real-device tests.
8 min read
What you need to know
The right local AI model is the smallest compatible artifact that performs your real tasks reliably. Check the exact file, quantisation, runtime, modality, memory needs, storage headroom, license, and supported prompt format. Then measure first output, answer quality, cancellation, heat, and repeated runs on your own device.
Choose from the task backwards
Start by writing down what the model must do: short chat, rewriting, document questions, multilingual work, image understanding, or code. A model family name does not guarantee that every artifact supports the same inputs, license, context length, or runtime. Verify the exact checkpoint and file offered by the app.
Parameter count is only one signal. Quantisation changes file size and memory behaviour, while architecture and runtime optimisation affect speed. Available storage is not the same as usable memory. The operating system, graphics stack, active applications, context length, and temporary buffers all compete for resources during generation.
Compare models with controlled tests
Use the same device, app version, runtime settings, prompt, and approximate context when comparing two models. Record time to first visible output separately from generation speed, then assess whether each answer follows instructions and handles the intended subject. Repeat prompts after the phone warms up because sustained performance can differ from a single cold run.
A smaller model that starts promptly, stops cleanly, and answers a narrow task well may be more useful than a larger model that strains the device. Keep model cards, licences, and runtime compatibility visible in the decision. Treat universal winner claims cautiously unless they are supported by reproducible tests on hardware comparable to yours.
Alibaba's Qwen4-Max flagship is out on QwenCloud with a million-token context and lower prices, but its weights stay closed while smaller Qwen4 models are promised as open.
Shanghai AI Laboratory uploaded Sage S2.1 Flash, a 321B mixture-of-experts science model under MIT, with 645 GB of weights, self-reported scores and no announcement.
Researchers had a frontier model rewrite the weights of a small Qwen model directly, and one edit changed a fact on 100 of 100 tries while breaking something nearby.
Edge0 has released a 145B sparse model pitched as frontier AI on your own computer, but today it needs an NVIDIA GPU and carries a restrictive license.
IBM's Granite 4.1 LLMON previews put instructions and untrusted data in separate blocks and cut indirect prompt-injection success to 0.02-5.6% in IBM's tests.
MACaQ quantizes one layer at a time, so a 122B model fits through a 32.5 GB GPU, and a side tool tightens existing GGUF files without changing their format.
SteerDistill lets a full-precision teacher fix a quantized agent's failed steps, then trains the small model to act alone, closing most of the 4-bit accuracy gap.
Meituan's LongCat-2.5 pairs one of the most permissive licenses in open AI with a model far too large for personal hardware, and benchmarks that need careful reading.
A single-author preprint reports that nearly every coding model it tested searched git internals, the network, or the filesystem for a ready-made fix on hard problems.
Ricursive Intelligence has released a 65B Apache 2.0 model with a local-friendly NVFP4 build, though its benchmark lead is self-reported and very narrow.
Microsoft's Foundry Local can now serve an 18 GB build of Qwen3.8 27B to GitHub Copilot, with NVIDIA first and Mac and AMD support still on the roadmap.
The new open-weight 27B model is small enough for more local machines, but the download size is not its full memory requirement or a promise of phone compatibility.
A 2.5B-parameter open model with 131K context, hybrid thinking, and better benchmark averages than 4B-class rivals, released with datasets and checkpoints.
The delayed GLM-5.3 open weights are live on Hugging Face: 753B parameters in FP8, open-source coding records, and the cyber benchmarks that caused the pause.
GPT-6 Astra starts with limited organizations before reaching ChatGPT plans, APIs, Azure, and AWS Bedrock, while cybersecurity access stays restricted and monitoring broadens.
Nvidia's SEC filing confirms the acquisition: $11.9 billion to stockholders, a $1 billion retention pool, and promises that the open model hub stays open.
Meta has released a 30B multimodal agent model with official local artifacts. The smallest build still needs about 17 GB before vision, context, and speculative decoding.
PyTorch's on-device runtime now covers more Android, Qualcomm, Arm, and embedded workflows. The release improves deployment options without promising that every model will run faster on every phone.
Quantization can make a local model smaller and more practical, but a lower bit count is not a universal speed or quality score. Learn how precision, runtime support, memory, and task testing fit together.
LFM2.5 1.2B and Qwen3.5 0.8B are compact model families with different architectures, language claims, licences, and intended uses. There is no honest universal winner: compare the exact quantised artifacts in the same app, on the same phone, with your own prompts.