Why Local AI Is Slow on a Phone and How to Diagnose It

Measure the two kinds of slowness

When someone says local AI is slow, the first question should be what part is slow. Time to first token measures the delay between pressing Send and seeing the first meaningful output. Generation rate describes how quickly later tokens appear. A phone can wait a long time and then stream at a usable rate, or begin quickly and produce the rest painfully slowly. Those patterns point to different bottlenecks.

The pre-output period can include runtime initialisation, model verification, reading weights from storage, allocating buffers, tokenising the conversation, and processing every input token. Generation then evaluates one or a small number of new positions repeatedly. The llama.cpp reference example explicitly separates prompt tokens from the later decode loop and exposes performance counters for the context. [1]

An interface should report these phases accurately. “Loading model,” “Processing conversation,” and “Generating” are more useful than one indefinite spinner. If nothing appears for minutes and output then streams slowly, do not rewrite that failure as a response that arrived all at once. Both prompt processing and generation may be underperforming.

Model loading and cold starts

A cold start happens when the model is not already resident in memory. The runtime must open the artifact, map or read its weights, initialise compute backends, create a context, and allocate working buffers. Storage speed, browser caching, file fragmentation, model format, and the amount of memory available to the app all influence this phase.

The model file's download size is not its complete runtime footprint. Weights compete with runtime code, temporary compute buffers, the key-value cache, imported document data, the browser or WebView, and every other process on the phone. Android explains that RAM is limited, that CPU and GPU access the same RAM, and that the operating system actively reclaims memory under pressure. [2]

Warm starts can be much faster when the runtime and model remain ready, but mobile operating systems do not promise that state. Backgrounding the app, switching among memory-heavy apps, locking the screen, or leaving the browser idle can cause resources to be reclaimed. A benchmark must therefore label cold and warm runs rather than averaging them into one misleading number.

Practical loading checks include restarting the app, noting the selected model, sending a short prompt, and comparing the first run with the second. If only the first run is slow, startup dominates. If every run begins slowly, inspect prompt growth, memory churn, and whether the runtime is being unnecessarily recreated for each message.

Prompt processing can dominate the wait

Before generating a reply, a transformer must process the input context. That context may include system instructions, conversation history, retrieved document passages, tool descriptions, and the latest user message. A short question can therefore create a long prompt when the application silently attaches a large history or many file excerpts.

Long advertised context windows are capacity limits, not free speed. More input tokens require more work and increase memory used by attention state. Runtime batch settings can trade memory for prompt throughput, but an aggressive batch that works on a desktop may cause pressure or failure on a phone. The llama.cpp server tracks prompt-processing time separately from token-generation time, which reflects the need to diagnose these phases independently. [3]

For interactive mobile chat, start with a deliberately small context budget. Summarise older turns, cap retrieved passages, avoid attaching entire documents, and disclose when a project or file search expands the prompt. A user asking “Reply with exactly OK” should not wait while the app processes thousands of irrelevant historical tokens.

Prompt caching can help when repeated requests share a stable prefix, but it is not universal. A cache may be invalidated by a changed template, model, context, or app lifecycle. Treat caching as an optimisation with observable fallbacks, not as the only reason the product is usable.

Model size, quantisation, and runtime fit

Larger models generally require more weight storage, more memory bandwidth, and more computation per generated token. Quantisation reduces the precision used to store weights and can make a model fit smaller hardware, but format compatibility and quality trade-offs matter. A highly compressed artifact that a runtime cannot accelerate may perform worse than a smaller, better-supported model.

Parameter count is not a complete speed ranking. Architecture, mixture-of-experts activation, tokenizer behaviour, context configuration, device backend, thread scheduling, and kernel quality all affect results. Compare exact artifacts on the same device and runtime. Do not transfer a tokens-per-second result from a flagship phone to a mid-range phone or from native code to a browser.

Runtime support is especially important in browsers. WebGPU can expose GPU computation, but availability and device limits vary by browser, GPU, driver, and secure context. A missing or lost GPU adapter can force a different path or stop the model. The W3C specification defines capability and limit negotiation rather than promising identical hardware. [4]

Choose the smallest model that can perform the task acceptably. A fast model that can be used consistently is often more valuable on a phone than a larger model whose cold starts, memory use, or heat make it impractical.

Heat, battery, and Android memory pressure

Sustained inference exercises compute units and memory bandwidth. The phone can warm up, consume battery quickly, and reduce clock speeds to remain within thermal limits. This means the fifth long generation may be slower than the first even when the software did not change. Charging, a case, ambient temperature, screen brightness, and other active applications can also affect a test.

Android's memory guidance says the system may reclaim memory or stop app processes to free resources for critical tasks. [5] Symptoms can include a model reloading after app switching, a browser tab disappearing, a generation stopping, or the entire app restarting. Storage capacity does not prevent these failures because storage and working RAM are different resources.

Close unrelated heavy apps before qualification, but do not design a product that only works in a laboratory. Test with realistic multitasking, low battery, a warm device, and repeated prompts. The interface should recover after interruption, preserve partial work where possible, and never leave the composer in Stop mode when no generation is active.

A practical diagnosis sequence

Use one exact prompt and record each phase. “Reply with exactly OK” is useful because it keeps expected output short. Capture the time when Send is pressed, when the status changes, when the first output appears, and when generation completes. Repeat once without closing the app to compare cold and warm behaviour.

Next, create a new empty conversation and run the same prompt. If it is much faster, history or retrieval is the likely cause. Then compare one smaller compatible model. If loading and generation improve together, model fit matters. If only the first turn improves, startup or context allocation deserves attention.

1. Verify the selected model and exact artifact.
2. Test one short prompt in a new conversation.
3. Separate cold-start time, prompt-processing time, and generation time.
4. Repeat immediately for a warm comparison.
5. Test after adding realistic history or one document.
6. Watch for heat, app restart, model reload, and Stop-state recovery.
7. Record device, OS, app, runtime, model, quantisation, and test date.

Avoid reporting a single “speed” number without this context. Time to first token and tokens per second answer different questions, and both can change across a session.

How CuriousLM should behave

CuriousLM treats long silence followed by slow streaming as a real local-inference regression. Interactive chat uses a bounded prompt budget, and the UI distinguishes model initialisation, verification, loading, thinking, streaming, stopped, and failed states. The selected model should remain selected after reopening the app so a test does not silently use a different artifact.

Model visibility is also separate from qualification. A catalogue entry can describe an artifact without promising that it runs well on every device. Download must move bytes or show a specific failure, Use must activate the model in the current app context, and cancellation must return the composer to Send once no generation remains active.

These behaviours do not make phone hardware unlimited. They make performance explainable. A user can choose a smaller model, shorten the conversation, remove large retrieval inputs, or retry after cooling the device only when the product identifies the actual phase instead of displaying vague status text.