Research Agent

Output

# Small language models running on-device in 2026

TL;DR — what the product team needs to know
- Running useful LLMs on-device is practical in 2026 for many product use-cases: 7B-class models are commonly run on consumer hardware after 3–4-bit quantization; 13B models are feasible with 4-bit or hybrid offload but require more memory/engineering [2][1].  
- The open-source ecosystem (llama.cpp, AutoGPTQ/GPTQ tooling, bitsandbytes, Hugging Face Transformers) provides production-ready conversion, quantization and inference stacks — including CPU, Apple Metal, Hexagon/Adreno, and CUDA backends — so you can ship local inference without proprietary cloud-only stacks [1][4][7].  
- Accuracy loss from modern weight-only and NF4-style 3–4-bit quantization is often small for instruction-following/chat tasks (many published benchmarks show near-FP16 quality), but quality depends on model, quantization method, and task; test for your task/metrics before committing [6][2].  
- Operational trade-offs: on-device models give lower end-to-end latency and stronger privacy/availability, but increase engineering complexity (model packaging, conversion, on-device memory and battery budgets, update/rollback, telemetry), and in some cases require device-specific kernel tuning or CPU/GPU hybrid offload [8][1][7].

Research questions I investigated
1. Which model sizes and quantization levels are realistically runnable on-device in 2026?  
2. What OSS tooling and runtimes exist to convert/quantize and run models on-device?  
3. What hardware/backends and performance tradeoffs (latency, battery, memory) should product teams expect?  
4. What operational and engineering considerations (packaging, updates, privacy, monitoring) matter for shipping on-device LLM features?

Findings organized by question

1) Which model sizes and quantization levels are realistically runnable on-device?
- Practical envelope in 2026:
  - Small models (100M–1.5B): trivial to run on-device (CPU or modest GPU) without aggressive quantization; typical mobile/laptop memory easily covers these. This is a general baseline supported by the inference toolkits and Transformers workflows [7][1].
  - 7B models: already commonly run locally after weight-only 3–4-bit quantization (GPTQ/AutoGPTQ, NF4). Community tool benchmarks show LLaMA‑7B in 4‑bit occupying roughly 4–5 GB of model memory (so fits on devices or GPU profiles with ~6–8 GB usable memory) and yielding competitive perplexity / generation quality in many tests [2][4].
    - Example: community GPTQ experiments list LLaMA‑7B (4‑bit GPTQ) memory ≈ 4.7 GB vs FP16 ≈ 13.9 GB in their tables — i.e., ~3× reduction vs FP16 [2]. AutoGPTQ/GPTQ tooling is used to create such 4‑bit artifacts [4][2].
  - 13B models: feasible on higher‑end devices or with careful 4‑bit quantization / offload. GPTQ tables show LLaMA‑13B in 4‑bit uses ≈ 8–8.5 GB; that makes it run on machines with ≥10–12 GB host/GPU RAM, or via hybrid CPU+GPU approaches [2][1].
  - 33B and above: require large memory even when quantized (tens of GB) or need hybrid multi-device/offload strategies; impractical for most single mobile devices without server assistance [2][1].
- Cross-check and caveats:
  - Quantized memory numbers and quality vary with quantization configuration (bits, group-size, "desc_act"/activation ordering) and conversion kernels. The community uses GPTQ/AutoGPTQ/RTN/NF4/etc. — results differ model‑by‑model; validate on your target models/devices [2][4][6].

Sources supporting this section: GPTQ community tables and AutoGPTQ docs for 4‑bit memory/benchmarks [2][4], and llama.cpp support for CPU/Metal/Hexagon backends and hybrid offload for models larger than device VRAM [1].

2) Tooling and runtimes — what’s available to build a product
- Major open-source pieces you can use right now:
  - llama.cpp / ggml: compact C/C++ inference library and ecosystem providing CPU-first inference, Apple Metal/Accelerate optimizations, Hexagon (Snapdragon) support, Vulkan/OpenCL backends and explicit hybrid CPU+GPU offload; supports many low-bit quantization formats (1.5–6–8 bits) and serves as the primary on-device runtime for many projects [1][8].
  - GPTQ / GPTQ‑for‑LLaMa and AutoGPTQ: weight-only post-training quantization toolkits (one-shot GPTQ and the AutoGPTQ packaging) widely used to create 3–4‑bit model files used for on-device inference; AutoGPTQ exposes APIs to quantize, save, and load models and integrates with the Hugging Face ecosystem [2][4].
  - bitsandbytes (and successors): 8‑bit optimizer/quantization tooling used in training/finetuning and memory reduction workflows; commonly used alongside transformers/QLoRA workflows [5][6].
  - Hugging Face Transformers and HF Hub: Transformers supports loading local model files, caching, and workflows for saving/loading quantized artifacts; you can push quantized models to HF or load them locally for offline operation [7][4].
- What these tools enable for a product team:
  - Convert a HF model → quantized artifact → device-friendly format (safetensors/gguf) → run via llama.cpp or vendor kernels (CUDA/Metal/Hexagon) [4][1][7].
  - Reuse community-built quantized models (many models on HF are distributed in quantized formats) and integrate them into an on-device runtime or a small local service [4][8].
- Evidence of ecosystem maturity:
  - Large community adoption: llama.cpp (very high GitHub stars), AutoGPTQ and GPTQ forks/usage show active community maintenance and many kernels/optimizations available for different hardware classes [1][4][2].

Sources: llama.cpp README and ecosystem pages [1], AutoGPTQ repo and docs [4], GPTQ-for-LLaMa [2], bitsandbytes repo [5], Transformers docs [7], llama.app product framing [8].

3) Hardware, performance, and tradeoffs (latency, battery, memory)
- Backends and device classes:
  - On desktop/servers: CUDA, HIP, Vulkan, SYCL, AVX/AMX optimized CPU code paths are available; quantized kernels (int4 * fp16 hybrids, exllama/Marlin kernels) accelerate inference on modern GPUs [1][4].
  - On Apple devices: Apple Metal/Accelerate kernels and Apple Silicon ARM NEON optimizations are supported by llama.cpp and related builds, enabling efficient CPU and GPU inference on MacBooks, iPads and iPhones that expose Metal compute [1][8].
  - On mobile: Hexagon (Snapdragon), OpenCL for Adreno, and other vendor backends are present in the runtimes (llama.cpp explicitly lists Hexagon, OpenCL/Adreno) so running models on modern flagship phones is possible in principle, subject to available memory and thermal/power constraints [1].
- Performance characteristics and practical tradeoffs:
  - Quantization yields a big memory and speed win: 4‑bit weight quantization typically reduces model memory by ~3× vs FP16 and often improves token throughput vs FP16 on quantization-optimized kernels; published community tables show substantial tokens/sec improvements using GPTQ/AutoGPTQ kernels [2][4].
  - Quality vs bit-width: NF4 / GPTQ 4‑bit and some 3‑bit variants often preserve generation quality close to FP16 for many tasks, but results are model- and task-dependent. Papers and community reports (QLoRA, GPTQ-based evaluations) show that finetuning/adapter methods plus NF4/GPTQ quantization can reach near full‑precision performance on typical chatbot benchmarks [6][2].
  - Power/thermal & latency: on-device inference reduces network latency and avoids round‑trip time to cloud, but inference is compute-intensive and will increase CPU/GPU load, affecting battery life and device temperature; kernel-level optimizations and dynamic throttling are often required. The runtimes document optimized kernels and performance troubleshooting to tune for throughput vs power [1][4].
- Hybrid and offload options:
  - If a target device lacks sufficient VRAM, hybrid CPU+GPU offload is supported (llama.cpp documents CPU+GPU hybrid inference to partially accelerate models larger than available VRAM) — this can enable larger models on constrained devices at cost of latency and implementation complexity [1].

Sources supporting this section: llama.cpp backends and hybrid offload support [1], AutoGPTQ performance/quantization kernels and throughput discussion [4], GPTQ/GPTQ‑for‑LLaMa quantization memory/perplexity tables [2], QLoRA paper on NF4/4-bit finetuning showing competitive quality [6].

4) Operational and engineering considerations for shipping on-device
- Packaging and distribution:
  - Use reproducible pipelines: quantize + export to a device format (safetensors / gguf), then bundle with the app or download-on-first-run using HF Hub caching. Transformers/HF tooling and AutoGPTQ explicitly support saving/pushing quantized artifacts to the Hub and loading them in-device or in a local service [7][4].
- Updates, security, and governance:
  - On-device model updates require signed/validated delivery (OTA or via app updates). If using third-party models, review licensing and provenance (many community models have different licenses). The HF tooling and transformer docs show the caching and local file mechanisms you would use; product teams must add update/version management and possibly remote model revocation workflows [7][4].
- Privacy and telemetry:
  - Running models locally gives strong privacy/availability benefits ("no API keys, no telemetry" is the explicit positioning of local-first runtimes), but also transfers responsibility for safe content handling, filtering, and local data management to your app [8][1].
- Monitoring and A/B testing:
  - On-device inference complicates centralized telemetry. You’ll need privacy-safe aggregated telemetry (or opt-in) to measure model failures and drift. For iterative model improvements, plan for collection endpoints and a mechanism to roll back problematic models.
- Engineering effort and device-specific tuning:
  - Expect nontrivial engineering work: (a) converting/validating quantized weights across kernel variants, (b) integrating vendor-specific compute kernels and fallbacks, (c) testing memory/thermal behavior across device fleet, (d) implementing update/rollback, and (e) evaluating task-specific quality loss from quantization. The open toolchains ease conversion, but per-device kernel tuning and QA remain necessary [1][4][7].

Sources: Transformers docs (model caching/loading) and AutoGPTQ save/load features [7][4], llama.app/llama.cpp statements about local-first and performance tuning [8][1].

Risks, unknowns, and where to be conservative
- Quality variability: quantization and kernel choices produce model‑ and task‑specific quality differences. Published community numbers and papers show good average results, but you must benchmark your tasks (classification, summarization, safety filters, conversation) on the actual quantized artifact you plan to ship [2][6][4].
- Device fragmentation: mobile SoCs and drivers vary widely. Although runtimes expose Hexagon/Metal/OpenCL backends, performance and stability will differ by device/driver — you must test across representative devices [1].  
- Energy and UX: battery and thermal impacts may degrade UX for continuous or heavy use — consider batching, on‑demand loading, short generation budgets, or hybrid cloud fallbacks for heavy workloads [1][4].  
- Licensing & model governance: many open models have distinct licenses and provenance; verify allowed usage and safety before distribution [4][7].

Practical recommendations for the product team (next steps)
1. Define target use-cases (latency, privacy, offline requirements, max tokens per session) and a device support matrix (min RAM, GPU/SoC families, OS versions). Use those to decide target model family (e.g., 7B quantized vs 13B quantized + offload). The model-memory figures from community experiments are a starting point for RAM targets [2][1].  
2. Prototype pipeline: pick one reference model (e.g., an open LLaMA-family or instruction‑tuned model), quantize with AutoGPTQ/GPTQ to 4‑bit, convert to gguf/safetensors, and run locally via llama.cpp on representative devices. Measure latency, tokens/s, battery/thermal curves, and end-to-end UX [4][2][1].  
3. Benchmark quality on your exact tasks: run the quantized model (and alternative bit-widths) against your metrics (accuracy, hallucination, safety) to determine acceptable tradeoffs [6][2].  
4. Plan ops: signing model artifacts, update/rollback paths, privacy-preserving telemetry, and device-specific kernel fallbacks. Use HF Hub or private artifact store for distribution and model versioning [7][4].  
5. Decide hybrid/cloud fallback policy for heavy queries or to support older devices that cannot meet latency/quality targets.

Sources (pages I opened while researching; every factual claim above traces to at least one opened page; I cross-checked claims where noted)
1. https://github.com/ggerganov/llama.cpp — README and supported backends, quantization and hybrid CPU+GPU inference, platform notes.  
2. https://github.com/qwopqwop200/GPTQ-for-LLaMa — community GPTQ quantization tables (memory, perplexity) for LLaMA sizes and bits; 4‑bit/3‑bit numbers referenced above.  
3. https://arxiv.org/abs/2306.15307 — (opened during research; paper is about CMOS sensors and is not used as an evidence source in the briefing).  
4. https://github.com/PanQiWei/AutoGPTQ — AutoGPTQ quantization/inference APIs, performance notes, saving/loading quantized models, integration notes.  
5. https://github.com/facebookresearch/bitsandbytes — bitsandbytes repo (8‑bit optimizer/quantization routines, referenced as part of the quantization/optimization ecosystem).  
6. https://arxiv.org/abs/2305.14314 — QLoRA paper (NF4, 4‑bit finetuning methods and results showing strong quality for quantized finetuning workflows).  
7. https://huggingface.co/docs/transformers/installation — Transformers docs about model loading, caching, and local-files use (explains local usage and artifact management).  
8. https://llama.app — product/overview page for local-first inference with llama.cpp and the rationale (local, privacy-first, hardware support).  

If you want, I can:
- Run a short feasibility checklist and an estimated device‑class matrix (e.g., minimum RAM/expected latency for 7B vs 13B quantized) tied to your target device models.  
- Produce a small prototype plan (exact commands and steps) to quantize a chosen model and measure on a reference device (macOS with Apple Silicon or an Android flagship with a given RAM/SoC).