Small Language Models: On-Device AI, Quantization & Distillation

Small Language Models

Small language models (SLMs) are compact LLMs — roughly a few hundred million to a handful of billion parameters — designed to run cheaply, fast, and often on-device rather than in a data center. The industry spent years chasing ever-larger frontier models; the counter-trend is realizing that for a great many tasks, a well-chosen small model is not just good enough but better — because it's faster, cheaper, private, and deployable where a giant model can't go.

The insight driving SLMs is that most production tasks aren't frontier-hard. Classification, extraction, routing, summarization of short text, and simple tool-calling don't need a model that can also prove theorems. Pairing a small model to a narrow task — sometimes sharpened by fine-tuning or distillation — routinely matches a frontier model at a fraction of the cost and latency, while opening up on-device and edge deployment that a large model rules out.

TL;DR

SLMs are compact models (hundreds of millions to a few billion parameters) built for speed, low cost, and on-device/edge deployment.
For narrow, well-defined tasks — classification, extraction, routing, simple summarization — a small model often matches a frontier one at a fraction of the cost and latency.
Quantization shrinks a model (e.g. 16-bit → 4-bit weights) so it fits and runs on modest hardware, with modest quality loss.
Distillation trains a small "student" to mimic a large "teacher," transferring capability into a compact model for a target task.
On-device inference brings privacy (data never leaves the device), offline capability, and zero per-call cost — at the price of limited capability.
Use SLMs as the workhorse and reserve frontier models for genuinely hard reasoning; route by task difficulty.

Quick Example

Running a small model locally — no API, no network, no per-call cost. With a local runtime like Ollama:

The model is a few gigabytes, runs on a laptop or edge device, answers in milliseconds, and the data never leaves the machine — for a task a frontier model would be overkill on.

Core Concepts

The Capability-vs-Cost Spectrum

Model choice is a spectrum, not a binary. At one end, frontier models offer maximum capability at maximum cost and latency; at the other, SLMs offer narrow capability at minimal cost and latency. The engineering skill is placing each task on that spectrum:

Reaching for a frontier model on a task an SLM handles is like renting a semi-truck to deliver a letter — it works, but you're paying enormously for capability you don't use.

Why Smaller Can Be Better

Cost — orders of magnitude cheaper per token, or free if self-hosted on-device.
Latency — fewer parameters means faster inference; often sub-100ms locally.
Privacy — on-device means data never leaves the device (no third-party sees it).
Offline — works with no network, on the edge, in the field.
Control — you own the model and its behavior; no provider deprecations or rate limits.

The Techniques That Make Models Small

Quantization — reduce the numeric precision of weights (16-bit → 8-bit → 4-bit). A 4-bit model is roughly a quarter the size and runs on far less memory, with a usually-modest quality drop. This is the main lever for fitting models onto consumer hardware.
Distillation — train a small "student" model to imitate a large "teacher" model's outputs, compressing much of the teacher's capability on a target distribution into a fraction of the size. See Fine-Tuning.
Pruning — remove weights or structures that contribute little, shrinking the model.
Task-specific fine-tuning — a small model fine-tuned on your narrow task often beats a general frontier model on that task, because it's specialized.

On-Device and Edge Inference

Running models locally — on phones, laptops, browsers (WebGPU), IoT, and edge servers — is where SLMs shine and large models can't follow.

Privacy by architecture. Sensitive data (health, personal messages, proprietary docs) never leaves the device. No data-processing agreement, no third-party exposure.
Zero marginal cost and no rate limits. After the model is on the device, inference is free and unlimited.
Offline and low-latency. Works on a plane, in a factory, in a car — with no round-trip to a server.

Runtimes like Ollama, llama.cpp, MLX (Apple), and in-browser WebGPU/WebLLM make local inference practical. The constraints are real: limited context, weaker reasoning, and device memory/thermal limits. Match the task accordingly.

Best Practices

Right-Size the Model to the Task

Prototype on a capable model to confirm the task is feasible, then push down the size ladder until quality drops below your bar. Many teams over-provision by defaulting to a frontier model everywhere; the savings from moving routine tasks to SLMs are large.

Route by Difficulty

In a multi-step system, use a small/cheap model for the easy majority of requests and escalate only the hard ones to a frontier model. An AI gateway or a simple classifier can do the routing. This "cascade" captures most of the cost savings while preserving quality where it matters.

Fine-Tune or Distill for Narrow Tasks

If a small model is close on your task, fine-tuning or distillation on your data often closes the gap — yielding a compact, specialized model that beats a general frontier model on that specific job. See Fine-Tuning.

Measure, Don't Assume

The only way to know if a smaller model suffices is to evaluate it on your real task. Build an eval set, run it across model tiers, and pick by the accuracy-vs-cost tradeoff. Intuition about "which model is good enough" is unreliable.

Choose Deployment for the Right Reasons

On-device is compelling for privacy, offline, and cost — but it's not free of engineering (model size, device constraints, update distribution). Use it when those benefits matter; use a hosted small model when you want the cost savings without shipping models to devices.

Common Mistakes

Defaulting to a Frontier Model Everywhere

Assuming Small Means Bad

Small models are narrow, not useless. On a task they're suited to — especially fine-tuned — they routinely match big models. Dismissing them without measuring leaves large savings on the table.

Ignoring Quantization Quality Loss

Aggressive quantization (very low bit-width) can degrade quality more than expected on some tasks. Don't assume a 4-bit model behaves identically to the full-precision one — evaluate the quantized model you'll actually deploy.

FAQ

When should I use a small model instead of a frontier one?

When the task is narrow and well-defined — classification, extraction, routing, short summarization, simple tool-calling — and doesn't need deep multi-step reasoning. There, a small (often fine-tuned or local) model matches a frontier model at a fraction of the cost and latency. Reserve frontier models for genuinely hard, open-ended, or agentic work.

What is quantization?

Quantization reduces the numeric precision of a model's weights — for example from 16-bit to 4-bit — shrinking its size and memory footprint by roughly 4x and letting it run on modest hardware like a laptop or phone. Quality loss is usually modest but task-dependent, so evaluate the quantized model you'll actually ship rather than assuming it matches full precision.

What is distillation?

Distillation trains a small "student" model to mimic a large "teacher" model's outputs, transferring much of the teacher's capability — on a target task — into a far smaller model. It's a common way to get a compact, specialized model that punches above its size on the distribution it was distilled for.

Why run a model on-device?

Three reasons: privacy (data never leaves the device), offline capability (no network needed), and cost (inference is free and unlimited after deployment). The tradeoffs are limited capability, smaller context, and device memory/thermal constraints — so on-device suits narrow tasks where those benefits outweigh the limits.

How do I decide which model size to use?

Empirically. Build an eval set for your real task, run it across frontier, mid-size, and small models, and pick by the accuracy-vs-cost tradeoff. A common pattern is to route — small model for the easy majority, escalate hard cases to a bigger one — capturing most savings without sacrificing quality where it counts.