AI / Small Language Models (SLMs) Interview questions

A Small Language Model (SLM) is a transformer-based language model with a modest parameter count, usually from about 100 million to roughly 10 billion, that is small enough to run on a laptop, phone, or a single modest GPU.

It uses the same basic architecture as a large language model. The difference is scale: fewer layers, narrower hidden dimensions, and less memory and compute per generated token.

SLMs are typically built with careful data curation, distillation, pruning, and quantization, so that they stay useful for focused tasks such as classification, extraction, summarization, and tool calling. There is no official size cutoff, so the term is relative and changes as hardware improves.

Most major labs now publish small open-weight models. Parameter counts are approximate and vary by release.

Older examples include TinyLlama (1.1B) and DistilBERT, a small encoder model often cited as an early distillation success. Check each model card for license terms before using one commercially.

The core advantage is cost and control. A small model needs less memory, less compute, and less energy for each token, which translates directly into lower latency and lower serving cost.

• Privacy: data can stay on the device or inside your network.
• Offline use: no network round trip is needed.
• Low latency: faster time to first token on local hardware.
• Cheap fine-tuning: LoRA on a 1B-3B model fits a single consumer GPU.
• Predictable cost: no per-token API bill at high volume.

These benefits matter most for narrow, high-volume tasks where a general-purpose frontier model is overkill.

Small models have less capacity, so they usually store less world knowledge and hallucinate more on obscure facts. Ask a 3B model about a niche API and it will often invent plausible details.

They also tend to struggle with long multi-step reasoning, complex instructions with many constraints, and robust multilingual or domain-specific tasks. Their output format is less reliable, so JSON or tool calls may break without constrained decoding.

Most of these gaps can be reduced with RAG, fine-tuning, tool use, or routing hard requests to a larger model. They are still real, so always evaluate on your own task before committing.

There is no formal definition, but most practitioners place SLMs between about 100M and 10B parameters. The most common sizes you will meet are 0.5B, 1B, 3B, and 7B-8B.

Rough hardware fit, using 4-bit weights:

• Under 1B: microcontrollers, browsers, and low-end phones.
• 1B-4B: modern phones and thin laptops.
• 7B-8B: laptops with 16 GB RAM or a consumer GPU.

Some vendors call 14B models small compared with frontier systems, so treat the label as relative rather than fixed.

In practice, pick the smallest size that passes your evaluation. Going from 3B to 8B roughly doubles memory use and roughly halves decode speed on the same hardware, so every extra billion has to earn its place.

Quantization stores model weights (and sometimes activations) in fewer bits, for example INT8 or INT4 instead of 16-bit floats. The model gets smaller and decoding gets faster because less data is read from memory.

A 7B model needs about 14 GB in FP16 but roughly 4 GB at 4-bit. Accuracy drops slightly, and the drop is usually acceptable at 8-bit and often at 4-bit for well-calibrated methods.

Common formats and methods include GGUF k-quants, GPTQ, AWQ, and bitsandbytes NF4. Always re-run your task evaluation after quantizing.

Knowledge distillation trains a small student model to imitate a larger teacher. Instead of learning only from hard labels, the student learns from the teacher's full probability distribution (soft labels), which carries more information per example.

A temperature setting softens the teacher's distribution so that relative probabilities of wrong answers still teach something. The loss is typically KL divergence between student and teacher outputs, often mixed with the normal cross-entropy loss.

DistilBERT is the classic example, and several recent small models, including Gemma 2 2B and the Llama 3.2 1B and 3B models, reportedly used larger-model outputs or logits during training.

Pruning removes parts of a trained network that contribute little to its output, such as individual weights, neurons, attention heads, or entire layers. The goal is a smaller or faster model with minimal accuracy loss.

After pruning, the model is normally retrained or distilled briefly to recover quality. Meta described using structured pruning plus distillation from larger Llama models to build the 1B and 3B Llama 3.2 models, and NVIDIA's Minitron work used the same idea.

Pruning is often combined with quantization, since the two attack different parts of the cost.

On-device inference means running the model directly on the user's hardware, such as a phone NPU, laptop CPU/GPU, or an embedded board, instead of calling a cloud API.

Benefits are privacy, offline operation, low network latency, and no per-token fee. The trade-offs are limited RAM, battery drain, thermal throttling, and the need to ship and update model files.

Typical runtimes include llama.cpp, MLX on Apple silicon, ONNX Runtime, ExecuTorch, and LiteRT (formerly TensorFlow Lite).

A good fit is a notes app that summarizes or rewrites text without uploading it. Plan for download size too, since even a 4-bit 3B model takes close to 2 GB of storage.

LoRA (Low-Rank Adaptation) freezes the base model and learns small low-rank matrices that are added to selected weight matrices. The update is written as ΔW = B × A where the rank r is small, such as 8 to 64.

Only the adapter matrices are trained, often well under 1% of the parameters. This cuts GPU memory and produces a tiny adapter file that can be swapped or merged into the base model later.

It is the default way to specialize a 1B-8B model on a single GPU.

SLMs fit tasks that are narrow, repetitive, or privacy-sensitive:

• Text classification and routing: ticket triage, intent detection, spam filtering.
• Information extraction: pulling fields from invoices, emails, or logs into JSON.
• Summarization of private notes or documents on the device.
• Code completion and small developer assistants running locally.
• Tool calling agents that pick a function and fill its arguments.
• Domain chatbots grounded by RAG over a company knowledge base.
• Embedded and IoT assistants with no reliable connection.

They are a poor first choice for open-ended research or tasks needing broad world knowledge.

GGUF is a single-file binary format used by llama.cpp and tools built on it, such as Ollama and LM Studio. It replaced the older GGML format.

One GGUF file holds the quantized tensors plus metadata: architecture, tokenizer, chat template, and hyperparameters. That makes models easy to download and run without extra config files.

File names usually carry the quantization type, for example Q4_K_M (4-bit k-quant, medium) or Q8_0 (8-bit). Q4_K_M is a popular balance of size and quality.

Because everything lives in one file, you can copy it to another machine and load it directly with llama-cli -m model.gguf. Higher-quality types such as Q6_K or Q8_0 give better accuracy at a larger file size.

RAG fetches relevant text from an external store at query time and puts it in the prompt, so the model answers from supplied evidence instead of memory. This matters more for SLMs because they hold less factual knowledge.

1. Split documents into chunks and embed them.
2. Store the vectors in an index.
3. Embed the user question and retrieve the top matching chunks.
4. Insert the chunks into the prompt with an instruction to answer only from them.
5. Generate the answer, optionally with citations.

Keep chunks short, because small models have limited context and get distracted by irrelevant passages.

Microsoft's 2023 paper Textbooks Are All You Need introduced phi-1, a 1.3B code model trained on a small amount of high-quality, textbook-style data rather than a huge raw web crawl.

The data mix combined filtered web content with synthetic textbooks and exercises generated by a larger model. The claim was that data quality and teaching value can compensate for fewer parameters and fewer tokens.

Later Phi releases, including Phi-3-mini and Phi-4-mini, extended the idea to general language and reasoning. A caveat is that strong benchmark scores do not guarantee broad world knowledge.

Tokens per second (tok/s) measures how fast the model produces output tokens during the decode phase. It is the main throughput number people quote for local models.

It is only half the picture. Time to first token (TTFT) reflects the prefill phase, where the whole prompt is processed before generation starts. Long prompts can make TTFT feel slow even when decode is fast.

A rule of thumb: about 5-10 tok/s feels readable for chat, and interactive code completion wants far lower latency. Always quote the hardware, quantization, and context length with the number.

Install Ollama, pull a model, then run it. Ollama downloads a quantized GGUF build and serves it with a local HTTP API.

ollama pull llama3.2:3b
ollama run llama3.2:3b "Summarize: SLMs run on modest hardware."

# REST API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Hello","stream":false}'

The server listens on port 11434 by default. You can also write a Modelfile to set a system prompt, temperature, or context size, and then create a custom model with ollama create.

Ollama picks a sensible default quantization for each tag and caches models locally, so the second run starts quickly. Use ollama list to see what is installed and ollama rm to free disk space.

Synthetic data is text produced by a stronger model or by rules instead of being collected from people. SLM builders use it to create clean explanations, exercises, reasoning traces, and instruction-response pairs.

It lets you target gaps, control difficulty, and avoid noisy web text. Microsoft's Phi models are the best-known example of leaning heavily on it.

Risks: the student inherits the teacher's mistakes and bias, diversity can collapse if prompts are repetitive, and the teacher's license may restrict how outputs are used. Always filter, deduplicate, and sample-check generated data.

Perplexity is the exponential of the average negative log-likelihood per token on a text sample. Intuitively it is how many equally likely options the model is choosing between at each step. Lower is better.

For SLMs it is mostly used to measure how much quality a change costs, for example comparing FP16 against a 4-bit quantized version on the same dataset.

Limits: scores are only comparable with the same tokenizer and test set, and low perplexity does not prove the model follows instructions or reasons well. Pair it with task-based evaluation.

The context window is the maximum number of tokens, prompt plus output, the model can attend to in one request.

Sizes vary: Llama 3.2 1B and 3B support 128K tokens, Gemma 2 uses 8K, and Phi-3-mini ships in 4K and 128K variants. A big advertised window does not mean it is practical on small hardware.

The KV cache grows linearly with context length, so a long window can use more RAM than the weights themselves. Quality on very long inputs also drops for small models, so test retrieval accuracy at your real length.

A pretrained base model just continues text. Instruction tuning fine-tunes it on prompt and response pairs so it follows directions and answers in a helpful format. The result is the instruct or chat variant.

Many pipelines add preference tuning on top, such as DPO or RLHF, to improve tone and safety. For SLMs, the quality of the instruction data matters more than the volume, since there is little capacity to waste.

Use a base model when you plan heavy custom fine-tuning, and an instruct model for chat or when you want to try prompting first. Remember to apply the model's correct chat template.

Both are transformer language models. The practical difference is the balance between capability and cost.

In production the two are often combined, with the SLM handling routine traffic and an LLM handling the hard cases.

A practical way to decide is to run your real prompts through both and compare accuracy, latency, and cost per thousand requests. The cheaper model that clears your quality bar wins.

Choose an SLM when the task is well-defined, high-volume, and cost, latency, or privacy sensitive, and when you can measure quality on your own data.

• The output space is narrow: labels, JSON fields, short rewrites.
• Requests run in the millions per day and API cost dominates.
• Data cannot leave the device or the private network.
• You need offline operation or response times under a few hundred milliseconds.
• You can afford to fine-tune and keep a small eval set.

Stay with an LLM when requests are open-ended, need broad knowledge, or involve long multi-step reasoning that you cannot decompose. A common compromise is a cascade where the SLM answers first and escalates uncertain cases.

A narrow task needs far less knowledge and skill than a general model carries. A large model spends most of its capacity on things your task never touches, such as other languages, trivia, and unrelated domains.

Fine-tuning points a small model's capacity at one input distribution and one output format. With a few thousand clean labeled examples, a 1B-8B model can often reach or exceed the accuracy of a much larger model used zero-shot, especially for classification and extraction.

The edge disappears when inputs drift outside the training distribution or when the task needs reasoning or knowledge the small model never learned. Verify with a held-out test set that mirrors production traffic.

Start with weights, then add the KV cache and overhead.

1. Weights = parameters × bytes per parameter. FP16 is 2 bytes, INT8 is 1, 4-bit is about 0.5 plus a little for scales.
2. KV cache = 2 × layers × KV heads × head dim × tokens × bytes per value.
3. Overhead for activations, buffers, and the runtime: budget roughly 10-20% extra.

Leave headroom for the operating system and other apps, or the device will swap and slow down sharply.

Post-training quantization (PTQ) converts an already trained model to lower precision, optionally using a small calibration set. Quantization-aware training (QAT) simulates low precision during training or fine-tuning so the weights learn to tolerate it.

Start with PTQ. Move to QAT only if the accuracy loss at your target bit-width is not acceptable. Some vendors now release official QAT checkpoints for small models.

These are often mixed up because they all appear in 4-bit model names, but they are not the same kind of thing.

So GPTQ and AWQ describe how weights are compressed, while GGUF describes how the model is packaged. Pick by runtime: GGUF for llama.cpp and Ollama, AWQ or GPTQ for GPU servers such as vLLM.

At batch size 1, each new token needs a pass through every weight, but each weight is used for only a couple of operations. The work is mostly reading memory, not doing math, so the chip waits on data.

That gives a handy upper bound: tokens/s ≈ memory bandwidth ÷ model size in bytes. A 4-bit 3B model is about 1.8 GB. On a laptop with 50 GB/s of bandwidth the ceiling is near 28 tok/s, and real numbers land lower.

This is why quantization speeds up decoding: fewer bytes per token. It also explains why a faster NPU alone may not help. Prefill, by contrast, processes many tokens at once and is compute-bound.

The KV cache stores the attention keys and values of every earlier token so they are not recomputed at each step. Its size grows linearly with context length and with the number of concurrent sequences.

Per token, the cost is 2 × layers × KV heads × head dim × bytes. For a 3B-class model with 28 layers, 8 KV heads, and head dim 128 in FP16, that is about 112 KB per token. At 8K tokens it is roughly 0.9 GB, and eight parallel sessions would need around 7.5 GB.

Ways to shrink it: grouped-query attention, 8-bit or 4-bit KV quantization, sliding-window attention, shorter contexts, and prefix sharing across requests.

In standard multi-head attention every query head has its own key and value head. Grouped-Query Attention (GQA) lets several query heads share one KV head, so fewer keys and values are stored.

With 24 query heads sharing 8 KV heads, the KV cache is 3 times smaller. Decoding reads less memory per token, which raises speed and allows longer contexts on the same device.

GQA sits between full multi-head attention and multi-query attention (a single KV head). It keeps quality close to multi-head while capturing most of the saving, and models such as Llama 3 and Mistral 7B use it.

Both train small adapter matrices. The difference is how the frozen base model is stored. LoRA keeps it in 16-bit. QLoRA keeps it in 4-bit NF4 with double quantization and trains the adapters in higher precision, with paged optimizers to absorb memory spikes.

from peft import LoraConfig
cfg = LoraConfig(r=16, lora_alpha=32,
                 target_modules=["q_proj", "v_proj"],
                 lora_dropout=0.05, task_type="CAUSAL_LM")

Use QLoRA when the GPU is the bottleneck, for example tuning a 7B-8B model on a 12-24 GB card.

Small models follow long, intricate prompts less reliably than large ones, and few-shot examples eat context and add latency on every call. Fine-tuning moves the behavior into the weights.

• Output format becomes consistent, so JSON and labels break less often.
• Prompts shrink to a sentence, cutting TTFT and cost.
• Domain terms and tone are learned rather than described.
• Quality on a narrow task often rises well above the prompted baseline.

Fine-tuning is poor at injecting fresh facts, so use RAG for knowledge. A sensible order is: prompt baseline, add RAG, then fine-tune once you have a few hundred to a few thousand quality examples and an eval set.

In speculative decoding, a small draft model quickly proposes several next tokens. The larger target model then checks all of them in one parallel forward pass and accepts the longest correct prefix.

With the standard acceptance rule, the output distribution is the same as running the target alone, so quality is unchanged. Speed-ups of roughly 2-3x are common when the draft agrees with the target often.

The draft must share the target's tokenizer, and it pays off only when its acceptance rate is high. Some methods drop the separate draft and use extra heads on the target instead.

sequenceDiagram
  participant D as Draft SLM
  participant T as Target model
  D->>T: propose k tokens
  T->>T: verify all k in one pass
  T-->>D: accept prefix, add one new token
  D->>T: propose next k tokens

Logit-based (token-level) distillation trains the student to match the teacher's full next-token probability distribution, usually with a KL divergence loss. It needs access to teacher logits and a shared vocabulary.

Sequence-level distillation simply fine-tunes the student on text generated by the teacher. It works even if the teacher is only available through an API.

A third option, on-policy distillation, has the student generate text and the teacher score it, which reduces train-test mismatch.

Chinchilla scaling suggests about 20 training tokens per parameter for the best result at a fixed training compute budget. That ignores what happens after release.

A deployed model is queried billions of times, so inference cost dominates. It is cheaper overall to train a smaller model for much longer than to serve a bigger one. Llama 3 8B was trained on about 15 trillion tokens, which is well over 1,000 tokens per parameter.

Returns diminish, but small models keep improving with more data. The result is a model that costs more to train yet is cheaper to run, which is exactly the trade an SLM vendor wants.

Unstructured pruning zeroes individual weights anywhere in a matrix. Structured pruning removes whole units such as attention heads, neurons, channels, or layers.

For edge SLMs, structured pruning is usually the practical choice because the result is a plain smaller model that any runtime can load. Pair it with distillation to win back lost accuracy.

Whichever you choose, measure real speed on the target runtime, because sparsity only saves time when the kernels actually exploit it.

The embedding matrix has vocab size × hidden dimension parameters, and an untied output head repeats it. In big models that is a rounding error. In small ones it is a large slice.

Llama 3.2 1B has a vocabulary of about 128K and hidden size 2048, so the embedding table is around 260M parameters, roughly a fifth of the model. Tied embeddings reuse the same matrix for input and output to avoid paying twice.

A bigger vocabulary shortens sequences and helps multilingual text, but it adds memory and a larger softmax at every step. Designers of SLMs weigh the two carefully.

Choose RAG when the problem is missing or changing knowledge. Choose fine-tuning when the problem is behavior, meaning format, tone, or a specialized skill.

SLMs hold little factual knowledge, so RAG is almost always part of a knowledge-heavy solution. Keep retrieved context short, since small models handle long, noisy context poorly.

Before building either, check how many failures come from missing facts and how many from wrong format. That split tells you which lever to pull first.

Public leaderboards are a weak guide for a 3B model on your workload. Build your own evaluation.

1. Collect a few hundred real, representative examples with gold answers, and keep them out of training.
2. Pick task metrics: exact match or F1 for extraction, accuracy for classification, JSON validity rate, pass@k for code.
3. Set baselines: your prompt-only SLM and a strong LLM.
4. Measure latency, memory, and tokens per second on the target hardware.
5. Re-run the suite after every quantization, prune, or fine-tune to catch regressions.
6. If you use an LLM judge, calibrate it against a human-labeled sample.

Use MMLU, GSM8K, or HumanEval only as a rough sanity check.

No single fix works, so layer several.

1. Ground it with RAG and instruct the model to answer only from the provided passages.
2. Allow refusal: tell it to say it does not know, and fine-tune on a few such examples.
3. Lower the temperature and use constrained or structured outputs.
4. Verify: ask for quoted evidence and check the quote exists in the source, or run a rule-based or NLI check.
5. Route low-confidence or out-of-domain queries to a larger model.

Measure the hallucination rate on a labeled set, since prompt tweaks that feel better often change nothing.

Small models also tend to be overconfident, so do not treat the model's own certainty as a reliable signal on its own.

At every step, the decoder masks tokens that would break a grammar, such as a JSON schema or regex, before sampling. The output is therefore guaranteed to parse.

This helps small models most, since they often drop a brace or invent a key. Tools include llama.cpp GBNF grammars, Outlines, XGrammar, and guided decoding in vLLM.

root ::= "{" ws "\"sentiment\"" ws ":" ws ("\"positive\"" | "\"negative\"" | "\"neutral\"") ws "}"
ws   ::= [ \t\n]*

It guarantees syntax, not correctness, so a valid JSON can still hold a wrong label. Keep the schema simple and still describe the task in the prompt.

A local request has two phases, and each one stresses different hardware.

1. The app applies the chat template and the tokenizer turns text into token IDs.
2. Prefill: the model processes the whole prompt in parallel and fills the KV cache. This is compute-bound and determines time to first token.
3. Decode: one token per step, each reading all weights and the cache. This is memory-bandwidth-bound.
4. A sampler picks the next token using temperature, top-p, and any grammar mask.
5. Tokens are detokenized and streamed to the UI until EOS or the length limit.

flowchart TD
  A["Prompt text"] --> B["Chat template and tokenizer"]
  B --> C["Prefill: fill KV cache"]
  C --> D["Decode one token"]
  D --> E["Sample next token"]
  E --> F{EOS or max tokens?}
  F -- No --> D
  F -- Yes --> G["Detokenize and stream output"]

First separate the symptoms: slow TTFT points to prefill, slow streaming points to decode. Then check, in order:

1. Fit: if weights plus KV cache exceed free RAM, the OS swaps and speed collapses. Use a smaller model or quantization.
2. Quantization: move from FP16 to Q4_K_M. Fewer bytes per token means faster decode.
3. Threads: match the physical core count (-t in llama.cpp). Oversubscribing hurts.
4. Build: confirm the binary uses AVX2, AVX-512, or NEON instructions.
5. Offload: push layers to an integrated or discrete GPU (-ngl).
6. Context: reduce -c and prompt length.
7. Power: plug in, disable battery saver, and watch for thermal throttling.

Memory bandwidth, such as single-channel versus dual-channel RAM, is often the hidden cap.

Work on three fronts: model, runtime, and UX.

• Model: choose a 1B-3B model at 4-bit, and consider a task-specific fine-tune over a bigger generalist.
• Runtime: use the NPU or GPU through Core ML, NNAPI or LiteRT, or vendor backends, and keep the model loaded so you skip cold starts.
• Prompt: keep it short, cache the static system prompt prefix, and cap max_new_tokens.
• Decoding: try speculative decoding or KV cache quantization.
• UX: stream tokens, show partial results, and fall back to the cloud on low battery or heat.

Test on a mid-range phone, not only the flagship, and measure sustained speed after a few minutes because thermal throttling lowers it.

Quality drops, and small models drop faster than large ones because they have less redundancy to spare. Below 4 bits the loss is often visible; at 2 bits it is frequently severe.

Typical signs: perplexity rising sharply, repetition loops, broken JSON, weaker math and code, and worse results in non-English languages. Reasoning and instruction following usually degrade before plain fluency does.

Mitigations: keep embeddings and the output layer at higher precision, use mixed k-quants like Q4_K_M or Q5_K_M, use an importance matrix with good calibration data, try QAT, or step up one bit-width. Whatever you pick, test on your own task.

In a cascade, the SLM answers first. A decision step estimates whether the answer is good enough. If not, the query is escalated to a larger model.

The decision can use token log-probabilities, a separate confidence classifier, agreement across several samples, a verifier, or a learned router such as RouteLLM. Rule-based triggers, like the query length or a topic filter, also work.

flowchart LR
  U["User query"] --> S["SLM answers"]
  S --> C{Confident or verified?}
  C -- Yes --> R["Return SLM answer"]
  C -- No --> L["Escalate to LLM"]
  L --> R2["Return LLM answer"]

Track the escalation rate, the cost saved, and the quality compared with always using the LLM. Beware that small models can be confidently wrong, and escalated requests pay the latency of both steps.

Catastrophic forgetting shows up when the tuned model nails your task but loses general skills such as following other instructions, basic math, or other languages.

1. Confirm: run a general benchmark or small probe set on the base and the tuned model.
2. Lower the learning rate and cut epochs, and use early stopping.
3. Switch to LoRA or a lower rank so the base weights stay untouched.
4. Mix in replay data: add 10-30% general instruction examples to the training set.
5. Merge or interpolate the tuned weights with the base to recover general ability.
6. Broaden the data if it is very narrow or repetitive.

Always keep a general regression suite next to your task eval.

None wins everywhere. Pick by target platform and the team's existing stack.

If you ship to phones from a PyTorch workflow, ExecuTorch is natural. For a Windows desktop app, ONNX Runtime is typical. For quick local use or Mac and Linux tooling, llama.cpp is hard to beat. Benchmark on your real devices before deciding.

Reasoning needs capacity to represent intermediate states and many learned procedures. A small model has less of both, and errors compound across steps, so a long chain fails if any link is wrong.

Ways to improve it:

• Distill reasoning traces from a stronger teacher. DeepSeek-R1 distilled into 1.5B-70B Qwen and Llama models and Phi-4-mini-reasoning follow this approach.
• Reinforcement learning with verifiable rewards on math and code.
• Decompose the task into small steps handled by separate calls.
• Use tools such as a calculator or code interpreter.
• Spend test-time compute: self-consistency or best-of-N with a verifier.

The cost is more tokens and latency, so measure the trade against simply escalating to a larger model.

On-device runs improve data privacy, but they move the model into an environment you do not control.

• Weights: anyone can extract a shipped model. Use encrypted storage and platform key stores to raise the bar, and rely on license terms rather than secrecy.
• Supply chain: download from trusted sources, verify hashes and signatures, and prefer safetensors or GGUF over pickle files.
• Prompt injection: treat retrieved or user-supplied text as untrusted. Give tools least privilege and confirm risky actions.
• Guardrails: add a small safety classifier such as Llama Guard and filter outputs.
• Local data: encrypt chat history, caches, and logs.
• Updates: sign model updates and keep a rollback path.

Running locally does not make a model safe by default, so test for jailbreaks and data leakage.

A dense model uses all of its parameters for every token. A Mixture-of-Experts (MoE) model has many expert feed-forward blocks, and a router activates only the top few for each token.

MoE gives quality closer to its total size while computing like its active size. But every expert must sit in memory, so it does not shrink the footprint. For tight RAM budgets a dense 1B-4B model is simpler. With plenty of memory but a speed target, a small-active MoE can be attractive.