AI / Edge AI Interview questions

Edge AI means running machine learning models directly on the device where the data is produced - a camera, phone, car, robot or sensor - instead of shipping that data to a remote cloud server for processing.

The model is usually trained in the cloud, then compressed and deployed to the device. From that point the device performs inference locally and acts on the result immediately.

A doorbell that detects a person on-device, or a phone that transcribes speech with no network, are typical examples. Only small results or occasional samples need to leave the device.

Edge AI moves inference next to the data, which gives five practical benefits:

• Low latency - no network round trip, so responses arrive in milliseconds.
• Bandwidth savings - a camera does not need to stream raw video upstream.
• Privacy - raw audio, images or health data can stay on the device.
• Offline reliability - the feature still works with a weak or missing connection.
• Lower cloud cost - you stop paying per-request inference fees at fleet scale.

The trade-off is that you must fit the model into tight memory, power and thermal limits.

Both run the same kind of models. The difference is where inference happens, and that choice changes latency, privacy, cost and model size.

Most real products end up hybrid: the edge handles the fast path, the cloud handles heavy or rare cases.

Edge AI shows up wherever decisions must be fast, private or made without a reliable network.

• Smart cameras - person detection, retail footfall, license plate reading.
• Automotive - driver monitoring and lane or obstacle detection.
• Industrial - predictive maintenance from vibration and temperature sensors.
• Voice - wake-word detection on phones, earbuds and speakers.
• Wearables - arrhythmia or fall detection on a watch.
• Agriculture and drones - crop or pest detection with no field connectivity.

The common thread is a continuous data stream, a need for an immediate decision, or poor connectivity. If none of those apply, a plain cloud API is usually simpler to build and maintain.

On-device inference is the act of running a trained model's forward pass on the local hardware, with no call to a remote service.

The device loads the model file into a runtime such as LiteRT (TFLite), ONNX Runtime or Core ML, then repeats a simple loop for each input:

1. Capture input (frame, audio window, sensor reading).
2. Preprocess it - resize, normalize, quantize.
3. Run the model through the runtime.
4. Postprocess the output - decode boxes, apply a threshold.
5. Act on the result.

No training happens here. Weights stay fixed until a new model version is deployed.

Edge hardware is a fraction of a server, and every constraint shapes model design.

• Memory - from a few KB on microcontrollers to a few GB on gateways.
• Compute - limited CPU, small or no GPU, fixed-function accelerators.
• Power - battery or a tight power budget, often milliwatts to a few watts.
• Thermal - passive cooling, so sustained loads cause throttling.
• Storage - small flash, which limits model size and update strategy.
• Connectivity - intermittent or metered links.

Because of this, model size, memory footprint and energy per inference matter as much as accuracy.

Different chips trade flexibility against efficiency. Pick based on the model, the power budget and the software stack.

Most devices combine several, so the runtime decides which operator goes where.

An NPU (Neural Processing Unit) is a hardware block built specifically for neural network math - mainly matrix multiplies and convolutions - usually at INT8, INT16 or FP16 precision.

Because it skips general-purpose overhead, an NPU delivers far more operations per watt than a CPU. Examples are Apple's Neural Engine, Qualcomm's Hexagon NPU and the NPU inside Rockchip RK3588.

The catch is flexibility. NPUs support a fixed list of operators and data types, and you normally reach them through a vendor SDK or a delegate. Anything unsupported falls back to the CPU.

TinyML is machine learning on microcontrollers - chips with tens to hundreds of KB of RAM, running on milliwatts, often on a coin cell for months.

Models are usually tens of KB, heavily quantized, and run through runtimes such as TensorFlow Lite for Microcontrollers or CMSIS-NN. There is typically no OS, so memory comes from a fixed, preallocated tensor arena.

Common applications are keyword spotting, gesture recognition, vibration anomaly detection and simple visual wake words.

A keyword-spotting model can fit in tens of KB at 8-bit. Platforms such as Edge Impulse help with data collection, training and exporting a C++ library for the target chip.

Quantization stores weights, and often activations, in lower precision such as INT8 instead of FP32, so the model is smaller and runs faster on integer hardware.

The common affine mapping is real = scale * (q - zero_point), where q is the stored integer. Scale and zero point are chosen from the observed value range.

Moving from FP32 to INT8 cuts model size about 4x and reduces memory traffic. The cost is some rounding error, which is why calibration or quantization-aware training is used to protect accuracy.

Pruning removes parameters that contribute little to the output - typically weights with the smallest magnitudes - to make the network sparser or smaller.

A common workflow is iterative: train, prune a fraction, fine-tune to recover accuracy, then repeat until you hit the target size or sparsity.

Pruning alone does not guarantee a speedup. Zeroed weights only help if the runtime or hardware can skip them, which is why many edge teams prefer removing whole channels or filters.

In practice pruning is combined with quantization, because the savings stack. TensorFlow's Model Optimization Toolkit and torch.nn.utils.prune both provide magnitude-based pruning.

Knowledge distillation trains a small student model to imitate a large teacher model, so the student keeps much of the teacher's accuracy at a fraction of the size.

The student learns from the teacher's soft labels - the full probability distribution, softened with a temperature - not just hard class labels. The loss usually blends the normal cross-entropy with a KL divergence term against the teacher.

DistilBERT is a well-known example: about 40% smaller and 60% faster than BERT while retaining around 97% of its language understanding.

TensorFlow Lite is Google's runtime for on-device inference on phones, embedded Linux and microcontrollers. Google has been rebranding it as LiteRT, but the .tflite format and APIs remain widely used.

The workflow has two parts. A converter turns a trained model into a compact FlatBuffer .tflite file, optionally quantized. A lightweight interpreter then loads that file on the device and runs it.

Hardware acceleration is added through delegates for GPUs, DSPs and NPUs, while the XNNPACK delegate speeds up the CPU path.

ONNX (Open Neural Network Exchange) is an open file format that describes a model as a graph of standard operators. ONNX Runtime (ORT) is the engine that executes that graph.

The point is portability: train in PyTorch, scikit-learn or another framework, export once to ONNX, then deploy to many targets.

ORT picks hardware through execution providers such as CUDA, TensorRT, OpenVINO, CoreML, NNAPI, QNN and XNNPACK, with CPU as the default fallback. There are also reduced-size builds for mobile.

ONNX Runtime also offers an ORT format for smaller binaries and a mobile package, plus tools to quantize an ONNX model to INT8 before you deploy it.

Latency is how long a single inference takes from input to output, measured in milliseconds. Throughput is how many inferences complete per second.

Edge applications are usually latency-driven. A camera at 30 FPS gives you a budget of roughly 33 ms per frame for the whole pipeline - capture, preprocessing, inference and postprocessing.

Pipelining stages can raise throughput without lowering the latency of any one frame. Always report p95 or p99 latency, not just the average, because spikes cause visible stutter.

MEC places compute and storage inside the telecom network - at base stations or aggregation sites - so applications run a few network hops from users rather than in a distant cloud region.

It is standardized by ETSI and often paired with 5G. It sits between the device and the cloud, which is why it differs from pure on-device inference.

Typical uses include AR and cloud gaming, video analytics across many cameras, and vehicle-to-everything services where a phone or sensor is too weak to run the model alone.

The Edge TPU is a small Google ASIC for fast, low-power inference. It is sold in the Coral line as a USB accelerator, M.2 and PCIe cards, and a dev board.

It delivers about 4 TOPS of INT8 compute at roughly 2 W. Models must be fully INT8 quantized and then compiled with the edgetpu_compiler.

The compiler maps supported operators to the TPU. Everything from the first unsupported operator onward runs on the CPU, so a model with an unsupported op in the middle may see little benefit.

Jetson is NVIDIA's family of embedded modules (Nano, Orin NX, AGX Orin and others) that put a CUDA GPU with Tensor cores, ARM CPUs and video engines on one board.

It targets heavier edge workloads such as multi-camera video analytics, robotics, autonomous machines and drones. The JetPack SDK provides CUDA, cuDNN and TensorRT, and DeepStream handles video pipelines.

Power modes are configurable, from single-digit watts to around 60 W depending on the module, so you trade performance for thermals.

Both produce a quantized model. The difference is when the model learns to tolerate lower precision.

QAT inserts fake-quantize nodes so the weights learn to survive rounding. Try PTQ first, and move to QAT only when accuracy falls short.

INT8 hits the sweet spot between savings and accuracy loss.

• 4x smaller weights than FP32, which helps tight flash and RAM.
• Less memory traffic, which is often the real bottleneck at the edge.
• Cheaper arithmetic - integer multiplies use less energy and silicon than floating point.
• Hardware support - most NPUs, DSPs and the Edge TPU are designed around INT8.

With a representative calibration set, CNN accuracy typically stays within a point or two of FP32. Lower precisions save more but need more care.

Using per-channel weight scales for convolutions usually recovers most of the accuracy lost by coarser per-tensor scaling, so it is a common default in converters.

Unstructured pruning zeroes individual weights anywhere in a tensor. Structured pruning removes whole units - filters, channels or attention heads.

Some hardware supports patterns such as NVIDIA's 2:4 sparsity, which sits in between. For most edge chips, structured pruning is the safer route.

As a rule of thumb, if you must hit a latency target on a phone NPU or Coral, remove channels and retrain. If the only goal is a smaller download, unstructured pruning plus compression of the sparse weights can still shrink the file.

MobileNet replaces standard convolutions with depthwise separable convolutions, which split the work in two steps.

1. A depthwise convolution filters each input channel separately.
2. A pointwise 1x1 convolution then mixes channels.

The cost falls to about 1/N + 1/K^2 of a standard convolution, where N is output channels and K the kernel size. For 3x3 kernels that is roughly 8-9x fewer multiply-adds.

MobileNetV2 adds inverted residual blocks, and V3 uses platform-aware architecture search and hard-swish. Width and resolution multipliers let you scale the model to the device.

The width multiplier thins every layer (0.5 halves the channels) and the resolution multiplier shrinks the input image. Together they give a family from tiny to fairly accurate for different devices.

Use tf.lite.TFLiteConverter on a SavedModel or Keras model. For full-integer INT8, supply a representative dataset so the converter can calibrate activation ranges.

import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def rep_data():
    for img in calib_ds.take(200):
        yield [img]

converter.representative_dataset = rep_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

open("model_int8.tflite", "wb").write(converter.convert())

Afterwards, load the file in the TFLite interpreter and compare its outputs and accuracy with the original model. Check that every operator is supported on your target.

If conversion fails, the usual cause is an operator with no TFLite builtin. You can enable select TF ops, but that links a much larger runtime, so replacing the op in the model is usually better on small devices.

Switch the model to eval mode, build a dummy input with the real shape, and call torch.onnx.export.

import torch

model.eval()
dummy = torch.randn(1, 3, 224, 224)

torch.onnx.export(
    model, dummy, "model.onnx",
    input_names=["input"], output_names=["logits"],
    opset_version=17,
)

Then validate. Run onnx.checker.check_model, and compare PyTorch and ONNX Runtime outputs on the same input with a small tolerance. Prefer static shapes when possible, since many edge runtimes and NPUs handle dynamic shapes poorly.

After export you can shrink the model further with ONNX Runtime's quantization tools, or hand the .onnx file to vendor converters such as TensorRT's trtexec, OpenVINO or Qualcomm's QNN tools.

Both optimize models for inference, but they target different hardware and use different strategies.

Choose TensorRT when you ship on Jetson or other NVIDIA GPUs. Choose TFLite when you need broad reach across phones and embedded boards.

The two also meet in the middle: ONNX Runtime can use TensorRT as an execution provider, so one ONNX model can reach NVIDIA GPUs and other targets.

Operator fusion merges adjacent operations - such as Conv + BatchNorm + ReLU - into one kernel, so intermediate results never travel back to main memory.

On edge hardware memory traffic often costs more time and energy than the math itself. Fewer kernel launches also reduce overhead.

Batch norm is a classic case: at inference it is just a scale and shift, so converters fold it into the convolution weights and the layer disappears entirely. Exporters and runtimes such as TFLite, TensorRT and ONNX Runtime apply common fusions automatically.

You can inspect the result in Netron. If a converted graph still shows separate Conv, BatchNorm and ReLU nodes, a fusion did not happen, which is worth investigating before blaming the hardware.

At load time the interpreter checks which parts of the graph a delegate can handle. Supported subgraphs are handed to the delegate, and everything else stays on the CPU.

Common delegates cover mobile GPUs, Qualcomm's Hexagon DSP, Core ML on iOS, and the Edge TPU. XNNPACK accelerates the CPU path by default.

Each hand-off between CPU and accelerator costs time for data copies, so a graph split into many small pieces can run slower than CPU alone. Here is how the Edge TPU delegate is attached on Linux:

from tflite_runtime.interpreter import Interpreter, load_delegate

interp = Interpreter(
    model_path="model_edgetpu.tflite",
    experimental_delegates=[load_delegate("libedgetpu.so.1")],
)
interp.allocate_tensors()

NPUs support a fixed operator list, specific data types and often static shapes. When the model steps outside that, the runtime either refuses to compile it or silently moves those operators to the CPU.

Frequent causes:

• Unsupported operators - custom ops, GELU, Swish or certain attention ops.
• Unsupported dtype - FP32 graphs on an INT8-only NPU.
• Dynamic shapes - variable input size or data-dependent control flow.
• Tensors too large for on-chip SRAM, forcing slow spills.

To fix it, check the vendor's op list, swap in friendlier operators (ReLU6 or hard-swish), freeze shapes, quantize properly, and profile to see where the graph splits.

Choose edge when the cost of the network - in time, money or risk - is higher than the cost of a smaller model.

• Latency budget under ~100 ms, such as safety or real-time control.
• Offline operation is required.
• Privacy or regulation restricts moving raw data.
• High data volume, like continuous video, makes uploading too costly.

Choose cloud when the model is too large for any device, you must iterate on it daily, or traffic is low and bursty. Many teams pick a hybrid: edge for the common case, cloud for hard ones.

A quick test: write down the worst-case latency, the offline requirement and the daily data volume per device. If any one of them rules out the cloud, start at the edge.

A hybrid design runs a small model on the device for the fast path, and escalates uncertain or complex inputs to a larger cloud model. The cloud also collects hard samples to retrain the edge model.

flowchart LR
  S["Sensor input"] --> E["On-device model"]
  E --> C{Confident?}
  C -- Yes --> A["Act locally"]
  C -- No --> CL["Cloud model"]
  CL --> A
  CL --> T["Hard samples for retraining"]
  T --> U["OTA model update"]
  U --> E

Most traffic is handled locally, so latency and cost stay low, while accuracy on difficult cases stays high. Define the confidence threshold carefully, since it controls the cloud load.

When inference runs locally, raw audio, video and health signals never have to leave the device. That follows the data minimization principle behind laws such as GDPR and HIPAA, and shrinks the amount of data a breach could expose.

It also builds user trust. A camera that only reports "person detected" is easier to accept than one streaming footage.

Edge is not automatically private, though. A compromised device can leak data, and a stolen model file can leak training information. Pair it with encryption, secure boot, and techniques like federated learning or differential privacy.

Regulation also shapes the design. For example, keeping biometric templates on the device can simplify consent and data-residency questions.

In federated learning, devices train a shared model on their own data and send only model updates to a server. The raw data never leaves the device.

sequenceDiagram
  participant S as Server
  participant D1 as Device 1
  participant D2 as Device 2
  S->>D1: Send global model
  S->>D2: Send global model
  D1->>D1: Train on local data
  D2->>D2: Train on local data
  D1->>S: Send weight update
  D2->>S: Send weight update
  S->>S: Aggregate with FedAvg
  S->>D1: Send improved model

FedAvg averages the updates, usually weighted by sample count. Challenges include non-IID data across devices, communication cost, and stragglers. Updates can still leak information, so secure aggregation and differential privacy are often added. Flower and TensorFlow Federated are common frameworks.

A well-designed edge system keeps working, because inference runs locally and does not depend on the network.

What changes is everything around it. The device should follow an offline-first pattern:

1. Keep making decisions with the on-device model and cached configuration.
2. Buffer telemetry and results in a bounded local queue (store-and-forward).
3. Timestamp events carefully, since clocks may drift without sync.
4. On reconnect, flush the queue with retry and exponential backoff.
5. Postpone model updates until the link is stable.

Define what to drop when the buffer fills, and choose safe default behavior for any feature that truly needs the cloud.

Test this explicitly by unplugging the network in staging. Most offline bugs - growing queues, expired tokens, startup blocked while waiting for a server - only show up then.

Treat a model update like a firmware release: signed, staged and reversible.

1. Package the model with a manifest (version, hash, compatible runtime).
2. Sign it, and have the device verify the signature before use.
3. Download efficiently - delta updates and resume on weak links.
4. Stage it in an inactive slot (A/B) and validate it with a quick self-test.
5. Roll out gradually - canary 1-5% of devices, watch metrics, then widen.
6. Roll back automatically if health checks fail.

Tools such as AWS IoT Greengrass, Azure IoT Edge, Mender and balena provide much of this pipeline.

Delta updates matter because a full model can be tens of MB over a metered cellular link. Sending only the changed parts can sometimes cut that sharply.

Split computing cuts a neural network into two parts. Early layers run on the device, and the intermediate activation is sent to a nearby edge server or the cloud, which runs the rest.

The best split point is where the activation tensor is small, such as a bottleneck layer, because early CNN layers often produce tensors larger than the input image. Total latency is roughly device compute + transfer time + server compute.

It helps weak devices and offers some privacy since raw data is not sent. It still depends on the network, so it works poorly offline.

Edge inputs usually arrive one at a time - a frame, an audio window, a sensor reading - and the response is needed immediately. Waiting to fill a batch would add latency.

Cloud GPUs batch requests to keep thousands of cores busy. Small edge accelerators have much less parallel headroom, so larger batches bring limited throughput gains and use scarce memory.

The exception is a gateway handling several camera streams, where batching frames across streams can raise utilization without hurting any one stream's deadline too much.

A related trick is to keep batch size at 1 but pipeline the stages across threads, so capture, inference and postprocessing overlap and the device stays busy without adding waiting time.

A layer is compute-bound when the processor's math units are the limit, and memory-bound when moving data to and from memory is the limit.

What decides it is arithmetic intensity - operations performed per byte moved. The roofline model compares that with the hardware's compute-to-bandwidth ratio.

• Large standard convolutions and big matrix multiplies tend to be compute-bound.
• Depthwise convolutions, element-wise ops and LLM token decoding tend to be memory-bound.

For memory-bound work, quantization, operator fusion and better data reuse help more than a faster ALU.

Profiling helps tell them apart. If raising the compute clock barely changes a layer's latency, it is probably memory-bound.

When a chip heats up, it lowers its clock speed to protect itself. For inference, that means latency creeps up and FPS drops after a few minutes of sustained load, even though the first runs looked fast.

This is common in fanless phones, sealed enclosures and outdoor cameras. Benchmarks that run for 30 seconds often miss it.

To handle it:

• Benchmark for sustained periods and watch temperature and clocks.
• Reduce work - lower precision, resolution or frame rate.
• Duty-cycle inference instead of running continuously.
• Improve heat dissipation, or choose a lower power mode.
• Design pipelines to tolerate variable latency.

Mobile operating systems expose thermal-state APIs, which let an app lower its frame rate before the system forces a drop.

Benchmark on the real target device, with realistic inputs and a fixed power mode. Results from a laptop say little about an NPU.

1. Run warm-up iterations first, so caches and clocks settle.
2. Run hundreds of iterations and report p50, p95 and p99 latency.
3. Time the full pipeline, including pre- and postprocessing.
4. Measure peak memory and, if possible, power or energy per inference.
5. Run long enough to expose thermal throttling.
6. Check accuracy on the device, not only latency.

Useful tools are TFLite's benchmark_model and NVIDIA's trtexec:

adb shell /data/local/tmp/benchmark_model --graph=model.tflite --num_threads=4 --use_gpu=true

trtexec --onnx=model.onnx --fp16 --saveEngine=model.plan

Fix the CPU governor and power mode where possible, such as nvpmodel and jetson_clocks on Jetson. Otherwise runs on the same board can differ by tens of percent.

They differ in bits per weight, and so in size, speed and accuracy risk.

Each step down halves memory traffic but raises accuracy risk. INT4 usually needs weight-only methods such as GPTQ or AWQ, or QAT, rather than naive rounding.

Between FP32 and INT8 sits FP16 or bfloat16: usually a near-free win on GPUs, though many NPUs lack floating point entirely. Mixed precision - keeping sensitive layers in INT8 or FP16 while the bulk goes lower - is how most teams ship INT4 models without wrecking accuracy.

Work from the model's outputs back to the specific layers causing damage.

1. Confirm the baseline - check FP32 accuracy and the preprocessing match on both.
2. Check calibration data - it must be representative, with a few hundred real samples.
3. Compare layer by layer - measure cosine similarity or SQNR between FP32 and INT8 activations to find the worst layers.
4. Try per-channel weight quantization instead of per-tensor, especially for depthwise convs.
5. Keep sensitive layers in higher precision - often the first, last, or attention softmax.
6. Switch to QAT if PTQ still falls short.

Activation outliers are a frequent culprit - a few huge values stretch the range and crush the resolution for everything else.

Document the final recipe - calibration set, per-channel settings and any layers kept in higher precision - so the result is reproducible each time the model is retrained.

Treat peak memory, not just file size, as the budget - activations and buffers often use more than weights.

1. Pick an efficient backbone such as MobileNet, EfficientNet-Lite or a small YOLO variant.
2. Reduce input resolution to the smallest size that keeps accuracy.
3. Quantize to INT8 to shrink both weights and activations.
4. Memory-map the model so weights are not duplicated in RAM.
5. Reuse buffers and avoid copying frames between stages.
6. Load one model at a time and skip frames when the pipeline falls behind.

Measure peak RSS on the real device and test under load, since the OS and camera stack also need their share.

As a sanity check, a 4 MB INT8 model can still need hundreds of MB at runtime once camera buffers, the OS and activations are counted, so leave 20-30% headroom.

A typical vision pipeline moves data through five stages, and the whole chain must fit in the latency budget.

flowchart LR
  A["Capture frame"] --> B[Preprocess]
  B --> C["Runtime and delegate inference"]
  C --> D[Postprocess]
  D --> E["Decision or action"]
  E --> F["Telemetry queue"]

1. Capture - the camera or sensor delivers a buffer.
2. Preprocess - resize, normalize, and quantize to the model's input type.
3. Inference - the runtime runs the graph on the CPU or a delegate.
4. Postprocess - decode boxes, apply non-maximum suppression and thresholds.
5. Act - trigger an alert, a motor command or a UI update.

Preprocessing and postprocessing are often on the CPU and can dominate total latency, so profile the whole chain.

A common optimization is to run capture and inference on separate threads with a small ring buffer, so a slow frame never blocks the camera. Drop old frames instead of queueing them - a stale detection is worse than a missing one.

Edge devices sit in places you do not control, so assume an attacker can touch the hardware.

• Secure boot and signed firmware - only trusted code runs.
• Signed, verified model updates - reject tampered files.
• Encrypt the model and data at rest, with keys in a hardware keystore or TrustZone.
• Use a TEE for sensitive inference where available.
• Encrypt traffic with mutual TLS and unique per-device identities.
• Limit retention - do not store raw data longer than needed.

Also plan for model extraction and adversarial inputs. Rate-limit outputs, and validate inputs, since a model on a device can be copied and probed.

Do not forget physical attacks. Disable debug ports such as JTAG and UART in production, and use anti-rollback counters so an attacker cannot flash an older, vulnerable model or firmware.

You rarely have labels on the device, so monitor proxies and send compact summaries instead of raw data.

• Input statistics - brightness, noise or feature histograms, compared with training data using PSI or a KS test.
• Prediction statistics - class mix and confidence distribution over time.
• Uncertainty rate - the share of low-confidence or escalated cases.
• Sampled review - upload a small, consented sample for labeling.
• Business signals - false alarm rate, user overrides.

Aggregate by device group and software version, so a drift caused by a new camera lens or firmware stands out. When drift crosses a threshold, retrain and ship an OTA update.

Keep the telemetry budget small - a few kilobytes of histograms per day per device is usually enough - and remember that privacy rules may limit what you can collect even for monitoring.

Small language models (roughly 1B to 8B parameters) fit on phones and laptops through aggressive quantization and optimized runtimes.

• 4-bit weights (GGUF Q4, GPTQ, AWQ) cut a 7B model from ~14 GB at FP16 to roughly 4 GB.
• Runtimes such as llama.cpp, ExecuTorch, MLC LLM and MediaPipe LLM Inference run them on CPU, GPU or NPU.
• KV cache grows with context length, so long prompts need extra memory beyond the weights.

Token generation (decode) is memory-bandwidth-bound: each token reads nearly all the weights, so speed is roughly memory bandwidth divided by model size. This is why 4-bit models also run faster.

Techniques such as speculative decoding, a smaller context window and sliding-window attention help further.

On phones, NPU or GPU backends can beat plain CPU decoding, but support varies by model architecture. Benchmark time to first token and tokens per second on the real device.

Treat every model as a versioned artifact, tracked in a model registry with its metadata.

1. Version it - semantic version, training data snapshot, metrics, and file hash.
2. Declare compatibility - the manifest lists the runtime, firmware and hardware it supports.
3. Keep the previous version on-device so rollback is instant and works offline.
4. Use shadow mode - run the new model beside the old one and compare outputs before switching.
5. Roll out in rings with automatic rollback on bad metrics.
6. Keep a kill switch to pin devices to a known-good version.

Report the active version in telemetry, so every complaint can be tied to the exact model that produced it.

Never promote a model to the full fleet on offline metrics alone. The new version may score better on the test set but run slower on older hardware, which only a staged rollout on real devices will reveal.

Energy per inference matters as much as latency on battery devices, so reduce both the work done and how often it is done.

• Cascade models - a tiny always-on detector wakes the larger model only when needed.
• Event-triggered inference - run on motion or sound, not constantly.
• Use the NPU or DSP - they deliver far more operations per joule than the CPU.
• Lower precision, resolution and frame rate where accuracy allows.
• Duty-cycle the sensor and radio, and batch network uploads.
• Measure energy per inference, not just speed.

A faster chip is not always more efficient. Track inferences per joule on the real device.

Radios are often the biggest drain, so on-device inference can save energy overall: sending a 100 KB frame over cellular can cost more energy than analyzing it locally.

Neural architecture search (NAS) automatically explores network designs. The hardware-aware version adds the target device's real latency or energy to the objective, not just accuracy.

This matters because FLOPs are a poor proxy for speed. Two networks with the same FLOPs can differ greatly in latency depending on memory access and operator support.

MnasNet measured latency on a phone during search, and MobileNetV3 used platform-aware NAS. Once-for-All trains one supernet from which many specialized sub-networks can be extracted for different devices. Latency lookup tables or predictors speed the search, but the search itself can still be expensive.

Most teams do not run NAS themselves. They start from a hardware-tuned family such as MobileNetV3 or EfficientNet-Lite, and search only when a device's operator set or latency budget is unusual.

In most products, the best answer is a small on-device model that escalates to the cloud only when unsure. It keeps typical latency and cost low, works offline for common cases, and still gets help on hard ones.

A heavy model forced onto the device usually means throttling, short battery life and slow responses. A cloud-first design fails the moment connectivity drops and adds latency to every request.

Decide with data: measure the share of cases the small model handles confidently.