AI / Traditional MLOps Interview questions

MLOps (machine learning operations) is a set of practices that brings DevOps discipline to the machine learning lifecycle. It covers building, testing, deploying, monitoring and retraining models in a repeatable, automated way.

The core idea is simple: a model is not finished when training ends. It has to be packaged, served, watched and updated as data changes. MLOps ties data engineers, data scientists and operations teams together so that happens reliably.

The usual building blocks are data and model versioning, automated pipelines, CI/CD, a model registry and production monitoring.

A classic ML lifecycle has six connected stages: data collection and ingestion, data preparation and feature engineering, model training and tuning, evaluation and validation, deployment, and monitoring.

It is a loop, not a straight line. Monitoring results (drift, falling accuracy, new data) feed back into data preparation and training, which triggers the next iteration.

flowchart LR
  A["Data collection"] --> B["Data prep and features"]
  B --> C["Training and tuning"]
  C --> D[Evaluation]
  D --> E[Deployment]
  E --> F[Monitoring]
  F -->|Drift or decay| B

MLOps targets the gap between a notebook that works once and a model that keeps working in production. Without it, teams hit the same issues again and again.

• Manual, slow releases - models wait weeks for hand-over to engineering.
• Irreproducible results - nobody can recreate last month's model because data, code or parameters were not recorded.
• Silent decay - accuracy drops as real-world data shifts and nobody notices.
• Training-serving mismatch - features are computed differently offline and online.
• No audit trail - hard to explain which model made which decision.

Pipelines, versioning, registries and monitoring each address one or more of these.

Model versioning means giving every trained model a unique, traceable identity so you can compare, deploy or roll back any of them later.

A useful version is more than a file. It links the model artifact to the code commit, training data snapshot, hyperparameters, metrics and environment that produced it.

Tools like MLflow and DVC, or a cloud model registry, usually store this. A model might be fraud-model v7 with a tag showing it trained on the March data snapshot. If v7 misbehaves, you can redeploy v6 within minutes.

Data versioning tracks changes to datasets the same way Git tracks code, so any training run can be tied to the exact data it used.

Datasets are too big for Git, so tools such as DVC, lakeFS or Delta Lake time travel store the data elsewhere. Git keeps only a small pointer file holding a content hash.

dvc add data/train.csv
git add data/train.csv.dvc .gitignore
git commit -m "Add training data v1"
dvc push

Checking out an older commit and running dvc pull brings back the matching data.

Experiment tracking is the practice of logging every training run's parameters, metrics, code version, data version and output artifacts in one searchable place.

Data scientists try dozens of variations. Without a log, results live in notebooks and memory. With tracking you can sort runs by validation AUC, compare two runs side by side and reproduce the best one.

import mlflow
with mlflow.start_run():
    mlflow.log_param("max_depth", 8)
    mlflow.log_metric("val_auc", 0.912)
    mlflow.sklearn.log_model(model, "model")

MLflow Tracking, Weights & Biases and Neptune are common choices.

A model registry is a central store that manages trained models and their lifecycle. Each registered model has versions, metadata, lineage and a stage or alias such as Staging or Production.

It acts as the hand-off point between training and deployment. The training pipeline registers a candidate, reviewers or automated tests promote it, and the deployment system pulls the approved version.

Examples are the MLflow Model Registry, SageMaker Model Registry and Vertex AI Model Registry. They also keep audit history, so you know who promoted which version and when.

A feature store is a shared system that stores, serves and manages ML features so they are computed once and reused across models and teams.

It usually has two parts: an offline store (large historical data used for training) and an online store (low-latency key-value lookups used at prediction time). Both are fed by the same feature definitions.

That shared definition is the main payoff. It avoids duplicate work and removes the most common cause of training-serving skew. Feast, Tecton and Databricks Feature Store are well-known options.

An ML pipeline is an automated, ordered series of steps that turns raw data into a deployable model. Each step takes defined inputs and produces defined outputs.

Typical steps are data ingestion, validation, preprocessing, feature engineering, training, evaluation and model registration. Because each step is a separate component, you can rerun only what changed and cache the rest.

Pipelines are defined in tools like Kubeflow Pipelines, Airflow, Vertex AI Pipelines or SageMaker Pipelines, and can run on a schedule or on a trigger such as new data arriving.

CI/CD for ML automates testing and delivery of the whole ML system, not just application code. It adds data and model checks on top of the usual build and deploy flow.

• CI - on each commit, run unit tests, data validation and a quick training check.
• CD - deploy the pipeline or the model service automatically once checks pass.
• CT (continuous training) - retrain models automatically when new data or drift triggers it.

CT is the piece that classic software CI/CD does not have.

There are four common patterns, picked by latency needs and traffic shape.

Many systems mix them, for example batch scores for most users plus online scoring for fresh events.

Model monitoring is the continuous observation of a deployed model's health. It checks that the service is up and that predictions are still good.

Two layers matter. Operational metrics cover latency, throughput, error rate and resource use. ML metrics cover input data drift, prediction distribution, and accuracy once true labels arrive.

Alerts on these signals let the team investigate or trigger retraining before users feel the damage. Prometheus and Grafana handle the operational side, while Evidently, WhyLabs or Arize are common for ML-specific checks.

Data drift (covariate shift) happens when the statistical distribution of input features in production differs from what the model saw in training, even if the relationship between inputs and the target has not changed.

For example, a loan model trained mostly on applicants aged 30-50 starts receiving many applications from people in their 20s. The model is now working on data it knows less about, so quality can fall.

It is detected by comparing production feature distributions against a training baseline using tests such as PSI, KS test or Jensen-Shannon distance.

Concept drift occurs when the relationship between input features and the target variable changes over time. The inputs may look the same, but the right answer for them is different now.

A classic case is fraud. Fraudsters change tactics, so a transaction pattern that was harmless last year may now signal fraud. The model's learned rules become outdated.

Because the inputs can look normal, concept drift often shows up only through falling accuracy or business metrics once labels arrive. The usual fix is retraining on recent data.

Training-serving skew is a difference between how features are produced during training and how they are produced at prediction time. The model gets inputs in production that it never really saw while learning.

Common causes are different code paths (SQL in training, Java in serving), different library versions, stale features online, or preprocessing steps that exist in only one place.

A typical fix is to share one transformation definition, for example through a feature store or by packaging preprocessing inside the model artifact. Logging served features and comparing them with training features also exposes skew.

Docker packages a model, its code, libraries and system dependencies into a container image that runs the same way on a laptop, a CI runner or a Kubernetes cluster.

This solves the 'works on my machine' problem. A training job and a serving endpoint can each have a pinned image, which also supports reproducibility.

FROM python:3.11-slim
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ /app
CMD ["uvicorn", "app.main:api", "--host", "0.0.0.0", "--port", "8080"]

Images are stored in a registry and deployed by orchestrators like Kubernetes.

Reproducibility means that someone else, or you months later, can rerun an experiment and get the same or statistically equivalent result.

It needs four things pinned: the code (Git commit), the data (versioned snapshot), the environment (library versions or container image) and the configuration (hyperparameters and random seeds).

If any one of these is missing, results can drift. Hardware differences such as GPU non-determinism can still cause tiny variations, so teams often accept results within a tolerance.

Tools are usually grouped by the job they do.

No one stack is mandatory. Pick based on cloud, team skills and scale.

Model retraining means training a model again on newer or additional data so it keeps up with changes in the real world.

It can be a full retrain from scratch or an incremental update on recent data. The new model should always go through the same evaluation and approval steps as the first one, and be compared against the current production model before replacing it.

Retraining can be triggered by a schedule (weekly), by new-data volume, or by monitoring alerts such as drift or accuracy loss.

Data validation is an automated check that incoming data matches expectations before it is used for training or prediction. It stops bad data from silently producing a bad model.

Typical checks cover schema (column names and types), missing values, value ranges, category sets and distribution shifts compared with a reference. If a check fails, the pipeline halts or raises an alert.

# Great Expectations style check
validator.expect_column_values_to_not_be_null("customer_id")
validator.expect_column_values_to_be_between("age", 18, 100)

TensorFlow Data Validation and Great Expectations are popular tools.

DevOps manages the lifecycle of code. MLOps manages the lifecycle of code, data and models together. That extra moving part changes testing, versioning and monitoring.

A model can pass every software test and still be wrong because the world changed. That is why MLOps adds continuous training and ML-specific monitoring on top of DevOps habits.

In ML, the same code on different data produces a different model. Git alone cannot tell you which data produced a given result, so code versioning by itself is not enough.

Data versioning closes that gap. It lets you rebuild any past model, debug a regression by diffing datasets, and prove for audits what a model was trained on.

• Reproduce - check out commit X and data version Y to rebuild the same model.
• Debug - if v8 is worse than v7, compare the two training datasets first.
• Audit - regulated industries must show training data lineage.

Without it, a silently refreshed table can change your results and leave no trace.

It forces training and serving to read features produced by the same definition. You write the transformation once, and the store materializes it to both an offline and an online store.

Without a feature store, a data scientist often writes feature logic in a notebook for training, and an engineer rewrites it for production. Small differences, such as how nulls are handled or how a window is cut, create skew.

The store adds two more safeguards. Point-in-time joins make sure training rows only use feature values that existed at that moment, avoiding leakage. Freshness tracking keeps the online values up to date.

flowchart LR
  D["Feature definition"] --> OFF["Offline store"]
  D --> ON["Online store"]
  OFF --> T["Training dataset"]
  ON --> S["Online prediction"]

Capture a reference dataset (usually the training or a healthy production window), log live features, and compare the two distributions on a schedule.

1. Log input features and predictions from the serving layer.
2. Pick a window (hourly, daily) and compute statistics per feature.
3. Run a test: KS test or Wasserstein distance for numeric features, chi-square or PSI for categorical or binned ones.
4. Alert when a score crosses a threshold, for example PSI above 0.2.

import numpy as np
def psi(expected, actual, bins=10):
    cuts = np.percentile(expected, np.linspace(0, 100, bins + 1))
    e = np.histogram(expected, cuts)[0] / len(expected) + 1e-6
    a = np.histogram(actual, cuts)[0] / len(actual) + 1e-6
    return np.sum((a - e) * np.log(a / e))

A common rule of thumb reads PSI below 0.1 as stable and above 0.25 as a major shift. Treat thresholds as starting points and tune them per feature.

Data drift is a change in the inputs. Concept drift is a change in the meaning of the inputs, that is, how they relate to the target.

Data drift does not always hurt performance, for instance if the shifted region is well covered by training. Concept drift almost always does, since the learned rule is now wrong.

Retrain when evidence says the current model is falling behind, not just because time has passed. The usual triggers fall into four groups.

• Performance - accuracy, precision or a business KPI drops below an agreed threshold.
• Drift - input or prediction distributions move beyond limits.
• Data - enough new labeled data has accumulated, or a data source changed.
• Schedule or event - weekly or monthly cadence, or a product or policy change.

Retraining has a cost in compute and review effort, so the trigger should balance that cost against the risk of staleness. Fast-moving domains like fraud may retrain daily. Stable ones like some image classifiers may need it a couple of times a year.

Choose batch inference when predictions are not needed instantly, and the inputs are known ahead of time. Choose online when each request needs an answer in milliseconds.

Examples for batch are nightly recommendation lists, weekly churn scores and credit-limit reviews. If the user context changes between runs and matters (fraud at checkout), batch will not work.

A hybrid is common: precompute heavy scores in batch, then add a light real-time adjustment.

Traditional CI/CD tests and ships code. ML CI/CD must also test and ship data and models, and the thing being deployed is often a training pipeline rather than a single service.

• Extra tests - data schema checks, feature tests, model quality gates against a baseline, fairness and latency checks.
• Extra triggers - new data and drift can start a run, not just commits.
• Extra artifacts - datasets, features and models are versioned alongside code.
• Continuous training - a pipeline that retrains and registers models automatically.

A passing build does not mean a good model. The quality gate, for example 'new AUC must be at least the production AUC minus 0.01', is the ML equivalent of a failing unit test.

A model moves through a controlled set of stages, with a gate between each one. Names vary by tool, but the flow is similar.

flowchart LR
  A["Training run"] --> B["Registered version"]
  B --> C[Staging]
  C -->|Tests and approval| D[Production]
  D -->|Replaced by newer version| E[Archived]
  C -->|Fails checks| E

1. Register - the pipeline logs a new version with metrics, lineage and artifacts.
2. Staging - integration tests, shadow traffic and review happen here.
3. Production - approved version serves live traffic.
4. Archived - older or rejected versions are kept for rollback and audit.

MLflow now favors aliases such as champion over fixed stages, but the idea is the same: serving code loads the model behind an alias, so promotion needs no redeploy.

Both release a new model gradually, but they differ on whether users see the new model's answers.

A typical path is shadow first, to catch crashes and odd outputs, then canary at 5%, 25% and 100% while watching error rate and business metrics. If metrics degrade, traffic goes back to the old model.

Rollback should be a routine, fast action, not an emergency script. That requires planning before the release.

1. Keep the previous version live-ready - old model stays in the registry with its image and config.
2. Switch traffic - move the alias or route back to the old version (or set the canary share to 0%).
3. Verify - confirm error rate, latency and prediction distributions return to normal.
4. Investigate - freeze the bad version, keep its logs and data, and run a post-mortem.

# MLflow alias example
client.set_registered_model_alias("churn-model", "champion", version=6)

Rollback only works well when the old model's feature inputs still exist. If the new release changed feature definitions, roll those back too.

A/B testing splits live traffic between the current model (A) and a candidate (B) and compares a business metric to decide which is better.

1. Randomly assign users, not requests, to A or B so each user gets a consistent experience.
2. Pick the primary metric first, such as conversion rate or fraud caught, plus guardrail metrics like latency.
3. Run until the sample size gives enough statistical power. Stopping early on a lucky streak is a classic mistake.
4. Test for significance. Promote B only if it wins on the primary metric without hurting guardrails.

Offline metrics like AUC do not always predict business impact, which is exactly why A/B tests exist. Bandit approaches can shift traffic gradually toward the winner when speed matters more than a clean answer.

Because the true answer often arrives late, or never. A model that predicts loan default won't learn if it was right for months. A recommender may never learn what the user would have clicked instead.

• Delayed labels - ground truth arrives days or months later.
• Missing labels - you only see outcomes for the cases the model allowed, a form of selection bias.
• Costly labels - human review is needed to label a sample.

Teams work around this with proxy signals and early warnings: input drift, prediction distribution shifts, confidence scores, and short-term business metrics like click rate. They also label a small random sample on a schedule, then compute real accuracy once labels land.

Proxies are hints, not proof. They tell you when to look, and delayed labels confirm what happened.

Start by measuring where the time goes: network, feature lookup, preprocessing or the model itself. Then attack the biggest slice.

• Model-level - quantization (FP32 to INT8), pruning, distillation into a smaller model, or exporting to ONNX / TensorRT.
• Serving-level - dynamic batching of requests, keeping models warm in memory, using GPU only when it pays off.
• Feature-level - precompute heavy features, use a low-latency online store such as Redis.
• System-level - place the service near the caller, autoscale before saturation, cache repeat predictions.

Always check accuracy after each change. Quantization, for example, usually costs a small drop that has to be acceptable. Track p95 and p99 latency, not only the average, since tail latency is what users notice.

Work from the most likely cause down. The gap is usually in the data, not the algorithm.

1. Check training-serving skew - log served features and compare them with training features for the same entities.
2. Check data drift - is live input different from the training distribution?
3. Check label leakage - did offline evaluation use information that is unavailable at prediction time?
4. Check pipeline bugs - wrong feature order, missing values, unit mismatches, stale features, wrong model version loaded.
5. Check the evaluation split - random splits can overstate quality when real data is time-ordered; use a time-based split.

Replaying a sample of production requests through the offline pipeline is a quick way to find which step produces different numbers.

An automated pipeline runs the same sequence every time, and each step can halt the run if a quality gate fails.

flowchart TD
  T["Trigger: schedule, new data or drift"] --> I["Ingest data"]
  I --> V{Data valid?}
  V -- No --> X["Stop and alert"]
  V -- Yes --> F["Feature engineering"]
  F --> TR["Train and tune"]
  TR --> E{Beats baseline?}
  E -- No --> X
  E -- Yes --> REG["Register model version"]
  REG --> D["Deploy to staging or canary"]

1. Trigger fires on a schedule, new data landing, or a drift alert.
2. Ingest and validate data. Bad schema or values stop the run.
3. Build features using the shared definitions.
4. Train and tune, logging each run to the experiment tracker.
5. Evaluate against a baseline and thresholds, including slice metrics.
6. Register the model with lineage, then deploy through a gated rollout.

Metadata such as data hash, code commit and metrics is stored at every step, which makes any run auditable and repeatable.

Pin every input that can change the outcome, and record it automatically instead of relying on memory.

import random, numpy as np, torch
random.seed(42); np.random.seed(42); torch.manual_seed(42)
torch.use_deterministic_algorithms(True)

Even then, some GPU kernels differ slightly between hardware generations. In that case, define a tolerance, for example metric within 0.2%, and treat results inside it as reproduced.

If nothing guards the pipeline, one of two bad things happens. It either crashes loudly, which is the good outcome, or it keeps running and feeds wrong data to the model, which is the dangerous one. A renamed column filled with nulls, or a field changing from dollars to cents, can quietly corrupt predictions.

A well-built pipeline handles it with several layers:

• Schema validation at ingestion stops the run when a column is missing, renamed or retyped.
• Data contracts between producer and consumer teams make changes a reviewed, versioned event.
• Distribution checks catch silent unit changes that keep the schema valid.
• Alerting and fallback keep the last good model serving while the team fixes the mapping.

The cheapest fix is organizational: make data producers notify consumers before changing a schema.

Real ML workflows have many dependent steps, run on different machines, and fail in the middle. An orchestrator handles that coordination so you do not write it by hand with cron and shell scripts.

• Dependencies - run step B only after step A succeeds (a DAG).
• Scheduling and triggers - time-based, sensor-based or event-based runs.
• Retries and failure handling - rerun a failed step, not the whole job.
• Caching and lineage - skip unchanged steps, record what ran with which inputs.
• Scaling - run steps in containers on Kubernetes or managed compute.

Airflow is a general workflow tool with a strong scheduler. Kubeflow Pipelines is Kubernetes-native and built around containerized ML components with artifact tracking.

They solve different problems and are often used together. MLflow tracks and manages experiments and models. Kubeflow orchestrates and runs ML workloads on Kubernetes.

A common setup is Kubeflow Pipelines to run the workflow, with each step logging to an MLflow tracking server and registering the final model there.

Google describes three levels of automation, from manual to fully automated CI/CD and continuous training.

Level 0 suits teams with a few models that rarely change. Level 1 fits models that need frequent retraining on fresh data. Level 2 pays off when many teams are experimenting and shipping pipeline changes often.

Moving up the ladder is incremental. Most teams start by automating the training pipeline and adding monitoring, not by building everything at once.

Governance means being able to answer: which model made this decision, who approved it, what data trained it, and was it fit for use. It is built from records created automatically along the pipeline.

• Lineage - link each model version to its code commit, data snapshot, features and parameters.
• Approvals - registry stage changes need a named reviewer and are logged.
• Model cards - document intended use, limits, metrics and slice performance.
• Access control - role-based permissions on data, models and endpoints.
• Prediction logging - store inputs, outputs and model version for later review, respecting privacy rules.
• Risk review - fairness, explainability and regulatory checks before production.

The key is automation. Audit records collected by the pipeline are far more trustworthy than documents written after the fact.

Neither wins everywhere. The better choice depends on how fast your data changes and how reliable your monitoring is.

Most mature teams combine them: a baseline schedule as a safety net plus triggers for early reaction. Triggers should start the same gated pipeline, so a retrain still has to beat the champion before it ships.

Package the model server in a container, deploy it as a Deployment behind a Service, and let Kubernetes add or remove replicas as load changes.

• Horizontal Pod Autoscaler (HPA) scales replicas on CPU, memory or custom metrics such as requests per second.
• Resource requests and limits keep pods from starving each other; GPU pods need node selectors or taints.
• Readiness probes keep traffic away until the model has loaded.
• KServe or Seldon add scale-to-zero, canary splits and model-aware routing.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: churn-model
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: churn-model}
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource: {name: cpu, target: {type: Utilization, averageUtilization: 65}}

Large models load slowly, so scale on a lead indicator and keep a minimum replica count to avoid cold-start spikes.

Most savings come from not wasting compute. Look at what runs, how long, and on what hardware.

• Spot or preemptible instances with checkpointing can cut compute cost sharply; resume from the last checkpoint if evicted.
• Early stopping and pruning of weak trials stop hopeless runs and hyperparameter search branches.
• Smarter search - Bayesian optimization or Hyperband instead of a big grid.
• Pipeline caching - skip steps whose inputs did not change.
• Right-sizing - match instance type to the job; avoid idle GPUs and auto-shutdown notebooks.
• Incremental training on new data where it is valid, instead of full retrains.
• Mixed precision to speed up training on supported GPUs.

Tag every job with team and project, so cost reports show who spends what. You cannot cut what you cannot see.

Treat the model like software with extra layers. Fast tests run on every commit, and heavier ones run before a release.

def test_no_regression():
    new_auc = evaluate(candidate, holdout)
    prod_auc = evaluate(champion, holdout)
    assert new_auc >= prod_auc - 0.01

Keep a small fixed 'golden' dataset in the repo for fast checks, and use a larger holdout in the full pipeline.

In a champion-challenger setup, the current production model is the champion and a new candidate is the challenger. The challenger must prove itself against the champion before it takes over.

The challenger can be scored offline on the same recent data, run in shadow mode on live traffic, or served to a small slice. Both models are compared on the same metrics, including business KPIs and latency.

1. Train the challenger from the pipeline.
2. Evaluate it side by side with the champion on a fresh holdout and live shadow traffic.
3. If it wins by the agreed margin, promote it; the old champion stays archived for rollback.
4. If not, keep the champion and learn from the failure.

It turns model updates into a measurable contest, rather than a hopeful release, and works well with automated retraining.

Aggregate accuracy can look fine while one group is treated much worse. Fairness monitoring splits metrics by group, then watches the gaps over time.

• Pick sensitive attributes and groups to track (where legally and ethically allowed), or proxies when the attribute itself is not collected.
• Compute group metrics - selection rate, true positive rate, false positive rate per group.
• Compare gaps using measures such as demographic parity difference, equal opportunity difference or the four-fifths rule.
• Alert on thresholds and review flagged models before retraining or promotion.

Labels often arrive late, so start with selection-rate and prediction-distribution checks, then add outcome-based metrics. Fairlearn and AIF360 are open-source toolkits for this. Metrics can conflict with each other, so the chosen definition should be agreed with legal and domain owners up front.

Security applies at each stage: data, training, artifacts and serving. The basics are the same as any production system, plus a few ML-specific risks.

• Access control - least-privilege roles on data stores, registries and pipelines; no shared admin keys.
• Secrets management - keep credentials in a vault, never in notebooks or images.
• Data protection - encrypt at rest and in transit; mask or tokenize PII; control training-data retention.
• Supply chain - scan container images, pin dependencies, and avoid loading untrusted pickle files, which can execute code.
• Endpoint hardening - authentication, TLS, rate limiting and input validation to blunt abuse and model extraction.
• Integrity and audit - sign model artifacts and log access and changes.

ML-specific threats include data poisoning, adversarial inputs and model theft through heavy querying. Rate limits, input validation and drift monitoring help detect them.

A real-time prediction passes through several components within a tight latency budget, often under 100 ms.

sequenceDiagram
  participant C as Client app
  participant G as API gateway
  participant S as Model service
  participant F as Online feature store
  participant L as Prediction log
  C->>G: Request with entity ID
  G->>S: Authenticated request
  S->>F: Fetch features for entity ID
  F-->>S: Feature vector
  S->>S: Preprocess and run model
  S-->>G: Prediction and model version
  G-->>C: Response
  S--)L: Log inputs, output, version

1. The gateway authenticates and routes the request.
2. The model service looks up precomputed features in the online store, and merges any fields sent in the request.
3. It applies the same preprocessing used in training and runs the model.
4. The response is returned, and the features, prediction and model version are logged asynchronously, so monitoring and later labeling can use them without adding latency.

The logged records are what let you compute drift, join true labels later and debug a specific bad prediction.