Cloud / Vertex AI Interview questions

Vertex AI is Google Cloud's managed platform for building, training, deploying and monitoring machine learning models and generative AI applications. Data prep, training, model registry, serving and MLOps tooling all sit behind one API (aiplatform.googleapis.com) and one console.

It covers classic ML (AutoML, or custom training with TensorFlow, PyTorch, scikit-learn and XGBoost) as well as generative AI through Gemini, Model Garden and agent tooling.

Naming note: since Google Cloud Next in April 2026, Vertex AI has been folded into the Gemini Enterprise Agent Platform. Existing workloads, SDKs and the API endpoint keep working, so interviewers still use both names.

Vertex AI is easiest to remember by lifecycle stage. Each stage has its own managed service, and you can adopt them one at a time.

Monitoring and governance features such as Model Monitoring and Model Armor cut across all of these.

Model Garden is a catalog of more than 200 foundation models that you can explore, test and use from one place. It includes Google's Gemini models, open-weight models such as Gemma, and partner models such as Claude.

Each model has a model card describing what it does. Depending on the model, you can call it as a managed API, deploy it to your own endpoint, tune it, or open a sample notebook.

Think of it as the starting point when you ask, "which model should I use for this job?"

Vertex AI Workbench provides managed JupyterLab notebook instances for data exploration and model development. Google handles the VM, the pre-installed ML frameworks and the JupyterLab setup.

Instances connect directly to BigQuery and Cloud Storage, can attach GPUs, and can be used to schedule notebook runs. You pay for the underlying VM while it runs, so stop idle instances. You choose the machine type and GPU at creation, can change them later, and control access through IAM and a custom service account.

If you want a lighter, Colab-style experience, Colab Enterprise offers notebooks in the same ecosystem without managing a VM yourself.

Vertex AI offers three practical ways to get a model, ordered from least to most control:

1. AutoML: you bring labeled data and Vertex AI picks the architecture and tunes it.
2. Foundation model tuning: you adapt a model such as Gemini with your own examples (supervised fine-tuning, and reinforcement-learning tuning on some models).
3. Custom training: you bring your own code and run it in a prebuilt or custom container on hardware you choose.

Pick the lowest-effort option that meets your accuracy and control requirements.

AutoML trains models from a labeled dataset without you writing model code. It handles feature engineering, architecture search and hyperparameter tuning, then produces a model you can deploy to an endpoint.

It is a good fit for small teams, quick baselines and standard problems such as tabular classification or regression, or image tasks.

The trade-offs are less control over the architecture and training loop, and billing by node hours, which can add up on large datasets. If you need a custom loss or a specific network, move to custom training.

Custom training lets you run your own training code (TensorFlow, PyTorch, scikit-learn, XGBoost or anything else) as a managed job. You choose the machine type, accelerators and number of replicas, and Vertex AI provisions them, runs the job and shuts them down.

You can submit a Python script with a prebuilt container, or package everything in your own custom container.

from google.cloud import aiplatform

aiplatform.init(project="my-project", location="us-central1",
                staging_bucket="gs://my-bucket")

job = aiplatform.CustomTrainingJob(
    display_name="churn-train",
    script_path="train.py",
    container_uri="us-docker.pkg.dev/vertex-ai/training/sklearn-cpu.1-0:latest",  # use a current image
    requirements=["pandas"],
)
job.run(replica_count=1, machine_type="n1-standard-4")

Write the trained model to the directory given by the AIP_MODEL_DIR environment variable and Vertex AI collects it in Cloud Storage. Run the job under a custom service account so it can read only the data it needs, and consider Spot VMs with checkpointing for long, cost-sensitive runs.

The Model Registry is a central place to store, version and manage your models, whether they came from AutoML, custom training or an import. Each model can hold multiple versions, and aliases such as default or champion point to the version you want to use.

From the registry you can deploy a version to an endpoint, run batch prediction, view evaluation metrics and track lineage back to the training job.

It replaces the habit of passing model files around in buckets with an auditable record of what was trained and what is live.

An endpoint is the HTTPS resource that serves online predictions. You deploy one or more models to it with a machine type, a minimum and maximum replica count, and a traffic percentage per model.

Clients send prediction requests to the endpoint and the platform routes them to a replica running your model container. Each deployed model gets its own replicas, so two versions on one endpoint scale independently.

endpoint = model.deploy(machine_type="n1-standard-2",
                        min_replica_count=1,
                        max_replica_count=3)
print(endpoint.predict(instances=[[34, 2, 0, 1]]).predictions)

Endpoints can be public or private (through Private Service Connect or VPC peering) when traffic must stay off the internet.

There are two main modes.

Online prediction is synchronous and low latency. You deploy a model to an endpoint and call it per request, for example to score a transaction as it happens.

Batch prediction is asynchronous. You point a job at a large input set in Cloud Storage or BigQuery, and it writes results back to Cloud Storage or BigQuery. No endpoint is needed, and compute exists only while the job runs.

Generative AI adds a third flavor: batch requests to Gemini, which run asynchronously at a discount compared with online calls.

Vertex AI Pipelines is a serverless service for running ML workflows defined with the Kubeflow Pipelines (KFP) SDK or TFX. A pipeline is a DAG of steps, such as prepare data, train, evaluate and deploy, and each step is a containerized component.

Inputs and outputs are tracked as artifacts in ML Metadata, so you get lineage for free. Results can be cached when a step's code and inputs haven't changed, and runs can be scheduled or triggered from CI/CD.

Its main value is repeatability: the same pipeline definition produces the same process every time.

Feature Store is a managed place to define, store and serve ML features so that training and serving use the same values. It tackles training-serving skew, where a feature is computed one way in the notebook and another way in production.

The current version is built on BigQuery: your feature data stays in BigQuery tables for offline training, and a feature view syncs the latest values to an online store for low-latency lookups. The older standalone version is considered legacy.

Teams also use it to share features across projects instead of rebuilding them.

Vertex AI Experiments tracks and compares training runs. For each run you log parameters (learning rate, batch size), metrics (accuracy, loss) and artifacts, then compare runs side by side in the console or in a notebook.

aiplatform.init(project="my-project", location="us-central1",
                experiment="churn-model")
aiplatform.start_run("run-01")
aiplatform.log_params({"lr": 0.01, "batch": 64})
aiplatform.log_metrics({"val_accuracy": 0.91})
aiplatform.end_run()

It integrates with Vertex AI TensorBoard for training curves, and the logged runs feed the same metadata store that pipelines use.

A practical convention is one experiment per problem (for example churn-model) and one run per training attempt. Pipelines can log into the same experiment, so manual notebook runs and automated runs end up in one comparison table.

Vector Search is a managed service for approximate nearest neighbor (ANN) search over embeddings. You store vectors in an index, deploy the index to an endpoint, and query it to find the most similar items in milliseconds, even across billions of vectors.

It was previously called Matching Engine and is built on Google's ScaNN technology. Typical uses are semantic search, recommendations and the retrieval step of RAG.

Indexes can be updated in batch or with streaming updates, and queries can be filtered by attributes such as category or region.

Vertex AI Studio is the console workspace for experimenting with generative models without writing code. In the new platform it appears as Agent Studio.

You can write and test prompts, adjust parameters such as temperature, compare models side by side, save and share prompts, and start a tuning job. When a prompt works, you can export it as code for the SDK.

It is meant for prototyping and prompt design. Production traffic should go through the API or an agent runtime.

Use the Google Gen AI SDK (google-genai) and set vertexai=True. Authentication goes through Application Default Credentials and IAM, so there is no API key to manage.

from google import genai

client = genai.Client(vertexai=True, project="my-project", location="global")
resp = client.models.generate_content(
    model="MODEL_ID",  # pick a current Gemini model from Model Garden
    contents="Explain feature drift in two lines.",
)
print(resp.text)

You can also call the REST API directly or try prompts first in Studio. Choose the region carefully if data residency matters: the global location works for many models, while a regional location keeps processing in that region.

Through the config argument you can add system instructions, temperature, a JSON response schema and safety settings. For multi-turn conversations, client.chats.create() keeps the history for you.

RAG Engine is a managed service that handles the plumbing of retrieval-augmented generation. It imports your documents, splits them into chunks, creates embeddings, stores them in a vector database and retrieves the best matches at query time.

The retrieved text is added to the prompt, so Gemini answers from your data rather than only from what it learned in training.

For storage you can use the built-in RagManagedDb or connect Vertex AI Vector Search and other vector stores. A serverless mode is in public preview, and schema-based metadata search lets you filter by document attributes.

ADK is an open-source, code-first framework for building AI agents. It is model-agnostic, so although it works well with Gemini you can plug in other models.

You define an agent with instructions and tools (Python functions, MCP servers, other agents), and ADK manages the loop of reasoning, tool calling and state. It supports multi-agent hierarchies and has built-in tooling for evaluation and local testing.

An ADK agent can run locally or be deployed to Agent Runtime for managed hosting. The adk web command starts a local dev UI where you can chat with the agent and trace each tool call.

Agent Runtime is the fully managed environment for hosting and scaling agents. It was previously named Agent Engine (and before that, Reasoning Engine).

You deploy agent code built with ADK or other frameworks, and the service handles scaling, IAM-based access and integration with Sessions and Memory Bank. The revamped runtime advertises sub-second cold starts and support for agents that run for long periods. Agents written with other frameworks, such as LangGraph, can be packaged for the runtime too.

The point is that you ship agent code and skip building the serving infrastructure yourself.

Access is controlled with IAM. Grant predefined roles such as Vertex AI User for day-to-day use, Vertex AI Viewer for read-only access and Vertex AI Administrator only for those who manage resources.

Training jobs and endpoints run as a service account. Use a dedicated one with only the permissions the job needs (for example, read one bucket) instead of the default or an owner-level account.

For stricter environments, add VPC Service Controls, private endpoints and customer-managed encryption keys on top of IAM.

Both expose Gemini through the same google-genai SDK. The difference is the surrounding platform: the Developer API is built for fast prototyping with an API key, while Vertex AI is built for production on Google Cloud with enterprise controls.

A common path is to prototype on the Developer API, then switch to vertexai=True with almost no code change when the app needs compliance and scale.

Choose AutoML when you have clean labeled data, a standard problem type and little time or ML expertise, and a solid baseline is more valuable than the last point of accuracy.

Choose custom training when you need a specific architecture, custom loss, special preprocessing, distributed training, or an exact framework version.

Many teams run AutoML first as a benchmark, then only invest in custom code if it clearly beats it.

Neither is better in general. The deciding question is how quickly a user or system needs the answer.

If predictions can be a few hours old, batch is usually much cheaper. A hybrid is common: score everyone nightly in batch, write results to BigQuery or Bigtable, and use online prediction only for the cases that need fresh data.

Google announced the change at Cloud Next in April 2026 to reflect a shift from a model-first platform to an agent-first one. The aim is to cover the whole agent lifecycle: build, scale, govern and optimize.

Vertex AI and Agent Builder were consolidated into one platform. New pieces include Agent Runtime, Memory Bank, Agent Registry for approved tools and agents, Agent Identity and Agent Gateway for governance, and agent observability.

For existing users the change is mostly branding and console navigation. Workloads continue to run, and the API endpoint is still aiplatform.googleapis.com. Check individual deprecation notices, since some modules (for example Vertex AI Extensions and Vertex Explainable AI) are being retired on their own schedules.

In a custom job you describe the cluster as a list of worker pools. Each pool has its own machine type, accelerator and replica count, and Vertex AI starts the replicas and injects the cluster configuration (such as TF_CONFIG or CLUSTER_SPEC) so your framework knows who is who.

Your code chooses the strategy: data parallelism (MirroredStrategy or MultiWorkerMirroredStrategy in TensorFlow, DistributedDataParallel in PyTorch), or model parallelism for very large networks.

For multi-node GPU training, Reduction Server can speed up gradient all-reduce by offloading it to dedicated CPU nodes. Use Spot VMs with regular checkpoints to cut cost, since preempted workers can resume from the last checkpoint.

The job succeeds when the primary replica exits cleanly, so make sure only the chief writes the final model.

Prebuilt containers cover TensorFlow, PyTorch, scikit-learn and XGBoost, for both training and serving. Start there: you supply a script or a saved model and skip building and maintaining an image.

Switch to a custom container when you need something they cannot provide:

• a framework or version that has no prebuilt image
• system libraries or native dependencies (for example OpenCV or a C++ extension)
• a custom serving stack, such as Triton or vLLM, or special pre and post-processing logic
• full control over the entrypoint and startup behavior

For serving, a custom container must run an HTTP server that answers the health and predict routes. If you only need light pre or post-processing, a custom prediction routine may be enough without writing the whole server.

A HyperparameterTuningJob runs your training code many times with different hyperparameter values and searches for the best result. By default it uses Bayesian optimization, powered by Vertex AI Vizier, so later trials learn from earlier ones. Grid and random search are also available.

from google.cloud.aiplatform import hyperparameter_tuning as hpt

tuning = aiplatform.HyperparameterTuningJob(
    display_name="lr-search",
    custom_job=custom_job,
    metric_spec={"val_accuracy": "maximize"},
    parameter_spec={
        "lr": hpt.DoubleParameterSpec(min=1e-4, max=1e-1, scale="log"),
        "batch": hpt.DiscreteParameterSpec(values=[32, 64, 128], scale="linear"),
    },
    max_trial_count=20,
    parallel_trial_count=4,
)
tuning.run()

Your training code must report the metric after each run, using the hypertune helper library. Note the trade-off in parallel_trial_count: more parallelism finishes sooner but gives Bayesian search less history to learn from.

Define the search space on a sensible scale: learning rates on a log scale, batch sizes as a few discrete values. Set a realistic max_trial_count for the size of the space, and use the best trial's parameters from the console to retrain the final model.

Every deployed model has a minimum and maximum replica count. The platform adds replicas when average utilization stays above a target and removes them when load drops, never going outside those bounds.

The default signal is CPU utilization (target around 60%). For GPU deployments you can scale on accelerator duty cycle instead, using a setting such as autoscaling_target_accelerator_duty_cycle.

Two practical points come up in interviews. First, new replicas take time to pull the image and load the model, so sudden spikes can see higher latency; raise the minimum replica count if that matters. Second, a standard endpoint keeps at least one replica running, so you pay for it even when idle.

Deploy the new version to the same endpoint as the current one and give it a small slice of traffic. The old model keeps serving the rest, so users never see an outage.

model_v2.deploy(endpoint=endpoint,
                machine_type="n1-standard-2",
                min_replica_count=1,
                traffic_percentage=10)   # v1 keeps the other 90%

1. Watch latency, error rate and business metrics for the 10% slice.
2. Increase the split step by step (10, 50, 100).
3. Undeploy the old version once the new one is stable.

Rollback is just moving the traffic percentage back to the old model. Keep both deployed during the test so that switching is instant.

Use the same machine type for both versions during the test so latency comparisons are fair, and agree in advance what "good" means (for example, error rate and p95 latency thresholds) so the promote-or-rollback call is not a matter of opinion.

You write components and a pipeline function with the KFP SDK, compile it to a pipeline spec, and submit it as a PipelineJob. Vertex AI then walks the DAG, launching each ready step as a containerized job.

from kfp import dsl, compiler
from google.cloud import aiplatform

@dsl.component
def clean(n: int) -> int:
    return n - 1

@dsl.component
def train(n: int) -> str:
    return f"trained on {n} rows"

@dsl.pipeline(name="demo", pipeline_root="gs://my-bucket/root")
def demo(n: int = 100):
    c = clean(n=n)
    train(n=c.output)

compiler.Compiler().compile(demo, "demo.json")
aiplatform.PipelineJob(display_name="demo-run", template_path="demo.json").run()

flowchart LR
  A["Compile pipeline to spec"] --> B["Submit PipelineJob"]
  B --> C["Resolve DAG order"]
  C --> D{Cached result for this step?}
  D -- Yes --> F["Reuse outputs"]
  D -- No --> E["Run component container"]
  E --> G["Write artifacts to pipeline root in GCS"]
  F --> H["Record lineage in ML Metadata"]
  G --> H
  H --> I["Next step or run complete"]

Before running a step, the service checks whether the same component with the same inputs already ran. If so, it reuses the earlier outputs instead of recomputing. You can turn this off with enable_caching=False.

Artifacts land in the pipeline root bucket and their lineage is recorded in ML Metadata, which is how you later trace a deployed model back to its data and code.

Conditional logic and parallel loops in the KFP SDK let a pipeline branch, for example deploying only when an evaluation metric passes a threshold. Each step runs as its own Vertex AI job, so it can request its own machine type or GPU.

Model Monitoring compares statistics of the data your model sees against a reference, and raises an alert when the distance crosses a threshold you set per feature.

For numerical features it measures distribution distance with Jensen-Shannon divergence, and for categorical features it uses L-infinity distance. Alerts can go to email or Cloud Monitoring.

It needs request data to be logged, usually with a sampling rate to control cost. Drift tells you something changed, not that accuracy has fallen, so pair it with ground-truth evaluation when labels arrive.

They solve different problems and often work together.

Choose Pipelines when the workflow is mostly ML: prepare, train, evaluate, register. Choose Composer when it spans many systems, such as loading from several sources, running Dataflow jobs and sending notifications.

A common pattern is Composer scheduling the business workflow and triggering a Vertex AI pipeline for the ML part.

The request is authenticated through IAM, then the endpoint applies its traffic split to choose a deployed model. A load balancer sends it to a healthy replica, where your model container's HTTP server runs inference and returns the result.

sequenceDiagram
  participant C as Client
  participant E as Endpoint
  participant R as Replica (model container)
  C->>E: POST :predict with instances
  E->>E: Check IAM, apply traffic split
  E->>R: Route to a healthy replica
  R->>R: Preprocess, run model, postprocess
  R-->>E: predictions
  E-->>C: JSON response

If request-response logging is enabled, a sample is written to BigQuery for monitoring. Latency has two parts you can inspect separately in Cloud Monitoring: time inside the model container and overhead added by the platform and network.

Custom containers read the AIP_PREDICT_ROUTE and AIP_HEALTH_ROUTE environment variables to learn which paths to serve. The endpoint only sends traffic to replicas whose health route succeeds, which is why a slow model load shows up as a not-ready period right after deployment.

Vector Search is built on ScaNN, Google's approximate nearest neighbor algorithm. The idea is to avoid comparing a query against every vector by narrowing the search in stages.

1. Partition: vectors are grouped into clusters (a tree structure). A query first picks the closest few clusters.
2. Compress and score: inside those clusters, vectors are stored in a compressed form (asymmetric hashing) so scoring is very fast.
3. Re-rank: the best candidates are re-scored using the full-precision vectors, and the top results are returned.

This trade accuracy for speed, and you tune it. Settings such as leaf_node_embedding_count and the percentage of leaf nodes searched control recall versus latency. The distance measure (dot product, cosine or squared L2) must match how the embeddings were trained.

Operationally, you create an index, deploy it to an index endpoint, and send queries. Updates arrive as batch rebuilds or streaming upserts, and token or numeric restricts filter results. For queries that need exact words as well as meaning, hybrid search combines dense and sparse signals.

Try the cheaper options first: better instructions, few-shot examples, then RAG or grounding. Fine-tune when those still fall short.

Tuning teaches behavior, not new facts, and it needs a few hundred clean, representative examples. If your data is thin or changes weekly, prompting plus retrieval is usually the safer route.

You give the model examples of the input and the exact output you want, and Vertex AI adjusts a small set of extra weights so the model reproduces that behavior. The base model weights stay frozen, which makes tuning faster and cheaper than full training (an adapter-based approach, similar in spirit to LoRA).

1. Prepare a JSONL file where each line is one conversation example, with a user turn and the ideal model turn.
2. Upload the training file (and a separate validation file) to Cloud Storage.
3. Start a tuning job with the source model, datasets and settings such as epochs and learning-rate multiplier.
4. Review training and validation loss in the console to spot overfitting.
5. Use the tuned model through its own endpoint, and compare it against the base model on a held-out set.

{"contents": [
  {"role": "user",  "parts": [{"text": "Classify: 'Refund still not received'"}]},
  {"role": "model", "parts": [{"text": "billing_issue"}]}
]}

Quality beats quantity. A few hundred clean, varied examples usually outperform thousands of noisy ones. Watch the validation curve: if it rises while training loss keeps falling, reduce epochs.

Some newer models also support reinforcement-learning tuning, where a reward signal replaces fixed target answers.

Use it when many requests start with the same large block of content, such as a long document, a codebase, a video or a big system prompt, and only the final question changes.

There are two flavors. Implicit caching is automatic: repeated prefixes may get a discount without any code. Explicit caching is something you create, with a time-to-live, so you get a predictable discount on the cached tokens.

from google.genai import types

cache = client.caches.create(
    model="MODEL_ID",
    config=types.CreateCachedContentConfig(
        contents=[big_document],
        system_instruction="Answer only from the document.",
        ttl="3600s",
    ),
)
resp = client.models.generate_content(
    model="MODEL_ID",
    contents="Summarize section 4.",
    config=types.GenerateContentConfig(cached_content=cache.name),
)

It is not worth it for one-off prompts or small contexts, because cached content adds a storage charge for as long as the cache lives and there is a minimum size.

Grounding connects the model to a source of truth at request time, so answers are built from retrieved evidence instead of only from training memory. The response includes grounding metadata that shows which sources supported which parts.

The main grounding sources are:

• Google Search for fresh public information
• Vertex AI Search or RAG Engine for your own documents
• Google Maps for place and location data

Because the citations come back with the answer, users and reviewers can verify claims. Grounding lowers the hallucination rate but doesn't eliminate it, so keep instructions like "answer only from the provided sources" and test with questions that have no answer in the data.

RAG has two phases: an offline ingestion phase that prepares your documents, and an online query phase that retrieves and answers.

sequenceDiagram
  participant U as User
  participant A as App
  participant R as RAG Engine
  participant V as Vector store
  participant G as Gemini
  Note over R,V: Ingestion (offline): import files, chunk, embed, store
  U->>A: Question
  A->>R: Retrieve for question
  R->>V: Embed query, similarity search
  V-->>R: Top matching chunks
  R-->>A: Chunks with sources
  A->>G: Prompt + chunks
  G-->>A: Grounded answer
  A-->>U: Answer with citations

1. Ingest: import from Cloud Storage, Drive or other sources into a corpus; files are split into chunks and each chunk is embedded.
2. Store: vectors go into the corpus's vector database, with optional metadata for filtering.
3. Retrieve: the user's query is embedded and compared with stored vectors, returning the top-k chunks, optionally re-ranked.
4. Generate: the chunks are added to the prompt and Gemini answers from them, returning sources.

Most quality problems come from chunk size, retrieval filters and top-k. Tune those before blaming the model.

Retrieval can be filtered by metadata such as department or document date, and the chunk sources become your citations. If nothing relevant is retrieved, instruct the model to say it does not know rather than guess.

The model never runs your code. You describe your functions, the model decides when one is needed and returns the function name with arguments, and your application executes it.

1. Declare functions with a name, description and parameter schema.
2. Send the user's prompt along with those declarations.
3. If the model chooses a function, it returns a call with arguments.
4. Your code runs the function and sends the result back as a function response.
5. The model uses the result to write the final answer.

from google.genai import types

get_weather = types.FunctionDeclaration(
    name="get_weather",
    description="Get current weather for a city",
    parameters={"type": "OBJECT",
                "properties": {"city": {"type": "STRING"}},
                "required": ["city"]},
)
resp = client.models.generate_content(
    model="MODEL_ID",
    contents="Weather in Pune?",
    config=types.GenerateContentConfig(
        tools=[types.Tool(function_declarations=[get_weather])]),
)
call = resp.function_calls[0]   # name and args; you execute it

A good description matters more than most people expect, because it is what the model uses to choose the tool. Validate arguments before running anything with side effects.

Use the Gen AI evaluation service. You build an evaluation dataset of prompts (and reference answers where you have them), generate responses from one or more models or prompts, and score them.

• Computation-based metrics such as exact match, BLEU and ROUGE, which need reference answers.
• Model-based metrics where a judge model scores quality, either pointwise (rate one response) or pairwise (pick the better of two).
• Custom metrics for criteria specific to your product, like tone or policy compliance.

It can also evaluate partner models such as Claude, so you can compare across providers on the same set. Judge models have biases, so spot-check their scores with human review, especially for high-stakes use.

Pay-as-you-go charges per token with shared capacity. It is flexible and cheap at low volume, but capacity is not reserved, so heavy shared demand can lead to throttling.

Provisioned Throughput reserves dedicated capacity for a specific model and region for a committed term. You pay a fixed fee, measured in generative AI scale units, and get predictable throughput and latency.

Requests above your reserved capacity can spill over to pay-as-you-go by default, or you can configure them to fail. Commit only after you have measured your steady-state load.

Cost control differs a little for generative AI and for classic ML, but the habit is the same: match the resource to the actual need.

For Gemini:

• Use a smaller model (Flash or Flash-Lite class) for simple tasks and reserve Pro-class models for hard ones.
• Send non-urgent work through batch requests, which are discounted versus online calls.
• Use context caching for repeated long prefixes, and keep prompts and max_output_tokens tight.

For ML:

• Train on Spot VMs with checkpointing, and right-size machine types.
• Set sensible minimum replicas, undeploy unused endpoints, and stop idle Workbench instances.
• Prefer batch prediction when latency doesn't matter, and add billing labels to every job for attribution.

Security is layered, and each layer covers a different risk.

VPC Service Controls are often the piece people forget: IAM says who may call the API, while the perimeter says from where and to what, so a stolen credential cannot copy a dataset to an outside project.

CMEK has limits. For example, Memory Bank and Sessions instances that use the global endpoint cannot use CMEK, so pick a regional endpoint if you need your own keys.

Add audit logging so access to models and data is traceable, and use Model Armor for prompt-level threats.

Model Armor screens the prompts going into a model and the responses coming out, based on a template of policies you configure. It does not rely on the model to police itself.

It can detect prompt injection and jailbreak attempts, sensitive data such as personal information (using Sensitive Data Protection), malicious URLs and harmful content, each with its own confidence threshold.

1. The app sends the user prompt to Model Armor before the model call.
2. Model Armor returns a verdict: allow, or block with the reason.
3. The app calls Gemini only if the prompt is clean.
4. The response is screened the same way before it reaches the user.

It also works with Agent Gateway so agent traffic is covered. Treat it as one layer: still limit tool permissions, since a screened prompt can slip through.

They split an agent's needs into three jobs: where it runs, what it remembers within one conversation, and what it remembers about a user over time.

• Agent Runtime hosts and scales the agent code.
• Sessions store the events of a single conversation (messages, tool calls, state). This is short-term context.
• Memory Bank stores long-term facts about a user across sessions, such as preferences or past decisions.

sequenceDiagram
  participant U as User
  participant A as Agent on Runtime
  participant S as Sessions
  participant M as Memory Bank
  U->>A: New message
  A->>S: Load session events
  A->>M: Retrieve relevant memories for user
  M-->>A: Matching memories
  A->>A: Reason with Gemini and tools
  A->>S: Append new events
  A-->>U: Response
  S-->>M: Trigger memory generation (event count or idle time)

Memory Bank doesn't save everything. It uses a model to extract the facts worth keeping, and generation can be triggered by criteria such as number of events or idle time. Retrieval is by relevance, and it keeps a version history of memories.

Keep memories scoped per user, so one person's facts never surface in another person's session.

Model Garden offers two ways to run non-Gemini models, and they differ in who runs the infrastructure.

Use the managed API for speed and low operational load. Self-deploy when you need an open model fine-tuned on your own data, specific hardware, strict network isolation, or steady high volume where dedicated machines are cheaper.

Work from the outside in: first confirm the job started, then check resources, then the code.

1. Open the job in the console and read the error state, then check Cloud Logging for the job's logs.
2. If it never started, check the image URI, the entrypoint and whether the service account can pull from Artifact Registry and read Cloud Storage.
3. Check quotas for the region (CPU, GPU, TPU). A job waiting or failing at provisioning is often a quota issue.
4. If it dies mid-run with exit code 137, the container ran out of memory. Use a bigger machine or a smaller batch size.
5. Reproduce locally with docker run on a small sample of data, and use the job's interactive shell for a live look.

Typical culprits are wrong bucket paths, missing dependencies in the container, version mismatches between training and the prebuilt image, and writing outputs to the wrong location (use AIP_MODEL_DIR).

First split the latency into model latency (time inside your container) and overhead latency (platform and network), which Cloud Monitoring reports separately. The answer decides where to look.

• High model latency: profile the model and its preprocessing, move to a GPU, quantize or use an optimized runtime, and batch requests inside the server.
• High overhead: check payload size, client region versus endpoint region, and consider a dedicated or private endpoint.
• Spikes under load: look at replica CPU or GPU utilization, and raise max_replica_count or lower the autoscaling target.
• Slow after idle periods: cold starts from replica scale-up, so raise the minimum replica count.

Also check whether heavy preprocessing runs per request that could be precomputed, such as tokenization or feature joins.

The goal is a loop where new data triggers training, bad models are blocked automatically and good ones are released gradually.

flowchart LR
  A["Data in BigQuery / GCS"] --> B["Feature Store"]
  B --> C["Pipeline: validate, train, evaluate"]
  C --> D{Beats current model?}
  D -- No --> X["Stop and alert"]
  D -- Yes --> E["Model Registry new version"]
  E --> F["Canary deploy to endpoint"]
  F --> G["Model Monitoring"]
  G -- Drift or quality drop --> C
  G -- Healthy --> H["Promote to 100%"]

Wrap the whole thing in CI/CD (for example Cloud Build): a code change compiles and runs the pipeline in a test project, and a merge to main promotes it. Keep dev, staging and production in separate projects with separate service accounts.

Finally, log metadata so any live prediction can be traced back to the exact data, code and parameters that produced the model.