Java / Azure Batch Interview questions

Azure Batch is a managed Azure service for running large-scale parallel and high-performance computing (HPC) workloads in the cloud. You describe the work as jobs and tasks, and Batch provisions the virtual machines, schedules the tasks onto them, and scales the pool up or down.

You don't build or maintain a scheduler or cluster manager yourself. There is no extra charge for Batch itself, and you pay only for the VMs, storage and networking the workload consumes.

Typical fits are embarrassingly parallel jobs such as rendering, Monte Carlo simulations, media transcoding and data processing.

An Azure Batch solution is built from a small set of resources that sit inside a Batch account.

Around these sit supporting features such as start tasks, application packages, resource files and job schedules.

A Batch account is the Azure resource that identifies your Batch service instance in a specific region. All pools, jobs and tasks belong to one account, and every API call is made against the account's endpoint.

The account also defines the pool allocation mode (Batch service or user subscription) and carries the compute quotas that cap how many cores you can use. It can be linked to an Azure Storage account for application packages.

You authenticate to it with Microsoft Entra ID or with shared keys.

A pool is the set of compute nodes on which a job's tasks run. When you create one you choose the VM size, the OS image, the node count (fixed or autoscaled), and optional settings such as a start task, virtual network and task slots per node.

Many jobs can share a pool, but a job is always attached to exactly one pool. Pools of the same account can use different VM sizes and images, so you can keep, say, a GPU pool beside a CPU pool.

Some properties, such as the VM size, can't be changed after creation. You create a new pool instead.

A compute node is a single Azure VM that belongs to a pool and runs tasks assigned to it. Each node runs the Batch node agent, which talks to the Batch service, downloads resource files, launches task command lines and reports state back.

A node has its own working directories and receives environment variables such as AZ_BATCH_NODE_ID. A node can be dedicated, or a Spot VM that Azure may reclaim.

Node states include idle, running, starting, startTaskFailed and unusable.

A job is a logical collection of tasks that all run on the same pool. It carries shared settings such as priority, constraints, and the optional job manager, job preparation and job release tasks.

Job priority ranges from -1000 to 1000, and higher-priority jobs are scheduled first. By default a job stays active after its tasks finish unless you set onAllTasksComplete to terminateJob.

A task is the unit of work in Batch: a command line (or a container run) that executes on a compute node. Each task has an ID that is unique within its job and can declare resource files to download, output files to upload, environment variables and constraints.

A task finishes with an exit code. Batch records it, along with timing and failure details, so you can decide whether to retry or treat the task as failed.

Tasks in a job normally run independently and in parallel, unless you configure task dependencies.

Batch suits workloads you can split into many independent pieces, or tightly coupled HPC runs that need several nodes at once.

• Rendering of frames for film and visual effects.
• Financial risk modelling and Monte Carlo simulation.
• Media transcoding of large video libraries.
• Genomics and scientific data processing.
• Batch ML scoring of large datasets.
• MPI-based engineering simulations using multi-instance tasks.

It is a poor fit for low-latency request/response services, where the node startup time and queueing model get in the way.

A start task is a command that runs on every node each time it joins the pool, and again whenever the node is rebooted. It prepares the node, for example by installing software, mounting a share or copying reference data.

If waitForSuccess is true, Batch won't schedule tasks on the node until the start task completes successfully. A failing start task leaves the node in the startTaskFailed state.

Keep it idempotent, because it can run more than once on the same node.

Application packages let you upload versioned zip archives of your executables to the Batch account and have them deployed to nodes automatically. The packages are stored in the linked Azure Storage account.

You reference a package and version on the pool (or task), and Batch downloads and unzips it on each node. The install location is exposed through an environment variable such as AZ_BATCH_APP_PACKAGE_<name>#<version>.

Use packages for application binaries, and resource files for task input data.

Resource files are files that Batch downloads onto a node before a task (or start task) runs. Each entry points to a source such as a blob URL, a container prefix or an HTTP URL, and a destination path on the node.

Access to private blobs is granted through a SAS URL or, preferably, a managed identity on the pool. Large downloads are included in the task's start-up time, so use them for per-task inputs rather than big shared data sets.

There are two modes, chosen when you create the Batch account.

Most teams start with Batch service mode and move to user subscription mode only when they need subscription-level control.

A job manager task is a special task that Batch starts first when a job is created. Its job is to create and coordinate the remaining tasks of the job, so the client doesn't have to stay connected.

It gets scheduling priority, can be relaunched if its node fails, and can end the job when the work is done. It is useful when the number of tasks depends on data discovered at run time.

It is optional, and a client can add all tasks directly instead.

You can work with Batch through several interfaces, and they all call the same underlying REST API.

• Azure portal for creating accounts and inspecting pools, jobs and tasks.
• Azure CLI (az batch) for scripting.
• SDKs for .NET, Python, Java and Node.js.
• Batch Explorer, a desktop client for browsing and debugging.
• ARM, Bicep and Terraform for provisioning accounts and pools as code.

You create a pool (or reuse one), create a job that points to it, then add tasks to the job. Batch schedules the tasks as nodes become free.

A minimal Python flow looks like this:

from azure.batch import BatchServiceClient
import azure.batch.models as m

client = BatchServiceClient(credentials, batch_url=BATCH_URL)

client.job.add(m.JobAddParameter(
    id="render-job",
    pool_info=m.PoolInformation(pool_id="render-pool")))

client.task.add_collection("render-job", [
    m.TaskAddParameter(id=f"frame-{i}",
        command_line=f"/bin/bash -c 'render.sh {i}'")
    for i in range(100)])

Afterwards you poll task states or use the SDK's wait helpers.

Spot VMs are spare Azure capacity offered at a steep discount in exchange for the risk that Azure can reclaim (preempt) the VM when it needs the capacity back. In Batch you set a target number of Spot nodes beside the dedicated node target.

They suit interruptible, restartable workloads. When a node is preempted, tasks running on it are requeued and run elsewhere.

Spot VMs replace the older low-priority nodes, which were retired on 30 September 2025.

A task moves through four states.

A task that fails and has retries left returns to active for another attempt, while a successful one goes straight to completed.

A job schedule creates jobs on a recurring basis, for example a nightly report run. You define a recurrence (interval, start and end times) and a job specification, and Batch creates a new job from it at each occurrence.

Only one job from a schedule is active at a time. Use it instead of an external cron when the pool and tasks already live in Batch.

Output files are a task setting that tells the node agent to upload files from the node to Azure Storage after the task runs. You give a file pattern, a destination container (SAS URL or managed identity) and an upload condition: taskSuccess, taskFailure or taskCompletion.

This matters because node disks are temporary. If the node is deleted or reimaged, anything not uploaded is lost. Uploading logs on failure also helps with later debugging.

Azure Functions is event-driven and short-lived, while Batch is built for long, compute-heavy jobs you control at the VM level.

A common pattern is a Function that reacts to a blob upload and submits a Batch job for the heavy processing.

With autoscale enabled, Batch evaluates a formula you supply at a fixed interval (minimum 5 minutes, default 15) and sets the target node counts from the result. It reads service metrics such as $PendingTasks, $ActiveTasks and $CPUPercent.

$samples = $PendingTasks.GetSamplePercent(TimeInterval_Minute * 5);
$tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1))
                      : max($PendingTasks.GetSample(1), avg($PendingTasks.GetSample(TimeInterval_Minute * 5)));
$TargetDedicatedNodes = min(20, $tasks);
$NodeDeallocationOption = taskcompletion;

While autoscale is on you can't resize the pool manually. Use the evaluate-formula API to test a formula before applying it.

Design tasks to be restartable. When a Spot node is preempted, Batch moves the node to the preempted state and requeues the tasks that were running on it, so they start again from the beginning on another node.

1. Make tasks idempotent so a rerun produces the same result.
2. Write checkpoints to Azure Storage, not the node disk.
3. Keep a small dedicated node target for critical work, such as the job manager.
4. Keep tasks short so little work is lost per eviction.

Batch replaces preempted nodes when capacity returns, as long as the pool target is unchanged.

Choose it when the VMs must live in your own subscription so existing governance applies. Typical reasons are using your subscription's reserved instance or savings plan discounts, applying your Azure Policy and network controls, or drawing on your subscription's VM quota instead of the Batch account quota.

The trade-off is extra setup. The Batch account needs access to an Azure Key Vault and the Batch service principal needs permissions on your subscription, and you take on more of the quota management.

If none of these needs apply, Batch service mode is simpler.

Batch assigns tasks from active jobs to nodes that have free task slots. Higher-priority jobs are considered first, and inside a job, tasks with higher priority go first, then in order of creation.

Two pool settings shape placement:

• taskSlotsPerNode sets how many task slots a node offers (up to 4 times the core count, capped at 256). A task can need several via requiredSlots.
• nodeFillType is spread (default, even across nodes) or pack (fill one node before the next).

Use pack with autoscale so idle nodes can be released sooner.

Both prepare a node, but they run at different scopes and times.

Put pool-wide setup in the start task, and job-specific setup in the job preparation task.

Task dependencies let a task wait until other tasks complete successfully. You must enable them by setting usesTaskDependencies to true when the job is created, and you can't switch it on later.

A task can then declare dependsOn with specific task IDs or a range of IDs. Batch keeps the task in the active state until its dependencies succeed, and runs it as soon as they do. By default, if a dependency fails, the dependent task is never run, though exit conditions can change that with dependencyAction.

A typical use is a final merge step that waits for many parallel tasks.

A multi-instance task runs across several nodes at once, which is how Batch supports MPI and other tightly coupled workloads. You set the number of instances and a coordination command line.

Batch picks a primary node and subtasks. Each node runs the coordination command first (for example, starting the MPI runtime), and once all are ready, the primary runs the application command line. The pool must have inter-node communication enabled.

Use them for simulations and solvers where processes exchange data. For independent work, use ordinary tasks.

Retries are controlled by maxTaskRetryCount in the task constraints. A task is retried when it exits with a non-zero code until the retry count is used up. Set it to 0 for no retries, or -1 for unlimited retries.

The count covers failures of the task itself. If a node is lost or preempted, the task is requeued independently of this count.

You can also use exit conditions to decide per exit code whether to retry, ignore the failure or terminate the job.

Start with the task's exit code and failureInfo, then read its output files on the node.

1. Check the task's state, executionInfo.exitCode and failureInfo (category and message) in the portal or API.
2. Read stdout.txt and stderr.txt from the task directory.
3. If the task never started, check for resource file download errors and start task failures.
4. If the node was lost, check the node's state and errors, and the pool's resize errors.
5. Log in with Batch Explorer, or RDP/SSH where allowed, to inspect the node directly.

A UserError failure category usually means your command or inputs, while ServerError points to the service.

Batch marks the node unusable and stops scheduling work on it. Tasks that were running there are requeued and run on another node, and this doesn't consume their retry count.

Common causes are a VM fault, a platform issue or an OS-level failure. You can reboot or reimage the node, or remove it from the pool and let a resize or autoscale replace it.

A node in startTaskFailed is a separate case: the node is healthy but your start task failed, so fix the script rather than the VM.

Look at the pool's resizeErrors first; they usually name the cause.

• Quota reached: not enough cores for the VM family or the account. Request a quota increase or use another size or region.
• Start task failed: nodes are created but never become usable.
• Resize timeout: the default timeout is 15 minutes and Batch stops allocating if it passes.
• Capacity or networking: the VM size isn't available in that region or zone, or the subnet has too few free IPs.

After fixing the cause, resize the pool again.

You can run tasks inside containers on a pool whose VM image supports containers, such as the Ubuntu container-ready images or a custom image with Docker installed.

1. Set containerConfiguration on the pool, optionally listing images to pre-fetch and registry credentials.
2. Set containerSettings on the task, with the image name and any run options.

Batch pulls the image, mounts the task working directory and runs the task command line inside the container. Pre-fetching images on the pool avoids a pull on each node's first task.

Layer several controls rather than relying on one.

• Authenticate with Microsoft Entra ID and Azure RBAC, and avoid shared keys where you can.
• Use managed identities on pools so tasks reach Storage and Key Vault without secrets in command lines.
• Deploy pools into a virtual network and restrict access with NSGs and a private endpoint for the account.
• Store secrets in Azure Key Vault, not in task definitions.
• Use customer-managed keys if your policy requires them, and enable diagnostic logs.

Specify a subnet ID in the pool's network configuration at creation time. The nodes then get private IPs from that subnet and can reach on-premises or peered resources.

Points to plan for:

• The subnet needs enough free addresses for the maximum pool size.
• Nodes must be able to talk to the Batch service, either through the default inbound rules or the newer node management private endpoint model.
• Nodes need outbound access to Azure Storage and any registry you use.
• You can disable public IP addresses on nodes, and then you need an outbound route such as NAT.

Network settings can't be changed on an existing pool, so create a new one.

Cost follows node-hours, so the goal is to run fewer and cheaper nodes for less time.

1. Use Spot VMs for interruptible work, keeping dedicated nodes for critical parts.
2. Enable autoscale and scale to zero when the queue is empty.
3. Pick the right VM size by measuring CPU and memory use, and pack more tasks per node with task slots.
4. Reduce startup time so nodes spend less billed time not working.
5. Delete pools and old jobs you no longer need, and use storage lifecycle rules for outputs.

Use a mix of metrics, logs and the task data Batch already stores.

• Azure Monitor metrics for node counts by state, task counts and pool resize events, with alerts on them.
• Diagnostic logs sent to Log Analytics for pool, task and service events.
• Application Insights from inside tasks for application-level telemetry.
• Batch Explorer for a heat map of node states and quick access to task logs.

A useful alert is on nodes stuck in startTaskFailed or unusable.

Batch runs the command line directly, not inside a shell. Operators such as $VAR, %VAR%, pipes and redirection are therefore not interpreted.

To use them, invoke a shell explicitly:

/bin/bash -c "echo $AZ_BATCH_TASK_WORKING_DIR > out.txt"
cmd /c "echo %AZ_BATCH_TASK_WORKING_DIR% > out.txt"

The same applies to chaining commands with &&. If a Linux task works in your terminal but fails in Batch with a missing file or odd argument, this is a likely cause.

Batch sets a number of AZ_BATCH_* variables on every task.

Use the shared directory to cache data that many tasks on the same node can reuse.

Both can run containers at scale, but they differ in what they manage for you.

If the work is a finite queue of jobs, Batch is usually simpler. If the team already standardises on Kubernetes, AKS keeps one platform.

Data Factory orchestrates; Batch does the compute. Use the Custom activity in a pipeline, which submits a task to a Batch pool through a linked service.

1. Create a Batch linked service with the account, pool name and credentials.
2. Create a storage linked service where the activity script and its folder live.
3. Add the Custom activity with the command to run and optional reference objects.

Data Factory waits for the task to finish and picks up its exit code, so pipeline dependencies and retries still work.

A job moves through a clear set of states, driven by your calls and by task completion.

stateDiagram-v2
  [*] --> active: create job
  active --> disabling: disable
  disabling --> disabled
  disabled --> enabling: enable
  enabling --> active
  active --> terminating: terminate or all tasks done
  terminating --> completed
  completed --> deleting: delete
  deleting --> [*]

While active the job accepts tasks and they are scheduled. Disabled stops new scheduling, and running tasks can be requeued, terminated or allowed to finish depending on the option chosen.

A job moves to completed only when terminated, or automatically when onAllTasksComplete is terminateJob. Deleting a job removes its tasks and their node data.

From the moment the scheduler assigns a task until its results are stored, the flow is as follows.

sequenceDiagram
  participant S as Batch Service
  participant A as Node Agent
  participant ST as Azure Storage
  S->>A: Assign task to node
  A->>ST: Download resource files
  A->>A: Run job preparation task if needed
  A->>A: Execute task command line
  A->>ST: Upload output files
  A->>S: Report exit code and state

If a resource file fails to download, the task fails before its command runs, and the failure is recorded in failureInfo. The task directory holding stdout and stderr stays on the node until the retention time passes or the task or job is deleted.

Batch is a control plane plus agents. Your client calls the Batch REST API, and the service keeps the state of accounts, pools, jobs and tasks.

flowchart LR
  C["Client or SDK"] --> API["Batch service API"]
  API --> SCH["Scheduler and pool manager"]
  SCH --> VM["Compute VMs from Azure"]
  VM --> AG["Node agent on each node"]
  AG --> API
  AG <--> ST["Azure Storage"]

The pool manager asks Azure compute for VMs matching the pool definition. Each VM boots an image that includes the node agent, which registers with the service, polls for assigned tasks, runs them and reports status.

The scheduler matches active tasks to free slots by job priority and task order. Because the agent initiates the communication, the service can track nodes without you managing any cluster software.

Throughput depends on how quickly nodes get busy and stay busy.

• Right-size tasks: very short tasks waste time on scheduling and file downloads, so group tiny work items into one task.
• Use task slots to match concurrency to CPU and memory.
• Add tasks in bulk with the collection API rather than one call per task.
• Cache shared inputs in the node shared directory or a start task instead of downloading per task.
• Avoid long dependency chains that leave nodes idle.

Check the node state heat map to see whether nodes are idle or saturated before tuning.

Startup time is the gap between a node being requested and it running your first task, and you pay for it.

1. Bake software into a custom image in Azure Compute Gallery instead of installing it in the start task.
2. Keep the start task short and avoid large downloads.
3. Pre-fetch container images in the pool's container configuration.
4. Use application packages of modest size, as they are downloaded to every new node.
5. Keep a small warm pool if workloads arrive in bursts.

Measure startup from the node's creation time to its first task start to see which step dominates.

Quotas cap how many cores you can use, and they are the most common reason a pool fails to scale.

In Batch service mode, quotas are per account and per VM family, with separate limits for dedicated and Spot cores, plus limits on pools and jobs. In user subscription mode, the subscription's regular VM quotas apply.

1. Check the quota page of the account before sizing pools.
2. Request increases through a support request well ahead of big runs.
3. Split workloads across VM families or regions if a family is exhausted.

A quota failure appears in the pool's resize errors.

By default, nothing. The job stays active and ready to accept more tasks, and it keeps showing in the job list.

To end it automatically, set onAllTasksComplete to terminateJob when you create the job, or update it afterwards. Be careful with this setting when you add tasks over time: if the job is momentarily empty or all current tasks are done, it will terminate before the next tasks arrive.

Task data stays on nodes until its retention time (7 days by default) passes, so save outputs to Storage first.

You can change some settings in place, but others need a new pool.

Updates to the start task or packages apply to nodes when they next reboot or when new nodes join, so existing nodes aren't changed immediately. To roll out a change fully, reboot or reimage the nodes, or create a new pool and move jobs to it.

A new pool with the same pattern lets you switch jobs over without disrupting running work.

Several older features are gone, and you shouldn't use them in new designs.

If you inherit older code, replace certificate-based secrets with Key Vault, and check that pools use Virtual Machine configuration and Spot nodes. Always confirm current status on Microsoft's retirement notices.

Neither is better in general. They answer different needs.

Pick Batch for new, cloud-native parallel work with minimal operations. Pick CycleCloud when you need to lift an existing Slurm or PBS environment with its scripts and users unchanged.