Cloud / Azure Databricks Interview questions

Azure Databricks is a managed analytics platform on Microsoft Azure built around Apache Spark. It gives data engineers, analysts and data scientists a single workspace for ETL, SQL analytics, streaming and machine learning on a lakehouse.

It is a first-party Azure service, so it plugs into Microsoft Entra ID for sign-on, ADLS Gen2 for storage, Azure Data Factory for orchestration and Power BI for reporting.

Compute runs on Azure VMs that scale automatically, and data sits in open formats, mainly Delta Lake, so you are not tied to a proprietary storage layer.

Typical use cases include batch ETL into Delta tables, streaming ingestion from Event Hubs, SQL dashboards on lakehouse data, and training ML models with MLflow tracking. Billing is in DBUs (Databricks Units) plus the Azure VM cost of the clusters.

The architecture is split into a control plane and a compute plane.

• Control plane: runs in Databricks' own subscription. It hosts the web UI, notebooks, job scheduler, cluster manager and REST APIs.
• Classic compute plane: the VMs for your clusters, created in your Azure subscription and VNet.
• Serverless compute plane: compute that Databricks manages for you for SQL warehouses, jobs and notebooks.
• Storage: a workspace storage account in a managed resource group, plus your own ADLS Gen2 accounts for data.

flowchart LR
  U["User or API call"] --> CP["Control plane: UI, notebooks, jobs, cluster manager"]
  CP --> CC["Classic compute: VMs in your subscription"]
  CP --> SC["Serverless compute: Databricks-managed"]
  CC --> ST[(ADLS Gen2 and workspace storage)]
  SC --> ST

Your data stays in your storage account; the control plane only stores metadata such as notebook code and job definitions.

A workspace is the environment where you organize and run your Databricks work. It holds notebooks, folders, clusters, jobs, SQL warehouses, dashboards and libraries.

In Azure, creating a workspace provisions an Azure resource in your subscription with a unique URL such as adb-<id>.<n>.azuredatabricks.net, plus a managed resource group for supporting resources.

Access inside a workspace is controlled by workspace permissions, and with Unity Catalog several workspaces can share one metastore so data governance stays consistent across them.

Workspaces are usually separated by environment (dev, test, prod) or by team. Permissions on notebooks, clusters and jobs are managed per workspace, while Unity Catalog manages data access at the account level.

Databricks offers a few compute types, chosen by workload.

• All-purpose clusters: shared, interactive clusters for notebooks and exploration. You start and stop them manually.
• Job clusters: created for a scheduled job run and terminated when it finishes.
• SQL warehouses: compute tuned for SQL and BI queries (classic, pro and serverless).
• Serverless compute: Databricks-managed compute for notebooks and jobs with no cluster configuration.
• Pools: idle, ready-to-use VMs that cut cluster start-up time.

Clusters can also be single-node (driver only) or multi-node with autoscaling workers.

Access mode matters too: standard mode isolates users on a shared cluster and supports Unity Catalog, while dedicated mode assigns the cluster to one user or group, which some ML libraries need.

A notebook is a web-based document made of cells that contain code, SQL, markdown or visualizations. It runs against an attached cluster or serverless compute.

Each cell can switch language with a magic command such as %python, %sql, %scala, %r, %md or %sh. %run executes another notebook inline.

Notebooks support real-time co-authoring, comments, revision history and Git integration, and they can be scheduled directly as job tasks.

Notebooks also use helpers through dbutils, for example dbutils.fs.ls to browse files and dbutils.notebook.run to call another notebook with parameters and capture its result. display(df) renders a DataFrame as an interactive table or chart.

Notebooks and jobs support Python (PySpark), SQL, Scala and R. Java is supported for compiled JAR jobs.

Python and SQL are by far the most common choices today. You can mix languages in one notebook through magic commands, and share data between them through temporary views or tables.

df = spark.range(5)
df.createOrReplaceTempView("nums")   # Python creates the view
# a %sql cell can now run: SELECT * FROM nums

PySpark and Spark SQL compile to the same optimized plans, so equivalent logic usually performs alike. Plain Python UDFs are the exception: they run row by row outside the JVM and are slower than built-in functions or pandas UDFs.

Delta Lake is an open-source storage layer that adds reliability to files in a data lake. Data is stored as Parquet, and a transaction log tracks every change.

That log gives you ACID transactions, schema enforcement, time travel, and the ability to run batch and streaming reads and writes against the same table.

In Azure Databricks, Delta is the default table format, so CREATE TABLE without a format creates a Delta table.

You can convert an existing Parquet folder with CONVERT TO DELTA, which builds the log without rewriting files. Delta Lake is open source, so the same tables can be read outside Databricks through connectors and Delta Sharing.

DBFS (Databricks File System) is a file-system abstraction over cloud object storage. Paths look like dbfs:/mnt/data or /dbfs/... on the driver.

The DBFS root is backed by the workspace's storage account. Databricks now discourages keeping production data there and discourages legacy mounts too.

The recommended approach is Unity Catalog: external locations for governed cloud paths and volumes for non-tabular files.

In notebooks you work with DBFS through dbutils.fs (for example dbutils.fs.ls('dbfs:/')). Direct cloud paths like abfss://container@account.dfs.core.windows.net/ bypass DBFS and are the norm with Unity Catalog.

A job (now part of Lakeflow Jobs, formerly Workflows) is a scheduled or triggered run of one or more tasks.

Tasks can be notebooks, Python scripts, JARs, SQL queries, dbt, or pipeline updates. You wire them into a dependency graph (a DAG), so downstream tasks wait for upstream ones.

Jobs support cron schedules, file-arrival and continuous triggers, retries, alerts, parameters, and repair runs that re-run only failed tasks.

Each task can use its own compute or share a job cluster with other tasks to avoid repeated start-up time. Small results can pass between tasks with dbutils.jobs.taskValues.

The Databricks Runtime (DBR) is the set of components installed on every cluster node: Apache Spark, Delta Lake, Java, Scala, Python, R, and curated libraries.

Variants include the standard runtime, the ML runtime (with TensorFlow, PyTorch, scikit-learn), and Photon-enabled runtimes. Each release has a version number, and some are marked LTS (long-term support).

For production jobs, pick an LTS version so you are not forced to upgrade every few months.

The runtime also fixes your Spark and Python versions, so libraries must match it. Release notes list the bundled library versions, and you can add wheels or PyPI packages at cluster level or per notebook with %pip install.

Unity Catalog is Databricks' unified governance layer for data and AI assets. It centralizes access control, auditing, lineage and discovery across workspaces.

Objects follow a three-level namespace: catalog.schema.table. Besides tables, it governs volumes, functions and ML models.

Permissions are granted with standard SQL GRANT statements, and everything is recorded in audit logs and column-level lineage.

Governance goes beyond tables: you can tag assets, mask columns, filter rows per user, and trace which notebooks and jobs produced a table through lineage. A metastore is tied to a region and shared by the workspaces assigned to it.

A DataFrame is a distributed collection of rows organized into named, typed columns, similar to a table. It is immutable: every transformation returns a new DataFrame.

Spark evaluates DataFrames lazily. Transformations like select and filter only build a plan; an action such as count() or write triggers execution.

df = spark.read.table("sales.orders")
big = df.filter("amount > 1000").select("order_id", "amount")  # lazy
big.count()                                                  # action runs the job

Pandas-style options exist too. toPandas() collects a DataFrame to the driver, so use it only on small results, while the pandas API on Spark scales the same syntax across the cluster.

Widgets add input controls to a notebook so one notebook can be reused with different parameters. Types are text, dropdown, combobox and multiselect.

dbutils.widgets.text("env", "dev")
env = dbutils.widgets.get("env")

When a notebook runs as a job task or from Azure Data Factory, the job parameters are passed into widgets with the same name.

Widgets are handy in backfills, for example a run_date widget lets one notebook process any day without code edits. Clear them with dbutils.widgets.removeAll() when they are no longer needed.

Databricks SQL is the SQL-first part of the platform for analysts and BI. It includes the SQL editor, SQL warehouses, dashboards, alerts and query history.

SQL warehouses are compute endpoints tuned for concurrent queries, available as classic, pro and serverless. BI tools such as Power BI and Tableau connect through the built-in connectors or JDBC/ODBC.

Queries run against Delta tables governed by Unity Catalog, so analysts get warehouse-style performance directly on lakehouse data.

Alerts can notify you when a query result crosses a threshold, and dashboards can refresh on a schedule. Serverless warehouses start in seconds, which suits spiky BI traffic better than keeping a classic warehouse running.

Git folders (formerly Repos) clone a remote Git repository into the workspace so you can version notebooks and files with real branches.

They work with Azure DevOps, GitHub, GitLab and Bitbucket. From the UI you can create branches, commit, push, pull and resolve conflicts, and jobs can run directly from a branch or tag.

They are the usual starting point for CI/CD because code review and release flow stay in Git instead of the workspace.

Keep notebooks as source files (.py) where possible so pull-request diffs stay readable. Credentials for the Git provider are stored per user as a personal access token or a linked account.

An all-purpose cluster is a long-lived, shared cluster for interactive work. A job cluster exists only for the duration of one job run.

For production pipelines, job clusters (or serverless jobs) are cheaper and give clean isolation between runs.

A common pattern is to develop on an all-purpose cluster and run the same notebook on a job cluster in production. Job clusters take a few minutes to start unless backed by a pool, so chained short tasks should share one job cluster or use serverless jobs.

Both can run Spark, but they are built around different centers. Databricks is a Spark and Delta lakehouse platform; Synapse bundles several engines under one Azure workspace.

Teams with heavy Spark, data science or streaming needs usually choose Databricks; T-SQL warehouse teams often prefer Synapse.

Portability differs as well. Delta tables written by Databricks are open and readable by other engines, while Synapse dedicated pools keep data in a proprietary warehouse format. Billing is DBU-based for Databricks and DWU-based for dedicated pools.

A mount uses a service principal with OAuth 2.0 and dbutils.fs.mount. The secret should come from a secret scope, never from plain text.

configs = {
  "fs.azure.account.auth.type": "OAuth",
  "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  &quot;fs.azure.account.oauth2.client.id&quot;: &quot;<app-id>&quot;,
  &quot;fs.azure.account.oauth2.client.secret&quot;: dbutils.secrets.get(&quot;kv-scope&quot;, &quot;sp-secret&quot;),
  &quot;fs.azure.account.oauth2.client.endpoint&quot;: &quot;https://login.microsoftonline.com/<tenant-id>/oauth2/token&quot;
}
dbutils.fs.mount(
  source=&quot;abfss://raw@mystorage.dfs.core.windows.net/&quot;,
  mount_point=&quot;/mnt/raw&quot;,
  extra_configs=configs)

Mounts are now a legacy pattern because everyone in the workspace can use them. For new work, use Unity Catalog external locations with an access connector.

To remove a mount use dbutils.fs.unmount('/mnt/raw'), and dbutils.fs.mounts() lists existing ones. A mount is visible to every user in the workspace with that identity's permissions, which conflicts with fine-grained Unity Catalog governance.

The difference is who owns the data files. A managed table stores data in a location Unity Catalog controls, and dropping the table removes the data. An external table points at a path you specify, and dropping it removes only the metadata.

CREATE TABLE main.sales.orders (id INT, amount DOUBLE);          -- managed

CREATE TABLE main.sales.orders_ext (id INT, amount DOUBLE)
LOCATION 'abfss://gold@mystorage.dfs.core.windows.net/orders'; -- external

Use external tables when other tools must read the same files or when data already lives at a fixed path. Dropped managed tables in Unity Catalog can be recovered for a limited window with UNDROP.

Choose managed tables for data the lakehouse owns, since Unity Catalog can then manage layout and run predictive optimization for you. Choose external tables for files produced or consumed by other systems, or when policy requires a specific storage account.

Delta Lake keeps an ordered transaction log in the _delta_log folder. Each commit is a JSON file listing the files added and removed. A table's state is simply the replay of that log.

1. A writer reads the latest table version.
2. It writes new Parquet data files.
3. It tries to commit a new log entry atomically.
4. If another writer committed a conflicting change first, Delta checks for conflicts and either retries or fails (optimistic concurrency).

Readers see only fully committed versions, so you never see half-written data. Checkpoint Parquet files are written periodically to speed up log replay.

Because each commit is atomic, a failed job leaves no partial output. Delta uses WriteSerializable isolation by default, which allows more concurrent writers; you can raise it to Serializable when strict ordering matters. Conflicting writers raise errors such as ConcurrentAppendException.

Time travel lets you query a table as it was at an earlier version or timestamp, because Delta keeps old data files referenced by older log entries.

SELECT * FROM sales.orders VERSION AS OF 12;
SELECT * FROM sales.orders TIMESTAMP AS OF '2026-10-01';
DESCRIBE HISTORY sales.orders;
RESTORE TABLE sales.orders TO VERSION AS OF 12;

It helps with audits, reproducing ML training data and undoing bad writes. History is limited by log retention (30 days by default) and by VACUUM, which deletes old files after 7 days by default.

Downstream jobs can pin reads to a version for reproducibility. To keep time travel useful, align delta.logRetentionDuration and delta.deletedFileRetentionDuration with how far back you need to go, and run VACUUM only after that window.

Create an Azure Key Vault-backed secret scope in the workspace, then read secrets with dbutils.secrets.get.

pwd = dbutils.secrets.get(scope="kv-scope", key="sql-password")

Databricks redacts secret values in notebook output, showing [REDACTED]. Grant the Databricks service access to the vault (via access policy or RBAC) and give users only READ on the scope.

A Databricks-backed scope also exists, but Key Vault-backed scopes centralize rotation and auditing in Azure.

Never print secrets or place them in logs. Pass them straight into connection options, for example the JDBC password option. For automation, grant the job's service principal access to the scope instead of a personal account.

Auto Loader incrementally ingests new files from cloud storage as they arrive. You use it through the cloudFiles source in Structured Streaming.

df = (spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schema")
      .load("abfss://landing@mystorage.dfs.core.windows.net/events"))

It tracks processed files in its checkpoint, infers and evolves schemas, and can find files by directory listing or by file notification using Event Grid and queues for very large directories. Bad records can be captured in a rescued data column.

A typical setup lands files in ADLS, runs Auto Loader into a bronze table with trigger(availableNow=True) on a schedule, and keeps a schema location. cloudFiles.schemaEvolutionMode decides whether new columns fail the stream, are added, or go to the rescued data column.

Both load files idempotently, but they scale differently and use different interfaces.

Databricks recommends Auto Loader for most new ingestion. COPY INTO is fine for small, simple SQL-only loads.

Auto Loader keeps file discovery cost flat as folders grow by using incremental listing or notifications. COPY INTO compares incoming files against the table's load history, which gets slower on very large histories. If you need a streaming table in a pipeline, only Auto Loader fits.

The medallion architecture organizes lakehouse data into three quality layers, each stored as Delta tables.

flowchart LR
  S[Sources] --> B["Bronze: raw, as ingested"]
  B --> SI["Silver: cleaned, deduplicated, conformed"]
  SI --> G["Gold: aggregated, business-ready"]
  G --> BI["BI, ML, APIs"]

• Bronze keeps raw data with ingest metadata, so you can always replay.
• Silver applies cleansing, typing, deduplication and joins.
• Gold holds aggregates and feature tables shaped for reports and models.

Because bronze is preserved, a bug in silver logic can be fixed and reprocessed without re-reading the source systems.

Not every pipeline needs exactly three layers. Some teams add a landing zone before bronze or a serving layer after gold. The idea is progressive quality improvement with each layer stored as Delta, which keeps checks, lineage and reprocessing simple.

There are three common ways, in order of preference for new setups.

1. Unity Catalog storage credential using an Access Connector for Azure Databricks (a managed identity), exposed through an external location. No secrets to rotate.
2. Service principal with OAuth 2.0, configured through Spark settings with the secret held in Key Vault.
3. Storage account key or SAS token. Works, but it is the least secure and is discouraged.

spark.conf.set("fs.azure.account.auth.type.mystorage.dfs.core.windows.net", "OAuth")
# ...plus provider type, client id, secret, and endpoint settings

Managed identity through an access connector is cleaner because access is governed by Unity Catalog grants on the external location.

Whatever you pick, apply least privilege: grant Storage Blob Data Contributor only on the containers needed, and avoid reusing one service principal across environments. Access connectors can be assigned per workspace to keep boundaries clear.

Parquet is a columnar file format. Delta Lake is a table format that stores its data in Parquet files and adds a transaction log on top.

Because Delta stays Parquet underneath, you keep columnar compression and predicate pushdown while gaining reliable table semantics.

That log is what enables MERGE, UPDATE, DELETE, data-skipping statistics and OPTIMIZE. An existing Parquet folder can be converted with CONVERT TO DELTA without rewriting data.

Both change the number of partitions, but repartition does a full shuffle while coalesce merges existing partitions without one.

df.repartition(200, "customer_id")   # balance and co-locate by key
df.coalesce(10).write.format("delta").save(path)  # fewer output files

A typical rule: use repartition before a wide operation when partitions are too few or uneven, and coalesce just before a write to cut output file count. With AQE on, post-shuffle partitions are also coalesced automatically, which reduces manual tuning.

Structured Streaming is Spark's stream-processing engine. It treats a live stream as an unbounded table that keeps getting new rows, so you write the same DataFrame code as for batch.

(spark.readStream.table("bronze.events")
   .groupBy("device_id").count()
   .writeStream.outputMode("complete")
   .option("checkpointLocation", "/Volumes/main/chk/events_count")
   .toTable("silver.device_counts"))

Queries run in micro-batches by default. Output modes are append, update and complete, and with a checkpoint plus an idempotent sink you get end-to-end exactly-once results. The availableNow trigger processes all pending data and stops, which suits scheduled incremental jobs.

Stateful operations such as aggregations, joins and deduplication keep state on the executors and use watermarks to drop old state so it does not grow forever. Data arriving later than the watermark is ignored for windowed aggregations.

A checkpoint is a storage location where a streaming query saves its progress: source offsets, committed batch IDs and any aggregation state.

If the query stops or fails, restarting it with the same checkpoint resumes exactly where it left off, which is how Spark delivers exactly-once processing with idempotent sinks.

• Use one unique checkpoint path per query.
• Never delete it unless you want a full reprocess.
• Changing the query's stateful logic or sources can make an old checkpoint incompatible.

Checkpoints can live on ADLS or in a Unity Catalog volume. In production treat them like data: protect the folder from cleanup jobs and keep one per stream so queries never overwrite each other's progress.

Delta Live Tables (DLT), now part of Lakeflow Declarative Pipelines, lets you define a pipeline by declaring the tables you want, and Databricks handles orchestration, dependencies, retries and infrastructure.

import dlt

@dlt.table
@dlt.expect_or_drop("valid_amount", "amount > 0")
def silver_orders():
    return spark.readStream.table("bronze_orders").select("order_id", "amount")

Expectations enforce data-quality rules and can warn, drop rows or fail the update. DLT also tracks lineage and data-quality metrics in an event log.

Pipelines run in triggered mode (process available data, then stop) or continuous mode (always on for low latency). Streaming tables handle append-only incremental ingestion, while materialized views compute and refresh aggregations and joins. Pipeline event logs expose expectation pass and fail counts per update.

MERGE INTO compares a source to a target on a key and applies different actions to matched and unmatched rows in one atomic operation.

MERGE INTO silver.customers AS t
USING updates AS s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.deleted = true THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

Make sure the source has one row per key; duplicates cause a 'multiple source rows matched' error. For large targets, add a partition or clustering predicate to the ON clause to limit the files scanned.

For slowly changing dimensions, one MERGE can close the old row and insert the new version. Newer runtimes support WHEN NOT MATCHED BY SOURCE to delete target rows missing from the source, and AUTO CDC (formerly APPLY CHANGES) in pipelines simplifies CDC handling.

A cluster policy is a JSON rule set that limits what users can configure when they create compute, such as allowed VM sizes, max workers, runtime versions and auto-termination.

{
  "node_type_id": {"type": "allowlist", "values": ["Standard_DS3_v2", "Standard_DS4_v2"]},
  "autotermination_minutes": {"type": "fixed", "value": 30},
  "custom_tags.team": {"type": "fixed", "value": "data-eng"}
}

Admins assign policies to users or groups. They are the main tool for controlling cost and enforcing tags and security settings without blocking self-service.

Policies also support defaults and limits, for example range for maximum workers or regex for allowed runtime versions. With the unrestricted-creation permission turned off for non-admins, they steer users to compute that fits budget and security rules.

ADF orchestrates Databricks through a linked service and the Databricks activities: Notebook, Python and JAR.

1. Create a Databricks linked service. Authenticate with an access token, managed identity or service principal.
2. Choose compute: a new job cluster per run or an existing cluster.
3. Add a Notebook activity and set baseParameters.
4. Read them in the notebook with widgets, and return a value with dbutils.notebook.exit().

The returned value is available to later ADF activities as @activity('name').output.runOutput. Many teams now use Databricks Jobs for orchestration and call them from ADF only when a wider cross-service pipeline is needed.

Keep tokens out of pipelines by referencing Key Vault in the linked service. Use a new job cluster for isolation and lower cost, and set retries on the activity so transient start-up failures do not break the whole pipeline.

Photon is Databricks' native vectorized query engine written in C++. It replaces parts of the JVM-based Spark execution for supported operations, speeding up SQL and DataFrame workloads on Delta and Parquet.

No code changes are needed. You enable it by selecting a Photon-enabled runtime or turning it on in the cluster settings. SQL warehouses use it by default.

Photon helps most with scans, joins, aggregations and large writes. It is billed at a higher DBU rate, so measure whether the faster runtime offsets the price for your workload.

Photon does not speed up Python UDFs or code that runs mostly on the driver. Check the query profile for Photon operators and compare run time before and after to confirm the gain is worth the higher DBU rate.

Each file costs a metadata lookup and a task. Thousands of tiny files mean slow listing, many small tasks and poor compression, so scans spend more time opening files than reading data.

OPTIMIZE rewrites many small files into fewer larger ones (target around 1 GB by default) and records the swap in the transaction log. Old files stay for time travel until VACUUM.

OPTIMIZE sales.orders;
ALTER TABLE sales.orders SET TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.autoOptimize.autoCompact'   = 'true');

Optimized writes and auto compaction prevent the problem at write time. Unity Catalog managed tables can also use predictive optimization to run maintenance automatically.

Streaming writes and frequent small appends are common sources of tiny files, so run OPTIMIZE after heavy ingestion or let auto compaction do it. Compaction creates new files, so storage grows until VACUUM removes the old ones; schedule both together. Partitioning by a high-cardinality column such as user_id also multiplies tiny files, another reason to prefer Liquid Clustering.

Both co-locate related rows so queries skip more files, but Liquid Clustering is the newer, more flexible approach that Databricks recommends for new Delta tables.

CREATE TABLE sales.events (ts TIMESTAMP, user_id BIGINT, ...)
CLUSTER BY (user_id, ts);

Liquid Clustering also handles data skew and high-cardinality keys better than static partitions.

For new tables, use CLUSTER BY and stop using PARTITIONED BY and ZORDER. Existing tables can be switched with ALTER TABLE ... CLUSTER BY on supported runtimes. Pick the columns most often used in filters, typically one to four, and on Unity Catalog managed tables consider automatic clustering to let Databricks choose keys from query history.

Adaptive Query Execution (AQE) re-optimizes a query plan at runtime using real statistics gathered after each shuffle stage, instead of relying only on estimates made before execution.

• Coalesces shuffle partitions: merges many tiny post-shuffle partitions into fewer.
• Switches join strategy: converts a sort-merge join into a broadcast join if one side turns out small.
• Splits skewed partitions: breaks oversized partitions in a join into smaller tasks.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

It is on by default in current runtimes. It helps most when statistics are missing or data volumes change between runs.

You can see whether AQE changed a plan in the Spark UI SQL tab, where the final plan shows AdaptiveSparkPlan with isFinalPlan=true. Useful settings include spark.sql.adaptive.coalescePartitions.enabled and spark.sql.adaptive.advisoryPartitionSizeInBytes. AQE cannot rescue a cartesian join or a cluster with too little memory, so sound design still matters.

Skew means a few keys hold most of the rows, so one task runs far longer than the rest. In the Spark UI you see one straggler task with huge shuffle read.

1. Enable AQE skew join handling so large partitions are split automatically.
2. Broadcast the smaller table if it fits in memory.
3. Filter out or isolate hot keys such as NULL or default values.
4. Apply salting: add a random suffix to the hot key on the big side and replicate the small side across the same suffix range.
5. Repartition on a more evenly distributed key.

from pyspark.sql import functions as F
salted = big.withColumn("salt", (F.rand() * 8).cast("int"))
small_rep = small.crossJoin(spark.range(8).withColumnRenamed("id", "salt"))
joined = salted.join(small_rep, ["key", "salt"])

Confirm the skew first with groupBy(key).count().orderBy(desc('count')) to find hot keys. Salting adds complexity, so try AQE and broadcast first. Another approach is to join the hot keys in a separate branch with a broadcast join and union it with the normal join output for the remaining keys.

A broadcast hash join sends the small table to every executor and joins locally with no shuffle of the big side. A sort-merge join shuffles and sorts both sides by key, then merges them.

from pyspark.sql.functions import broadcast
df = orders.join(broadcast(dim_country), "country_id")

Broadcasting a table that is too large can cause driver or executor out-of-memory errors.

Spark also has a shuffle hash join and, for non-equi conditions, a broadcast nested loop join, which can be very slow on large inputs. You can hint a strategy with /*+ BROADCAST(t) */ or /*+ MERGE(t) */. With AQE, a sort-merge join can convert to a broadcast join at runtime when a filtered side turns out small enough.

First find where memory ran out: the driver or the executors.

Open the Spark UI and check the stage with the failure: look for skewed task sizes, shuffle spill and high GC time. Cluster metrics show whether memory is the actual bottleneck.

Raising partition counts (or relying on AQE) reduces data per task, and a memory-optimized VM family gives more RAM per core. Large cached DataFrames add pressure, so call unpersist() when done. If logs show lost executors or GC overhead errors, inspect the data per task before simply adding nodes.

When you run an action, Spark turns your code into a job and executes it in stages.

flowchart LR
  A["Action called"] --> B["Logical plan"]
  B --> C["Catalyst optimization"]
  C --> D["Physical plan"]
  D --> E["Split into stages at shuffle boundaries"]
  E --> F["Tasks per partition sent to executors"]
  F --> G["Results returned or written"]

1. The driver builds a logical plan from your DataFrame operations.
2. Catalyst optimizes it (predicate pushdown, column pruning) and selects a physical plan.
3. The DAG scheduler splits it into stages wherever a shuffle is needed.
4. Each stage becomes tasks, one per partition, run on executor cores.
5. Results go back to the driver or are written to storage.

With AQE on, later stages may be re-planned using runtime statistics.

VACUUM permanently deletes data files that are no longer referenced by the current table version and are older than the retention period. It frees storage but removes the ability to time travel to those versions.

VACUUM sales.orders;                 -- default 7 days (168 hours)
VACUUM sales.orders RETAIN 240 HOURS;
VACUUM sales.orders DRY RUN;

The 7-day default protects long-running readers and streaming jobs that may still read older files. Databricks blocks values below it unless you disable spark.databricks.delta.retentionDurationCheck.enabled, which is risky. Use DRY RUN to preview what would be deleted.

Avoid short retention on tables that streaming jobs or long queries still read, since they can fail with file-not-found errors. Predictive optimization can vacuum Unity Catalog managed tables automatically. VACUUM removes only data files; old log entries are cleaned up separately according to log retention.

Access follows the hierarchy metastore, catalog, schema, table. A user needs USE CATALOG and USE SCHEMA on the parents plus a privilege such as SELECT on the table.

GRANT USE CATALOG ON CATALOG main TO `analysts`;
GRANT USE SCHEMA  ON SCHEMA  main.sales TO `analysts`;
GRANT SELECT      ON TABLE   main.sales.orders TO `analysts`;

Privileges inherit downward, so granting SELECT on a schema covers all current and future tables in it. Grant to groups synced from Entra ID rather than individuals, and use row filters and column masks for fine-grained control.

To review permissions, use SHOW GRANTS ON TABLE main.sales.orders. Owners and metastore admins can grant privileges, and ownership can be transferred. For external storage you also need privileges such as READ FILES on the external location. Row filters and column masks restrict sensitive data without making copies.

Delta Sharing is an open protocol for sharing live data with other organizations or platforms without copying it. The provider publishes tables in a share, and a recipient reads them.

CREATE SHARE retail_share;
ALTER SHARE retail_share ADD TABLE main.sales.orders;
CREATE RECIPIENT partner_a;
GRANT SELECT ON SHARE retail_share TO RECIPIENT partner_a;

Recipients can be on Databricks (Unity Catalog to Unity Catalog) or use any client that speaks the protocol, such as pandas, Spark or Power BI. Access is time-limited by tokens or federated identity, and shares are audited.

Delta Sharing also supports views, volumes and notebooks in some cases, and Databricks-to-Databricks sharing needs no token management. No data is replicated, so providers pay for storage and recipients for their own compute, though cross-region reads may incur cloud egress charges.

The modern approach is Databricks Asset Bundles: you describe jobs, pipelines and clusters in a databricks.yml file that lives in Git and deploy it with the CLI.

flowchart LR
  D["Developer pushes to Git"] --> PR["Pull request: tests and bundle validate"]
  PR --> M["Merge to main"]
  M --> S["Deploy to staging target"]
  S --> T["Integration tests"]
  T --> P["Deploy to prod target"]

databricks bundle validate -t staging
databricks bundle deploy   -t staging
databricks bundle run nightly_etl -t staging

Run this from Azure DevOps or GitHub Actions using a service principal, define dev, staging and prod as bundle targets, and unit-test transformation code with pytest before deploy.

A good setup keeps notebooks light and moves reusable logic into tested Python packages. Secrets stay in Key Vault or pipeline variable groups, and bundle targets override cluster size, schedules and catalog names per environment. Use mode: development in dev targets so resources get a user prefix and schedules are paused.

Cost is DBUs plus the underlying Azure VM and storage costs, so levers exist on both sides.

• Use job clusters or serverless jobs for pipelines instead of leaving all-purpose clusters running.
• Set auto-termination and enable autoscaling.
• Use spot VMs for fault-tolerant workers, with an on-demand driver.
• Apply cluster policies and mandatory tags to cap size and attribute cost.
• Right-size VMs based on Spark UI and cluster metrics.
• Fix inefficiencies: compact small files, avoid full shuffles, use Photon where it pays off.
• Consider a pre-purchase plan (DBCU) for steady usage.

Cost views by tag in Azure Cost Management and system billing tables in Unity Catalog show who spends what.

Cost governance is continuous: review system billing tables or Azure Cost Management weekly, look for low-utilization clusters, and flag all-purpose clusters left idle. Serverless reduces idle waste and start-up time, but compare cost per workload because its DBU rate is higher than classic compute.

Change Data Feed (CDF) records row-level changes (inserts, updates, deletes) made to a Delta table so downstream consumers can read only what changed.

ALTER TABLE silver.customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT * FROM table_changes('silver.customers', 15, 20);

Each row carries _change_type (insert, update_preimage, update_postimage, delete), _commit_version and _commit_timestamp. It is useful for incremental loads to gold tables, audit trails and syncing to external systems. Only changes made after enabling it are captured.

A downstream job stores the last processed _commit_version and reads from that version onward. CDF adds some storage and write overhead, so enable it only on tables that need it. Pair it with MERGE on the target to keep gold tables or external systems in sync.

By default Delta enforces the schema: a write with an unexpected column or incompatible type fails. You opt in to evolution when you want it.

• mergeSchema on an append or overwrite adds new columns.
• overwriteSchema replaces the schema during a full overwrite.
• spark.databricks.delta.schema.autoMerge.enabled enables evolution for MERGE.
• Column mapping allows renaming and dropping columns without rewriting data.

(df.write.format("delta")
   .mode("append")
   .option("mergeSchema", "true")
   .saveAsTable("bronze.events"))

Allow evolution in bronze where sources drift. Keep silver and gold strict so breaking changes are caught early.

Be careful with type changes: widening such as INT to BIGINT is supported on newer runtimes, but narrowing needs a table rewrite. Use ALTER TABLE ADD COLUMNS for planned changes, and review the Auto Loader rescued data column for unexpected fields before promoting them to silver.

Security is layered across network, identity, data and secrets.

The Premium tier is required for features such as role-based access controls and Unity Catalog-grade governance.

Review security regularly: enable audit logging and system tables to see who accessed what, restrict cluster creation through policies, turn off legacy features such as the DBFS browser where possible, and keep production in a separate workspace so scheduled jobs are not shared with ad-hoc experimentation.