BigData / Apache Airflow Interview Questions
What is dynamic task mapping in Airflow?
Dynamic task mapping, introduced in Airflow 2.3, allows you to create a variable number of task instances at runtime based on data rather than defining a fixed set of tasks at DAG parse time.
from airflow.decorators import dag, task @dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False) def dynamic_example(): @task def process(item: str): print(f'Processing {item}') # Creates one task instance per item in the list process.expand(item=['a', 'b', 'c', 'd']) dynamic_example()
You can also map over the output of an upstream task with .expand(item=upstream_task()), which makes the parallelism truly data-driven.
Which Airflow version introduced dynamic task mapping?
More Related questions...
