BigData / Apache Airflow Interview Questions
What is branching in Airflow and how is BranchPythonOperator used?
Branching lets a DAG conditionally execute one or more downstream paths based on runtime logic. The BranchPythonOperator runs a Python callable that returns the task_id (or list of task_ids) of the branch(es) to follow. All other branches are skipped.
from airflow.operators.python import BranchPythonOperator def choose_branch(): import random return 'branch_a' if random.random() > 0.5 else 'branch_b' branch = BranchPythonOperator( task_id='choose', python_callable=choose_branch ) branch >> [task_a, task_b]
Tasks not selected by the branch get a skipped state, so downstream join tasks often need trigger_rule='none_failed_min_one_success'.
More Related questions...