BigData / Apache Airflow Interview Questions
What is catchup in Airflow and how does it work?
Catchup is a DAG parameter that controls whether Airflow should backfill all DAG Runs from the start_date up to the current date when a DAG is first enabled or its schedule is changed. When catchup=True (default), Airflow will create a DAG Run for every missed schedule interval. When catchup=False, only the most recent interval is scheduled.
with DAG( 'my_dag', start_date=datetime(2024, 1, 1), schedule='@daily', catchup=False # do not backfill past runs ) as dag: ...
For most production pipelines, setting catchup=False is safer to avoid unexpected mass backfill on re-activation.
What happens when catchup=True and a DAG has been paused for 30 days?
More Related questions...
