BigData / Apache Airflow Interview Questions
What is backfilling in Apache Airflow?
Backfilling is the process of running a DAG for historical date ranges that were not previously executed. You can trigger a backfill from the CLI:
airflow dags backfill --start-date 2024-01-01 --end-date 2024-03-31 my_dag
This creates DAG Runs for every schedule interval between the start and end dates. Backfilling is useful when you add a new DAG with start_date in the past and want to process historical data, or when a pipeline was down and you need to reprocess missed intervals.
More Related questions...