BigData / Apache Airflow Interview Questions
What is idempotency in the context of Airflow tasks?
An idempotent task produces the same result regardless of how many times it is executed for the same input. In Airflow, tasks are often retried or re-run (via clear), so designing them to be idempotent prevents duplicate data or inconsistent state.
Techniques to ensure idempotency:
- Use
INSERT ... ON CONFLICT DO NOTHINGor upserts instead of plain inserts. - Partition output by
{{ ds }}and overwrite the partition on each run. - Write to intermediate staging tables and swap atomically.
- Delete existing data for the date range before inserting.
Why is idempotency important for Airflow tasks?
More Related questions...
