BigData / Apache Airflow Interview Questions
What are best practices for writing efficient Airflow DAGs?
Key best practices for production-quality DAGs:
- Keep DAG files lightweight — avoid heavy imports or database calls at parse time; the scheduler parses DAG files continuously.
- Use top-level constants only — don't call APIs or read files at module level; do it inside operators/callables.
- Set catchup=False unless backfilling is intentional.
- Prefer TaskFlow API for clarity and automatic XCom passing.
- Use sensors in reschedule mode for long waits.
- Keep tasks idempotent and atomic.
- Use Pools to protect downstream systems from overload.
- Set email_on_failure and SLAs for alerting.
- Avoid using Variables at top-level — each call hits the DB at parse time.
Why should you avoid calling Airflow Variables at the top level of a DAG file?
More Related questions...
