BigData / Apache Airflow Interview Questions
What are common Airflow anti-patterns to avoid?
Common pitfalls that hurt reliability and performance:
- Top-level DB calls — calling
Variable.get()orConnection.get_connection_from_secrets()at parse time stresses the scheduler. - Non-idempotent tasks — retries create duplicate data.
- Giant XCom payloads — overloads the metadata DB.
- Too many small tasks — scheduling overhead grows linearly; batch micro-tasks where possible.
- Dynamic DAG generation at parse time — if the generation is slow (API calls, DB queries), the scheduler lags.
- Using SubDAGs — causes deadlocks; use TaskGroups instead.
- Sensors in poke mode for long waits — blocks worker slots; use reschedule mode.
- Hardcoded credentials — always use Connections or secrets backend.
Why should you avoid using SubDAGs in Airflow?
More Related questions...
