BigData / Apache Airflow Interview Questions
What is a Deferrable Operator (async operator) in Airflow?
Deferrable Operators (introduced in Airflow 2.2) allow tasks to suspend themselves, release the worker slot, and wait for an external event via a lightweight Trigger component. This is more efficient than sensors in poke mode because no worker thread is held while waiting.
The flow is:
- Task starts, then defers itself by calling
self.defer(...). - A Triggerer process monitors the event (file arrival, HTTP response, etc.).
- When the event fires, the Triggerer resumes the task on a worker.
Use deferrable operators for long-polling scenarios (e.g., waiting hours for a cloud job to finish) to avoid tying up workers.
What Airflow process monitors events for deferrable operators?
More Related questions...
