BigData / Apache Airflow Interview Questions
What is Airflow on Kubernetes (KEDA) autoscaling?
When running CeleryExecutor on Kubernetes, KEDA (Kubernetes-based Event Driven Autoscaler) can automatically scale Celery worker pods based on the number of tasks in the queue. This removes the need for static worker counts and makes the cluster cost-efficient.
How it works:
- KEDA is deployed alongside Airflow.
- A ScaledObject is configured to watch the Celery broker queue length.
- When tasks accumulate, KEDA scales up worker Deployment replicas.
- When the queue drains, workers scale back down to zero.
Helm chart support: set workers.keda.enabled=true in the official Airflow Helm chart.
More Related questions...