BigData / Apache Airflow Interview Questions
What is XCom in Airflow and how is it used?
XCom (cross-communication) is a mechanism that lets tasks exchange small amounts of data. A task can push a value into XCom and a downstream task can pull it.
def push_func(**context): context['ti'].xcom_push(key='result', value=42) def pull_func(**context): val = context['ti'].xcom_pull(task_ids='push_task', key='result') print(f'Received: {val}')
XCom values are stored in the metadata database, so they should be used for small payloads (strings, numbers, short dicts). Passing large dataframes through XCom is an anti-pattern — use a shared storage layer such as S3 or GCS instead.
What is the recommended maximum size of data to pass via XCom?
More Related questions...
