# XComs and Their Limits — Apache Airflow: Orchestrate Data Pipelines

Source: https://www.geekswithgeeks.com/en/airflow/data-xcom

> Use XComs for small metadata and pass large data through external storage.

## Pass pointers, not payloads

An **XCom** ("cross-communication") is a small value stored in the metadata database and handed from one task to another: a file path, a row count, an ID. It is **not** meant for large data such as DataFrames or files: that bloats the database and slows the scheduler. For big data, write it to object storage (S3, GCS) or a warehouse table, and pass only the **location** through XCom. TaskFlow does this plumbing automatically for return values; with classic operators use `ti.xcom_push` / `ti.xcom_pull`.

## Small values, big ideas

Tasks share small values through XComs, use templates for dates and parameters, and read secrets from connections.

![Three tools: XCom, templates, connections.](assets/figures/airflow/section-3-map.svg) — Figure 3.1 — XCom, templates and connections.

## Pass the location, not the data

The first task writes the large file and returns only its path; the second task receives the path. (Illustrative; `write_parquet` and `read_parquet` stand for your own code.)

```python
@task
def extract(ds=None) -> str:
    path = f"s3://lake/orders/dt={ds}/raw.parquet"
    write_parquet(fetch_orders(ds), path)      # big data goes to storage
    return path                                 # only a short string goes to XCom

@task
def transform(path: str) -> str:
    df = read_parquet(path)
    out = path.replace("raw", "clean")
    write_parquet(clean(df), out)
    return out
```

## Watch XCom size

If you see slow UI pages or a growing metadata database, check for large XCom values. Keep each under a few kilobytes as a rule of thumb.

**Quiz:** What should you pass through XCom for a 2 GB dataset?

- [ ] Nothing, XCom is forbidden
- [ ] The whole dataset
- [x] Only its storage location
- [ ] A screenshot

*Answer:* Only its storage location. XCom is for small metadata; large data belongs in external storage.