XComs and Their Limits

Pass pointers, not payloads

An XCom ("cross-communication") is a small value stored in the metadata database and handed from one task to another: a file path, a row count, an ID. It is not meant for large data such as DataFrames or files: that bloats the database and slows the scheduler. For big data, write it to object storage (S3, GCS) or a warehouse table, and pass only the location through XCom. TaskFlow does this plumbing automatically for return values; with classic operators use ti.xcom_push / ti.xcom_pull.

Small values, big ideas

Tasks share small values through XComs, use templates for dates and parameters, and read secrets from connections.

Pass the location, not the data

The first task writes the large file and returns only its path; the second task receives the path. (Illustrative; write_parquet and read_parquet stand for your own code.)

@task
def extract(ds=None) -> str:
    path = f"s3://lake/orders/dt={ds}/raw.parquet"
    write_parquet(fetch_orders(ds), path)      # big data goes to storage
    return path                                 # only a short string goes to XCom

@task
def transform(path: str) -> str:
    df = read_parquet(path)
    out = path.replace("raw", "clean")
    write_parquet(clean(df), out)
    return out

Watch XCom size

If you see slow UI pages or a growing metadata database, check for large XCom values. Keep each under a few kilobytes as a rule of thumb.

Quick check: What should you pass through XCom for a 2 GB dataset?

• Nothing, XCom is forbidden
• The whole dataset
• Only its storage location
• A screenshot