Deferrable Operators & Triggers

Documentation

Concepts

There are some terms and concepts that are important to understand:

1. asyncio: This library is core to deferrable operator’s functionality, and is used when writing triggers.
2. Triggers: These are small, asynchronous pieces of Python code. Due to their asynchronous nature, they coexist efficiently in a single process known as the triggerer.
3. Triggerer: This is a new airflow service (like a scheduler or a worker) that runs an asyncio event loop in your Airflow environment. Running a triggerer is essential for using deferrable operators.
4. Deferred: This is a new Airflow task state (medium purple color) introduced to indicate that a task has paused its execution, released the worker slot, and submitted a trigger to be picked up by the triggerer process.

Note

The terms «deferrable» and «async» or «asynchronous» are often used interchangeably. They mean the same thing in this context.

Using Deferrable Operators

Use pre-written Deferrable Operators, such as TimeSensorAsync, which comes with Airflow.

Currently, the following deferrable operators are available in Airflow:

1. TimeSensorAsync — Waits until the specified time of the day, freeing up a worker slot while it is waiting.
2. DateTimeSensorAsync — Waits until the specified datetime, deferring itself to avoid taking up a worker slot while it is waiting.
3. TimeDeltaSensorAsync — A drop-in replacement for TimeDeltaSensor that defers itself to avoid taking up a worker slot while it is waiting.

Note

You cannot yet use the deferral ability from inside custom PythonOperator/TaskFlow Python functions; it is only available to traditional, class-based Operators at the moment.