apache / apache/airflow

Provide feedback when accessing undeclared outlet at runtime

Open
#50,003 2 comments 0 reactions 0 assignees View on GitHub
area:data-aware-scheduling kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Currently, operations toward a non-declared outlet to `outlet_events` is simply ignored:

```python
@task
def emit_alias_event():
context = get_current_context()
context["outlet_events"][AssetAlias("a")].add(Asset("new"))
# Line above does nothing since AssetAlias("a") is not an outlet.
```

The same also applies to `yield Metadata`.

This is a bit confusing to the user since they did something, and reasonably expect it’s an Airflow bug if nothing seems to happen.

I can think of a couple of ways to resolve this:

#### Automatically add outlets when you do this

If you access something, it automatically becomes an outlet. However, Airflow would not be able to detect such a dynamic outlet in many other places, such as the dag dependency graph (which is generated statically). Would this only shift the confusion elsewhere?

#### Raise a KeyError

Or a more detailed error with a hint. More explicit, but would it be awkward for certain workflows?

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the runtime handling of context["outlet_events"] and yielded Metadata in the task example. Decide between automatic outlet registration and raising a KeyError or more detailed error, then define the expected behavior for undeclared outlets and verify it in the relevant runtime tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.