apache / apache/airflow

Unexpected primitive based Enum (de)serialization

Open
#51,016 9 comments 2 reactions 0 assignees View on GitHub
area:core area:serialization kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

main (development)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

When enumeration inherits some primitive type (e.g. str, int), deserialization makes the value primitive as well.
```python
import enum
from airflow.serialization.serde import deserialize, serialize

class MyEnum(enum.IntEnum):
A = 1
B = 2

value = MyEnum.A
res = deserialize(serialize(value))
assert res is MyEnum.A, f'{type(res)=}, {res=}'
# AssertionError: type(res)=, res=1
```

### What you think should happen instead?

This is little unexpected for me as a user and for mypy
```python
@task
def my_task(v: MyEnum):
assert is instance(v, MyEnum), 'fine for mypy, fails in runtime'
v = MyEnum(v)
...
my_task(MyEnum.A)
```

The other issue is that if I add airflow compatible custom serialization the result is incorrect too
```python
class MyEnum(enum.IntEnum):
A = 1
B = 2

def serialize(self):
return f'{self.name}.{self.value}'

res = serialize(MyEnum.A)
assert res == 'A.1', f'{type(res)=}, {res=}'
# AssertionError: type(res)=, res=1
```

I would expect one of
- implement default serialization and deserialization for enums
- fail with `TypeError: cannot serialize object of type ` unless it implements the interface and is registered for deserialization (as it happens for enums that do not inherit primitive)

### How to reproduce

See the snippet above

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the serialize and deserialize entry points in airflow.serialization.serde and reproduce the IntEnum examples from the issue. Resolve which behavior is intended for primitive-based enums, then verify that serialization preserves the enum type or consistently rejects unsupported custom serialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.