Dynamic and Discriminated Unions

When modeling semi-structured data, discriminated unions are an invaluable tool in your arsenal.
It allows you to model a variable unit of data, where each unit concisely specifies its type and payload. This is a common pattern in event-driven architectures, where you have a stream of "events", "commands", or "actions", each of which is a discriminated union of event types.

I'd like to propose two new features to the typing module that formalize dynamic and discriminated unions.

1️⃣ Dynamic Unions

Current state

Assume we have a Status model that is a discriminated union of StatusA and StatusB:

from typing import Literal, Union
from pydantic import BaseModel


class StatusBase(BaseModel):
    status: str


class StatusA(StatusBase):
    status: Literal['a']


class StatusB(StatusBase):
    status: Literal['b']

To define a union that is understood by the type-checker at static-analysis time, you declare it by mentioning every member of the union:

Status = Union[StatusA, StatusB]
reveal_type(Status)  # Type of "Status" is "type[StatusA] | type[StatusB]"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

Whenever adding a new subclass of StatusBase, this declaration must be updated.

I often abuse a pattern of lazily declaring a dynamic union instead of maintaining a static one, by tying all the members of the union to the subclasses of a parent class (in this case StatusBase):

Status = Union[tuple(StatusBase.__subclasses__())]
reveal_type(Status)  # Type of "Status" is "Unknown"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

The type-checker is not able to infer the type of Status, because it's only compiled at runtime.

Proposed feature

I propose a new feature that would allow the type-checker to infer a dynamic union of all StatusBase subclasses, and finally error if more subclasses are declared afterward.

from typing import DynamicUnion  # Currently not a thing

Status = DynamicUnion[StatusBase]
reveal_type(Status)  # Type of "Status" is "type[StatusA] | type[StatusB]"
print(Status)  # typing.Union[__main__.StatusA, __main__.StatusB]

(Maybe call it SubclassUnion instead of DynamicUnion?)

The type-checker should detect if a subclass of StatusBase is declared after Status, and raise an error.

Status = DynamicUnion[StatusBase]

class StatusC(StatusBase):  # Error: Subclass of StatusBase declared after declaration of DynamicUnion[StatusBase]
    status: Literal['c']

This could work like the @sealed decorator was proposed to work in the Sealed Typing PEP draft. More useful information found on this email chain.

The @sealed decorator was designed as part of PEP 622 (structural pattern matching). Think of it as confining a type and its subclasses to a single module, so that the type-checker can dynamically infer a union of the subclasses. These are also known as algebraic data types.

I'd like to also explore the possibility of scoping algebraic data types to a package, or a dynamic scope bound between the parent class declaration and the union declaration.

2️⃣ Discriminated unions

Current state

For lack of a formalization of discriminated unions in core python, data (de)serialization libraries turn to their own objects. For example, pydantic specifies a discriminator value with Field, and uses Annotated to tag the union:

from typing import Annotated
from pydantic import Field

AnnotatedStatus = Annotated[
    Status,
    Field(discriminator='status')
]

This does not raise an error at runtime if any member of the Status union does not have a status field UNTIL AnnotatedStatus is used as a type on the field of a pydantic model declaration.

Annotated is an "escape hatch" that allows clients to express things that type-checkers
might not understand.

However, there is a push by third-party libraries for objects in the typing module that would canonically be used with the Annotated type. It signals a willingness to move from redundant proprietary internals toward ubiquitous core python. See PEP 727 and related discussion.

Proposed feature

I propose a new feature that would allow the type-checker to infer the type of AnnotatedStatus, by declaring it as a dynamic discriminated union:

from typing import DiscriminatedUnion  # Currently not a thing

AnnotatedStatus = DiscriminatedUnion[Status, 'status']
reveal_type(AnnotatedStatus)  # Type of "AnnotatedStatus" is "type[StatusA] | type[StatusB]"
print(AnnotatedStatus)  # typing.Union[__main__.StatusA, __main__.StatusB]

This would grant data (de)serialization libraries unified plumbing to declare discriminated unions, and would allow the type-checker to throw an error at static-analysis time if any member of the Status union does not have a status field, or if any member of the Status union has a status field with a value that is not unique among all members of the union.

The latter hinges on type-checkers being able to implicitly detect disjoint unions. See pyright#5933 for more information on mutable field invariance and implicit disjoint unions.

Motivation

When I'm writing JSONSchemas or deserializing in event-driven architectures, and I'm iterating fast, I gravitate toward easily being able to add new members to a union.

I usually do this in one of two ways. Either:

• all in one file, declaring parent, subclasses, then the union (as shown in the example above), or
• all in one directory, declaring the parent in base.py, each of the subclasses in a separate file, and the union in the module’s __init__.py

While this approach feels more magical than maintaining a static union yourself, it facilitates an accessible plug-in architecture, where users can implement new subclasses of the parent class in a new file, and have it automatically become part of the union without deeply understanding the internals of the library.

Beginne mit dem Lesen der Vorschläge zu DynamicUnion und DiscriminatedUnion und folge anschließend dem referenzierten Entwurf des Sealed Typing PEP, PEP 622 und pyright#5933. Als abgeschlossen gilt die Aufgabe, wenn ein abgestimmtes Typing-Design vorliegt, das den Zeitpunkt der Unterklassenbildung, die Validierung des Discriminators und das Verhalten des Type-Checkers abdeckt; das Issue nennt keine Implementierungsdateien oder Tests.