dlt-hub / dlt-hub/dlt

Support generic types in config fields

Open
#465 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

Currently we don't support `TypeVar`s in config fields, so generic types need to be defined `Any`

We should allow typevar hints on fields which are part of generic classes.

This would allow incremental config to be typed and validated correctly:

```python
TCursorValue = TypeVar("TCursorValue", bound=Any)

@configspec
class Incremental(BaseConfiguration, Generic[TCursorValue]):
cursor_path: str = None
initial_value: Optional[TCursorValue] = None

...

@configspec
class SomeResourceConfig(BaseConfiguration):
updated_at: Incremental[pendulum.DateTime]
```

### Implementation notes:

* `base_configuration.is_valid_hint` should allow `TypeVar` as a hint

Best would be to check whether the spec class is a generic with the same typevar and not allow unbound typevars.
E.g. this is fine:

```
class Incremental(BaseConfiguration, Generic[T]):
initial_value: Optional[T]
```

This is not

```
class Incremental(BaseConfiguration):
initial_value: Optional[T]
```

* `get_resolvable_fields` when called on a concrete subclass or a generic alias, such as `Incremental[str]` or `class SomeClass(Incremental[str])`

Should match the typevar of the fields with the typevars on the generic base and return the concrete type for each field. There may be multiple generic types in the class.

* When `get_resolvable_fields` is called on the generic baseclass, if typevars can't be mapped to concrete types, either:

1. Raise an exception -> force user to specify concrete type (may be breaking change)
2. Allow it but replace unmatched typevar hints with `Any`
3. Start with 2. but with deprecation warnings and change to 1. in later release

### Testing

* All existing tests should pass unchanged
* Test generic with 1 and multiple typevars
* Test generic baseclass as a config
* Test subclass of a concrete type

Contributor guide

Open the contributing guide

Research direction

Start in the implementations of base_configuration.is_valid_hint and get_resolvable_fields, then trace how generic config classes and aliases expose their type variables. Decide how unmapped typevars on a generic base should behave before implementing resolution. Done means existing tests remain unchanged and coverage includes one and multiple typevars, generic base configs, and subclasses of concrete generic types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.