feat(typing): consistent top-level types
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
`dlt` has several top-level constructs: source, resource, transformation, destination, pipeline, schema, dataset, relation, configuration, credentials, items.
Currently, it's difficult for `dlt` users to type annotate their code because:
- naming is inconsistent
- `DltSource`, `DltResource`, `DltTransformationResource`, `Destination`, `Pipeline`
- import paths are inconsistent
- `dlt.sources.DltSource`, `dlt.sources.DltResource`, `dlt.common.destination.Destination`, `dlt.Pipeline`, `dlt.Schema`, `dlt.Dataset`; `DltSource` is exposed both in `dlt.extract` and `dlt.sources`
- requires importing from private modules and some code snippets suggesting to do so
## Proposed solution
Have consistent naming for top-level constructs that can be imported from a single location. Having core constructs under `dlt.Type` would help humans and LLMs write well-annotated code.
```python
# intended API
from dlt import (
Source,
Resource,
Destination,
Pipeline, # done
Schema, # done
Dataset, # done
Relation, # done
)
from dlt.hub import Transformation
```
### Required changes
This can be a relatively small change, but requires good testing. There are two approaches possible
#### 1) `DltSource -> Source`
1. Create the new type `Source`
2. Add alias `Source = DltSource`
3. make `dlt.Source` available.
4. Add tests that `Source is DltSource`
This approach is simple, requires few lines of code, but creates a mismatch: we tell our users to use `dlt.Source`, but we use `dlt.sources.DltSource` everywhere in our code base.
#### 2) `Source -> DltSource`
1. Rename `DltSource` to `Source` everywhere in the code base (it's almost exclusively used in type annotations and `isinstance` check; low risk of breaking things)
2. Addd deprecation notices
3. Add alias `DltSource = Source`
4. Make `dlt.Source` available
5. Add tests to ensure that `DltSource` exists and is not removed from the code base. Otherwise, `DltSource` (the new one) will appear unused and could be deleted, thus breaking a massive backwards compatibility issue
Review this PR commit-by-commit #2983 ([in particular this one](https://github.com/dlt-hub/dlt/pull/2983/commits/093c1f2a44080055b67d9c4b9963274505b15c37))
> [!IMPORTANT]
> Maintain a clean and atomic git commit history to make the reviewers life easy.
## Alternative
It's likely that adding imports to `dlt/__init__.py` creates circular import errors. Adding `from __future__ import annotations` could fix the problem... or not. If it proves complicated, we can move types to `dlt/typing.py`.
Contributor guide
Research direction
Review the existing public exports in dlt/__init__.py and the current definitions and imports under dlt.sources, dlt.extract, and dlt.common.destination. Compare the two proposed Source/DltSource approaches with PR #2983, especially its referenced commit, and check for circular imports. Done means consistent top-level imports, backwards-compatible aliases or deprecations, and tests covering the aliases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, developer-experience
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100