dlt-hub / dlt-hub/dlt

feat(typing): consistent top-level types

Open
#3,344 0 comments 0 reactions 0 assignees View on GitHub
enhancement QoL tech-debt
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

`dlt` has several top-level constructs: source, resource, transformation, destination, pipeline, schema, dataset, relation, configuration, credentials, items.

Currently, it's difficult for `dlt` users to type annotate their code because:
- naming is inconsistent
- `DltSource`, `DltResource`, `DltTransformationResource`, `Destination`, `Pipeline`
- import paths are inconsistent
- `dlt.sources.DltSource`, `dlt.sources.DltResource`, `dlt.common.destination.Destination`, `dlt.Pipeline`, `dlt.Schema`, `dlt.Dataset`; `DltSource` is exposed both in `dlt.extract` and `dlt.sources`
- requires importing from private modules and some code snippets suggesting to do so

## Proposed solution
Have consistent naming for top-level constructs that can be imported from a single location. Having core constructs under `dlt.Type` would help humans and LLMs write well-annotated code.

```python
# intended API
from dlt import (
Source,
Resource,
Destination,
Pipeline, # done
Schema, # done
Dataset, # done
Relation, # done
)
from dlt.hub import Transformation
```

### Required changes
This can be a relatively small change, but requires good testing. There are two approaches possible

#### 1) `DltSource -> Source`
1. Create the new type `Source`
2. Add alias `Source = DltSource`
3. make `dlt.Source` available.
4. Add tests that `Source is DltSource`

This approach is simple, requires few lines of code, but creates a mismatch: we tell our users to use `dlt.Source`, but we use `dlt.sources.DltSource` everywhere in our code base.

#### 2) `Source -> DltSource`
1. Rename `DltSource` to `Source` everywhere in the code base (it's almost exclusively used in type annotations and `isinstance` check; low risk of breaking things)
2. Addd deprecation notices
3. Add alias `DltSource = Source`
4. Make `dlt.Source` available
5. Add tests to ensure that `DltSource` exists and is not removed from the code base. Otherwise, `DltSource` (the new one) will appear unused and could be deleted, thus breaking a massive backwards compatibility issue

Review this PR commit-by-commit #2983 ([in particular this one](https://github.com/dlt-hub/dlt/pull/2983/commits/093c1f2a44080055b67d9c4b9963274505b15c37))

> [!IMPORTANT]
> Maintain a clean and atomic git commit history to make the reviewers life easy.

## Alternative
It's likely that adding imports to `dlt/__init__.py` creates circular import errors. Adding `from __future__ import annotations` could fix the problem... or not. If it proves complicated, we can move types to `dlt/typing.py`.

Contributor guide

Open the contributing guide

Research direction

Review the existing public exports in dlt/__init__.py and the current definitions and imports under dlt.sources, dlt.extract, and dlt.common.destination. Compare the two proposed Source/DltSource approaches with PR #2983, especially its referenced commit, and check for circular imports. Done means consistent top-level imports, backwards-compatible aliases or deprecations, and tests covering the aliases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, developer-experience
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.