when yielding data from a resource, mark given value to be normalized
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
**Goal**
We want to mark certain string values to be normalized according to naming convention used later in the pipeline. Compare such feature to `dlt.with_table` where we mark whole dictionary to be routed to table with given type.
**Motivation**
Quite often some lineage metadata is loaded with the data itself. For example in case of `pipedrive` we load a table with definitions of user defined fields. Those fields are present in `dlt` schema and their names are normalized like any other column names.
**Tasks**
1. * [ ] add `normalize` function in `dlt.mark` module. that will wrap a `str` in a string like object that will be serialized in `typed_dump` as a special type (like for example `Decimal` or `datetime`)
2. * [ ] when deserialized in normalizer, a `normalize_identifier` method from the current schema context will be applied to the value
Contributor guide
Research direction
Start in the dlt.mark module and trace how typed_dump serializes special values such as Decimal or datetime. Then follow deserialization in the normalizer to the current schema context and its normalize_identifier method. Done means marked strings survive serialization and are normalized according to that schema context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100