recognize uris as data locations in pipeline.run method and auto use core sources
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
**Background**
With working `fsspec` we may recognize and automatically load data from various uris. We can combine this with a few additional types like pandas frame.
- accept strings as dlt data if they are uris to resources
- recognize fsspec uris
- allow loading the following data formats (by extension): json, jsonl, csv ... from those uris
- accept gzipped files
~- accept panda frames~
- allow to stream large json files, recognize files containing lists of objects and a few other streamable cases (we have a concept code)
**Implementation Outline**
Extent our internal sources by merging the following into main library
~- pandas source (enumerate pandas frames)~
- json and jsonl sources
- jsonl streaming source with format autodetection
- we can use pandas for csv, xml, xls etc.
~**Future Work**
At some point we want to change how the normalizer works so it can deal with (serialized) panda frames (ie. feather), parquet files etc. directly to not be forces to convert all of them into python objects and back~
Contributor guide
Research direction
Start at the pipeline.run method and inspect the existing internal sources and fsspec-related data loading paths. The issue proposes URI recognition, format-based loading, gzip support, and streaming JSON/JSONL, but names no files or tests; done would require agreeing on scope and covering the selected URI formats with tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100