dlt-hub / dlt-hub/dlt

remove the OS filename length restriction impacting the identifiers length

Open
#191 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

**Background**
extract and normalize stage use files to pass intermediary data (extracted and normalized files). the metadata on that files is stored in file name. it contains schema name, table name and other information
the os file name limit is ~255 characters (but may be much less on some NFSses etc.)
that impact our ability to create long table names on bigquery/duckdb even with #157

**Tasks**
We need to remove metadata information from file name or find a way to shorten the file name ie. by using hashes. File names should be human readable though.

see test: `test_run_with_table_name_exceeding_path_length` which must start passing

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.