remove the OS filename length restriction impacting the identifiers length
Open
bug
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
**Background**
extract and normalize stage use files to pass intermediary data (extracted and normalized files). the metadata on that files is stored in file name. it contains schema name, table name and other information
the os file name limit is ~255 characters (but may be much less on some NFSses etc.)
that impact our ability to create long table names on bigquery/duckdb even with #157
**Tasks**
We need to remove metadata information from file name or find a way to shorten the file name ie. by using hashes. File names should be human readable though.
see test: `test_run_with_table_name_exceeding_path_length` which must start passing
Contributor guide
Assessment
This issue has not been assessed yet.