Tabular Ingest Column Limit
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Greetings,
I wanted to start a tread regarding the idea of limiting the ingest of file depending on the number of column.
There is already a limitation based on file size :TabularIngestSizeLimit.
We use 500 Mo for this limitation.
We experienced too long running ingest job on one CSV file of 430 Mo and 500k columns, maybe others are facing similars issues regarding the number of columns.
I really don't know if there is a way to avoid the ingest of a "too big file to ingest".
If that can help to think about an idea, here are some ingest time tested or estimated :
| Type | Size | Number of columns | Ingest time |
|---|---|---|---|
| CSV | 18 Mo | 1170 | 13 minutes |
| CSV | 20 Mo | 41723 | 18 hours |
| CSV | 430 Mo | 481844 | Estimated 420 days |
Related post : https://groups.google.com/u/1/g/dataverse-community/c/yMWryN52pyE
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing TabularIngestSizeLimit configuration linked in the issue and the tabular ingest path that handles wide CSV files. Determine whether a column-count limit is feasible, what threshold and failure behavior are needed, and how the reported wide-CSV case should be covered; the issue is not yet specific about files or tests.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100