Possibility to load all data-sizes into datastore
- Dominant language
- Python
- Stars
- 58
- Forks
- 67
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 8
Description
It would be great if we had a possibility to configure xloader to load all sizes of data into the datastore.
Right now, the default value for `ckanext.xloader.max_content_length` is 1GB which makes sense in my eyes but I would like to add a way to have no limit at all.
@davidread do you have concerns about my idea? I was thinking to maybe check if `ckanext.xloader.max_content_length` was set to `0` we would just go forward and load the data.
If anyone has concerns or a better idea to implement this, let me know. If there is interest/no backlash, I will hopefully be able to spend some time on this next month.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating where ckanext.xloader.max_content_length is defined and enforced, then trace the path that loads data into the DataStore. Confirm the intended configuration for an unlimited value and add coverage for that behavior; done means files larger than the default limit can load without changing the existing bounded behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100