DataStore to CSV service, for download of large resources.
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Explanation:
I would like to use the DataStore via the API as primary data-source. This works without a problem already.
However, if people wants to download the entire resource as a CSV, via /dump/, it only downloads 100K records (this is hardcoded into CKAN).
It also takes quite a long time to generate the CSV file.
I have resources with over 10+ mio. Rows and would like to offer a complete download via CSV. But changing the hardcoded 100K row limit puts a lot of pressure on the system.
It would be very nice to have a feature where, using the API for the DataStore, would update a corresponding CSV-file for download. So download wouldn’t need to generate the file.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the existing DataStore API and the /dump/ endpoint, including the hardcoded 100K-record limit and how CSV downloads are generated. Define how API changes would update a corresponding CSV for resources over 10 million rows, and verify that downloads no longer require generating the file on demand.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100