Smart caching of datastore dumps
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
With the new datastore dump formats (https://github.com/ckan/ckan/pull/3390), and even the regular CSV dump, the files are created dynamically, which is an expensive operation, with large tables timing out with errors.
It'd be great if the writers can implement some smart caching mechanism where the files are stored in the file system (or the cloud, if using ckanext-cloudstorage) and the cached files are automatically expired when the datastore is updated.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the datastore dump formats described in PR #3390 and identify the writers that create CSV and other dump files dynamically. Define the caching behavior for filesystem or ckanext-cloudstorage storage, including expiration after datastore updates, and verify that large-table dumps no longer time out.
Written by the indexing model from the issue text.
Assessment
- Domain
- cloud, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100