euroargodev / euroargodev/argopy
Caching in the facade and after pre-processing in Argo stores ?
- Dominant language
- Python
- Stars
- 229
- Forks
- 52
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
At this point, caching is used at the Argo store lowest level, where data are directly fetched from any sources using fsspec methods.
But they are 2 other levels of the argopy data workflow that should be implementing cache because operations may be long:
1. at the Argo store level, during the preprocessing step
2. at the facade level, once data sources have post-processed data returned by Argo stores
This would significantly increase the cache size, and may be expiration time should be different (#196), but the performance gain could be worth it !
Contributor guide
Research direction
Start by tracing the existing cache at the Argo store's lowest level, then inspect the preprocessing step and facade post-processing flow. Review issue #196 for expiration-time considerations. Done means cache behavior is defined and implemented at both proposed levels without changing the existing source-fetching behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100