euroargodev / euroargodev/argopy

Caching in the facade and after pre-processing in Argo stores ?

Open
#204 4 comments 0 reactions 0 assignees View on GitHub
internals performance stale
Dominant language
Python
Stars
229
Forks
52
Avg merge
2d 10h
Merged PRs (30d)
8

Description

At this point, caching is used at the Argo store lowest level, where data are directly fetched from any sources using fsspec methods.

But they are 2 other levels of the argopy data workflow that should be implementing cache because operations may be long:

1. at the Argo store level, during the preprocessing step
2. at the facade level, once data sources have post-processed data returned by Argo stores

This would significantly increase the cache size, and may be expiration time should be different (#196), but the performance gain could be worth it !

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing cache at the Argo store's lowest level, then inspect the preprocessing step and facade post-processing flow. Review issue #196 for expiration-time considerations. Done means cache behavior is defined and implemented at both proposed levels without changing the existing source-fetching behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.