live update on hdfstores in blaze server
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
@mrocklin @jreback
I would like to augment the blaze server so that if someone updates an hdfstore, the blaze server can do compute on the updated file.
Here is my plan, I'd like your feedback on it
- subclass hdfstore such that if timestamps change, the pytables file handle will be reopened if it's stale
- add an entry to resource to return this subclass
(maybe we call it livehdfstore?)
The other option is to look for a kwarg in the resource call (live=True?) and dispatch based on that to either the original hdfstore, or the subclass?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing HDFStore implementation and the resource dispatch entry point. Determine how an externally updated HDF store can be detected and reopened safely, then add coverage showing that the Blaze server computes against updated data without requiring a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100