fsspec / fsspec/filesystem_spec
[QUESTION] cache policy control
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
We're trying to use fsspec through dask/rapids/etc (dd.read_parquet('filecache:://az//...', storage_options={'filecache': ...})) but are unsure what parameters are valid, especially for controlling eviction policy + storage representation. I couldn't tell from the API docs as storage_options is untyped in them and did not see examples here, so any pointers would be appreciated!
In particular, we're thinking about:
- eviction policy: "LRU", "X items", "Y mb", "Z seconds"
- data rep: ensuring raw mirror vs compression, ...
We've been looking forward to using for awhile, so are excited!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the filecache storage_options API documentation and the existing filecache entry point; verify which parameters control eviction policy and storage representation. Document the supported values and add examples for the dask/rapids usage shown here, so users can determine how to configure those options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100