Time to list a big reference set
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
When xarray opens a zarr dataset, the first thing it does is ls("") on the root, to find out the "directories" contained, each of which will be an array. If each array has many chunks, this takes a long time, since we consider every single key in building the directory structure, with string processing on each.
Suggestion: to store the set of directories (or a tree, if deep) in the reference set, to avoid having to do this every time. This would require changing the spec and updating referenceFS.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the current reference-set specification and the referenceFS implementation, then trace the root ls("") path that builds the directory structure. Determine how directory or tree metadata would fit the specification and implementation. Done means the specification and referenceFS agree on the stored metadata and root listing no longer processes every key individually.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100