Include storage options in references?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Currently, we include only the URLs of references in the generated JSON files. With https://github.com/fsspec/filesystem_spec/pull/867 , this becomes more complicated, as we can have multiple remote filesystems, each with their own set of optional storage_options.
Since the notebook cells instantiating a ReferenceFileSystem were already getting rather large (although you can encode these in an Intake entry), perhaps we could encode these options directly into the references? On the downside, this makes is less convenient to relocate the original data, by providing an override for URL templates - but I don't know if that feature was being used in practice.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the notebook cells that instantiate ReferenceFileSystem and the generated JSON reference structure, then examine how Intake entries and URL-template overrides are currently used. Determine whether storage_options should be encoded per reference or supplied separately, and document the chosen behavior and its effect on relocating original data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100