concatenating files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm not sure if this should go here or to the main fsspec repo.
I'd like to use the reference filesystem to concatenate pieces. That is, for one reference, I'd like to specify a list of things to join together for making up a new file in stead of just having a single pointer. This could roughly look like:
"refs": {
"key0": ["data", ["http://target_url", 10000, 100]],
"key1": [["http://target_url", 10000, 100], ["http://{{u}}", 10000, 100]],
}
etc... Using that method, one could completely rearrange existing files.
In my current application, I'd like to join existing chunks of uncompressed netCDF file into a single larger chunk to be used within zarr.
A potential issue might be, that the following would become ambiguous:
"refs": {
"key0": ["https://test"],
}
This could refer to either just a single piece of raw data containing the text "https://test" or a reference to the entire object behind the link. However, it should be possible to disambiguate this, by defining that single-element raw data blocks must always be written without the list.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the reference filesystem and the existing reference format handling described in this issue. Trace how a reference currently resolves a single pointer, then determine how concatenated pieces and the single-element ambiguity should be represented. Done means the format and behavior are specified well enough to support rearranging referenced data, including the netCDF-to-zarr use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100