fsspec / fsspec/kerchunk

Kerchunk for representing subset of a dataset

Open
#185 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Is it possible, or how could we enable, using kerchunk to create files that represent a subset of a dataset? In other words, given that you know which chunks you would like to include, and where those chunks are located, can you use kerchunk to create a file that zarr can use to read in a portion of a single dataset?

The paricular use case I have in mind is creating a kerchunk json file that represents a single burst within a Sentinel-1 SLC dataset. Bursts are subsets of SLC files and the SLC files come with metadata stating where each burst is located within each SLC image. SLCs come in GeoTIFFs that are chunked by line, and the metadata provides information stating which lines the bursts begin and end on. I've done some digging and here is a rough idea of how this might work.

  1. Obtain the line location information for a burst
  2. Convert this line information to byte offset and length
  3. Pass this information to fsspec/tifffile
  4. Write this information to a kerchunk json file

I have already completed step 1, but could use assistance with the last three. At this point, I'm not concerned with including metadata with the subsetted data. As an example, here is a link to download a Sentinel-1 SLC from the Alaska Satellite Facility's archive, and I've included the relevant file and line location information for one burst below:

Burst Numer: 1
PATH: S1A_IW_SLC__1SDV_20211229T231926_20211229T231953_041230_04E66A_3DBE.SAFE/measurement/s1a-iw2-slc-vv-20211229t231926-20211229t231951-041230-04e66a-005.tiff
Line Start: 0
Line End: 1509

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing how fsspec and tifffile expose the GeoTIFF chunk locations and how kerchunk JSON references are consumed by zarr. Use the Sentinel-1 example and the provided line range to trace the proposed byte-offset and length conversion. Done means a documented or implemented path can generate a kerchunk file for the selected burst, without requiring its metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.