fsspec / fsspec/kerchunk

Difference in access time for shorter vs longer kerchunk files

Open
#538 19 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Hi! I use kerchunk all the time and love it, so thank you! I have a question. I find pretty different access times that get longer the longer the kerchunk file is.

1 year kerchunk file:

Image

24 year kerchunk file (but same number of times and values accessed as in the 1 year kerchunk file):

Image

(The comparisons I am running in each plot are trying out subchunking since these files are uncompressed netCDF4 files.)

Is this expected behavior? Is there anything I can do to counteract this effect? For example, would it be better if I used 24 1-year kerchunk files? I've read through a bunch of issues here and I wonder if there is a flag that would help with this.

Thank you for any help!

Edited to add: These are parquet kerchunk files.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported access-time difference using the one-year and 24-year parquet kerchunk files, keeping the accessed times and values and subchunking comparison consistent. Start by tracing the relevant kerchunk access path and measure whether splitting the data into yearly files changes the result. Done means explaining whether the scaling is expected and identifying a supported mitigation, if one exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.