Difference in access time for shorter vs longer kerchunk files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Hi! I use kerchunk all the time and love it, so thank you! I have a question. I find pretty different access times that get longer the longer the kerchunk file is.
1 year kerchunk file:
24 year kerchunk file (but same number of times and values accessed as in the 1 year kerchunk file):
(The comparisons I am running in each plot are trying out subchunking since these files are uncompressed netCDF4 files.)
Is this expected behavior? Is there anything I can do to counteract this effect? For example, would it be better if I used 24 1-year kerchunk files? I've read through a bunch of issues here and I wonder if there is a flag that would help with this.
Thank you for any help!
Edited to add: These are parquet kerchunk files.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported access-time difference using the one-year and 24-year parquet kerchunk files, keeping the accessed times and values and subchunking comparison consistent. Start by tracing the relevant kerchunk access path and measure whether splitting the data into yearly files changes the result. Done means explaining whether the scaling is expected and identifying a supported mitigation, if one exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100