Implement dask.sizeof for xarray.core.indexing.ImplicitToExplicitIndexingAdapter
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
I'm looking at a pangeo gallery workflow that suffers from poor load balancing because objects of type xarray.core.indexing.ImplicitToExplicitIndexingAdapter are being interpretted as 48B when in fact, I suspect, they are significantly larger to move around.
I'm seeing number of processing tasks charts that look like the following, which is a common sign of the load balancer not making good decisions, which is most commonly caused by poor data size measurements:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting xarray.core.indexing.ImplicitToExplicitIndexingAdapter and how Dask measures object sizes. Reproduce the reported load-balancing behavior from the linked Pangeo gallery workflow, then add coverage for the adapter's size measurement and verify that Dask no longer treats it as 48B.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100