`.fillna()` slower than expected for sparse data arrays with `fill_value=nan`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
Expected behavior
.fillna(0) should be near instantaneous when applied to a sparse DataArray with fill_value=nan
Why
.fillna(0) only needs to update the fill_value to 0.
Current behaviour
The normal .where() operation is applied on the DataArray instead of using the shortcut described above.
Question
What would be required to improve the performance of fillna()? I'm happy to try taking a stab at it if pointed in the right direction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DataArray .fillna() entry point and trace the normal .where() operation for sparse arrays with fill_value=nan. Confirm that .fillna(0) can update the sparse fill_value directly, then verify that the operation avoids the slower path while preserving the expected values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100