mars-project / mars-project/mars
Use pydata/sparse to replace current SparseMatrix and SparseVector
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Currently, the sparse operand's execution is driven by mars/lib/sparse in which SparseNDArray is the most important part, however, SparseNDArray only has two inheritance, SparseMatrix and SparseVector which represents two and one dimensional sparse data. SparseMatrix and SparseVector take advantage of `scipy.sparse` and `cupyx.scipy.sparse` to handle the sparse data which cannot be over two dimensions.
Maybe we can just introduce [pydata/sparse](https://github.com/pydata/sparse) to handle the sparse data beyond two dimensions. One downside is that this library does not support GPU yet.
Welcome to bring up other choices for sparse support.
Contributor guide
Research direction
Start with mars/lib/sparse and inspect SparseNDArray, SparseMatrix, and SparseVector, including their use of scipy.sparse and cupyx.scipy.sparse. Review pydata/sparse for multidimensional support and its GPU limitation, then check the surrounding sparse execution paths. Done requires an agreed backend choice and a defined migration scope before implementation can begin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100