mars-project / mars-project/mars

Use pydata/sparse to replace current SparseMatrix and SparseVector

Open
#256 0 comments 0 reactions 0 assignees View on GitHub
mod: tensor type: feature
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

Currently, the sparse operand's execution is driven by mars/lib/sparse in which SparseNDArray is the most important part, however, SparseNDArray only has two inheritance, SparseMatrix and SparseVector which represents two and one dimensional sparse data. SparseMatrix and SparseVector take advantage of `scipy.sparse` and `cupyx.scipy.sparse` to handle the sparse data which cannot be over two dimensions.

Maybe we can just introduce [pydata/sparse](https://github.com/pydata/sparse) to handle the sparse data beyond two dimensions. One downside is that this library does not support GPU yet.

Welcome to bring up other choices for sparse support.

Contributor guide

Open the contributing guide

Research direction

Start with mars/lib/sparse and inspect SparseNDArray, SparseMatrix, and SparseVector, including their use of scipy.sparse and cupyx.scipy.sparse. Review pydata/sparse for multidimensional support and its GPU limitation, then check the surrounding sparse execution paths. Done requires an agreed backend choice and a defined migration scope before implementation can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.