Handle basic sparsity
Open
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Many data sets can be extremely sparse, and currently datatable has no form of compression. Even a simple compression or handle of sparsity would make dt much more efficient. Some datasets can't even be on disk or memory at all because they are so large in dense representation.
Then DAI could treat this as sparse, do special sparse aware engineering if using mostly dt for operations, and then pass along a scipy sparse representation to xgboost, lightgbm, scikit routines.
Contributor guide
Assessment
This issue has not been assessed yet.