h2oai / h2oai/datatable

Handle basic sparsity

Open
#1,644 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Many data sets can be extremely sparse, and currently datatable has no form of compression. Even a simple compression or handle of sparsity would make dt much more efficient. Some datasets can't even be on disk or memory at all because they are so large in dense representation.

Then DAI could treat this as sparse, do special sparse aware engineering if using mostly dt for operations, and then pass along a scipy sparse representation to xgboost, lightgbm, scikit routines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.