apache / apache/gluten

[VL] Refine Documents about big features

Open
#9,004 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently our documents isn't clear enough, we need to go through all the big features and document what we have implemented, what's the limitation and to do next. The issue to track all the big feature list:

- iceberg (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxIceberg.md)
- UDF (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxUDF.md)
- port udf to c++
- partial project (https://github.com/apache/incubator-gluten/blob/main/docs/developers/PartialProject.md)
- Functions (https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-scalar-function-support.md)
- Operators (https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md)
- BHJ
- SMJ
- C2R/R2C
- Write
- Data types
- Scan
- in mem cache
- local ssd cache (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxLocalCache.md)
- background I/O
- Memory management and config
- Storage memory
- global memory
- dynamic off-heap sizing (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxDynamicSizingOffheap.md)
- rdd cache
- spill
- Config (https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md)
- pyspark
- Arrow UDF (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxUDF.md#pandas-udfs-aka-vectorized-udfs)
- microbenchmark(https://github.com/apache/incubator-gluten/blob/main/docs/developers/MicroBenchmarks.md)
- operator trace (https://github.com/apache/incubator-gluten/blob/main/docs/developers/QueryTrace.md)
- stagelevel resource (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxStageResourceAdj.md)
- shuffle
- hash shuffle
- sort shuffle
- Dictionary support
- Remote shuffle service
- Development
- Profile

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.