[VL] Refine Documents about big features
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Currently our documents isn't clear enough, we need to go through all the big features and document what we have implemented, what's the limitation and to do next. The issue to track all the big feature list:
- iceberg (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxIceberg.md)
- UDF (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxUDF.md)
- port udf to c++
- partial project (https://github.com/apache/incubator-gluten/blob/main/docs/developers/PartialProject.md)
- Functions (https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-scalar-function-support.md)
- Operators (https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md)
- BHJ
- SMJ
- C2R/R2C
- Write
- Data types
- Scan
- in mem cache
- local ssd cache (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxLocalCache.md)
- background I/O
- Memory management and config
- Storage memory
- global memory
- dynamic off-heap sizing (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxDynamicSizingOffheap.md)
- rdd cache
- spill
- Config (https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md)
- pyspark
- Arrow UDF (https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxUDF.md#pandas-udfs-aka-vectorized-udfs)
- microbenchmark(https://github.com/apache/incubator-gluten/blob/main/docs/developers/MicroBenchmarks.md)
- operator trace (https://github.com/apache/incubator-gluten/blob/main/docs/developers/QueryTrace.md)
- stagelevel resource (https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxStageResourceAdj.md)
- shuffle
- hash shuffle
- sort shuffle
- Dictionary support
- Remote shuffle service
- Development
- Profile
Contributor guide
Assessment
This issue has not been assessed yet.