apache / apache/hugegraph-toolchain
loader导入parquet文件的建议
Open
enhancement
- Dominant language
- Java
- Stars
- 121
- Forks
- 125
- Avg merge
- 10d 10h
- Merged PRs (30d)
- 3
Description
### Feature Description (功能描述)
目前loader读取parquet文件时,所有的列都读出来了,没有考虑field_mapping只用到了其中部分列的情况
例如如果原始表有 100 列, 但是实际入图只需要 10 列,全部读取再从里面去把需要的10列捞出来,这样发挥不出来 `parquet` 列存储的优势,读取很慢,而且传递的 map 消耗了许多不必要的内存
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the loader's Parquet-reading entry point and follow how field_mapping determines the columns needed for graph loading. Confirm the current path reads all columns, then define completion as reading only mapped columns while preserving the existing loader behavior and reducing unnecessary map data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100