apache / apache/hugegraph-toolchain

loader导入parquet文件的建议

Open
#225 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
121
Forks
125
Avg merge
10d 10h
Merged PRs (30d)
3

Description

### Feature Description (功能描述)

目前loader读取parquet文件时,所有的列都读出来了,没有考虑field_mapping只用到了其中部分列的情况

例如如果原始表有 100 列, 但是实际入图只需要 10 列,全部读取再从里面去把需要的10列捞出来,这样发挥不出来 `parquet` 列存储的优势,读取很慢,而且传递的 map 消耗了许多不必要的内存

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the loader's Parquet-reading entry point and follow how field_mapping determines the columns needed for graph loading. Confirm the current path reads all columns, then define completion as reading only mapped columns while preserving the existing loader behavior and reducing unnecessary map data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.