apache / apache/gluten

[VL] Track Jvm based libhdfs performance gap issue

Open
#7,746 20 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently, we supported JVM-based libhdfs in both the [Velox ](https://github.com/facebookincubator/velox/pull/9835)and [Gluten ](https://github.com/apache/incubator-gluten/pull/6172)communities. However, the performance in Jenkins tests has not met our expectations.

- Static linking with libhdfs3.so: 340%

- Dynamic linking with JVM: 214.75% (requires further investigation)

- Dynamic linking with libhdfs3: 323.85% (only replacing libhdfs.so with libhdfs3.so)

- Expected performance for static linking in [PR#7697](https://github.com/apache/incubator-gluten/pull/7697): 340%

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.