apache / apache/gluten

[CORE][VL] Use protobuf to pass metrics from C++ to Java

Open
#9,637 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

The current way to fetch metrics from Velox task to Java is in way complicated:

https://github.com/apache/incubator-gluten/blob/e436fa4899eee052a5ed019542d8ce5a4183ffd5/cpp/core/jni/JniWrapper.cc#L497-L536

https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala

https://github.com/apache/incubator-gluten/blob/main/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala

The long array stores a traversal of the metric tree Velox returns, then we actually tend to reconstruct the tree from the traversal array in Java side.

To simplify code, we can store the metrics returned by Velox directly in a structured object (perhaps, using protobuf), then perform Velox-to-Gluten metrics conversion from Java side in one go.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.