apache / apache/hudi

[SUPPORT] Hudi Spark WEB UI [SQL / DataFrame] Details for Query, It does not display detailed indicator information

Open
#9,944 8 comments 0 reactions 0 assignees View on GitHub
area:metrics engine:spark type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

if I insert the hive table , spark web ui can display indicator information.

![image](https://github.com/apache/hudi/assets/16716138/0d171ac0-4bdf-4f8d-baa7-b8fc2a1ad597)

but if I insert hudi table , The DAG diagram is not expanded and it does not display detailed indicator information

![image](https://github.com/apache/hudi/assets/16716138/c7d32462-849c-4bad-9f68-ffef70f0b765)

**To Reproduce**

**Expected behavior**

**Environment Description**

* Hudi version :

master

* Spark version :
3.3.1

* Hive version :

3.1.3

* Hadoop version :
3.2.1

* Storage (HDFS/S3/GCS..) :
HDFS

* Running on Docker? (yes/no) :
No

**Additional context**
I change InsertIntoHoodieTableCommand super class to extends DataWritingCommand.
image

The DAG diagram is expanded. However, the other operator indicators are still not shown.

![image](https://github.com/apache/hudi/assets/16716138/4e4d6282-4dfa-4f20-81a1-2bc5ccbf7f46)

My question is that I can make the InsertIntoHoodieTableCommand node have metrics and just initialize the Metric. but why are other Nodes, such as Exchange not showing the metrics that they already have. if I can figure this out, I can mention a PR to add a feature.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Hudi and Hive insert comparison in Spark Web UI with Hudi master, Spark 3.3.1, and HDFS. Inspect InsertIntoHoodieTableCommand, DataWritingCommand, and the Exchange nodes to understand why the DAG and existing operator indicators differ. Done means the Hudi query expands in the UI and displays detailed indicators for the relevant operators.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.