apache / apache/hudi

Why can't I obtain the metric data in Flink?

Open
#18,141 1 comment 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
When I was using Flink to write data to Hudi, I wanted to obtain the metric information during the writing process.
The only information I obtained in the Metric class is key = "sink_table.index.type", value = 6.Why can't we obtain the metric information?
flink1.20.3
hudi 1.1.1
**What you expected:**
I will eventually write the metric information to MySQL.
**Steps to reproduce:**
public class MetricNew extends AbstractUserDefinedMetricsReporter {

private static final Logger log = LogManager.getLogger(MetricNew.class);

private final ScheduledExecutorService exec =
Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r, "hudi-user-metrics-reporter");
t.setDaemon(true);
return t;
});

public MetricNew(Properties props, MetricRegistry registry) {
super(props, registry);
}

/**
* ⚠️ Flink 场景下必须自己调度 report
*/
@Override
public void start() {
// 延迟 10s 后开始,每 30s 打印一次
exec.scheduleAtFixedRate(this::report, 10, 30, TimeUnit.SECONDS);
log.info("MetricNew started, schedule metrics report");
}

/**
* 只做一件事:打印 registry 里已有的 metrics
*/
@Override
public void report() {
try {
if (getRegistry() == null || getRegistry().getGauges().isEmpty()) {
log.info("No gauges found in MetricRegistry");
return;
}

getRegistry().getGauges().forEach((key, value) -> {
Object v;
try {
v = value.getValue();
} catch (Exception e) {
v = "N/A";
}
log.info("[HUDI-METRIC] key=" + key + ", value=" + v);
});
} catch (Throwable t) {
log.error("MetricNew report failed", t);
}
}

@Override
public void stop() {
exec.shutdownNow();
log.info("MetricNew stopped");
}
}

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the AbstractUserDefinedMetricsReporter implementation and the MetricRegistry usage shown in MetricNew, then reproduce the report with Flink 1.20.3 and Hudi 1.1.1. Check why the registry exposes only sink_table.index.type and determine what configuration or integration is needed for the expected metrics to be available for writing to MySQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.