Why can't I obtain the metric data in Flink?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
When I was using Flink to write data to Hudi, I wanted to obtain the metric information during the writing process.
The only information I obtained in the Metric class is key = "sink_table.index.type", value = 6.Why can't we obtain the metric information?
flink1.20.3
hudi 1.1.1
**What you expected:**
I will eventually write the metric information to MySQL.
**Steps to reproduce:**
public class MetricNew extends AbstractUserDefinedMetricsReporter {
private static final Logger log = LogManager.getLogger(MetricNew.class);
private final ScheduledExecutorService exec =
Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r, "hudi-user-metrics-reporter");
t.setDaemon(true);
return t;
});
public MetricNew(Properties props, MetricRegistry registry) {
super(props, registry);
}
/**
* ⚠️ Flink 场景下必须自己调度 report
*/
@Override
public void start() {
// 延迟 10s 后开始,每 30s 打印一次
exec.scheduleAtFixedRate(this::report, 10, 30, TimeUnit.SECONDS);
log.info("MetricNew started, schedule metrics report");
}
/**
* 只做一件事:打印 registry 里已有的 metrics
*/
@Override
public void report() {
try {
if (getRegistry() == null || getRegistry().getGauges().isEmpty()) {
log.info("No gauges found in MetricRegistry");
return;
}
getRegistry().getGauges().forEach((key, value) -> {
Object v;
try {
v = value.getValue();
} catch (Exception e) {
v = "N/A";
}
log.info("[HUDI-METRIC] key=" + key + ", value=" + v);
});
} catch (Throwable t) {
log.error("MetricNew report failed", t);
}
}
@Override
public void stop() {
exec.shutdownNow();
log.info("MetricNew stopped");
}
}
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the AbstractUserDefinedMetricsReporter implementation and the MetricRegistry usage shown in MetricNew, then reproduce the report with Flink 1.20.3 and Hudi 1.1.1. Check why the registry exposes only sink_table.index.type and determine what configuration or integration is needed for the expected metrics to be available for writing to MySQL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100