apache / apache/hudi

HoodieMetadataMetrics registers metric names without the table prefix, so CloudWatch cannot derive a Table dimension

Open
#19,507 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Describe the problem

`HoodieMetadataMetrics` registers gauges whose names do not carry the table prefix that every other Hudi metric carries, and in some cases carry no table name at all. This is the producer-side cause of the CloudWatch failures in #12182 / #13051.

`HoodieMetadataMetrics` receives a `HoodieMetricsConfig` and discards it:

```java
public HoodieMetadataMetrics(HoodieMetricsConfig metricsConfig, HoodieStorage storage, boolean detailedMetricsEnabled) {
this.metrics = Metrics.getInstance(metricsConfig, storage); // config not retained
```

so `setMetric`/`incrementMetric` call `metrics.registerGauge(action, value)` directly, bypassing the `hoodie.metrics.reporter.metricsname.prefix` that `Metrics#registerGauges` applies. Two consequences:

**1. Names with no dot at all** — nothing can derive a table from them:

- `lookup_meta_index_bloom_filters_file_count` — `BaseTableMetadata.java:212`, on the normal bloom-index read path
- `_bootstrap_error` — `HoodieBackedTableMetadataWriter.java:482`

**2. Names with a dot but no table** — a consumer that derives a table from the first segment gets the wrong answer:

- `.count` / `.totalDuration` — `HoodieMetadataMetrics.java:156-157`, giving `initialize.count`, `lookup_partitions.totalDuration`, ...
- `.{baseFileCount,logFileCount,totalBaseFileSizeInBytes,totalLogFileSizeInBytes}` — `HoodieMetadataMetrics.java:137-140`, giving `files.baseFileCount`, `column_stats.logFileCount`, ...

These are registered in practice — `TestJavaHoodieBackedMetadata` asserts the registry contains `initialize.count`, `files.baseFileCount` and friends.

## Why it matters beyond CloudWatch

`AWSGlueCatalogSyncClient`... (no) — `CloudWatchReporter` derives its `Table` dimension by splitting the metric name on the first dot, so category 1 cannot be reported at all and category 2 is reported under a wrong `Table`. CloudWatch bills per unique metric name plus dimension set, so the wrong-dimension metrics are billed as distinct custom metrics.

Other reporters are unaffected in behaviour but still emit inconsistent names.

## Suggested direction

Retain the `HoodieMetricsConfig` in `HoodieMetadataMetrics` and apply `getMetricReporterMetricsNamePrefix()` in `setMetric` and `incrementMetric`. That fixes both categories, for every reporter at once, in one place.

**This renames metrics**, so it breaks existing Graphite / Prometheus / JMX / Datadog dashboards and needs a release note — which is why #19476 deliberately did not do it. Prior fixes for the same symptom went producer-side (`1a5a9f7f03ec` [HUDI-4439] #6164, and `100e9ac47590` [HUDI-9068] #12873), so this is the direction with precedent.

## Context

Raised from review on #19476, which makes `CloudWatchReporter` stop losing an entire batch when it meets one of these names. That is the "stop the bleeding" half; this issue is the cause.

Credit to @voonhous for tracing the producer sites and the billing implication.

## Related

- #19476
- #12182, #13051

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in HoodieMetadataMetrics and inspect the metric registrations at the cited lines, then compare them with Metrics#registerGauges and the retained HoodieMetricsConfig prefix. Check BaseTableMetadata.java, HoodieBackedTableMetadataWriter.java, and TestJavaHoodieBackedMetadata for affected names. Done means registered names consistently include the table prefix, the registry assertions and reporter behavior are correct, and the metric rename has a release note.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.