apache / apache/hudi

[SUPPORT] Experiencing Hudi CloudWatchReporter Exception with Hudi 0.15.0

Open
#13,051 1 comment 0 reactions 0 assignees View on GitHub
area:aws area:metrics
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I recently enabled metrics reporting from Hudi via Cloudwatch. When I did that, the job completes successfully but no metrics are published and in the logs I see the following error:

```
25/03/27 23:07:34 INFO CloudWatchReporter: Reporting Metrics to CloudWatch.
25/03/27 23:07:34 ERROR ScheduledReporter: Exception thrown from CloudWatchReporter#report. Exception was suppressed.
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.stageMetricDatum(CloudWatchReporter.java:281)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.lambda$processGauge$2(CloudWatchReporter.java:250)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.processGauge(CloudWatchReporter.java:250)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:189)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```

**To Reproduce**

Steps to reproduce the behavior:
When running an upsert/delete job on EMR-6.15.0, with the below mentioned Hudi version and scalaVersion=2.12.19, enable reporting by enabling the following hudi options (as per documentation):
```
HoodieMetricsConfig.TURN_METRICS_ON.key() -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key() -> MetricsReporterType.CLOUDWATCH.toString,
```

Packages included:
```
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.15.0,org.apache.hudi:hudi-aws-bundle:0.15.0
```

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.4.1

* Hive version :

* Hadoop version : 2.7.5

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional context**

NA

**Stacktrace**

```
25/03/27 23:07:34 INFO CloudWatchReporter: Reporting Metrics to CloudWatch.
25/03/27 23:07:34 ERROR ScheduledReporter: Exception thrown from CloudWatchReporter#report. Exception was suppressed.
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.stageMetricDatum(CloudWatchReporter.java:281)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.lambda$processGauge$2(CloudWatchReporter.java:250)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.processGauge(CloudWatchReporter.java:250)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:189)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in CloudWatchReporter.java at stageMetricDatum, then trace the processGauge and report calls shown in the stack trace. Reproduce with Hudi 0.15.0 on Spark 3.4.1 using the listed CloudWatch metrics settings, and verify that the job publishes metrics without the ArrayIndexOutOfBoundsException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, scala, spark
Domain
cloud, data-engineering, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.