[SUPPORT] Experiencing Hudi CloudWatchReporter Exception with Hudi 0.15.0
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I recently enabled metrics reporting from Hudi via Cloudwatch. When I did that, the job completes successfully but no metrics are published and in the logs I see the following error:
```
25/03/27 23:07:34 INFO CloudWatchReporter: Reporting Metrics to CloudWatch.
25/03/27 23:07:34 ERROR ScheduledReporter: Exception thrown from CloudWatchReporter#report. Exception was suppressed.
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.stageMetricDatum(CloudWatchReporter.java:281)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.lambda$processGauge$2(CloudWatchReporter.java:250)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.processGauge(CloudWatchReporter.java:250)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:189)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
**To Reproduce**
Steps to reproduce the behavior:
When running an upsert/delete job on EMR-6.15.0, with the below mentioned Hudi version and scalaVersion=2.12.19, enable reporting by enabling the following hudi options (as per documentation):
```
HoodieMetricsConfig.TURN_METRICS_ON.key() -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key() -> MetricsReporterType.CLOUDWATCH.toString,
```
Packages included:
```
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.15.0,org.apache.hudi:hudi-aws-bundle:0.15.0
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.15.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version : 2.7.5
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
NA
**Stacktrace**
```
25/03/27 23:07:34 INFO CloudWatchReporter: Reporting Metrics to CloudWatch.
25/03/27 23:07:34 ERROR ScheduledReporter: Exception thrown from CloudWatchReporter#report. Exception was suppressed.
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.stageMetricDatum(CloudWatchReporter.java:281)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.lambda$processGauge$2(CloudWatchReporter.java:250)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.processGauge(CloudWatchReporter.java:250)
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:189)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in CloudWatchReporter.java at stageMetricDatum, then trace the processGauge and report calls shown in the stack trace. Reproduce with Hudi 0.15.0 on Spark 3.4.1 using the listed CloudWatch metrics settings, and verify that the job publishes metrics without the ArrayIndexOutOfBoundsException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, scala, spark
- Domain
- cloud, data-engineering, observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100