[SUPPORT] - Hudi metrics for commitFreshnessInMS commitLatencyInMs not populating in cloudwatch
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Number of documented [metrics](https://hudi.apache.org/docs/metrics/#list-of-metrics) aren't showing up in cloudwatch when the documented configuration is added.
**To Reproduce**
Steps to reproduce the behavior:
1. Added config to produce metrics
2. Checked cloudwatch for metric (i.e commitFreshnessInMS commitLatencyInMs) but metric is not produced.
Here are the metrics we do see populated.



**Expected behavior**
Added config to produce the metrics documented.
**Environment Description**
* Hudi version : 0.12.1
* EMR version : 6.8.9
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Configuration**
We've added the hoodie.payload.event.time.field config and relevant metrics configuration as shown below
``` "--hoodie-conf", "hoodie.database.name={{ database }}_raw",
"--hoodie-conf", "hoodie.table.name={{ table }}",
"--hoodie-conf", "hoodie.datasource.write.recordkey.field={{ primary_key }}",
"--hoodie-conf", "hoodie.datasource.write.precombine.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.payload.event.time.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator",
"--hoodie-conf", "hoodie.datasource.write.partitionpath.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS",
"--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd",
"--hoodie-conf", "hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true",
"--hoodie-conf", "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer",
# AWS Glue Data Catalog config
"--hoodie-conf", "hoodie.datasource.hive_sync.enable=true",
"--hoodie-conf", "hoodie.datasource.hive_sync.database={{ database }}_raw",
"--hoodie-conf", "hoodie.datasource.hive_sync.table={{ table }}",
"--hoodie-conf", "hoodie.datasource.hive_sync.partition_fields=_event_date",
# Hudi Metrics
"--hoodie-conf", "hoodie.metrics.on=true",
"--hoodie-conf", "hoodie.metrics.reporter.type=CLOUDWATCH",
```
When checking the commit log we see that the minAndMaxEventTime is not being populated:
```
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalLogFilesCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 1768,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Hudi metrics documentation and the CLOUDWATCH reporter configuration described in the issue; then trace how commitFreshnessInMS and commitLatencyInMs relate to the missing minAndMaxEventTime value in the commit log. Reproduce on Hudi 0.12.1 with EMR 6.8.9 and S3, and confirm the documented metrics appear in CloudWatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100