apache / apache/hudi

[SUPPORT] - Hudi metrics for commitFreshnessInMS commitLatencyInMs not populating in cloudwatch

Open
#8,427 0 comments 0 reactions 0 assignees View on GitHub
area:aws area:metrics priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

Number of documented [metrics](https://hudi.apache.org/docs/metrics/#list-of-metrics) aren't showing up in cloudwatch when the documented configuration is added.

**To Reproduce**

Steps to reproduce the behavior:

1. Added config to produce metrics
2. Checked cloudwatch for metric (i.e commitFreshnessInMS commitLatencyInMs) but metric is not produced.

Here are the metrics we do see populated.

![image](https://user-images.githubusercontent.com/89083438/231150466-126a720e-f269-47a1-8c15-2955b4dfdc0c.png)
![image](https://user-images.githubusercontent.com/89083438/231150492-4041654b-7b89-433e-818a-377ef48ee211.png)
![image](https://user-images.githubusercontent.com/89083438/231150506-2e13edef-c9f9-443e-8f92-20711a1b37ef.png)

**Expected behavior**

Added config to produce the metrics documented.

**Environment Description**

* Hudi version : 0.12.1

* EMR version : 6.8.9

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Configuration**
We've added the hoodie.payload.event.time.field config and relevant metrics configuration as shown below

``` "--hoodie-conf", "hoodie.database.name={{ database }}_raw",
"--hoodie-conf", "hoodie.table.name={{ table }}",
"--hoodie-conf", "hoodie.datasource.write.recordkey.field={{ primary_key }}",
"--hoodie-conf", "hoodie.datasource.write.precombine.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.payload.event.time.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator",
"--hoodie-conf", "hoodie.datasource.write.partitionpath.field={{ timestamp_field }}",
"--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS",
"--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd",
"--hoodie-conf", "hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true",
"--hoodie-conf", "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer",
# AWS Glue Data Catalog config
"--hoodie-conf", "hoodie.datasource.hive_sync.enable=true",
"--hoodie-conf", "hoodie.datasource.hive_sync.database={{ database }}_raw",
"--hoodie-conf", "hoodie.datasource.hive_sync.table={{ table }}",
"--hoodie-conf", "hoodie.datasource.hive_sync.partition_fields=_event_date",
# Hudi Metrics
"--hoodie-conf", "hoodie.metrics.on=true",
"--hoodie-conf", "hoodie.metrics.reporter.type=CLOUDWATCH",
```
When checking the commit log we see that the minAndMaxEventTime is not being populated:

```
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalLogFilesCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 1768,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Hudi metrics documentation and the CLOUDWATCH reporter configuration described in the issue; then trace how commitFreshnessInMS and commitLatencyInMs relate to the missing minAndMaxEventTime value in the commit log. Reproduce on Hudi 0.12.1 with EMR 6.8.9 and S3, and confirm the documented metrics appear in CloudWatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.