apache / apache/hudi

[SUPPORT] Should we introduce partition-level metrics ?

Open
#12,197 4 comments 0 reactions 0 assignees View on GitHub
area:metrics type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Can we provide partition-level metrics? In many scenarios where partitions are used, such as `p_date` and `p_product`, which are separated by time or type, the data before the partition is quite different. Can we provide a partition dimension metrics to reflect some metrics, for example, p99 latency of compaction operation for specified partition? This will help a lot when doing performance optimization.

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1.
2.
3.
4.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :

* Spark version :

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

No implementation files, tests, or entry points are identified. First clarify the desired partition-level metrics and their scope, including p99 compaction latency for partitions such as p_date and p_product, then locate the existing metrics and compaction paths; done means an agreed design and validated partition-dimension metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.