apache / apache/opendal

new feature: provide object level metrics

Open
#7,358 0 comments 2 reactions 0 assignees View on GitHub
enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Hi team, in the metrics layer, I found we collect and report two layers of metrics:
- operation level (metrics prefixed with `opendal_operation_`), which tracks at `Access` trait boundary
- http request level (metrics prefixed by `opendal_http_requests_`), which tracks each individual HTTP requests

Both tiers correspond to the internal mechanisms of the IO library, what's missing is a user-facing, object-level tier which captures the complete lifecycle of a user's API call (i.e., `read(file)`).

**Pain point**

Today, when a user calls op.read("large_file.bin") with concurrent(4) and chunk(8MB), OpenDAL internally splits this into multiple concurrent chunk reads. The metrics layer wraps each `Access::read()` invocation independently via `MetricsWrapper`, so a single user-level object download appears as N separate operation-level read metrics, each measuring a chunk. The `OperationDurationSeconds` reflects chunk read time, not the end-to-end object download latency.

In other words, I think operation is an opendal internal concept, for different operations it "behaves differently".
- For multipart concurrent write, OperationDurationSeconds records the latency for the whole object upload
- For concurrent read, OperationDurationSeconds records one chunk read latency

**Propose to add object level metrics**

Object level metrics are important. Usually there're three layers in terms of IO path: library user, IO library (like opendal) and storage service (like S3), with each of belonging to different companies and departments.
The object level metrics and dashboards are useful and important to prove the effectiveness for IO library, and it's different from server-side metrics.

A few object-level metrics I would expect, which is similar to what we do for operation and http metrics
- counter for requests
- gauge for ongoing requests
- histogram for access latency
- counter for errors encounter

### Problem and Solution

"object" is a term commonly seen in object storage.
I understand the goal for this library is for "all storage backend", the object I refer to generally means "the object to access from opendal's APIs".

Now to get object-level metrics, I have to add a thin layer around opendal and collect metrics myself.

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the metrics layer around the Access trait boundary and MetricsWrapper, then compare it with the user-facing read API and HTTP request metrics. Clarify the lifecycle and semantics of object-level requests across chunked reads and multipart writes, and define the requested request, ongoing-request, latency, and error metrics before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.