(feat) Implement Iceberg REST Catalog Metrics API for Compute Audit Correlation
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Is your feature request related to a problem? Please describe.
Implement the Apache Iceberg REST Catalog Metrics API (`POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics`) to enable compute engines (Spark, Trino, Flink) to report query-level metrics back to the catalog. This enables correlation between catalog operations and compute-side query execution for comprehensive audit trails.
**Motivation:**
Currently, the Polaris catalog provides audit logging for catalog-level operations (table metadata access, namespace management, credential vending). However, **critical query-level metrics are not visible at the catalog layer**:
| Data Point | Available at Catalog? | Why Not? |
|------------|:---------------------:|----------|
| Rows read/written | No| Data flows directly from compute to storage |
| Query text (SQL) | No | Queries are composed in compute engines |
| Bytes processed | No | File scanning happens in compute engines |
| Query duration | No | Catalog only handles metadata lookups |
This gap prevents:
* Fine-grained audit trails (table → actual data access)
* Cost allocation by catalog/namespace/table based on actual usage
* Security forensics for data access patterns
* Compliance reporting with complete access records
### Describe the solution you'd like
**Proposed Solution:**
Implement the Iceberg REST Catalog Metrics endpoint as defined in the [Apache Iceberg REST Catalog OpenAPI Specification](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) (line 1316):
```
POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics
```
The endpoint accepts `ReportMetricsRequest` payloads containing either:
- **ScanReport**: For read operations (snapshot-id, filter, projected fields, scan metrics)
- **CommitReport**: For write operations (snapshot-id, sequence-number, operation, commit metrics)
Correlation with catalog events is achieved via `trace-id` passed in the `metadata` field of the metrics report.
**Acceptance Criteria:**
- [ ] Implement `POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics` endpoint
- [ ] Accept `ReportMetricsRequest` schema (ScanReport and CommitReport)
- [ ] Validate request against Iceberg OpenAPI specification
- [ ] Return HTTP 204 No Content on success (per Iceberg spec)
- [ ] Return standard `IcebergErrorResponse` on errors (400, 401, 403, 404, 5XX)
- [ ] Extract `trace-id` from `metadata` field for event correlation
- [ ] Emit audit event with metrics data for downstream correlation
- [ ] Support OAuth2 and Bearer token authentication (per Iceberg security schemes)
- [ ] Documentation updated with endpoint usage and examples
- [ ] Unit and integration tests added
### Describe alternatives you've considered
**1. Custom audit reporting endpoint**
Pros: Full control over schema and behavior
Cons: Non-standard; requires custom client implementations; not compatible with existing Iceberg clients
**2. Extend existing catalog events with compute metrics**
Pros: Single event stream
Cons: Catalog doesn't have access to compute-side metrics; would require invasive changes to compute engines
**3. External correlation via timestamp**
Current approach: Join audit logs and compute logs by time window
Cons: Non-deterministic; fails with concurrent requests; complex queries; no guaranteed correlation
**4. Use AWS CloudTrail/S3 access logs for correlation**
Pros: Captures actual S3 access
Cons: Requires AWS STS session tags (see #3325); doesn't capture Iceberg-specific metrics like snapshot-id, filter expressions
### Additional context
**Dependencies:**
* Requires compute engines (Spark, Trino, Flink) to implement metrics reporting via their respective listener interfaces
* Compute engines must propagate `trace-id` from catalog responses to metrics reports
**Related:**
* [Apache Iceberg REST Catalog OpenAPI Specification](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) - Line 1316
* #3325 - AWS STS Session Tags for CloudTrail Correlation (complementary feature for S3-level audit)
* `INCLUDE_PRINCIPAL_NAME_IN_SUBSCOPED_CREDENTIAL` feature flag
**Example ScanReport Payload:**
```json
{
"report-type": "scan-report",
"table-name": "analytics.user_events",
"snapshot-id": 3497810964824022504,
"filter": { "type": "eq", "term": "event_date", "value": "2025-12-22" },
"schema-id": 1,
"projected-field-ids": [1, 2, 3, 5],
"projected-field-names": ["id", "user_id", "event_type", "timestamp"],
"metrics": {
"total-planning-duration": { "count": 1, "time-unit": "nanoseconds", "total-duration": 2644235116 },
"result-data-files": { "unit": "count", "value": 47 },
"total-file-size-bytes": { "unit": "bytes", "value": 5368709120 }
},
"metadata": {
"trace-id": "abc123def456789012345678901234ab",
"compute-engine": "spark-3.5.0",
"cluster-id": "emr-cluster-abc123"
}
}
```
**Example CommitReport Payload:**
```json
{
"report-type": "commit-report",
"table-name": "analytics.user_events",
"snapshot-id": 3497810964824022505,
"sequence-number": 42,
"operation": "append",
"metrics": {
"total-duration": { "count": 1, "time-unit": "nanoseconds", "total-duration": 1523456789 },
"added-data-files": { "unit": "count", "value": 12 },
"added-records": { "unit": "count", "value": 1500000 }
},
"metadata": {
"trace-id": "abc123def456789012345678901234ab",
"compute-engine": "spark-3.5.0"
}
}
```
Contributor guide
Research direction
Start with the Apache Iceberg REST Catalog OpenAPI Specification linked in the issue and the proposed POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics entry point. Review the ScanReport and CommitReport requirements and acceptance criteria; done means the endpoint, validation, authentication, audit correlation, documentation, and unit and integration tests are all covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100