apache / apache/iceberg

Core: Add table-level filtering for MetricsReporter implementations

Open
#16,573 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Add a built-in mechanism that lets users restrict which tables' `ScanReport`s and `CommitReport`s are forwarded to a configured `MetricsReporter`, applied uniformly across all reporter implementations (`LoggingMetricsReporter`, `RESTMetricsReporter`, `OtelMetricsReporter`, and custom user-supplied ones).

#### Motivation

In deployments with many tables, users frequently want to emit metrics for only a subset:

- Only tables in production databases (e.g., `prod.*`), not staging or sandbox
- Only specific business-critical tables, excluding intermediate ones
- Exclude noisy test or scratch tables (`tmp.*`, `*.bench_*`)

Existing per-reporter knobs only partially address this. The `iceberg.otel.metrics.attributes` allowlist added in #16250 controls *which attributes* an OTel metric carries — useful for cardinality but does not stop metrics from being emitted for tables the user doesn't care about. Cardinality-control mechanisms in time-series backends (OTel Views, Prometheus relabel rules, etc.) are reporter-specific and require host-side knowledge.

Table-level filtering is a cross-cutting concern that belongs above any single reporter. Putting it inside each reporter implementation would lead to repeated, slightly inconsistent flag sets per reporter. Putting it once in the framework layer means every existing and future `MetricsReporter` benefits without re-implementation.

#### Proposal

Introduce two catalog properties recognized by the catalog when constructing the reporter pipeline:

```
metrics-reporter-impl=org.apache.iceberg.metrics.OtelMetricsReporter
metrics-reporter.table-name.include=prod\..*
metrics-reporter.table-name.exclude=.*\.tmp_.*
```

Values are Java regex patterns matched against `ScanReport.tableName()` / `CommitReport.tableName()`. The catalog wraps the user's reporter in a filtering layer when either property is present. When both are present, `exclude` wins over `include` (an explicit deny overrides an include). When neither is set, behavior is identical to today (pass-through, with no runtime overhead).

#### Behavior

- `include` only set: forward reports whose table name matches; drop others.
- `exclude` only set: drop reports whose table name matches; forward others.
- Both set: drop if `exclude` matches; otherwise forward only if `include` matches.
- Neither set: forward everything (current behavior).
- Empty value (`metrics-reporter.table-name.include=`) is treated as "not set" rather than "match nothing" to avoid accidentally silencing all metrics on misconfiguration.

#### Relationship to existing work

- #16169, #16250 — surfaced this concern during discussion of per-table cardinality of the OTel reporter. This proposal complements `iceberg.otel.metrics.attributes` (attribute pruning) by giving users a way to also drop entire reports for uninteresting tables.
- dev@ DISCUSS for #16250: https://lists.apache.org/thread/vn4gglocg2g40p69mfrrh86qzkn1rr4b

### Query engine

None — applies to all engines that consume `MetricsReporter`.

### Willingness to contribute

- [X] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start at the catalog's reporter-construction pipeline and trace the MetricsReporter interface with ScanReport.tableName() and CommitReport.tableName(). Review LoggingMetricsReporter, RESTMetricsReporter, and OtelMetricsReporter to understand the shared boundary. Done means include and exclude properties apply uniformly, empty values pass through, exclude wins, and no-filter behavior remains unchanged across built-in and custom reporters.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.