apache / apache/pulsar

[Bug] Compaction counters do not cover failures that happen before the raw reader is created

Open
#26,487 0 comments 0 reactions 1 assignee Claimed by @lhotari View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I reproduced the issue on the current `master` branch.

### User environment

- Broker version: 4.2.4
- Present on `master` (`9ba61bd95de`); line numbers below are `master`.

### Issue Description

`pulsar_compaction_succeed_count` and `pulsar_compaction_failed_count` do not account for the whole compaction attempt. `Compactor.compact()` creates the raw reader **outside** the block that records the counters:

```java
public CompletableFuture compact(String topic) {
return RawReader.create(pulsar, topic, COMPACTION_SUBSCRIPTION, false, false) // :58
.thenComposeAsync(this::compactAndCloseReader, scheduler);
}

private CompletableFuture compactAndCloseReader(RawReader reader) {
CompletableFuture promise = new CompletableFuture<>();
mxBean.addCompactionStartOp(reader.getTopic()); // :64
...
mxBean.addCompactionEndOp(reader.getTopic(), false); // :76
```

`Compactor.java:58-80` → `CompactorMXBeanImpl.addCompactionEndOp()` → `CompactionRecord.compactionFailedCount`.

Any failure of `RawReader.create()` — lookup, connection, authorization, topic-not-found, a client-side error — happens before `addCompactionStartOp`, so it increments **neither** counter. The only trace is the `Compaction failure.` WARN in `PersistentTopic`'s `currentCompaction.whenComplete(...)` (`PersistentTopic.java:4921` on `master`).

#### Why it matters

`pulsar_compaction_failed_count == 0` is routinely read as "no compaction attempt has failed", and it is used that way when triaging compaction problems. It does not mean that. A topic whose compaction never gets as far as opening a reader looks identical to a topic that is not being compacted at all — both show zero for both counters.

The same applies to any subclass that does work before delegating to `Compactor.compact()`.

### Error messages

```text
[] Compaction failure.
```
(WARN, from `PersistentTopic.java:4921`; no counter moves)

### Reproducing the issue

Analysis is from code:

1. Trigger compaction on a topic where `RawReader.create()` will fail — for example revoke the broker client's permission to subscribe, or point compaction at a topic that is being deleted concurrently.
2. Observe `pulsar_compaction_failed_count` for that topic stays at 0 while the run clearly failed.

### Additional information

Suggested fix: move `mxBean.addCompactionStartOp(topic)` ahead of `RawReader.create()` (the topic name is already available in `compact(String topic)`), and record a failure if the reader cannot be created. Alternatively add a distinct counter for setup failures so they are not silently invisible.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.