redpanda-data / redpanda-data/console
Message count estimation on compacted topics
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 4.3k
- Forks
- 432
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 40
Description
We estimate the number of messages by calculating highWatermark - lowWatermark. This returns the correct number of messages for topics whose cleanup.policy is set to delete but it's not reliable and possibly very wrong for compacted topics.
The only way to get the exact number of messages per topic and per partition is to actually consume all these messages. Since this may take too long we elaborate other options:
Idea 1: Estimating the number of messages by using the partition size
We know each partitions log dir size as this can be queried via the Kafka API. The idea is to consume some messages in order to calculate a representative average message size and use that along with the partition size to calculate the number of messages in the compacted topic.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing how topic message counts are currently calculated, then review the Kafka API data available for partition log sizes and compacted topics; done means providing a defensible count estimate without consuming every message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100