[Bug] Topic "peek" (admin/v2/persistent/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}) management API returns undocumented and inconsistent data when batch messages are accessed
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.10.3
### Minimal reproduce step
1. Produce a message containing a bytes-schema payload `abc123` to a persistent topic with a *non-batched* (`batchingEnabled=false`) producer.
2. Retrieve that message from the topic using the v2 admin API's `GET admin/v2/persistent/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}` functionality.
3. Observe that the returned payload exactly matches the string `abc123`.
4. Produce the exact same message with `batchingEnabled=true`.
5. Repeat step 2.
6. Observe that the returned payload no longer matches `abc123`.
### What did you expect to see?
Something I can use to extract individual messages from a batch. That could include any of the below ideas, or something totally different:
- (Easiest) documentation on how to parse a batch of messages client-side such that I can extract an individual message. I checked `PulsarApi.proto` against the batch data and nothing added up, so I suspect this is something internal, but may be wrong.
- A change to the API to support a parameter indicating batch message index, such that only messages with that index would be returned.
- Variable length HTTP headers indicating where each message content lies (offset and length) within the returned batch blob.
- A change to the API to return multipart HTTP responses, one per batch message.
### What did you see instead?
An undocumented blob of binary (I think this is a raw chunk of message data from the ledger) that looks like it contains some info re: properties/etc. up front, and then concatenated message+metadata entries for each message in the batch.
### Anything else?
Many of my proposed fixes break backwards compatibility, so this may be better suited as a feature request.
However, in the short term, I'd love to find a reference on how to extract individual messages from the batch in a non-Java environment. I control all my admin API accesses in my environment, so I can add parsing logic to those wrappers--I just need to know how to parse the data.
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the v2 admin API entry endpoint described in the issue and compare its non-batched and batched responses. Read PulsarApi.proto alongside the observed batch blob to determine whether it documents the format. Done requires either documenting reliable individual-message extraction or defining an agreed API response change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100