apache / apache/iggy

Python SDK: expose the consumer offset APIs (store / get / delete)

Open
#3,997 0 comments 0 reactions 1 assignee Claimed by @ethanlin01x View on GitHub
python
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

The Python client has no consumer offset methods. `poll_messages` can commit through `auto_commit=True`, but on a schedule the caller does not control, and `IggyConsumer.store_offset()` only works for a consumer group built through `consumer_group()`. A plain `poll_messages` caller cannot commit an offset it chooses, and nothing in the binding can read a stored offset back. Every other SDK exposes the `ConsumerOffsetClient` methods on the client.

This blocks polling with `auto_commit=False` and committing after a batch is processed. It also makes consumer lag impossible to check from Python.

### Affected area / component

Python SDK

### Proposed solution

Add the three `ConsumerOffsetClient` methods to `IggyClient` in `foreign/python/src/client.rs`:

```python
await client.store_consumer_offset(
stream="s", topic="t", consumer=Consumer.Single("my-app"), offset=42, partition_id=0
)

info = await client.get_consumer_offset(
stream="s", topic="t", consumer=Consumer.Single("my-app"), partition_id=0
)
if info is not None:
print(info.partition_id, info.current_offset, info.stored_offset)

await client.delete_consumer_offset(
stream="s", topic="t", consumer=Consumer.Single("my-app"), partition_id=0
)
```

`partition_id` stays optional on all three, like on `poll_messages`, so a consumer group can resolve the member's assigned partition on the server.

`get_consumer_offset` needs a new `ConsumerOffsetInfo` pyclass with `partition_id`, `current_offset` and `stored_offset` getters. It returns `None` when the server has no offset for that consumer, matching `get_consumer_group`.

### Alternatives considered

_No response_

### Contribution

- [x] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.