apache / apache/iggy

Python SDK: expose consumer shutdown and offset drain timeout

Open
#4,165 0 comments 0 reactions 0 assignees View on GitHub
good first issue python
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

The Rust high-level consumer exposes an asynchronous `shutdown()` method that drains pending offset work, performs final commits when enabled, leaves the consumer group, and stops its connection watcher.

The Python `IggyConsumer` currently has no corresponding method. Its `consumer_group()` builder also does not expose the Rust builder's `offset_drain_timeout`, which controls how long shutdown waits for background offset tasks.

This covers B22 and the currently applicable part of B23 in the Python SDK parity work tracked in #3893.

The `commit_failed_messages` setting listed in B23 is not present in the current Rust SDK and should not be added to the Python API.

### Affected area / component

Python SDK

### Proposed solution

- Add `IggyConsumer.shutdown()` as an asynchronous method that locks the wrapped Rust consumer, calls its `shutdown()`, and maps any error to `RuntimeError`.
- Preserve the Rust method's idempotent behavior.
- Add an optional `offset_drain_timeout: datetime.timedelta | None` argument to `IggyClient.consumer_group()`.
- When provided, convert it to `IggyDuration` and pass it to `IggyConsumerBuilder::offset_drain_timeout()`.
- When omitted, preserve the Rust builder default of five seconds.
- Regenerate `foreign/python/apache_iggy.pyi` and document the supported shutdown call order.
- Extend `foreign/python/tests/test_consumer_group.py` to verify:
- A consumer can be shut down after its consumption loop exits.
- Repeated shutdown calls succeed.
- A custom offset-drain timeout is accepted and propagated.
- Shutdown leaves a consumer group promptly so another member can receive its partitions.

Adding a replacement for the removed `commit_failed_messages` option is outside this issue.

### Alternatives considered

Continue relying only on the `asyncio.Event` accepted by `consume_messages()`. That event stops one consumption loop but does not expose the Rust consumer's resource and offset cleanup operation.

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [x] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Research direction

Start with the Python SDK's IggyConsumer and IggyClient.consumer_group() entry points, then compare the Rust high-level consumer shutdown() and offset_drain_timeout() behavior. Review foreign/python/tests/test_consumer_group.py and regenerate foreign/python/apache_iggy.pyi. Done means shutdown is idempotent, timeout propagation and prompt group departure are covered, and the supported shutdown call order is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.