fix(python): read consumer metadata without the consumer lock - #3888
Open
ethanlin01x wants to merge 2 commits into
Open
fix(python): read consumer metadata without the consumer lock#3888ethanlin01x wants to merge 2 commits into
ethanlin01x wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3888 +/- ##
============================================
- Coverage 83.84% 83.79% -0.05%
Complexity 1358 1358
============================================
Files 1212 1212
Lines 166843 166869 +26
Branches 134306 134454 +148
============================================
- Hits 139890 139830 -60
+ Misses 23312 23256 -56
- Partials 3641 3783 +142
🚀 New features to boost your workflow:
|
ethanlin01x
force-pushed
the
fix/python-consumer-metadata-deadlock
branch
from
August 15, 2026 09:28
8d0d07d to
b73cf67
Compare
ethanlin01x
marked this pull request as ready for review
August 15, 2026 09:41
Contributor
Author
|
/request-review @hubcio |
Contributor
Author
|
/ready |
ethanlin01x
force-pushed
the
fix/python-consumer-metadata-deadlock
branch
from
August 20, 2026 13:19
7b7a74e to
5763540
Compare
The synchronous getters on IggyConsumer took the consumer mutex with blocking_lock() while holding the GIL, and consume_messages holds that mutex for the whole consumption run. Reading an attribute during consumption hung the interpreter; reading one from a callback panicked inside the Tokio runtime. None of those getters need exclusive access. The name, stream and topic are fixed at construction, and the partition id and offsets live behind Arcs that IggyConsumerState now exposes as a cloneable view. IggyConsumer owns that state and delegates to it, so the Python wrapper reads metadata without the lock. Found while reviewing apache#3776.
ethanlin01x
force-pushed
the
fix/python-consumer-metadata-deadlock
branch
from
August 20, 2026 13:40
5763540 to
e9c641e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR address?
Relates to #3776 (Found while reviewing)
Rationale
The synchronous getters on the Python
IggyConsumertook the consumer mutex withblocking_lock()while holding the GIL, andconsume_messagesholds that mutex for the whole consumption run. Reading an attribute during consumption hung the interpreter; reading one from a callback panicked inside the Tokio runtime. Neither is recoverable from Python.What changed?
None of those getters need exclusive access.
name,streamandtopicnever change after construction, so the Python wrapper snapshots them. The partition id and offsets do change, but already live behindArcs in the Rust SDK, which now hands them out asIggyConsumerStateviaIggyConsumer::state(). The wrapper keeps a clone and reads them as atomic loads, never taking the lock.Additive on the Rust side. On the Python side
stream()andtopic()now return the identifier directly instead of aPyResult.Local Execution
AI Usage