libp2p / libp2p/jvm-libp2p

dropLowPriority() doesn't reclaim state control parts, and queued subscriptions grow unbounded

Open
#519 2 comments 0 reactions 1 assignee View on GitHub

@lucassaldanha is already working on this.

Since Aug 14, 2026.

bug
Dominant language
Kotlin
Stars
366
Forks
85
Avg merge
2d 39m
Merged PRs (30d)
6

Description

Found this while soak testing 1.3.6-RC2 with Teku on a hoodi node. Not urgent, but I think there are two related problems with the outbound parts queue.

dropLowPriority() only frees BULK parts

DefaultGossipRpcPartsQueue.dropLowPriority() drops priorityPartLists[BULK_PRIORITY], so publish, ihave and iwant. Subscription, graft and prune parts live in STATE_CONTROL_PRIORITY and never get dropped.

Those parts still count towards estimateMaxSerializedSize(), which is what GossipRouter.trackSlowPeers() compares against slowPeerPendingBytesThreshold. So for a peer whose queue is mostly state control parts, the queue stays above the threshold, notifySlowPeer fires every slowPeerHeartbeatThreshold heartbeats, trimOutboundQueue drops the bulk parts (possibly none), and the queue never drops back under the threshold. The bound added in 5a68b18a doesn't cover those parts.

Queued subscriptions keep growing

addSubscription appends a new part every time, so subscribe/unsubscribe churn on the same topic just piles up. Teku churns attestation subnet subscriptions every epoch, so there's a steady stream of them. Only the final state per topic actually matters on the wire, so I'd expect the queue to collapse them rather than keep every transition. Keying subscriptions by topic and overwriting the status would bound this to the number of topics.

Numbers from heap histograms on the node (single Teku process, ~200 peers, process uptime in the header):

                            24.4h       28.6h       34.6h       48.2h
SubscriptionPart          213,214     241,118     314,541     444,718
IHavePart                 650,983     782,518     787,547     674,320
TTLSeenCache$TimedMessage 3,150,008   3,243,836   2,905,139   3,075,955
WBytes                    1,691,278   1,815,221   1,629,733   1,674,305

SubscriptionPart is the only one that just goes up, roughly 9.7k/hour. Everything else oscillates with gossip load. That's about 2,200 per peer at the moment so it's a long way off the 4 MiB threshold.

Nothing else looks off on the node. Post GC old gen floor is flat over 48h, direct buffers flat at ~230MB, and one DroppedRpcPartsException in two days, which is the slow peer path working as intended.

Both of these come from the size aware batching in 02410783 rather than the threshold default in 5a68b18a. Before batching, takeMerged() cleared the whole queue on every flush so nothing could be retained.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.