ingest v2: WAL truncation blocks persist/init_shards, causing ingest timeouts under load

Describe the bug

In ingest v2, the ingester's shard-positions gossip handler holds the ingester's global lock across an entire WAL-truncation pass, which blocks ingestion on that node while the pass runs.

EventSubscriber<ShardPositionsUpdate>::handle_event acquires the full ingester lock once (lock_fully) and holds it while it truncates/deletes every shard in the update.

persist and init_shards take that same lock. So the truncation pass is effectively a stop-the-world for that ingester, each truncate/delete performs WAL disk I/O, so on a node with many shards and/or a WAL on slower storage it can hold the lock for tens of seconds.

During that window:

• incoming persist requests queue behind the lock and (our case) exceed client ingest timeouts;

• a concurrent init_shards, auto-scale-up triggered by the same traffic surge, also blocks and times out, so the source cannot add capacity exactly when it needs to.

Steps to reproduce (if applicable)

This is a contention/timing issue, so it reproduces under load. Conditions that trigger it:

1. Run an ingester (ingest v2) that leads many shards, ideally with the WAL on slower storage.
2. Drive a sustained ingestion surge into a source so the control plane attempts to scale the source up while shard positions are actively gossiped (frequent ShardPositionsUpdate truncations).
3. Observe on that ingester: WAL lock-acquire warnings, ShardPositionsUpdate handler-budget warnings, failed to init shards ... request timed out.

Expected behavior

Background WAL cleanup driven by gossiped shard positions must not block ingestion for the duration of the whole pass. Ingest latency (and init_shards) should not spike to seconds/tens-of-seconds (up to 50 in our case) because a truncation pass is in progress. The cleanup should yield the ingester lock frequently enough that queued persist/init_shards calls can interleave, performing small units of work rather than the entire update.

Configuration:

quickwit:d33dd237f8250899c223bb2cd7997f005ed616fe

Start at EventSubscriber::handle_event and trace lock_fully, persist, and init_shards, then reproduce the contention under sustained ingestion with frequent shard-position updates. The fix is done when WAL cleanup yields the ingester lock between small units of work and persist/init_shards no longer time out during truncation; verify with the relevant ingest and concurrency tests.