[Bug][MP] FSL2Adapter serialises every key inside a load batch (2.2x slower on 24 MiB chunks, measured)

Label

bug, performance, mp_mode, backend — also relevant to fs_connector.

---

Describe the bug

FSL2Adapter never parallelises the keys inside a submit_load_task batch. Each key's I/O
is awaited before the next begins, so a batch of N keys takes N sequential round trips.

lmcache/v1/distributed/l2_adapters/fs_l2_adapter.py, _execute_load:

for i, key in enumerate(keys):
    ...
    if self._use_odirect:
        num_read = await self._loop.run_in_executor(   # awaited in-loop
            None, self._read_with_odirect, file_path, dst_buf,
        )
        ...
        continue
    async with aiofiles.open(file_path, "rb") as f:
        num_read = await _async_readinto_full(f, dst_buf)   # awaited in-loop

Dispatching to run_in_executor does not help, because the result is awaited immediately.
The same shape appears in the store path (for key, obj in zip(keys, objects, strict=True),
which awaits exists → write → replace per key, i.e. three sequential round trips each)
and in the lookup path (for i, key in enumerate(keys) awaiting _key_exists_on_disk).

There is no outer concurrency to compensate. submit_load_task schedules exactly one
coroutine per batch, and the caller puts a whole request into one batch —
storage_controllers/prefetch_controller.py:931, "Step 7: submit load tasks per adapter":

for adapter_idx, bitmap in trimmed_plan.items():
    per_adapter_keys = bitmap.gather(request.keys)
    ...
    task_id = self._l2_adapters[adapter_idx].submit_load_task(
        per_adapter_keys, per_adapter_objs
    )

So all of a request's keys for a given adapter are serialised. Concurrency exists only
across adapters, which does nothing for the common single-fs-adapter deployment.

Measurements

lmcache bench l2 on XFS over a 4-drive NVMe RAID-0. The experiment holds total work
constant and moves concurrency from within a batch to across batches, so the only
variable is where the parallelism sits.

24 MiB keys (a real KV chunk: Qwen3-Coder-30B-A3B, chunk_size: 256, TP=1 → 98,304
B/token × 256 = 24 MiB), 32 keys total, use_odirect: true:

Layout	Lookup	Store	Load
32 keys, 1 batch	2.43 ms	82.84 ms	68.75 ms (11,202 MB/s)
4 keys × 8 batches	4.79 ms	79.11 ms	31.60 ms (24,308 MB/s)

Load is 2.2× slower when the same 32 keys arrive as one batch instead of eight.

Store is not materially affected at this size (1.05×) — it appears write-bandwidth-bound
on the device, so the serialisation is masked. Lookup is too small here to measure
meaningfully. The defect is in all three code paths, but only Load demonstrably suffers
at this size.

The mechanism is confirmed by a batch-size sweep at --in-flight 1 (256 KB keys), where
duration scales linearly with batch size — the signature of a serial loop:

--num-keys      1      4      8     16     32     64    128
duration     0.34   1.87   6.63   9.02  16.00  25.91  46.16   ms

128× the keys, 136× the duration.

Reproducing

# one batch of 32
lmcache bench l2 --l2-adapter '{"type":"fs","base_path":"/mnt/nvme/l2bench","use_odirect":true}' \
  --num-keys 32 --in-flight 1 --data-size-kb 24576 --l1-align-bytes 4096 --rounds 3

# eight batches of 4 -- identical total work
lmcache bench l2 --l2-adapter '{"type":"fs","base_path":"/mnt/nvme/l2bench","use_odirect":true}' \
  --num-keys 4  --in-flight 8 --data-size-kb 24576 --l1-align-bytes 4096 --rounds 3

Note lmcache bench l2 needs the native_storage_ops extension (it imports Bitmap);
NO_GPU_EXT=1 python setup.py build_ext --inplace is enough to get it.

Expected behavior

A batch API should exploit the batch. Loading N keys should overlap the N I/Os, as the
other adapters already do:

• nixl_store_l2_adapter builds one vectorised descriptor list and issues a single
  transfer;
• raw_block_l2_adapter uses dedicated ThreadPoolExecutors (_load_pool, _store_pool,
  _lookup_pool);
• s3_l2_adapter gathers per-key futures with asyncio.gather.

fs is the odd one out, and it is the default file-backed adapter.

The fix looks small: build the per-key coroutines and await asyncio.gather(*tasks)
instead of awaiting in the loop, with a bound on in-flight I/O so a large batch cannot
swamp the executor. Worth noting from the data that unbounded is not automatically better
— at 24 MiB keys, 1 key × 32 batches measured worse than 4 keys × 8 (65.72 ms vs
33.37 ms), so a concurrency limit belongs in the design rather than a bare gather.

Caveats

• Buffered numbers are page-cache reads, not disk. The same comparison with
  use_odirect: false shows a much larger spread (121.83 ms vs 19.09 ms, 6.4×), but the
  bench writes then reads 768 MiB on a host with ample RAM, so those figures are memory
  bandwidth. The O_DIRECT row above is the one that reflects storage, and 2.2× is the
  figure I would rely on.
• 3 measurement rounds, 1 warmup, single host. Enough to establish the effect and its
  rough size, not a precise characterisation.
• The 1 key × 32 O_DIRECT anomaly noted above suggests executor or queue-depth limits at
  high concurrency; I have not investigated it.
• I have not measured the end-to-end TTFT impact in a live MP deployment — only the
  adapter in isolation.

Environment

• LMCache v0.5.1.dev52, upstream/dev
• Ubuntu 26.04, kernel 7.0.0-28-generic
• torch 2.11.0+cu130, CUDA 13.0
• XFS on /dev/md0, software RAID-0 over 4 × Dell PM1735a MU 1.6 TB NVMe

Related

Same defect class as the in-process LocalDiskBackend, whose batched_get_blocking
defaulted to a serial for key in keys: get_blocking(key) and was given a thread pool
(#3961, hardened further in #4348). The MP interface removed that particular footgun — it
is batch-native, with no single-key get to loop over and a Bitmap result for partial
success — but FSL2Adapter's implementation reintroduces the behaviour internally.

Separately, #4364 proposes consolidating O_DIRECT handling and also concerns this file;
that is a different defect and a different fix.

Start in lmcache/v1/distributed/l2_adapters/fs_l2_adapter.py, especially _execute_load and the store and lookup loops; compare their awaits with the batched patterns in the other adapters. Reproduce the issue with the two lmcache bench l2 commands, then verify that per-key I/O overlaps within a batch with a bounded concurrency limit across load, store, and lookup paths.