Add IBM Storage Scale (IBM_SCALE) to NIXL backend support and fix NIXL transfer performance
- Dominant language
- Python
- Stars
- 11.9k
- Forks
- 1.9k
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 141
Description
**Label**
onboarding, enhancement, storage
**Summary**
Add IBM Storage Scale (`IBM_SCALE`) as a supported NIXL file backend in LMCache,
and fix several latency bugs in the shared NIXL transfer path that are exposed by fast
`io_uring`-backed backends. The companion upstream plugin PR is tracked at
[ai-dynamo/nixl#2134](https://github.com/ai-dynamo/nixl/issues/2134).
**Details**
IBM Storage Scale (formerly GPFS) is a high-performance parallel filesystem widely used
in AI/ML infrastructure. A new `IBM_SCALE` backend is being contributed to the upstream
NIXL project. On the LMCache side, several changes are needed:
- Allowlist gap — `_VALID_DYNAMIC_BACKENDS`, `_VALID_NIXL_BACKENDS`, and
`_FILE_BACKENDS` do not include `IBM_SCALE`, so LMCache refuses to instantiate it even
when the plugin is installed.
- 10 ms polling floor — `post_non_blocking` and `_post_non_blocking` unconditionally
sleep 10 ms between `check_xfer_state` calls. `io_uring`-backed transfers (IBM_SCALE,
GDS) typically complete in <1 ms; this sleep dominates TTFT.
- Sequential store path — `_execute_store_in_the_loop` awaits each file's DMA write
one at a time, preventing the `io_uring` submission queue from being saturated. The load
path already uses `asyncio.gather`; the store path should match.
- Wrong open flags — `_open_flags()` opens files `O_RDWR` for both reads and writes.
The IBM_SCALE plugin infers write intent from the fd access mode to select the correct
GPFS prefetch hint; `O_RDWR` is ambiguous and causes the wrong hint to fire. `O_WRONLY`
on store and `O_RDONLY` on load is also better POSIX practice (principle of least
privilege).
- `os.stat` under the lock — `_secondary_lookup_locked` calls `os.stat` while holding
`_lock`, blocking all concurrent store and load operations on slow or remote filesystems.
**Steps / Reproduction (if applicable)**
1. Install the `IBM_SCALE` NIXL plugin (from [ai-dynamo/nixl#2134](https://github.com/ai-dynamo/nixl/issues/2134)).
2. Configure LMCache with `backend = "IBM_SCALE"` in a `DynamicNixlStoreL2AdapterConfig`.
3. LMCache raises `ValueError: backend must be one of ('GDS', 'GDS_MT', 'POSIX', 'HF3FS')` before any transfer is attempted.
4. Even after patching the allowlist, observe that TTFT is dominated by the 10 ms sleep floor in `_post_non_blocking` for transfers that complete in <1 ms.
**Expected Outcome / Goal**
- `IBM_SCALE` is accepted as a valid backend in both the static and dynamic NIXL adapters.
- The NIXL polling loop adds minimal overhead for fast backends: spin-check first, then fall back to a 1 ms sleep.
- Store batches are issued concurrently via `asyncio.gather`, matching the load path.
- Files are opened with the minimal required access mode (`O_WRONLY` / `O_RDONLY`).
- Secondary lookup does not hold `_lock` during filesystem calls.
**Actual Outcome (if applicable)**
- `IBM_SCALE` is rejected at config validation time.
- When the allowlist is patched manually, TTFT is inflated by ~10 ms per transfer due to the unconditional sleep.
- Store throughput is limited by sequential DMA writes despite the backend supporting concurrent submission.
**Additional Context**
- Upstream NIXL tracking issue: [ai-dynamo/nixl#2134](https://github.com/ai-dynamo/nixl/issues/2134)
- Performance improvements (polling fix, concurrent store, open flags, lock-free stat) benefit _all_ fast NIXL backends, not just IBM_SCALE — GDS users will also see reduced latency.
- This issue tracks the LMCache side of the work; the NIXL plugin itself is contributed separately.
Contributor guide
Research direction
Start at the NIXL adapter entry points and the named functions: _VALID_DYNAMIC_BACKENDS, _VALID_NIXL_BACKENDS, _FILE_BACKENDS, post_non_blocking, _post_non_blocking, _execute_store_in_the_loop, _open_flags(), and _secondary_lookup_locked. Trace the load path's asyncio.gather usage and existing backend validation and transfer tests. Done means IBM_SCALE is accepted, fast polling and concurrent stores work, access flags are correct, and filesystem stat calls do not hold the lock.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100