ceph / ceph/ceph-nvmeof

Optimize ANA state transitions using a batch RPC to eliminate high IPC latency at scale

Open
#2,016 0 comments 0 reactions 1 assignee Claimed by @leonidc View on GitHub
Dominant language
Python
Stars
132
Forks
69
Avg merge
1d 9h
Merged PRs (30d)
52

Description

**Problem**
In multi-gateway Ceph NVMe-oF deployments operating at scale (e.g., 100+ subsystems, 8 gateways so 8 ANA groups per gateway ), setting or updating ANA states triggers an excessive number of JSON-RPC requests over the management socket to SPDK.

Currently, ANA states are updated on a per-subsystem, per-ANA-group basis. As a result, changing ANA states across 100 subsystems and 8 ANA groups requires 800 individual JSON-RPC invocations per failover/fencing event.

**Impact & Bottlenecks**
Severe Management Socket Overhead: Executing hundreds/thousands of sequential JSON-RPC calls introduces significant Unix domain socket context switches, JSON serialization/parsing overhead, and round-trip IPC delays.

Failover Timeout Risk: The IPC processing time alone can take 1 to 2+ seconds, consuming a massive portion of the time-sensitive failover window and increasing the risk of host NVMe io_timeout expirations.

**Proposed Solution**
Introduce a Batch SPDK RPC API:
Add a single batch RPC (e.g., nvmf_set_ana_states_all_subsystems) that accepts an array or map of ANA groups and target states in a single JSON-RPC payload.

Optimizations:
Inside the C implementation of the batch handler, check the current state against the requested state before initiating a transition
Iterate through all target subsystems internally in C memory during a single socket invocation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.