bytedance / bytedance/InfiniStore

[BUG] InfiniStore returns `Unexpected opcode: 129`

Open
#197 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
437
Forks
44
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
After setting up the InfiniStore server, I tried to run the `infinistore/example/client.py` to test whether the setup is successful. Then, the error encountered here is

```
[2026-01-08: 10:08:18.897] [infini] [info] Connecting to
[2026-01-08: 10:08:18.899] [infini] [info] found device irdma0
[2026-01-08: 10:08:18.900] [infini] [warning] [../rdma.cpp:100] RoCE choose user specified gid index 1
[2026-01-08: 10:08:18.900] [infini] [info] gid index 1, gid
[2026-01-08: 10:08:18.157] [infini] [debug] remote rdma_conn_info: psn: 0, qpn: 92, gid: 10.102.196.102, enum mtu: 3
[2026-01-08: 10:08:18.157] [infini] [debug] local rdma_conn_info: psn: 0, qpn: 91, gid: 10.102.196.102, enum mtu: 3 rdma connection: cpu -> cuda: 0
[2026-01-08: 10:08:18.404] [infini] [info] register mr done for base_ptr : 97375080146304
[2026-01-08: 10:08:18.405] [infini] [debug] read cache/allocated/commit msg request send 97374976098784
[2026-01-08: 10:08:18.405] [infini] [error] [../libinfinistoer.cpp:164] Unexpected opcode 129
```

The script above is run on the same node as the InfiniStore server, meaning the `server-ip` is actually the localhost IP. Then, to verify the RDMA connection is successful, `ib_write_bw` is also run and the benchmark results has the appropriate output as well. Therefore, the RDMA connection does not have any issue.

After the client has completed the request, the server log may/may not show the following log (Invalid header)

Image

**To Reproduce**

Install the InfiniStore by building from source
```
apt install libuv1-dev
apt install libflatbuffers-dev
apt install libspdlog-dev libfmt-dev
apt install ibverbs-utils libibverbs-dev
apt install libboost-dev libboost-stacktrace-dev
pip install --no-build-isolation -e .
pip install pre-commit
pre-commit install
```

Then, set up the InfiniStore server using

```
infinistore --manage-port 8088 --service-port 12345 --dev-name irdma0 --hint-gid-index 1 --link-type Ethernet --prealloc-size 2
```

Afterwards, run the test script provided in this repository after pointing the IP to the correct IP Address.

```
python3 infinistore/example/client.py
```

**System info**
- OS: Ubuntu 22.04
- GPU count and types: 1 machine with 8 * 6000 Ada
- Python version: 3.10.12
- InfiniStore version: pulled latest at `main` branch

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.