[Ask For Help] metadata operations works but failed to write data into storage servers.
- Dominant language
- C++
- Stars
- 10.2k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
I have deployed 3FS in a production Kubernetes environment (RoCE RDMA container network). Components like Mgmtd and Meta appear normal, and the `list-nodes` command shows proper results. After mounting FUSE, metadata operations like directory listing work correctly, but data writes `echo hello > /xxxx/hello.txt` fail with the following errors. RDMA network connectivity between FUSE clients and Storage nodes has been verified
### list node output

### stderr
```
[2025-04-13T07:18:58.223072746+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops 0x7f1fbc792758 after #3 retries, elapsed time: 60001ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223156990+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] #1/1 Failed batchRead operation 0x7f1fbc7641f0 with id RequestId(0), routing target: ChainId(900100014)@ChainVersion(2)@RoutingInfoVersion(18):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000003-E0040000-00000000), user: Uid(0)/Gid(0), usercall: #1, status: StorageClient::NotAvailable(7005)
[2025-04-13T07:18:58.223172010+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] Finally collected 1/1 failed batchRead operations 0x7f1fbc77d768 issued by user Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223277021+00:00 hf3fs_fuse_main:235453 FuseOps.cc:1544 ERROR] hf3fs_read error, ino 0x000000000003e004, size 4096, off 0, err -7005
```
### stdout

Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the reported locations in StorageClientImpl.cc:1173 and :1637, then FuseOps.cc:1544, and reproduce the failure after mounting FUSE with `list-nodes` and the reported write command. Done means data writes complete successfully without the StorageClient::NotAvailable(7005) and hf3fs_read errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100