deepseek-ai / deepseek-ai/3FS

[Ask For Help] metadata operations works but failed to write data into storage servers.

Open
#239 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

I have deployed 3FS in a production Kubernetes environment (RoCE RDMA container network). Components like Mgmtd and Meta appear normal, and the `list-nodes` command shows proper results. After mounting FUSE, metadata operations like directory listing work correctly, but data writes `echo hello > /xxxx/hello.txt` fail with the following errors. RDMA network connectivity between FUSE clients and Storage nodes has been verified

### list node output

![Image](https://github.com/user-attachments/assets/9748f64f-c403-4c58-8bd2-f26e46cc1f98)

### stderr
```
[2025-04-13T07:18:58.223072746+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops 0x7f1fbc792758 after #3 retries, elapsed time: 60001ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223156990+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] #1/1 Failed batchRead operation 0x7f1fbc7641f0 with id RequestId(0), routing target: ChainId(900100014)@ChainVersion(2)@RoutingInfoVersion(18):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000003-E0040000-00000000), user: Uid(0)/Gid(0), usercall: #1, status: StorageClient::NotAvailable(7005)
[2025-04-13T07:18:58.223172010+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] Finally collected 1/1 failed batchRead operations 0x7f1fbc77d768 issued by user Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223277021+00:00 hf3fs_fuse_main:235453 FuseOps.cc:1544 ERROR] hf3fs_read error, ino 0x000000000003e004, size 4096, off 0, err -7005
```
### stdout

![Image](https://github.com/user-attachments/assets/14528708-ef08-47aa-b569-96510cd33411)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the reported locations in StorageClientImpl.cc:1173 and :1637, then FuseOps.cc:1544, and reproduce the failure after mounting FUSE with `list-nodes` and the reported write command. Done means data writes complete successfully without the StorageClient::NotAvailable(7005) and hf3fs_read errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.