Unable to Deploy 3FS Without RDMA – Consider Virtualizing an RDMA Network

In our standard network deployment environment, it is not possible to deploy **3FS** directly because **RDMA networking is mandatory**. Therefore, we can consider **virtualizing an RDMA network environment** to meet the requirements.

**Steps to Set Up a Virtual RDMA Environment**

​ 1. **Install Required Packages**

```
sudo apt update
sudo apt install -y rdma-core infiniband-diags ibverbs-utils
```

​ 2. **Load the rxe Kernel Module**

```
modprobe rdma_rxe
```

​ 3. **Verify That the Module Is Loaded**

```
lsmod | grep rxe
```

Expected output:

```
rdma_rxe 139264 0
ip6_udp_tunnel 16384 1 rdma_rxe
udp_tunnel 20480 1 rdma_rxe
ib_uverbs 163840 2 rdma_rxe,rdma_ucm
ib_core 393216 6 rdma_cm,rdma_rxe,iw_cm,rdma_ucm,ib_uverbs,ib_cm
```

​ 4. **Check Available Network Interfaces**

```
ip link show
```

Example output:

```
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:71:4b:d8 brd ff:ff:ff:ff:ff:ff
3: docker0: mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 3e:25:8e:73:c6:9a brd ff:ff:ff:ff:ff:ff
```

​ • **lo**: Loopback interface for local communication (127.0.0.1), not a physical NIC.

​ • **enp1s0**: The primary network interface (Ethernet), status **UP**, MAC address 52:54:00:71:4b:d8.

​ • **docker0**: Docker virtual bridge, currently **DOWN**, meaning no active container connections.

​ 5. **Add enp1s0 as an RDMA Device**

```
rdma link add rxe0 type rxe netdev enp1s0
```

​ 6. **Verify RDMA Device Creation**

```
rdma link show
```

If successful, you should see the new **rxe0** device, which serves as the virtual RDMA version of **enp1s0**.

**Run 3FS with the Virtualized RDMA Network**

Once the virtual RDMA environment is set up, you can proceed with starting **3FS**:

```
./bin/monitor_collector_main -cfg ./etc/monitor_collector_main.toml
```

This should now work as expected in an environment where physical RDMA hardware is unavailable.