deployment crashed at data placement part need expert to look into

Open

#416 opened on Jun 7, 2026 1 comment 0 reactions 0 assignees View on GitHub

Dominant language: C++
Stars: 10.2k
Forks: 1.1k
PR merge metrics: No merged PRs in 30d

Description

hello team

below is error

```
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml
DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:14500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName:
nodes:
3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry: Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout: ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s}
INFO[0000] Running task CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data: NODE=""
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts: NODE=""
INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql NODE=local TASK=CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0 NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: e166fa5bf7c46930cd28ac238f6d6a8cb1d42bff0938cf274fb75964865f8226 NODE=""
DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response NODE=""
DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2 NODE=local TASK=CreatePgClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections NODE=""
INFO[0001] Started postgresql container 3fs-postgres successfully NODE=local TASK=CreatePgClusterTask
INFO[0001] Running task CreateFdbClusterTask
DEBU[0001] fdb cluster file content: 3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data: NODE=""
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs: NODE=""
DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63 NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: 9bace2adf85cd6e1f1c5e6971171570ce8f864411ee588f6e3e0f89bb8e9f59e NODE=""
INFO[0001] Started fdb container 3fs-fdb successfully NODE=local TASK=CreateFdbClusterTask
INFO[0001] Initializing fdb cluster NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd' NODE=""
^C
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# vim cluster.yml
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml
DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:4500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName:3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry:Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout: ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s}
INFO[0000] Running task CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data: NODE=""
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts: NODE=""
INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql NODE=local TASK=CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0 NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: fa11a8f1cc8e28698f5616daae0c09694097d86744decce0f330ca763f54b849 NODE=""
DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE=""
DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response NODE=""
DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2 NODE=local TASK=CreatePgClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections NODE=""
INFO[0001] Started postgresql container 3fs-postgres successfully NODE=local TASK=CreatePgClusterTask
INFO[0001] Running task CreateFdbClusterTask
DEBU[0001] fdb cluster file content: 45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data: NODE=""
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs: NODE=""
DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63 NODE=""
DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: d11b3ff5ade923a38c7408a59c3e2322902d34678701143071ee0dc77fc04087 NODE=""
INFO[0001] Started fdb container 3fs-fdb successfully NODE=local TASK=CreateFdbClusterTask
INFO[0001] Initializing fdb cluster NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd' NODE=""
DEBU[0004] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd': Database created NODE=""
INFO[0004] Waiting for fdb cluster initialized NODE=local TASK=CreateFdbClusterTask
DEBU[0004] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal' NODE=""
DEBU[0005] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available, but has issues (type 'status' for more information). NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal' NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available. NODE=""
INFO[0010] Initialized fdb cluster NODE=local TASK=CreateFdbClusterTask
INFO[0010] Running task CreateClickhouseClusterTask
DEBU[0010] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX: /tmp/3fs-clickhouse.jCQ5nO NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql: NODE=""
DEBU[0010] Run command: sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy NODE=""
DEBU[0010] Output of sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy: e63b4ffd3e8cf6f8be7d72d88f40ffe8a3c80d9502fe5d83945e899613a77861 NODE=""
INFO[0015] Started clickhouse container 3fs-clickhouse successfully NODE=local TASK=CreateClickhouseClusterTask
INFO[0015] Initializing clickhouse cluster NODE=local TASK=CreateClickhouseClusterTask
DEBU[0015] Run command: sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql" NODE=""
DEBU[0015] Output of sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql": NODE=""
INFO[0015] Initialized clickhouse cluster NODE=local TASK=CreateClickhouseClusterTask
DEBU[0015] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO NODE=""
DEBU[0015] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO: NODE=""
INFO[0015] Running task CreateGrafanaServiceTask
INFO[0015] Generating datasource.yaml NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources NODE=""
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources: NODE=""
INFO[0015] Generating dashboard.yaml NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards NODE=""
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards: NODE=""
INFO[0015] Generating 3fs.json NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards NODE=""
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards: NODE=""
DEBU[0015] Run command: sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0 NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0: 849e83e1e65ee4aeb3f4e18efdf5c1cad63ba2ce9d562e07ee03c9d4145691cc NODE=""
INFO[0016] Started grafana container 3fs-grafana successfully, service endpoint is http://10.74.40.86:3000, login with username "admin" and password "admin" NODE=local TASK=CreateGrafanaServiceTask
INFO[0016] Running task CreateMonitorTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX: /tmp/3fs-monitor.stFhHX NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml: 475e4148833690ba401f2855e1742f3e8fb71bb2e375deea3bf6aefdbcc93acc NODE=""
INFO[0016] Started monitor container 3fs-monitor successfully NODE=local TASK=CreateMonitorTask
DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX: NODE=""
INFO[0016] Running task CreateMgmtdServiceTask
DEBU[0016] Node ID map: map[local:1] NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Admin cli config template data: map[ClusterID:open3fs-single MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]] NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.DoAci2 NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2 NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2: NODE=""
INFO[0016] Create mgmtd_main config dir /mnt/nvme3n1/3fs-lab/mgmtd NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd: NODE=""
DEBU[0016] Template data of mgmtd_main_app.toml.tmpl: map[NodeID:1] NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main_app.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_app.toml NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Config of mgmtd_main_app.toml: allow_empty_node_id = true
node_id = 1 NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Template data of mgmtd_main_launcher.toml.tmpl: map[ClusterID:open3fs-single HostMountpoint:/mnt/3fsmount MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]] NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main_launcher.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_launcher.toml NODE=local TASK=CreateMgmtdServiceTask

clusterFile = '/opt/3fs/etc/fdb.cluster'
default_backoff = 0
enableMultipleClient = false
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json'

[ib_devices]
allow_no_usable_devices = false
allow_unknown_zone = true
default_network_zone = 'UNKNOWN'
default_pkey_index = 0
default_roce_pkey_index = 0
default_traffic_class = 0
device_filter = []
fork_safe = true
prefer_ibdevice = true
skip_inactive_ports = true
skip_unusable_device = true
subnets = []

[kv_engine]
use_memkv = false

[kv_engine.fdb]
casual_read_risky = false
clusterFile = ''
default_backoff = 0
enableMultipleClient = false
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json' NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Template data of mgmtd_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8000 TCPListenPort:9000] NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main.toml NODE=local TASK=CreateMgmtdServiceTask

[common.monitor]
collect_period = '1s'
num_collectors = 1

[[common.monitor.reporters]]
type = 'monitor_collector'

[common.monitor.reporters.monitor_collector]
remote_ip = "10.74.40.86:10000"

[server.base.independent_thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.base.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[[server.base.groups]]
check_connections_interval = '1min'
connection_expiration_time = '1day'
network_type = 'RDMA'
services = [ 'Mgmtd' ]
use_independent_thread_pool = false

[server.service]
allow_heartbeat_from_unregistered = true
authenticate = false
bootstrapping_length = '2min'
bump_routing_info_version_interval = '5s'
check_status_interval = '10s'
client_session_timeout = '20min'
enable_routinginfo_cache = true
extend_lease_check_release_version = true
extend_lease_interval = '10s'
heartbeat_fail_interval = '1min'
heartbeat_ignore_stale_targets = true
heartbeat_ignore_unknown_targets = false
heartbeat_timestamp_valid_window = '30s'
lease_length = '1min'
new_chain_bootstrap_interval = '2min'
only_accept_client_uuid = false
retry_times_on_txn_errors = -1
send_heartbeat = true
send_heartbeat_interval = '10s'
suspicious_lease_interval = '20s'
target_info_load_interval = '1s'
target_info_persist_batch = 1000
target_info_persist_interval = '1s'
try_adjust_target_order_as_preferred = false
update_chains_interval = '1s'
update_metrics_interval = '1s'
validate_lease_on_write = true

[server.service.retry_transaction]
max_backoff = '1s'
max_retry_count = 10

[server.service.user_cache]
buckets = 127
exist_ttl = '5min'
inexist_ttl = '10s' NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Save admin cli config to /tmp/prepare-3fs-config.DoAci2/admin_cli.toml NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating fdb.cluster to /tmp/prepare-3fs-config.DoAci2/fdb.cluster NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Copying mgmtd_main configs from /tmp/prepare-3fs-config.DoAci2 to local /mnt/nvme3n1/3fs-lab/mgmtd/config.d NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2 NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log: NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16' NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16': Init filesystem, root directory layout: chain table ChainTableId(1), chunksize 1048576, stripesize 16

Init config for MGMTD version 1 NODE=""
INFO[0016] Cluster initialization success NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Starting mgmtd_main container 3fs-mgmtd NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml: cec202504223f22561a2b9a90f37f918146ca0389adb2ced6d7c79fcd6968934 NODE=""
INFO[0016] Started mgmtd_main container 3fs-mgmtd successfully NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX NODE=""
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX: /tmp/3fs-mgmtd.65wChW NODE=""
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW: NODE=""
DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh: NODE=""
DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW: NODE=""
INFO[0017] Running task CreateMetaServiceTask
DEBU[0017] Node ID map: map[local:100] NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.57VFbH NODE=""
DEBU[0017] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH: NODE=""
INFO[0017] Create meta_main config dir /mnt/nvme3n1/3fs-lab/meta NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta: NODE=""
DEBU[0017] Template data of meta_main_app.toml.tmpl: map[NodeID:100] NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating meta_main_app.toml to /tmp/prepare-3fs-config.57VFbH/meta_main_app.toml NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Config of meta_main_app.toml: allow_empty_node_id = true
node_id = 100 NODE=local TASK=CreateMetaServiceTask

[client.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[mgmtd_client]
accept_incomplete_routing_info_during_mgmtd_bootstrapping = true
auto_extend_client_session_interval = '10s'
auto_heartbeat_interval = '10s'
auto_refresh_interval = '10s'
enable_auto_extend_client_session = false
enable_auto_heartbeat = true
enable_auto_refresh = true
mgmtd_server_addresses = ["RDMA://10.74.40.86:8000"]
work_queue_size = 100 NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Template data of meta_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8001 TCPListenPort:9001] NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating meta_main.toml to /tmp/prepare-3fs-config.57VFbH/meta_main.toml NODE=local TASK=CreateMetaServiceTask

[server.background_client.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[[server.base.groups]]
check_connections_interval = '1min'
connection_expiration_time = '1day'
network_type = 'RDMA'
services = [ 'Me
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json'

[server.meta]
acl_cache_time = '15s'
allow_directly_move_to_trash = false
allow_owner_change_immutable = false
allow_stat_deleted_inodes = true
authenticate = false
batch_stat_by_path_concurrent = 4
batch_stat_concurrent = 8
check_file_hole = false
dynamic_stripe = false
dynamic_stripe_growth = 2
dynamic_stripe_initial = 16
enable_new_chunk_engine = false
grv_cache = false
idempotent_record_clean = '1min'
idempotent_record_expire = '30min'
idempotent_remove = true
idempotent_rename = false
iflags_chain_allocation = false
iflags_chunk_engine = true
ignore_length_hint = false
inodeId_abort_on_duplicate = false
inodeId_check_unique = true
list_default_limit = 128
max_batch_operations = 4096
max_directory_depth = 64
max_remove_chunks_per_request = 32
max_symlink_count = 10
max_symlink_depth = 4
operation_timeout = '5s'
otrunc_replace_file = true
otrunc_replace_file
rdma_connect_timeout = '5s'
read_write_rdma_in_event_thread = false
read_write_tcp_in_event_thread = false
tcp_connect_timeout = '1s'
wait_to_retry_send = '100ms'

[server.storage_client.net_client.io_worker.connect_concurrency_limiter]
max_concurrency = 4

[server.storage_client.net_client.io_worker.ibsocket]
buf_ack_batch = 8
buf_signal_batch = 8
buf_size = 16384
drain_timeout = '5s'
drop_connections = 0
event_ack_batch = 128
max_rd_atomic = 16
max_rdma_wr = 128
max_rdma_wr_per_post = 32
max_sge = 1
min_rnr_timer = 1
record_bytes_per_peer = false
record_latency_per_peer = false
retry_cnt = 7
rnr_retry = 0
send_buf_cnt = 32
sl = 0
start_psn = 0
timeout = 14

[server.storage_client.net_client.io_worker.transport_pool]
max_connections = 1

[server.storage_client.net_client.processor]
enable_coroutines_pool = true
max_coroutines_num = 256
max_processing_requests_num = 4096
response_compression_level = 1
response_compression_threshold = '128KB
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.read]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.remove]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.truncate]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.write]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true NODE=local TASK=CreateMetaServiceTask
INFO[0017] Save admin cli config to /tmp/prepare-3fs-config.57VFbH/admin_cli.toml NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating fdb.cluster to /tmp/prepare-3fs-config.57VFbH/fdb.cluster NODE=local TASK=CreateMetaServiceTask
INFO[0017] Copying meta_main configs from /tmp/prepare-3fs-config.57VFbH to local /mnt/nvme3n1/3fs-lab/meta/config.d NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d: NODE=""
DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH: NODE=""
INFO[0017] Upload meta_main main config NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml' NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml': Succeed
ConfigVersion 1 NODE=""
INFO[0017] Service meta_main main config uploaded NODE=local TASK=CreateMetaServiceTask
INFO[0017] Starting meta_main container 3fs-meta NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml NODE=""
DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml: 05d1527d43e491187ab14a66170e261b3bbb109aa78cb8fab5de0e8567900902 NODE=""
INFO[0017] Waiting for service meta_main active... NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE=""
DEBU[0018] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee
100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE=""
INFO[0020] Service meta_main is not active NODE=local TASK=CreateMetaServiceTask
INFO[0020] Started meta_main container 3fs-meta successfully NODE=local TASK=CreateMetaServiceTask
INFO[0020] Running task CreateStorageServiceTask
DEBU[0020] Node ID map: map[local:10001] NODE=local TASK=CreateStorageServiceTask
INFO[0020] Start to run script disk_tool.sh on node NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX: /tmp/remote-run-script.pwn6yQ NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ: NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage: NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.LO9bgmrBXL NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL: NODE=""
INFO[0020] Scp /tmp/remote-run-script.pwn6yQ/tmp_script.sh to /tmp/tmp.LO9bgmrBXL NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL: NODE=""
INFO[0020] Run disk_tool.sh with [/mnt/nvme3n1/3fs-lab/storage 1 dir prepare] NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare: + '[' 4 -lt 4 ']'
+ MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata
+ DISK_NUM=1
+ DISK_TYPE=dir
+ ACTION=prepare
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata
+ [[ dir == \n\v\m\e ]]
+ [[ prepare == \p\r\e\p\a\r\e ]]
+ prepare_dir_disk
++ seq 0 0
+ for i in $(seq 0 $((DISK_NUM-1)))
+ MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs NODE=""
DEBU[0020] Run disk_tool.sh output: + '[' 4 -lt 4 ']'
+ MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata
+ DISK_NUM=1
+ DISK_TYPE=dir
+ ACTION=prepare
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata
+ [[ dir == \n\v\m\e ]]
+ [[ prepare == \p\r\e\p\a\r\e ]]
+ prepare_dir_disk
++ seq 0 0
+ for i in $(seq 0 $((DISK_NUM-1)))
+ MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs NODE=local TASK=CreateStorageServiceTask
INFO[0020] Run disk_tool.sh success NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL: NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ: NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.2QLDXp NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp: NODE=""
INFO[0020] Create storage_main config dir /mnt/nvme3n1/3fs-lab/storage NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage NODE=""

[client.rdma_control]
max_concurrent_transmission = 64

[server.base.groups.io_worker.ibsocket]
buf_ack_batch = 8
buf_signal_batch = 8
buf_size = 16384
drain_timeout = '5s'
drop_connections = 0
event_ack_batch = 128
max_rd_atomic = 16
max_rdma_wr = 128
max_rdma_wr_per_post = 32
max_sge = 1
min_rnr_timer = 1
record_bytes_per_peer = false
record_latency_per_peer = false
retry_cnt = 7
rnr_retry = 0
send_buf_cnt = 32
sl = 0
start_psn = 0
timeout = 14

[server.base.groups.io_worker.transport_pool]
max_connections = 1

[server.base.groups.listener]
domain_socket_index = 1
filter_list = []
listen_port = 8002
listen_queue_depth = 4096
rdma_accept_timeout = '15s'
rdma_listen_ethernet = true
reuse_port = false

[server.base.groups.processor]
enable_coroutines_pool = true
max_coroutines_num = 256
max_processing_requests_num = 4096
re
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.coroutines_pool_default]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_read]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_sync]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_update]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.dump_worker]
dump_interval = '1day'
dump_root_path = ''
high_cpu_usage_threshold = 100

[server.forward_client]
default_compression_level = 0
default_compression_threshold = '128KB'
default_log_long_running_threshold = '0ns'
default_report_metrics = false
default_send_retry_times = 1
default_timeout = '1s'
enable_r
enable_work_stealing = false
queue_size = 1024

[server.targets]
allow_disk_without_uuid = true
collect_all_fds = true
create_engine_path = true
space_info_cache_timeout = '5s'
target_num_per_path = 0
target_paths = ["/mnt/3fsdata/data0/3fs"]

[server.targets.storage_target]
force_persist = true
kv_path = ''
migrate_kv_store = false
mutex_num = 257
point_query_strategy = 'NONE'

[server.targets.storage_target.file_store]
preopen_chunk_size_list = []

[server.targets.storage_target.kv_store]
create_if_missing = false
integrate_leveldb_log = false
leveldb_block_cache_size = '8GB'
leveldb_iterator_fill_cache = true
leveldb_shared_block_cache = true
leveldb_sst_file_size = '16MB'
leveldb_write_buffer_size = '16MB'
rocksdb_avoid_flush_during_recovery = false
rocksdb_avoid_flush_during_shutdown = false
rocksdb_avoid_unnecessary_blocking_io = false
rocksdb_block_cache_size = '8GB'
rocksdb_block_size = '4KB'
rocksdb_bloom_filter_bits_per_key = 10
rocksdb_compression = 'kNoCompression'
rocksdb_enable_bloom_filter = true
rocksdb_enable_pipelined_write = false
rocksdb_enable_prefix_transform = true
rocksdb_keep_log_file_num = 10
rocksdb_level0_file_num_compaction_trigger = 4
rocksdb_lowest_used_cache_tier = 'kNonVolatileBlockTier'
rocksdb_max_manifest_file_size = '64MB'
rocksdb_num_levels = 7
rocksdb_prepopulate_block_cache = 'kDisable'
rocksdb_readahead_size = '2MB'
rocksdb_shared_block_cache = true
rocksdb_stats_dump_period = '2min'
rocksdb_target_file_size_base = '64MB'
rocksdb_target_file_size_multiplier = 1
rocksdb_threads_num = 8
rocksdb_unordered_write = false
rocksdb_wal_recovery_mode = 'kTolerateCorruptedTailRecords'
rocksdb_write_buffer_size = '16MB'
sync_when_write = true
type = 'LevelDB'

[server.targets.storage_target.meta_store]
allocate_size = '256MB'
punch_hole_batch_size = 16
recycle_batch_size = 256
removed_chunk_expiration_time = '3day'
removed_chunk_force_recycled_time = '1h' NODE=local TASK=CreateStorageServiceTask
INFO[0020] Save admin cli config to /tmp/prepare-3fs-config.2QLDXp/admin_cli.toml NODE=local TASK=CreateStorageServiceTask
INFO[0020] Generating fdb.cluster to /tmp/prepare-3fs-config.2QLDXp/fdb.cluster NODE=local TASK=CreateStorageServiceTask
INFO[0020] Copying storage_main configs from /tmp/prepare-3fs-config.2QLDXp to local /mnt/nvme3n1/3fs-lab/storage/config.d NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d: NODE=""
DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp NODE=""
DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp: NODE=""
INFO[0020] Upload storage_main main config NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml' NODE=""
DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml': Succeed
ConfigVersion 1 NODE=""
INFO[0021] Service storage_main main config uploaded NODE=local TASK=CreateStorageServiceTask
INFO[0021] Starting storage_main container 3fs-storage NODE=local TASK=CreateStorageServiceTask
DEBU[0021] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml NODE=""
DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml: bc8b2bb426eabe682f7c232948fad7f057e1a59c6874d727371b4fa60878c9a4 NODE=""
INFO[0021] Waiting for service storage_main active... NODE=local TASK=CreateStorageServiceTask
DEBU[0021] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE=""
DEBU[0021] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee
100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE=""
DEBU[0023] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE=""
DEBU[0023] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee
100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE=""
DEBU[0025] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee
100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee
10001 STORAGE HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:19 1(UPTODATE) 250228-dev-1-999999-ee9a5cee NODE=""
INFO[0026] Service storage_main is not active NODE=local TASK=CreateStorageServiceTask
INFO[0026] Started storage_main container 3fs-storage successfully NODE=local TASK=CreateStorageServiceTask
INFO[0026] Creating mount-3fs-disks script and mount-3fs-disks.service service... NODE=local TASK=CreateStorageServiceTask
DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.P7fFFvwgTu NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.CIlnSMAAuU NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU: NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service: Created symlink /etc/systemd/system/multi-user.target.wants/mount-3fs-disks.service → /usr/lib/systemd/system/mount-3fs-disks.service. NODE=""
DEBU[0026] Run command: sudo -S /bin/bash -c systemctl daemon-reload NODE=""
DEBU[0026] Output of sudo -S /bin/bash -c systemctl daemon-reload: NODE=""
DEBU[0026] Created dir type disk: /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0, index: 0 NODE=local TASK=CreateStorageServiceTask
INFO[0026] Running task InitUserAndChainTask
DEBU[0026] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add--root --admin 0 root" NODE=""
DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add --root --admin 0 root": Uid 0
Name root
Token AAD1pfWU8QAgLseW2wDulnc0(Expired at N/A)
IsRootUser true
IsAdmin true
Gid 0
SupplementaryGids NODE=""
DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt" NODE=""
DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt": NODE=""
DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1 NODE=""
DEBU[0028] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1: NODE=""
DEBU[0028] Command failed stacktrace: exit status 1
Traceback (most recent call last):
File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in
main()
File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main
).run(
File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run
num_loops = self.max_recovery_traffic_on_peer*2
File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer
return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1))
ZeroDivisionError: division by zero

pkg/external/manager.go:50:(*externalBase).run: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1]
pkg/external/docker.go:168:(*dockerExternal).Exec:
pkg/mgmtd/steps.go:239:(*initUserAndChainStep).initChainFiles: run data_placement.py
pkg/mgmtd/steps.go:177:(*initUserAndChainStep).Execute:
pkg/task/task.go:101:(*BaseTask).ExecuteSteps.(*BaseTask).newStepExecuter.func1:
pkg/task/task.go:139:(*BaseTask).ExecuteSteps:
pkg/task/runner.go:252:(*Runner).Run: run task InitUserAndChainTask
cmd/m3fs/cluster.go:211:createCluster: create cluster
2026/06/07 04:20:24 create cluster: run task InitUserAndChainTask: run data_placement.py: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -typeCR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1]: exit status 1
Traceback (most recent call last):
File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in
main()
File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main
).run(
File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run
num_loops = self.max_recovery_traffic_on_peer*2
File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer
return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1))
ZeroDivisionError: division by zero
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs#
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

deployment crashed at data placement part need expert to look into

Description

Contributor guide

Assessment

Get new issues in your inbox