agentscope-ai / agentscope-ai/AgentTeams

bug(manager): Manager crash loop on NFS-backed workspaces where the container user cannot chown to uid 0

Open
#1,236 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.6k
Forks
692
Avg merge
5d 4h
Merged PRs (30d)
23

Description

### Related

Fix: PR #1163

### Affected deployment scenario

The Manager workspace lives on an **NFS export**, and the container user (root) **cannot chown files to uid 0** on that export. Two common configurations hit this:

1. **NFS with `root_squash`** (the default on TrueNAS, Synology, and most NAS exports): the container's root is squashed to `nobody` (65534) on the export, so chown to uid 0 fails with `EPERM`.
2. **NFS with root mapped to a non-root user** (e.g. NFSv4 `maproot`/idmap or a server-side root→user mapping to a uid such as 1000). This mapping is often adopted deliberately to keep stateful writes (e.g. MinIO IAM state) consistent under a non-root owner, but root still cannot chown to uid 0.

Any filesystem with the same property — the invoking user lacks the privilege to chown to the source owner — behaves the same way.

### Symptoms

Manager dies during the plugin-install step of startup:

```
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.json': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.py': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools': Operation not permitted
```

`start-qwenpaw-manager.sh` runs under a global `set -e` (`manager/scripts/lib/base.sh:5`), so this `cp` failure aborts the entire startup script. The container exits seconds after boot and is restarted by the Controller, entering a **crash loop** (RestartCount grows without bound). The `agentteams-manager-tools` plugin (projectflow / taskflow / message / filesync) is never installed, so the Manager is unusable.

### Root cause

`cp -a` expands to `--preserve=all`, which includes `--preserve=ownership`: after copying, `cp` attempts to `chown` the file to the **source owner** — `root` (uid 0), the owner of `/opt/agentteams/plugins/` inside the image. On an export where the caller is not root (squashed or mapped), that `chown` returns `Operation not permitted`, `cp` exits non-zero, and the global `set -e` turns it into a fatal startup abort.

The plugin-install loop implicitly assumes the workspace is a local filesystem (or that the container has full chown privilege); an NFS-backed workspace is a reasonable deployment requirement for persisting Manager state across container rebuilds.

### Reproduction

```bash
# Export a share from any NFS server with default root_squash
# (or with root mapped to a non-root uid), then mount it as the Manager workspace:
docker run --rm \
-v /Manager:/root/manager-workspace \
--env-file manager-env.txt \
agentteams-manager-qwenpaw:v1.2.3
# -> cp: failed to preserve ownership ...: Operation not permitted
# -> startup script exits, container enters a restart loop
```

### Affected scope

- **Component**: Manager container only — `manager/scripts/init/start-qwenpaw-manager.sh:311` (the plugin-copy loop). Worker and Controller startup paths are unaffected.
- **Versions**: v1.2.2 and v1.2.3 (identical line at 311 in both tags); the line is still present on `main` (verified at `f65d6e1a`).
- Note: the sibling `migrate-copaw-state.sh` already guards its `cp -a` calls with `if ! cp ...; then`, so this crash loop is specific to the unguarded copy in the plugin-install step.

### Fix

PR #1163 applies a one-line change so the plugin copy does not preserve ownership:

```diff
- cp -a "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
+ cp -a --no-preserve=ownership "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
```

Mode, timestamps, and symlinks are still preserved (`-a` otherwise unchanged); only the chown attempt is skipped. Option order matters: `--no-preserve=ownership` must come **after** `-a`, because `-a` (`--preserve=all`) would otherwise re-enable preservation.

### Compatibility expectations

- **Local disk / fully-privileged environments**: no observable change. The container runs as root and the source files are also owned by root, so resulting ownership is identical to today; permission bits, timestamps, and links continue to be preserved.
- **NFS with root_squash / non-root maproot**: Manager boots and plugins install. The plugin files end up owned by the container's effective uid — the same user that runs the Manager process — so they remain fully readable by the QwenPaw PluginLoader; no functional change.
- **Upgrades**: no data migration needed. The loop `rm -rf`s each plugin directory before copying, so the next startup after the fix re-materializes the plugins under the correct ownership semantics.
- **No API / CRD / protocol changes**; no effect on Worker, Controller, or MinIO behavior.

Validated in a production-like deployment: an NFSv4 export (TrueNAS) with root mapped to a non-root user — with the fix, Manager starts and stays up (RestartCount = 0, plugin-install step completes without errors).

---

### 关联

修复:PR #1163

### 受影响的部署场景

Manager 工作区位于 **NFS 导出**上,且容器用户(root)在导出上**无法 chown 到 uid 0**。两种常见配置都会触发:

1. **带 `root_squash` 的 NFS**(TrueNAS、Synology 及大多数 NAS 导出的默认行为):容器内 root 被压缩为 `nobody`(65534),chown 到 uid 0 报 `EPERM`。
2. **root 映射到非 root 用户的 NFS**(如 NFSv4 `maproot`/idmap 或服务端 root→user 映射到 uid 1000 之类)。这种映射常是为了让有状态写入(如 MinIO IAM 状态)在非 root 属主下保持一致而刻意配置的,但 root 依然无法 chown 到 uid 0。

任何具备相同属性的文件系统——调用者没有把文件 chown 到源属主的权限——行为一致。

### 现象

Manager 在启动的插件安装步骤死亡:

```
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.json': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.py': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools': Operation not permitted
```

`start-qwenpaw-manager.sh` 运行在全局 `set -e` 之下(`manager/scripts/lib/base.sh:5`),因此这次 `cp` 失败会中止整个启动脚本。容器在启动几秒后退出、被 Controller 反复拉起,进入**崩溃循环**(RestartCount 持续增长)。`agentteams-manager-tools` 插件(projectflow / taskflow / message / filesync)永远安装不上,Manager 不可用。

### 根因

`cp -a` 展开为 `--preserve=all`,其中包含 `--preserve=ownership`:复制后 `cp` 会尝试把文件 `chown` 为**源属主**——镜像内 `/opt/agentteams/plugins/` 的属主 `root`(uid 0)。在调用者非 root(被压缩或映射)的导出上,该 `chown` 返回 `Operation not permitted`,`cp` 非零退出,全局 `set -e` 将其变成致命的启动中止。

插件安装循环隐含假设工作区是本地文件系统(或容器拥有完整 chown 权限);而 NFS 挂载的工作区是「容器重建后持久保留 Manager 状态」的合理部署需求。

### 复现

```bash
# 从任意 NFS 服务器导出一个共享(默认 root_squash,
# 或把 root 映射到非 root uid),将其挂载为 Manager 工作区:
docker run --rm \
-v /Manager:/root/manager-workspace \
--env-file manager-env.txt \
agentteams-manager-qwenpaw:v1.2.3
# -> cp: failed to preserve ownership ...: Operation not permitted
# -> 启动脚本退出,容器进入重启循环
```

### 影响范围

- **组件**:仅 Manager 容器——`manager/scripts/init/start-qwenpaw-manager.sh:311`(插件复制循环)。Worker 与 Controller 启动路径不受影响。
- **版本**:v1.2.2 与 v1.2.3(两个 tag 上同一行 311 完全相同);`main` 上该行仍存在(已在 `f65d6e1a` 核验)。
- 注:同目录的 `migrate-copaw-state.sh` 已用 `if ! cp ...; then` 给 `cp -a` 加了容错,所以该崩溃循环特指插件安装步骤里无容错的这次复制。

### 修复

PR #1163 一行改动,使插件复制不保留属主:

```diff
- cp -a "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
+ cp -a --no-preserve=ownership "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
```

权限位、时间戳、符号链接仍然保留(`-a` 其余语义不变),只是跳过 chown 尝试。选项顺序很关键:`--no-preserve=ownership` 必须放在 `-a` **之后**,否则会被 `-a`(`--preserve=all`)重新覆盖。

### 兼容性预期

- **本地磁盘 / 完全特权环境**:无可见变化。容器以 root 运行、源文件属主也是 root,结果属主与现状完全一致;权限位、时间戳、链接继续保留。
- **root_squash / 非 root maproot 的 NFS**:Manager 正常启动、插件正常安装。插件文件属主 = 容器有效 uid(即运行 Manager 进程的用户),QwenPaw PluginLoader 可完整读取,功能无变化。
- **升级**:无需数据迁移。循环每次先 `rm -rf` 各插件目录再复制,修复后的下次启动即按正确属主语义重新落盘。
- **无 API / CRD / 协议变化**;对 Worker、Controller、MinIO 行为无影响。

已在类生产环境验证:NFSv4 导出(TrueNAS)root 映射到非 root 用户——应用修复后 Manager 正常启动并保持稳定(RestartCount = 0,插件安装步骤无错完成)。

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.