agentscope-ai / agentscope-ai/AgentTeams
bug(manager): Manager crash loop on NFS-backed workspaces where the container user cannot chown to uid 0
- Dominant language
- Go
- Stars
- 5.6k
- Forks
- 692
- Avg merge
- 5d 4h
- Merged PRs (30d)
- 23
Description
### Related
Fix: PR #1163
### Affected deployment scenario
The Manager workspace lives on an **NFS export**, and the container user (root) **cannot chown files to uid 0** on that export. Two common configurations hit this:
1. **NFS with `root_squash`** (the default on TrueNAS, Synology, and most NAS exports): the container's root is squashed to `nobody` (65534) on the export, so chown to uid 0 fails with `EPERM`.
2. **NFS with root mapped to a non-root user** (e.g. NFSv4 `maproot`/idmap or a server-side root→user mapping to a uid such as 1000). This mapping is often adopted deliberately to keep stateful writes (e.g. MinIO IAM state) consistent under a non-root owner, but root still cannot chown to uid 0.
Any filesystem with the same property — the invoking user lacks the privilege to chown to the source owner — behaves the same way.
### Symptoms
Manager dies during the plugin-install step of startup:
```
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.json': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.py': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools': Operation not permitted
```
`start-qwenpaw-manager.sh` runs under a global `set -e` (`manager/scripts/lib/base.sh:5`), so this `cp` failure aborts the entire startup script. The container exits seconds after boot and is restarted by the Controller, entering a **crash loop** (RestartCount grows without bound). The `agentteams-manager-tools` plugin (projectflow / taskflow / message / filesync) is never installed, so the Manager is unusable.
### Root cause
`cp -a` expands to `--preserve=all`, which includes `--preserve=ownership`: after copying, `cp` attempts to `chown` the file to the **source owner** — `root` (uid 0), the owner of `/opt/agentteams/plugins/` inside the image. On an export where the caller is not root (squashed or mapped), that `chown` returns `Operation not permitted`, `cp` exits non-zero, and the global `set -e` turns it into a fatal startup abort.
The plugin-install loop implicitly assumes the workspace is a local filesystem (or that the container has full chown privilege); an NFS-backed workspace is a reasonable deployment requirement for persisting Manager state across container rebuilds.
### Reproduction
```bash
# Export a share from any NFS server with default root_squash
# (or with root mapped to a non-root uid), then mount it as the Manager workspace:
docker run --rm \
-v /Manager:/root/manager-workspace \
--env-file manager-env.txt \
agentteams-manager-qwenpaw:v1.2.3
# -> cp: failed to preserve ownership ...: Operation not permitted
# -> startup script exits, container enters a restart loop
```
### Affected scope
- **Component**: Manager container only — `manager/scripts/init/start-qwenpaw-manager.sh:311` (the plugin-copy loop). Worker and Controller startup paths are unaffected.
- **Versions**: v1.2.2 and v1.2.3 (identical line at 311 in both tags); the line is still present on `main` (verified at `f65d6e1a`).
- Note: the sibling `migrate-copaw-state.sh` already guards its `cp -a` calls with `if ! cp ...; then`, so this crash loop is specific to the unguarded copy in the plugin-install step.
### Fix
PR #1163 applies a one-line change so the plugin copy does not preserve ownership:
```diff
- cp -a "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
+ cp -a --no-preserve=ownership "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
```
Mode, timestamps, and symlinks are still preserved (`-a` otherwise unchanged); only the chown attempt is skipped. Option order matters: `--no-preserve=ownership` must come **after** `-a`, because `-a` (`--preserve=all`) would otherwise re-enable preservation.
### Compatibility expectations
- **Local disk / fully-privileged environments**: no observable change. The container runs as root and the source files are also owned by root, so resulting ownership is identical to today; permission bits, timestamps, and links continue to be preserved.
- **NFS with root_squash / non-root maproot**: Manager boots and plugins install. The plugin files end up owned by the container's effective uid — the same user that runs the Manager process — so they remain fully readable by the QwenPaw PluginLoader; no functional change.
- **Upgrades**: no data migration needed. The loop `rm -rf`s each plugin directory before copying, so the next startup after the fix re-materializes the plugins under the correct ownership semantics.
- **No API / CRD / protocol changes**; no effect on Worker, Controller, or MinIO behavior.
Validated in a production-like deployment: an NFSv4 export (TrueNAS) with root mapped to a non-root user — with the fix, Manager starts and stays up (RestartCount = 0, plugin-install step completes without errors).
---
### 关联
修复:PR #1163
### 受影响的部署场景
Manager 工作区位于 **NFS 导出**上,且容器用户(root)在导出上**无法 chown 到 uid 0**。两种常见配置都会触发:
1. **带 `root_squash` 的 NFS**(TrueNAS、Synology 及大多数 NAS 导出的默认行为):容器内 root 被压缩为 `nobody`(65534),chown 到 uid 0 报 `EPERM`。
2. **root 映射到非 root 用户的 NFS**(如 NFSv4 `maproot`/idmap 或服务端 root→user 映射到 uid 1000 之类)。这种映射常是为了让有状态写入(如 MinIO IAM 状态)在非 root 属主下保持一致而刻意配置的,但 root 依然无法 chown 到 uid 0。
任何具备相同属性的文件系统——调用者没有把文件 chown 到源属主的权限——行为一致。
### 现象
Manager 在启动的插件安装步骤死亡:
```
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.json': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools/plugin.py': Operation not permitted
cp: failed to preserve ownership for '/root/manager-workspace/.qwenpaw/plugins/agentteams-manager-tools': Operation not permitted
```
`start-qwenpaw-manager.sh` 运行在全局 `set -e` 之下(`manager/scripts/lib/base.sh:5`),因此这次 `cp` 失败会中止整个启动脚本。容器在启动几秒后退出、被 Controller 反复拉起,进入**崩溃循环**(RestartCount 持续增长)。`agentteams-manager-tools` 插件(projectflow / taskflow / message / filesync)永远安装不上,Manager 不可用。
### 根因
`cp -a` 展开为 `--preserve=all`,其中包含 `--preserve=ownership`:复制后 `cp` 会尝试把文件 `chown` 为**源属主**——镜像内 `/opt/agentteams/plugins/` 的属主 `root`(uid 0)。在调用者非 root(被压缩或映射)的导出上,该 `chown` 返回 `Operation not permitted`,`cp` 非零退出,全局 `set -e` 将其变成致命的启动中止。
插件安装循环隐含假设工作区是本地文件系统(或容器拥有完整 chown 权限);而 NFS 挂载的工作区是「容器重建后持久保留 Manager 状态」的合理部署需求。
### 复现
```bash
# 从任意 NFS 服务器导出一个共享(默认 root_squash,
# 或把 root 映射到非 root uid),将其挂载为 Manager 工作区:
docker run --rm \
-v /Manager:/root/manager-workspace \
--env-file manager-env.txt \
agentteams-manager-qwenpaw:v1.2.3
# -> cp: failed to preserve ownership ...: Operation not permitted
# -> 启动脚本退出,容器进入重启循环
```
### 影响范围
- **组件**:仅 Manager 容器——`manager/scripts/init/start-qwenpaw-manager.sh:311`(插件复制循环)。Worker 与 Controller 启动路径不受影响。
- **版本**:v1.2.2 与 v1.2.3(两个 tag 上同一行 311 完全相同);`main` 上该行仍存在(已在 `f65d6e1a` 核验)。
- 注:同目录的 `migrate-copaw-state.sh` 已用 `if ! cp ...; then` 给 `cp -a` 加了容错,所以该崩溃循环特指插件安装步骤里无容错的这次复制。
### 修复
PR #1163 一行改动,使插件复制不保留属主:
```diff
- cp -a "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
+ cp -a --no-preserve=ownership "${_plugin_src}" "${PLUGINS_TARGET}/${_plugin_name}"
```
权限位、时间戳、符号链接仍然保留(`-a` 其余语义不变),只是跳过 chown 尝试。选项顺序很关键:`--no-preserve=ownership` 必须放在 `-a` **之后**,否则会被 `-a`(`--preserve=all`)重新覆盖。
### 兼容性预期
- **本地磁盘 / 完全特权环境**:无可见变化。容器以 root 运行、源文件属主也是 root,结果属主与现状完全一致;权限位、时间戳、链接继续保留。
- **root_squash / 非 root maproot 的 NFS**:Manager 正常启动、插件正常安装。插件文件属主 = 容器有效 uid(即运行 Manager 进程的用户),QwenPaw PluginLoader 可完整读取,功能无变化。
- **升级**:无需数据迁移。循环每次先 `rm -rf` 各插件目录再复制,修复后的下次启动即按正确属主语义重新落盘。
- **无 API / CRD / 协议变化**;对 Worker、Controller、MinIO 行为无影响。
已在类生产环境验证:NFSv4 导出(TrueNAS)root 映射到非 root 用户——应用修复后 Manager 正常启动并保持稳定(RestartCount = 0,插件安装步骤无错完成)。
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.