agentscope-ai / agentscope-ai/QwenPaw

[Question]: Full-backup is excessively slow and needs immediate optimization

未关闭
#4,678 2 条评论 0 个 reaction 已指派 1 人 已被 @jinglinpeng 认领 在 GitHub 查看
question
主要语言
Python
星标
34.9k
派生
3.1k
平均合并
1 天 15 小时
30 天内合并 PR
225

描述

## Question or topic

The full-backup is extremely slow after my data reach 800+MB(the .zip backup file is about 300MB), sometimes it takes several minutes to do the job.
I think there are some opportunities to optimize the backup process:
---

Parallelization Opportunity: Agent Workspace Backup is Strictly Serial

**Current implementation** (`backup/_ops/create_helpers.py:add_agent_workspaces()`):

```python
for i, (aid, ref) in enumerate(valid_agents): # ← purely serial
ws = Path(ref.workspace_dir).expanduser().resolve()
if ws.is_dir():
for entry in sorted(ws.rglob("*")): # ← scan all files
if entry.is_file():
zf.write(entry, ...) # ← compress & write
```

The entire backup pipeline runs in a single background thread (`asyncio.to_thread` → `_compress_to_tmp` → `add_files_to_zip`). Each agent's workspace is traversed (`rglob`), then each file is compressed and written to the zip **one after another**. With N agents and hundreds of MB of data, the total time is the **sum of all agents' processing time**.

The key technical constraint: **`zipfile.ZipFile` is not thread-safe**, so we cannot simply wrap the loop with a thread pool.

### Proposed Approaches

All three approaches break the serial dependency and let agents be processed concurrently:

| # | Approach | How it works | Pros | Cons |
|---|----------|-------------|------|------|
| **1** | **Per-agent temp zips → merge** | Each agent is compressed into its own temporary `.zip` by a `ProcessPoolExecutor` worker. A final pass merges them into the single backup zip. | Maximum parallelism; no GIL contention; pure Python zip merge is cheap. | ~2x disk space during merge; needs a merge step. |
| **2** | **Parallel file scan + serial compress** | Use `ThreadPoolExecutor` to parallelize the `rglob` file discovery per agent. Collect all file paths first, then serialize the `zf.write()` calls. | Minimal code change; no intermediate files; avoids slow filesystem walks blocking each other. | Compression (CPU-bound) still serial; speedup depends on I/O vs CPU ratio. |
| **3** | **Per-agent in-memory streams → final zip** | Each agent compresses into its own `BytesIO` buffer in parallel threads/processes, then the buffers are stitched into the final zip at the end. | No temp files; fully parallel. | High memory pressure (~300MB additional RAM); still need to handle the "merge into one zip" part. |

## Context

QwenPaw version: v1.1.8 post1

## Tried so far

N/A

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。