agentscope-ai / agentscope-ai/QwenPaw
[Question]: Full-backup is excessively slow and needs immediate optimization
- 主要语言
- Python
- 星标
- 34.9k
- 派生
- 3.1k
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 225
描述
## Question or topic
The full-backup is extremely slow after my data reach 800+MB(the .zip backup file is about 300MB), sometimes it takes several minutes to do the job.
I think there are some opportunities to optimize the backup process:
---
Parallelization Opportunity: Agent Workspace Backup is Strictly Serial
**Current implementation** (`backup/_ops/create_helpers.py:add_agent_workspaces()`):
```python
for i, (aid, ref) in enumerate(valid_agents): # ← purely serial
ws = Path(ref.workspace_dir).expanduser().resolve()
if ws.is_dir():
for entry in sorted(ws.rglob("*")): # ← scan all files
if entry.is_file():
zf.write(entry, ...) # ← compress & write
```
The entire backup pipeline runs in a single background thread (`asyncio.to_thread` → `_compress_to_tmp` → `add_files_to_zip`). Each agent's workspace is traversed (`rglob`), then each file is compressed and written to the zip **one after another**. With N agents and hundreds of MB of data, the total time is the **sum of all agents' processing time**.
The key technical constraint: **`zipfile.ZipFile` is not thread-safe**, so we cannot simply wrap the loop with a thread pool.
### Proposed Approaches
All three approaches break the serial dependency and let agents be processed concurrently:
| # | Approach | How it works | Pros | Cons |
|---|----------|-------------|------|------|
| **1** | **Per-agent temp zips → merge** | Each agent is compressed into its own temporary `.zip` by a `ProcessPoolExecutor` worker. A final pass merges them into the single backup zip. | Maximum parallelism; no GIL contention; pure Python zip merge is cheap. | ~2x disk space during merge; needs a merge step. |
| **2** | **Parallel file scan + serial compress** | Use `ThreadPoolExecutor` to parallelize the `rglob` file discovery per agent. Collect all file paths first, then serialize the `zf.write()` calls. | Minimal code change; no intermediate files; avoids slow filesystem walks blocking each other. | Compression (CPU-bound) still serial; speedup depends on I/O vs CPU ratio. |
| **3** | **Per-agent in-memory streams → final zip** | Each agent compresses into its own `BytesIO` buffer in parallel threads/processes, then the buffers are stitched into the final zip at the end. | No temp files; fully parallel. | High memory pressure (~300MB additional RAM); still need to handle the "merge into one zip" part. |
## Context
QwenPaw version: v1.1.8 post1
## Tried so far
N/A
贡献指南
评估
这个 Issue 还没有评估数据。