agentscope-ai / agentscope-ai/agentscope

IndexWorker blocks the event loop: lifespan never wires up parser_executor (CPU parsers run in the loop thread)

オープン
#2,176 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
31.5k
フォーク
3.5k
平均マージ
1日 23時間
マージ済み PR(30日)
95

説明

## Summary

`IndexWorker` already accepts a `parser_executor: ProcessPoolExecutor | None` parameter (the `_parse` method routes to `loop.run_in_executor(...)` when it's not `None`), but `agentscope.app._lifespan.lifespan` constructs `IndexWorker` without passing `parser_executor`. As a result, CPU-heavy parsers (PDF / Office with image extraction) run synchronously **in the event-loop thread**, blocking the entire FastAPI app for tens of seconds on a large document.

## Reproduction

1. Start an AgentScope app with the default lifespan (`create_app(...)` + `enable_index_worker=True`).
2. Upload a 19MB PPTX file containing images (with `PPTParser(include_image=True)` registered) via `POST /as/knowledge_bases/{kb_id}/documents`.
3. While the upload returns `201` immediately and the worker starts indexing, hit `GET /api/health` (or any other endpoint) in another request — observe it hangs until parsing finishes.

```
2026-07-25 23:35:57,351 INFO → POST /as/knowledge_bases/.../documents (19MB pptx)
2026-07-25 23:35:57,457 INFO ← POST status=201
# ... no further request is served for ~30s, health check included
```

`lsof -ti:8765` shows the backend process is alive, but `curl --max-time 10 /api/health` returns `HTTP 000 time=10.00s`.

## Expected vs Actual

**Expected** (per the `IndexWorker` docstring):

> `parser_executor` (`ProcessPoolExecutor | None`, optional): Process pool used to off-load CPU-intensive parses (PDF, Office). `None` runs parses in the event-loop thread, which is fine for plain text but unsafe for third-party byte-oriented parsers. Injected so a single pool can be shared across the app (**built in lifespan**).

**Actual**: `lifespan` does not build that pool, so `parser_executor` stays `None`. The docstring's own "unsafe for third-party byte-oriented parsers" warning is the behavior we get in production.

## Root Cause

`agentscope/app/_lifespan.py`:

```python
if enable_index_worker:
node_id = f"{socket.gethostname()}:{uuid.uuid4().hex[:8]}"
worker = IndexWorker(
storage=storage,
blob_store=blob_store,
knowledge_base_manager=knowledge_base_manager,
parsers=app.state.knowledge_parsers,
chunker=app.state.knowledge_chunker,
node_id=node_id,
) # ❌ no parser_executor=...
```

`agentscope/app/_service/_index_worker.py`:

```python
async def _parse(self, parser, file_bytes, filename):
if self._parser_executor is None:
return await parser.parse(file_bytes, filename) # ← blocks the loop
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._parser_executor, _run_parser_sync, ...)
```

## Impact

- Uploading any non-trivial PPTX / DOCX / PDF blocks the event loop for the whole parse duration. During that window **no** request can be served — health checks, document status polling, SSE streams, the scheduler sweeper, everything stalls.
- The frontend shows an indefinite spinner because the polling endpoint itself can't respond.
- This makes the embedded worker mode (`enable_index_worker=True`) effectively unusable for the Office/PDF upload use case, which is the primary RAG scenario.

## Suggested Fix

Build a shared `ProcessPoolExecutor` in `lifespan`, register its shutdown on the `AsyncExitStack`, and pass it to `IndexWorker`:

```python
from concurrent.futures import ProcessPoolExecutor

if enable_index_worker:
parser_pool = ProcessPoolExecutor(max_workers=4, thread_name_prefix="as-parser")
stack.callback(parser_pool.shutdown, wait=False)

worker = IndexWorker(
storage=storage,
blob_store=blob_store,
knowledge_base_manager=knowledge_base_manager,
parsers=app.state.knowledge_parsers,
chunker=app.state.knowledge_chunker,
node_id=node_id,
parser_executor=parser_pool, # ✅
)
```

The `IndexWorker` side already supports it — there's nothing to change there.

## Environment

- agentscope: `2.0.4.post1`
- Python: 3.12
- OS: macOS (Apple Silicon)
- Vector store: Milvus Lite
- Parsers: `TextParser`, `PDFParser`, `WordParser(include_image=True)`, `PPTParser(include_image=True)`, `ExcelParser`

## Workaround

For anyone hitting this now, monkey-patch `IndexWorker.__init__` to inject a shared pool when `parser_executor` is not provided:

```python
from concurrent.futures import ProcessPoolExecutor
from agentscope.app._service import IndexWorker

_orig_init = IndexWorker.__init__
_shared_pool: ProcessPoolExecutor | None = None

def _patched_init(self, *args, **kwargs):
global _shared_pool
if kwargs.get("parser_executor") is None:
if _shared_pool is None:
_shared_pool = ProcessPoolExecutor(max_workers=4, thread_name_prefix="as-parser")
kwargs["parser_executor"] = _shared_pool
_orig_init(self, *args, **kwargs)

IndexWorker.__init__ = _patched_init
```

Happy to open a PR if the maintainers agree on the approach — the change is small and local to `lifespan`.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。