agentscope-ai / agentscope-ai/agentscope
IndexWorker blocks the event loop: lifespan never wires up parser_executor (CPU parsers run in the loop thread)
- 主要言語
- Python
- スター
- 31.5k
- フォーク
- 3.5k
- 平均マージ
- 1日 23時間
- マージ済み PR(30日)
- 95
説明
## Summary
`IndexWorker` already accepts a `parser_executor: ProcessPoolExecutor | None` parameter (the `_parse` method routes to `loop.run_in_executor(...)` when it's not `None`), but `agentscope.app._lifespan.lifespan` constructs `IndexWorker` without passing `parser_executor`. As a result, CPU-heavy parsers (PDF / Office with image extraction) run synchronously **in the event-loop thread**, blocking the entire FastAPI app for tens of seconds on a large document.
## Reproduction
1. Start an AgentScope app with the default lifespan (`create_app(...)` + `enable_index_worker=True`).
2. Upload a 19MB PPTX file containing images (with `PPTParser(include_image=True)` registered) via `POST /as/knowledge_bases/{kb_id}/documents`.
3. While the upload returns `201` immediately and the worker starts indexing, hit `GET /api/health` (or any other endpoint) in another request — observe it hangs until parsing finishes.
```
2026-07-25 23:35:57,351 INFO → POST /as/knowledge_bases/.../documents (19MB pptx)
2026-07-25 23:35:57,457 INFO ← POST status=201
# ... no further request is served for ~30s, health check included
```
`lsof -ti:8765` shows the backend process is alive, but `curl --max-time 10 /api/health` returns `HTTP 000 time=10.00s`.
## Expected vs Actual
**Expected** (per the `IndexWorker` docstring):
> `parser_executor` (`ProcessPoolExecutor | None`, optional): Process pool used to off-load CPU-intensive parses (PDF, Office). `None` runs parses in the event-loop thread, which is fine for plain text but unsafe for third-party byte-oriented parsers. Injected so a single pool can be shared across the app (**built in lifespan**).
**Actual**: `lifespan` does not build that pool, so `parser_executor` stays `None`. The docstring's own "unsafe for third-party byte-oriented parsers" warning is the behavior we get in production.
## Root Cause
`agentscope/app/_lifespan.py`:
```python
if enable_index_worker:
node_id = f"{socket.gethostname()}:{uuid.uuid4().hex[:8]}"
worker = IndexWorker(
storage=storage,
blob_store=blob_store,
knowledge_base_manager=knowledge_base_manager,
parsers=app.state.knowledge_parsers,
chunker=app.state.knowledge_chunker,
node_id=node_id,
) # ❌ no parser_executor=...
```
`agentscope/app/_service/_index_worker.py`:
```python
async def _parse(self, parser, file_bytes, filename):
if self._parser_executor is None:
return await parser.parse(file_bytes, filename) # ← blocks the loop
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._parser_executor, _run_parser_sync, ...)
```
## Impact
- Uploading any non-trivial PPTX / DOCX / PDF blocks the event loop for the whole parse duration. During that window **no** request can be served — health checks, document status polling, SSE streams, the scheduler sweeper, everything stalls.
- The frontend shows an indefinite spinner because the polling endpoint itself can't respond.
- This makes the embedded worker mode (`enable_index_worker=True`) effectively unusable for the Office/PDF upload use case, which is the primary RAG scenario.
## Suggested Fix
Build a shared `ProcessPoolExecutor` in `lifespan`, register its shutdown on the `AsyncExitStack`, and pass it to `IndexWorker`:
```python
from concurrent.futures import ProcessPoolExecutor
if enable_index_worker:
parser_pool = ProcessPoolExecutor(max_workers=4, thread_name_prefix="as-parser")
stack.callback(parser_pool.shutdown, wait=False)
worker = IndexWorker(
storage=storage,
blob_store=blob_store,
knowledge_base_manager=knowledge_base_manager,
parsers=app.state.knowledge_parsers,
chunker=app.state.knowledge_chunker,
node_id=node_id,
parser_executor=parser_pool, # ✅
)
```
The `IndexWorker` side already supports it — there's nothing to change there.
## Environment
- agentscope: `2.0.4.post1`
- Python: 3.12
- OS: macOS (Apple Silicon)
- Vector store: Milvus Lite
- Parsers: `TextParser`, `PDFParser`, `WordParser(include_image=True)`, `PPTParser(include_image=True)`, `ExcelParser`
## Workaround
For anyone hitting this now, monkey-patch `IndexWorker.__init__` to inject a shared pool when `parser_executor` is not provided:
```python
from concurrent.futures import ProcessPoolExecutor
from agentscope.app._service import IndexWorker
_orig_init = IndexWorker.__init__
_shared_pool: ProcessPoolExecutor | None = None
def _patched_init(self, *args, **kwargs):
global _shared_pool
if kwargs.get("parser_executor") is None:
if _shared_pool is None:
_shared_pool = ProcessPoolExecutor(max_workers=4, thread_name_prefix="as-parser")
kwargs["parser_executor"] = _shared_pool
_orig_init(self, *args, **kwargs)
IndexWorker.__init__ = _patched_init
```
Happy to open a PR if the maintainers agree on the approach — the change is small and local to `lifespan`.
コントリビューションガイド
評価
この issue はまだ評価されていません。