Hit testing fails with "Can't operate on closed transaction inside context manager"
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.16.0-rc1
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
1. Deploy Dify `1.16.0-rc1` locally using Docker Compose.
2. Create a knowledge base.
3. Upload a document and wait for indexing to complete.
4. Open the knowledge base retrieval test page.
5. Enter a query that matches one or more indexed segments.
6. Run the retrieval test.
The API returns HTTP 500. The relevant API log is:
```text
sqlalchemy.exc.InvalidRequestError: Can't operate on closed transaction inside context manager.
Please complete the context manager before emitting further commands.
File "/app/api/services/hit_testing_service.py", line 188, in retrieve
return cls.compact_retrieve_response(query, all_documents, session=session)
File "/app/api/services/hit_testing_service.py", line 244, in compact_retrieve_response
"records": cls._dump_retrieval_records(session, records)
File "/app/api/services/hit_testing_service.py", line 72, in _dump_retrieval_records
for document in session.scalars(
select(DatasetDocument).where(DatasetDocument.id.in_(document_ids))
).all()
```
The issue is reproducible when the retrieval returns matching document segments.
### ✔️ Expected Behavior
The retrieval test should complete successfully and display the matched document segments without returning an internal server error.
### ❌ Actual Behavior
The vector retrieval returns matching segments, but the API responds with HTTP 500 when it attempts to load the related document information.
`HitTestingService.retrieve()` calls `session.commit()` and then calls `compact_retrieve_response()`, which executes another query using the same session. Because the controller manages the session inside a transaction context, the explicit commit closes the transaction before the document query is executed.
Changing the internal `session.commit()` to `session.flush()` allows the outer transaction context to manage the final commit and prevents the error.
Contributor guide
Research direction
Start in api/services/hit_testing_service.py, especially retrieve(), compact_retrieve_response(), and _dump_retrieval_records(), then inspect how the controller manages the SQLAlchemy session transaction. Reproduce with the Docker Compose retrieval test using a query that matches indexed segments. Done means the test returns matched document segments without the closed-transaction HTTP 500.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlalchemy
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100