GET /messages silently drops retriever_from/page/doc_metadata/title/files from citations
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the Contributing Guide.
- [x] This is a bug report, not a question.
- [x] I have searched for existing issues, including closed ones.
- [x] Submitted in English.
### Dify version
1.17.0 (also reproduced by reading `main`'s current source — not fixed there either)
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
1. Create a chat app whose knowledge retrieval returns citations carrying extra metadata beyond
`RetrieverResource`'s declared fields — e.g. an external knowledge API (or a custom retrieval
node) that returns `retriever_from`, `page`, `doc_metadata`, `title`, or `files` in a citation's
`RetrievalSourceMetadata` (`api/core/rag/entities/citation_metadata.py`, which declares all
five of these fields).
2. Send a message through the streaming endpoint (`POST /chat-messages`). The response's citation
correctly includes all fields, `doc_metadata` included — confirmed by inspecting the raw SSE
response.
3. Reload the conversation (or call `GET /messages` / `GET /installed-apps/.../messages` / the
equivalent service-api endpoint directly) for that same message.
4. Inspect the `retriever_resources` entry for the same citation in the response.
### ✔️ Expected Behavior
`GET /messages` returns the same citation data the streaming response did — `retriever_from`,
`page`, `doc_metadata`, `title`, and `files` all present when they were populated at
message-completion time.
### ❌ Actual Behavior
`GET /messages` silently drops `retriever_from`, `page`, `doc_metadata`, `title`, and `files` from
every citation. Root cause, confirmed by reading source directly (not inferred):
- The full citation, including these five fields, **is correctly stored** in
`messages.message_metadata` (a JSON column) at write time — nothing is lost in storage. Verified
directly against a Postgres row for a real citation with a populated `doc_metadata` dict.
- `RetrieverResource` (`api/fields/message_fields.py`), the Pydantic response model used by
`GET /messages` and every other `MessageListItem`-derived history endpoint, declares none of
these five fields — while `RetrievalSourceMetadata`, the model that actually produces each
citation, declares all five.
- `ResponseModel`'s `model_config` (`api/fields/base.py`) sets `extra="ignore"`. When
`controllers/web/message.py`'s `MessageListApi.get()` (and the console/service-api equivalents)
re-validates the stored `Message.retriever_resources` property (`api/models/model.py`, itself
just `json.loads`-ing `message_metadata`) against `RetrieverResource`, Pydantic silently drops
every key the response model doesn't declare.
- The streaming path doesn't go through `RetrieverResource` and isn't affected, which is why the
gap is easy to miss — a client that branches on `doc_metadata` (e.g. rendering a video player
keyed off `doc_metadata.start_seconds` for a video citation, our case) works during a live
answer and silently degrades to a plain citation the moment the same conversation reloads
through history.
I have a fix and a PR ready — a straightforward field-parity addition to `RetrieverResource`, no
migration or backfill needed since the underlying data was never actually lost. Will link it here.
From Claude Code
Contributor guide
Research direction
Start with api/fields/message_fields.py and compare RetrieverResource with RetrievalSourceMetadata, then trace response validation through api/fields/base.py, controllers/web/message.py, and api/models/model.py. Verify the message-history endpoints preserve retriever_from, page, doc_metadata, title, and files already present in message_metadata, with no migration needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100