langgenius / langgenius/dify

GET /messages silently drops retriever_from/page/doc_metadata/title/files from citations

Open
#41,860 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the Contributing Guide.
- [x] This is a bug report, not a question.
- [x] I have searched for existing issues, including closed ones.
- [x] Submitted in English.

### Dify version

1.17.0 (also reproduced by reading `main`'s current source — not fixed there either)

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

1. Create a chat app whose knowledge retrieval returns citations carrying extra metadata beyond
`RetrieverResource`'s declared fields — e.g. an external knowledge API (or a custom retrieval
node) that returns `retriever_from`, `page`, `doc_metadata`, `title`, or `files` in a citation's
`RetrievalSourceMetadata` (`api/core/rag/entities/citation_metadata.py`, which declares all
five of these fields).
2. Send a message through the streaming endpoint (`POST /chat-messages`). The response's citation
correctly includes all fields, `doc_metadata` included — confirmed by inspecting the raw SSE
response.
3. Reload the conversation (or call `GET /messages` / `GET /installed-apps/.../messages` / the
equivalent service-api endpoint directly) for that same message.
4. Inspect the `retriever_resources` entry for the same citation in the response.

### ✔️ Expected Behavior

`GET /messages` returns the same citation data the streaming response did — `retriever_from`,
`page`, `doc_metadata`, `title`, and `files` all present when they were populated at
message-completion time.

### ❌ Actual Behavior

`GET /messages` silently drops `retriever_from`, `page`, `doc_metadata`, `title`, and `files` from
every citation. Root cause, confirmed by reading source directly (not inferred):

- The full citation, including these five fields, **is correctly stored** in
`messages.message_metadata` (a JSON column) at write time — nothing is lost in storage. Verified
directly against a Postgres row for a real citation with a populated `doc_metadata` dict.
- `RetrieverResource` (`api/fields/message_fields.py`), the Pydantic response model used by
`GET /messages` and every other `MessageListItem`-derived history endpoint, declares none of
these five fields — while `RetrievalSourceMetadata`, the model that actually produces each
citation, declares all five.
- `ResponseModel`'s `model_config` (`api/fields/base.py`) sets `extra="ignore"`. When
`controllers/web/message.py`'s `MessageListApi.get()` (and the console/service-api equivalents)
re-validates the stored `Message.retriever_resources` property (`api/models/model.py`, itself
just `json.loads`-ing `message_metadata`) against `RetrieverResource`, Pydantic silently drops
every key the response model doesn't declare.
- The streaming path doesn't go through `RetrieverResource` and isn't affected, which is why the
gap is easy to miss — a client that branches on `doc_metadata` (e.g. rendering a video player
keyed off `doc_metadata.start_seconds` for a video citation, our case) works during a live
answer and silently degrades to a plain citation the moment the same conversation reloads
through history.

I have a fix and a PR ready — a straightforward field-parity addition to `RetrieverResource`, no
migration or backfill needed since the underlying data was never actually lost. Will link it here.

From Claude Code

Contributor guide

Open the contributing guide

Research direction

Start with api/fields/message_fields.py and compare RetrieverResource with RetrievalSourceMetadata, then trace response validation through api/fields/base.py, controllers/web/message.py, and api/models/model.py. Verify the message-history endpoints preserve retriever_from, page, doc_metadata, title, and files already present in message_metadata, with no migration needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.