langgenius / langgenius/dify

Multimodal rerank maps results to the wrong document after skipping a missing image upload

Open
#41,479 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for a bug report. Questions belong in [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I searched existing issues, including closed ones.
- [x] I am submitting this report in English.
- [x] I did not modify the required fields of the bug report template.

### Dify version

Current `main` at `09054169c42100b8ff5ebc1163636220fa7162bc`.

### Cloud or Self Hosted

Self Hosted (Source)

### Steps to reproduce

1. Configure a multimodal dataset with a vision-capable reranking model.
2. Leave an image vector candidate whose `UploadFile` row no longer exists. This state is production-reachable when vector/keyword cleanup in `clean_document_task` fails: the task explicitly continues PostgreSQL and storage cleanup while leaving vector orphans for later reaping.
3. Run image-query semantic retrieval with the missing image candidate followed by a valid text candidate.
4. Have the reranking provider return result index `0` for the only document it received.

A minimal mocked reproduction on the commit above produces:

```text
model_input_count=1
model_input_content=valid text
returned_doc_id=missing-image
```

`RerankModelRunner.fetch_multimodal_rerank()` omits the missing image from `docs`, but still appends its source `Document` to `unique_documents`. `RerankModelRunner.run()` later resolves the provider's result index against that longer, shifted list.

### ✔️ Expected Behavior

A candidate that cannot be included in the model input should also be omitted from the parallel source-document list. Result index `0` should therefore map to the valid text document that was actually sent to the model.

### ❌ Actual Behavior

The missing image remains at index `0` in `unique_documents`, so the valid text result receives the missing image's metadata and provider. Later candidates are similarly shifted.

### Additional context

The existing multimodal reranker test named `test_fetch_multimodal_rerank_skips_missing_image_upload` uses a text-query mock and accepts the stale source-document list, so it does not cover the image-query result mapping.

I can submit a minimal fix and focused regression test if the maintainers agree with this direction.

AI assistance was used to inspect the code paths, build the isolated reproduction, and draft this report. I reviewed and verified the technical claims against the referenced commit.

Contributor guide

Open the contributing guide

Research direction

Start at RerankModelRunner.fetch_multimodal_rerank() and RerankModelRunner.run(), then read the existing test named test_fetch_multimodal_rerank_skips_missing_image. Add an image-query regression case using a missing image followed by valid text, and run the multimodal reranker tests; done means provider result indexes map to the documents actually sent to the model.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
ai, backend, databases, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.