Switching the embedding model re-indexes with the previous model (vector index task dispatched before the dataset row commits)

3. Wait for the automatic re-index to finish (all documents return to `completed`).
4. Read one stored vector back from the vector store, and separately embed the same chunk text with `A` and with `B`. Compare by cosine similarity.

The switch can be repeated in either direction. We ran it three times, alternating `A → B`, `B → A`, `A → B`, and every round behaved the same way: the declared model became the requested one, while the stored vectors were produced by the model that had been declared before the request.

### ✔️ Expected Behavior

After the automatic re-index triggered by the model switch completes, the stored vectors are produced by the newly declared embedding model, so that queries (embedded with the newly declared model) and stored vectors live in the same vector space.

### ❌ Actual Behavior

The declaration and the stored vectors disagree, permanently and silently.

- `datasets.embedding_model` / `embedding_model_provider` hold the requested model.
- Every stored vector was produced by the previously declared model.
- Retrieval still embeds the query with the newly declared model, so queries are matched against vectors from a different model.

Measured on one knowledge base after switching:

| Comparison | Cosine similarity |
| --- | --- |
| Stored vector vs. the same text embedded with the previously declared model | 0.99 |
| Stored vector vs. the same text embedded with the newly declared model | 0.40 |

Retrieval quality drops without any error. Taking the first 180 characters of a chunk verbatim as the query, the chunk's own similarity score fell from 0.74 to 0.60 after the switch. Nothing in the console, the document status, or the logs indicates a problem, and re-issuing the same `PATCH` does not repair it.

#### Root cause

`DatasetService._update_internal_dataset` writes the new embedding model with `session.execute(update(Dataset)...)` followed by `session.flush()`, and then dispatches `deal_dataset_vector_index_task.delay(...)` while the transaction is still open.

- [`api/services/dataset_service.py#L825-L836` (1.16.1)](https://github.com/langgenius/dify/blob/6f8ed69ee15f9a2e7189ca066275e973d091d1e9/api/services/dataset_service.py#L825-L836)
- [`api/services/dataset_service.py#L826-L837` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/services/dataset_service.py#L826-L837)

```python
session.execute(update(Dataset).where(Dataset.id == dataset.id).values(**filtered_data))
session.flush()
...
if action:
deal_dataset_vector_index_task.delay(dataset.id, action)
```

`flush()` rather than `commit()` is deliberate: the session is owned by the caller, and committing here closed the caller's context manager early (#39191). The consequence is that the commit happens after the request handler returns, in the `with_session` decorator.

- [`api/controllers/common/session.py#L48-L56` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/controllers/common/session.py#L48-L56)

The worker does not receive the new model as an argument. It opens its own session and re-reads the `datasets` row.

- [`api/tasks/deal_dataset_vector_index_task.py#L30-L32` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/tasks/deal_dataset_vector_index_task.py#L30-L32)

The embedding model used for the re-index is then resolved from that row.

- [`api/core/rag/datasource/vdb/vector_factory.py#L64-L74` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/core/rag/datasource/vdb/vector_factory.py#L64-L74)

So the ordering is: dispatch, worker reads the pre-update row, worker re-embeds with the old model, and only then does the web request commit the new model. Formally this is a race, but the worker wins in practice: `delay()` publishes immediately, while the request handler still has permission lookups and response serialization to do before its commit.

Two details make the outcome worse:

- The result is not self-healing. `_update_embedding_model_settings` returns the `update` action only when the requested model differs from the row it reads, so once the declaration has moved to `B`, sending the same `PATCH` again is a no-op and never re-dispatches the task.
- [`api/services/dataset_service.py#L1079-L1082` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/services/dataset_service.py#L1079-L1082)
- `regenerate_summary_index_task` is dispatched from the same place and reads the same row, so summary vectors are affected the same way.

This also explains why the bug is not universal. The only `commit()` that can happen between the `flush()` and the dispatch is the one inside `_update_pipeline_knowledge_base_node_data`, and that helper returns immediately unless `runtime_mode` is `RAG_PIPELINE`. So a RAG-pipeline knowledge base can commit in time and see the new model, while every other knowledge base cannot.

- [`api/services/dataset_service.py#L858-L859` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/services/dataset_service.py#L858-L859)
- [`api/services/dataset_service.py#L928-L929` (`main`, 8d18931)](https://github.com/langgenius/dify/blob/8d18931125782b475b7a9c54b99ced9549e3ddb9/api/services/dataset_service.py#L928-L929)

#### How we verified it

1. Read the raw stored vector for a known chunk out of the vector store.
2. Embed that chunk's text with model `A` and with model `B` through the same provider Dify uses.
3. Compute cosine similarity of the stored vector against both. The stored vector matches the model that was declared *before* the `PATCH` (0.99) and not the one declared after it (0.40).

Comparing declared metadata alone cannot detect this, because the declaration is the half that is correct.

#### Workaround we used

There is no way to re-drive the re-index once the two have diverged, so we restored consistency in the other direction: setting `embedding_model_provider`, `embedding_model` and `retrieval_model` on the `datasets` row back to the previously declared model, directly in the database. That does not trigger a re-index, so the declaration ends up matching the vectors that are actually stored. It is a repair, not a way to complete the switch.

#### Suggested direction

Dispatching after the commit, or passing the intended embedding model to the task instead of having the worker re-read the row, would both remove the dependency on commit timing. Given that `commit()` inside the service caused #39191, deferring the dispatch until after the transaction commits (for example via an `after_commit` hook on the session, or by having the controller dispatch it) looks like the smaller change.