Indexing: token counting sends the whole document in one plugin request (no max_chunks batching)

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

1.11.2 (self-hosted); code path unchanged on main as of 2026-07-25 (latest release 1.16.0)

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

> **Environment note.** Our deployment is a self-hosted Kubernetes one running an Enterprise build based
> on core 1.11.2. The affected code is OSS core, and I verified it is unchanged on `main`; Dify support
> confirmed it is not fixed upstream and suggested filing it here. **This is not a plugin bug** — see the
> note about #15035 below: that report was worked around inside the TEI plugin, but core still emits the
> unbounded request for every provider.

### The core problem

During indexing, the **token-counting** call sends **every chunk of the document in a single request**
to the embedding plugin, with no `max_chunks` batching — while the far heavier **embedding** call is
already batched. The size of that one request therefore grows linearly with the document, unbounded.

* `api/core/indexing_runner.py:117` / `:202`
`token_counts = calculate_segment_token_counts(dataset=dataset, documents=documents)`
* `api/core/rag/embedding/token_counter.py:25`
`return embedding_model.get_text_embedding_num_tokens([document.page_content for document in documents])`
* `api/core/model_manager.py:256-270`
`get_text_embedding_num_tokens(texts)` forwards `texts` to the plugin as-is.

versus the batched embedding path:

* `api/core/rag/embedding/cached_embedding.py:60-66`
`max_chunks = model_schema.model_properties[MAX_CHUNKS]` → `for i in range(0, len(texts), max_chunks)`

In our case an 11.4MB xlsx expands to ~25MB of extracted text / 9,981 chunks, so this one call carries
a **25MB** body, while every embedding call stays at ~97KB (32 chunks). Note this is provider-independent:
token counting is done locally by the tokenizer inside the plugin, so switching the embedding backend
does not help.

### How to reproduce

1. Create a Knowledge Base with **high_quality** indexing.
2. Upload a document large enough to produce thousands of chunks.
3. Observe the single `get_text_embedding_num_tokens` request whose body is the entire document.

Whether this is merely wasteful or fatal depends on what sits downstream of that request:

* **Tokenizer backend with a batch/body cap — reproducible on Community**: this is exactly #15035, where
TEI's `/tokenize` returns `413 Payload Too Large`, with `batch size 213 > maximum allowed batch size 128`
in the TEI log. That issue was worked around on the plugin side; the core behaviour is unchanged, so
the same class of failure keeps resurfacing per provider.
* **Any proxy in front of the plugin runtime with a body-size limit** (our deployment has an nginx sidecar
with `client_max_body_size 20M`): the request is rejected with **HTTP 413** and an HTML error page.
plugin-daemon expects JSON and surfaces it as:

```
PluginDaemonInnerError ... invalid character '<' looking for beginning of value
... status: 413 ... original response: 413 Request Entity Too Large ...
```

* **A memory-constrained plugin-daemon** (ours: 256Mi limit): it is **OOMKilled (exit 137)** while buffering
the ~25MB body, before any error can be logged — indexing fails silently with nothing in the UI.

In all three cases indexing fails **before any embedding request is made**, and the document ends up with
0 segments and NULL `word_count` / `tokens`.

### Verification we ran

From an api pod, using Dify's own `ModelManager` against the same plugin path as indexing:

| Call | Payload | Result |
|---|---|---|
| `invoke_text_embedding`, 32 real chunks | 58KB | 200 OK |
| `invoke_text_embedding`, 32 synthetic chunks | 89KB | 200 OK |
| `invoke_text_embedding`, 1 large chunk | 586KB | 200 OK |
| `invoke_text_embedding`, 64 chunks | 192KB | 200 OK |
| **`get_text_embedding_num_tokens`, all chunks** | **25.2MB** | **413 — identical to the UI error** |

The failure is reproduced only by the unbatched token-count call. Splitting the source file into smaller
documents is the only workaround we have. We also confirmed it fails identically with two different
embedding providers (Azure OpenAI and a self-hosted xinference model), consistent with token counting
never reaching the embedding backend.

### ✔️ Expected Behavior

Token counting during indexing should be chunked the same way embedding already is — i.e. split the
input into `max_chunks`-sized batches (and additionally bound the serialized payload size) — so that a
single plugin request stays bounded regardless of document size.

Indexing a large document with high_quality indexing should succeed without the user having to split
the source file, and without depending on the body-size limit of whatever proxy sits in front of the
plugin runtime.

### ❌ Actual Behavior

The whole document is sent in one request. The request size scales with the document (25MB for an
11.4MB xlsx), so indexing fails once it exceeds a body-size limit (HTTP 413, surfaced as a JSON parse
error against an HTML error page) or exhausts plugin-daemon memory (OOMKilled, silent failure with no
log line). Raising the proxy limit or the memory limit only moves the ceiling; a larger document fails
again.

Suggested fix (happy to submit a PR if the direction is agreed):

* batch in `calculate_segment_token_counts` / `get_text_embedding_num_tokens` using the same
`ModelPropertyKey.MAX_CHUNKS` logic as `cached_embedding.py`, concatenating the per-batch results;
* additionally cap each batch by **serialized JSON payload size**, so that a single oversized chunk —
or CJK text that expands after JSON escaping — cannot exceed the limit on its own.

Start by reading api/core/indexing_runner.py at the token-counting calls, then follow calculate_segment_token_counts in api/core/rag/embedding/token_counter.py and get_text_embedding_num_tokens in api/core/model_manager.py. Compare that path with the batching logic in api/core/rag/embedding/cached_embedding.py; done means large-document indexing uses bounded token-count requests and succeeds without exceeding plugin or proxy body limits.

Written by the indexing model from the issue text.