cloudflare / cloudflare/developer-platform

🐛 BUG: AI Search — R2-sourced chunks never reach Vectorize (stuck at "completed" without embedding)

Open
#34 0 comments 0 reactions 0 assignees View on GitHub
awaiting-response:cloudflare
Dominant language
No language data
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

### What version of Wrangler are you using?

4.110.0

### What operating system are you using?

Windows

### Describe the Bug

Using `wrangler ai-search` (AI Search / AutoRAG), R2-sourced items are crawled, parsed, and
chunked successfully (`"status": "completed"`, non-zero `chunks_count`, no errors), but their
vectors are never inserted into Vectorize. Every subsequent indexing job sees the items as
already "completed" and silently skips re-embedding them, with no self-serve way to force a
retry (deleting the item via the Items API returns `item_not_found` for R2-origin items).

**Account ID:** 82df962258ddb15f8bd78b6c5e440548
**Instance:** `ai_search_alsinor` (namespace: default)
**Data source:** R2 bucket `alsinor-ai-search-docs` (`"type": "r2"`)

This instance was recreated on 2026-07-22 to fix an earlier issue where `"type"` was `null`
despite `source`/`source_params` pointing at the R2 bucket (that part is fixed — `"type": "r2"`
correctly set on the new instance via `POST /accounts/{account_id}/ai-search/namespaces/default/instances`).

#### Evidence

1. **Items list** (`GET .../instances/ai_search_alsinor/items`) — all 6 R2-sourced items show:
- `"source_id": "r2:alsinor-ai-search-docs"`
- `"status": "completed"`, `"error": null`, `"next_action": null`
- non-zero `chunks_count` (totaling 20 chunks across the 6 files)

2. **Stats** (`GET .../instances/ai_search_alsinor/stats`):
```json
{
"completed": 81, "error": 0, "queued": 0, "running": 0,
"file_embed_errors": {},
"engine": { "vectorize": { "vectorsCount": 91 } }
}
```
`vectorsCount: 91` exactly matches the sum of `chunks_count` across only the 75
**builtin**-sourced items (91). The R2 items' 20 chunks are completely absent from
`vectorsCount`. `file_embed_errors` is empty — no error is recorded anywhere.

3. **Search is empty for R2-unique content** — `wrangler ai-search search ai_search_alsinor
--query "..."` with terms unique to the R2 documents (company name, addresses, phone
numbers) returns zero relevant results, confirming the vectors genuinely aren't searchable
(not just a relevance/threshold issue — `hybrid_search_enabled: false`, pure vector search).

4. **Job logs, compared side by side** — this is the clearest signal:

Job 1 (`45977ed8-6957-451b-8209-4dc7ccfeaa61`, first reindex after the type fix):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Processing embedding batch of 1 items
Checking 4 RUNNING file(s) for ingestion recovery
Finished embedding files
4 file(s) still pending Vectorize ingestion confirmation
```

Job 2 (`eed866de-ebc8-4903-9a5b-2843339738cb`, retriggered ~30 min later via
`wrangler ai-search jobs create ai_search_alsinor`):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Finished embedding files
```

Job 2 has **no** "Processing embedding batch of N items" line at all — all 6 R2 items are
already marked `"status": "completed"` from job 1, so the pipeline skips them entirely on
every subsequent run. Job 1's "4 file(s) still pending Vectorize ingestion confirmation"
never actually resolved; it just stopped being mentioned once the items were marked complete.

5. **No way to force a retry** — `DELETE .../instances/ai_search_alsinor/items/{id}` for any of
the 6 R2-sourced item IDs (confirmed to exist via the same-second GET) returns:
```json
{"errors":[{"code":7041,"message":"item_not_found"}]}
```
R2-origin items appear to be managed exclusively by the R2 sync pipeline and cannot be
deleted via the generic Items API the way builtin-storage items can.

#### Ruled out

- No active Cloudflare incident affects this: the Workers AI incident active at the time only
impacted `@cf/moonshotai/kimi-k2.6` / `kimi-k2.7-code`, not our embedding model
(`@cf/qwen/qwen3-embedding-0.6b`). Vectorize showed no incidents.
- Not a service-token permission issue as far as I can tell — the same token/instance
successfully embeds all 75 builtin-storage items without issue; only R2-origin items are
affected.
- Not a caching artifact — confirmed with multiple distinct, R2-content-specific queries, and
by directly comparing `vectorsCount` against the known chunk-count breakdown by source.

### Steps to Reproduce

1. Create an AI Search instance with an R2 bucket as the data source (`"type": "r2"`).
2. Add a handful of markdown files to the R2 bucket.
3. Also populate the instance's builtin storage via the Items API (optional, but is what
surfaced the discrepancy here).
4. Run `wrangler ai-search jobs create ` to trigger indexing.
5. Check `wrangler ai-search stats --json` — `engine.vectorize.vectorsCount` will
only reflect the builtin-storage chunks, never the R2-sourced ones, even though the R2 items
show `"status": "completed"` in `wrangler ai-search get`/items listing.
6. Re-running `jobs create` does not fix it — subsequent runs skip the already-"completed" R2
items entirely (see job log comparison above).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.