cloudflare / cloudflare/developer-platform

🐛 BUG: AI Search — R2-sourced chunks never reach Vectorize (stuck at "completed" without embedding)

Ouverte
#34 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
awaiting-response:cloudflare
Langage dominant
Aucune donnée de langage
Étoiles
1
Forks
0
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

### What version of Wrangler are you using?

4.110.0

### What operating system are you using?

Windows

### Describe the Bug

Using `wrangler ai-search` (AI Search / AutoRAG), R2-sourced items are crawled, parsed, and
chunked successfully (`"status": "completed"`, non-zero `chunks_count`, no errors), but their
vectors are never inserted into Vectorize. Every subsequent indexing job sees the items as
already "completed" and silently skips re-embedding them, with no self-serve way to force a
retry (deleting the item via the Items API returns `item_not_found` for R2-origin items).

**Account ID:** 82df962258ddb15f8bd78b6c5e440548
**Instance:** `ai_search_alsinor` (namespace: default)
**Data source:** R2 bucket `alsinor-ai-search-docs` (`"type": "r2"`)

This instance was recreated on 2026-07-22 to fix an earlier issue where `"type"` was `null`
despite `source`/`source_params` pointing at the R2 bucket (that part is fixed — `"type": "r2"`
correctly set on the new instance via `POST /accounts/{account_id}/ai-search/namespaces/default/instances`).

#### Evidence

1. **Items list** (`GET .../instances/ai_search_alsinor/items`) — all 6 R2-sourced items show:
- `"source_id": "r2:alsinor-ai-search-docs"`
- `"status": "completed"`, `"error": null`, `"next_action": null`
- non-zero `chunks_count` (totaling 20 chunks across the 6 files)

2. **Stats** (`GET .../instances/ai_search_alsinor/stats`):
```json
{
"completed": 81, "error": 0, "queued": 0, "running": 0,
"file_embed_errors": {},
"engine": { "vectorize": { "vectorsCount": 91 } }
}
```
`vectorsCount: 91` exactly matches the sum of `chunks_count` across only the 75
**builtin**-sourced items (91). The R2 items' 20 chunks are completely absent from
`vectorsCount`. `file_embed_errors` is empty — no error is recorded anywhere.

3. **Search is empty for R2-unique content** — `wrangler ai-search search ai_search_alsinor
--query "..."` with terms unique to the R2 documents (company name, addresses, phone
numbers) returns zero relevant results, confirming the vectors genuinely aren't searchable
(not just a relevance/threshold issue — `hybrid_search_enabled: false`, pure vector search).

4. **Job logs, compared side by side** — this is the clearest signal:

Job 1 (`45977ed8-6957-451b-8209-4dc7ccfeaa61`, first reindex after the type fix):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Processing embedding batch of 1 items
Checking 4 RUNNING file(s) for ingestion recovery
Finished embedding files
4 file(s) still pending Vectorize ingestion confirmation
```

Job 2 (`eed866de-ebc8-4903-9a5b-2843339738cb`, retriggered ~30 min later via
`wrangler ai-search jobs create ai_search_alsinor`):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Finished embedding files
```

Job 2 has **no** "Processing embedding batch of N items" line at all — all 6 R2 items are
already marked `"status": "completed"` from job 1, so the pipeline skips them entirely on
every subsequent run. Job 1's "4 file(s) still pending Vectorize ingestion confirmation"
never actually resolved; it just stopped being mentioned once the items were marked complete.

5. **No way to force a retry** — `DELETE .../instances/ai_search_alsinor/items/{id}` for any of
the 6 R2-sourced item IDs (confirmed to exist via the same-second GET) returns:
```json
{"errors":[{"code":7041,"message":"item_not_found"}]}
```
R2-origin items appear to be managed exclusively by the R2 sync pipeline and cannot be
deleted via the generic Items API the way builtin-storage items can.

#### Ruled out

- No active Cloudflare incident affects this: the Workers AI incident active at the time only
impacted `@cf/moonshotai/kimi-k2.6` / `kimi-k2.7-code`, not our embedding model
(`@cf/qwen/qwen3-embedding-0.6b`). Vectorize showed no incidents.
- Not a service-token permission issue as far as I can tell — the same token/instance
successfully embeds all 75 builtin-storage items without issue; only R2-origin items are
affected.
- Not a caching artifact — confirmed with multiple distinct, R2-content-specific queries, and
by directly comparing `vectorsCount` against the known chunk-count breakdown by source.

### Steps to Reproduce

1. Create an AI Search instance with an R2 bucket as the data source (`"type": "r2"`).
2. Add a handful of markdown files to the R2 bucket.
3. Also populate the instance's builtin storage via the Items API (optional, but is what
surfaced the discrepancy here).
4. Run `wrangler ai-search jobs create ` to trigger indexing.
5. Check `wrangler ai-search stats --json` — `engine.vectorize.vectorsCount` will
only reflect the builtin-storage chunks, never the R2-sourced ones, even though the R2 items
show `"status": "completed"` in `wrangler ai-search get`/items listing.
6. Re-running `jobs create` does not fix it — subsequent runs skip the already-"completed" R2
items entirely (see job log comparison above).

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par reproduire le problème avec `wrangler ai-search jobs create`, puis comparez `wrangler ai-search stats --json` et la liste des éléments pour les chunks provenant de R2. Suivez le chemin d’indexation et d’ingestion dans Vectorize responsable de marquer les éléments comme terminés, en utilisant les journaux du job comme point d’entrée observé. Le travail est terminé lorsque les chunks R2 produisent des vecteurs, apparaissent dans `vectorsCount` et peuvent être réessayés lorsque la confirmation de l’ingestion échoue.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Domaine
cloud, search
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
42/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.