cloudflare / cloudflare/developer-platform

🐛 BUG: AI Search — R2-sourced chunks never reach Vectorize (stuck at "completed" without embedding)

Abierto
#34 0 comentarios 0 reacciones 0 asignados Ver en GitHub
awaiting-response:cloudflare
Lenguaje dominante
Sin datos de lenguaje
Estrellas
1
Forks
0
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### What version of Wrangler are you using?

4.110.0

### What operating system are you using?

Windows

### Describe the Bug

Using `wrangler ai-search` (AI Search / AutoRAG), R2-sourced items are crawled, parsed, and
chunked successfully (`"status": "completed"`, non-zero `chunks_count`, no errors), but their
vectors are never inserted into Vectorize. Every subsequent indexing job sees the items as
already "completed" and silently skips re-embedding them, with no self-serve way to force a
retry (deleting the item via the Items API returns `item_not_found` for R2-origin items).

**Account ID:** 82df962258ddb15f8bd78b6c5e440548
**Instance:** `ai_search_alsinor` (namespace: default)
**Data source:** R2 bucket `alsinor-ai-search-docs` (`"type": "r2"`)

This instance was recreated on 2026-07-22 to fix an earlier issue where `"type"` was `null`
despite `source`/`source_params` pointing at the R2 bucket (that part is fixed — `"type": "r2"`
correctly set on the new instance via `POST /accounts/{account_id}/ai-search/namespaces/default/instances`).

#### Evidence

1. **Items list** (`GET .../instances/ai_search_alsinor/items`) — all 6 R2-sourced items show:
- `"source_id": "r2:alsinor-ai-search-docs"`
- `"status": "completed"`, `"error": null`, `"next_action": null`
- non-zero `chunks_count` (totaling 20 chunks across the 6 files)

2. **Stats** (`GET .../instances/ai_search_alsinor/stats`):
```json
{
"completed": 81, "error": 0, "queued": 0, "running": 0,
"file_embed_errors": {},
"engine": { "vectorize": { "vectorsCount": 91 } }
}
```
`vectorsCount: 91` exactly matches the sum of `chunks_count` across only the 75
**builtin**-sourced items (91). The R2 items' 20 chunks are completely absent from
`vectorsCount`. `file_embed_errors` is empty — no error is recorded anywhere.

3. **Search is empty for R2-unique content** — `wrangler ai-search search ai_search_alsinor
--query "..."` with terms unique to the R2 documents (company name, addresses, phone
numbers) returns zero relevant results, confirming the vectors genuinely aren't searchable
(not just a relevance/threshold issue — `hybrid_search_enabled: false`, pure vector search).

4. **Job logs, compared side by side** — this is the clearest signal:

Job 1 (`45977ed8-6957-451b-8209-4dc7ccfeaa61`, first reindex after the type fix):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Processing embedding batch of 1 items
Checking 4 RUNNING file(s) for ingestion recovery
Finished embedding files
4 file(s) still pending Vectorize ingestion confirmation
```

Job 2 (`eed866de-ebc8-4903-9a5b-2843339738cb`, retriggered ~30 min later via
`wrangler ai-search jobs create ai_search_alsinor`):
```
Starting job
Retrieving job configurations
Starting indexing data source...
Getting source page/batch 1
Finished indexing data source, got 1 pages/batches, 6 files seen
Finished embedding files
```

Job 2 has **no** "Processing embedding batch of N items" line at all — all 6 R2 items are
already marked `"status": "completed"` from job 1, so the pipeline skips them entirely on
every subsequent run. Job 1's "4 file(s) still pending Vectorize ingestion confirmation"
never actually resolved; it just stopped being mentioned once the items were marked complete.

5. **No way to force a retry** — `DELETE .../instances/ai_search_alsinor/items/{id}` for any of
the 6 R2-sourced item IDs (confirmed to exist via the same-second GET) returns:
```json
{"errors":[{"code":7041,"message":"item_not_found"}]}
```
R2-origin items appear to be managed exclusively by the R2 sync pipeline and cannot be
deleted via the generic Items API the way builtin-storage items can.

#### Ruled out

- No active Cloudflare incident affects this: the Workers AI incident active at the time only
impacted `@cf/moonshotai/kimi-k2.6` / `kimi-k2.7-code`, not our embedding model
(`@cf/qwen/qwen3-embedding-0.6b`). Vectorize showed no incidents.
- Not a service-token permission issue as far as I can tell — the same token/instance
successfully embeds all 75 builtin-storage items without issue; only R2-origin items are
affected.
- Not a caching artifact — confirmed with multiple distinct, R2-content-specific queries, and
by directly comparing `vectorsCount` against the known chunk-count breakdown by source.

### Steps to Reproduce

1. Create an AI Search instance with an R2 bucket as the data source (`"type": "r2"`).
2. Add a handful of markdown files to the R2 bucket.
3. Also populate the instance's builtin storage via the Items API (optional, but is what
surfaced the discrepancy here).
4. Run `wrangler ai-search jobs create ` to trigger indexing.
5. Check `wrangler ai-search stats --json` — `engine.vectorize.vectorsCount` will
only reflect the builtin-storage chunks, never the R2-sourced ones, even though the R2 items
show `"status": "completed"` in `wrangler ai-search get`/items listing.
6. Re-running `jobs create` does not fix it — subsequent runs skip the already-"completed" R2
items entirely (see job log comparison above).

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza reproduciendo el problema con `wrangler ai-search jobs create` y luego compara `wrangler ai-search stats --json` con el listado de elementos para los chunks obtenidos de R2. Sigue el flujo de indexación e ingesta en Vectorize responsable de marcar los elementos como completados, usando los logs del job como punto de entrada observado. Se considera terminado cuando los chunks de R2 producen vectores, aparecen en `vectorsCount` y se pueden reintentar cuando falla la confirmación de la ingesta.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Área
cloud, search
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
42/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.