`scraper_runner.py`: Failed embedding batches silently upsert zero-vector items
- 主要语言
- TypeScript
- 星标
- 0
- 派生
- 0
- 平均合并
- 16 分钟
- 30 天内合并 PR
- 1
描述
### Problem
When an embedding batch fails (Azure OpenAI timeout, quota exhaustion, etc.), `scraper_runner.py` fills the embedding field with an empty list `[]` and continues to upsert those items:
**File:** `PluckIt.Processor/agents/scraper_runner.py:219`
```python
embeddings = [] # fallback on failure
# items are still upserted below with empty embeddings
```
Items upserted with a zero/empty vector become silently unfindable via semantic search. There is no log of which batches failed or which item IDs are affected.
### Impact
- Scraped items that hit a transient embedding failure are permanently broken for vector search
- No observability — operators have no way to identify or reprocess affected items
- Silent data corruption: Cosmos has the item, but search never surfaces it
### Proposed Fix
1. Log failed item IDs with `logger.warning("Embedding failed for items: %s", item_ids)`
2. Skip the upsert for items with failed embeddings (or mark them with `embeddingStatus: "failed"`)
3. Optionally retry the batch with exponential backoff before giving up
### Functionality Impact
Items with failed embeddings are already broken for search — skipping their upsert prevents the silent corruption. A retry makes the scraper more resilient to transient quota limits.
贡献指南
评估
这个 Issue 还没有评估数据。