elastic / elastic/semantic-code-search-indexer
feat: add startup health check and intelligent auto-retry for failed documents
- Dominant language
- TypeScript
- Stars
- 19
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Documents can fail during indexing due to transient issues (Elasticsearch not ready, network blips, inference service warming up). The current retry mechanism exhausts all 3 attempts in under a second - before transient issues resolve.
```
Current Behavior
════════════════
Indexer starts immediately, ES might not be ready:
┌─────────────────┐ ┌─────────────────┐
│ Indexer Start │────►│ Process Batch │──── ES not ready ────► FAIL
└─────────────────┘ └─────────────────┘ │
▼
Retry 1 (immediate) ──► FAIL
│
▼
Retry 2 (immediate) ──► FAIL
│
▼
Retry 3 (immediate) ──► FAIL
│
▼
Mark as FAILED permanently
(ES became ready 2 seconds later)
All 3 retries exhaust in < 1 second
═══════════════════════════════════
Transient issues typically resolve in 5-30 seconds
```
While `queue:retry-failed` exists for manual recovery, this requires human intervention - not practical for CI, automated pipelines, or single-run workflows.
## Proposed Solution
### 1. Startup Health Check
Wait for Elasticsearch before processing:
```
Proposed: Health Check Gate
═══════════════════════════
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Indexer Start │────►│ Health Check │────►│ Process Batch │
└─────────────────┘ └────────┬────────┘ └─────────────────┘
│
▼
┌────────────────────────┐
│ ES cluster.health() │
└────────────┬───────────┘
│
┌────────────┴───────────┐
│ │
▼ ▼
✓ Ready ✗ Not Ready
│ │
│ ▼
│ Wait 5s, retry (up to 10x)
│ │
▼ ▼
Continue Fail after 50s total
```
### 2. Intelligent Auto-Retry
After main indexing completes, automatically retry failed documents:
```
Auto-Retry Flow
═══════════════
┌──────────────────┐
│ Main Index Done │
│ Failed: 1000 │
└────────┬─────────┘
│
▼
┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 1 │────►│ Result: 1000 → 100 failed │
└──────────────────┘ │ Recovered: 900 (90%) │
│ Threshold: 100 (10% of 1000) │
│ 900 ≥ 100? YES ──► Continue │
└─────────────────────────────────────┘
│
▼
┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 2 │────►│ Result: 100 → 8 failed │
└──────────────────┘ │ Recovered: 92 (92%) │
│ Threshold: 10 (10% of 100) │
│ 92 ≥ 10? YES ──► Continue │
└─────────────────────────────────────┘
│
▼
┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 3 │────►│ Result: 8 → 0 failed │
└──────────────────┘ │ All recovered ──► Exit 0 │
└─────────────────────────────────────┘
```
### Termination Conditions
```
When does auto-retry STOP?
══════════════════════════
┌─────────────────────────────────────────────────────────────────┐
│ │
│ 1. ALL RECOVERED ─────────────────────────────► Exit 0 ✓ │
│ Failed count reaches 0 │
│ │
│ 2. INSUFFICIENT PROGRESS ─────────────────────► Exit 1 ✗ │
│ Recovered < max(10% of previous, 3 docs) │
│ │
│ Example: 50 failed → 48 failed │
│ Recovered: 2 │
│ Threshold: max(5, 3) = 5 │
│ 2 < 5 → STOP (not making progress) │
│ │
│ 3. MAX ATTEMPTS ──────────────────────────────► Exit 1 ✗ │
│ 3 consecutive retry rounds exhausted │
│ (regardless of progress) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
### Example Scenarios
```
Scenario A: Full Recovery
═════════════════════════
Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 1000→100 │ 900 │ 100 │ ✓ Continue
2 │ 100→8 │ 92 │ 10 │ ✓ Continue
3 │ 8→0 │ 8 │ 3 │ ✓ Done, exit 0
Scenario B: Stuck Failures (persistent errors)
══════════════════════════════════════════════
Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 50→45 │ 5 │ 5 │ ✓ Continue
2 │ 45→44 │ 1 │ 5 │ ✗ Stop, exit 1
(only 1 recovered, need 5 - these are likely permanent failures)
Scenario C: Slow Progress, Hit Max Attempts
═══════════════════════════════════════════
Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 100→90 │ 10 │ 10 │ ✓ Continue
2 │ 90→81 │ 9 │ 9 │ ✓ Continue
3 │ 81→73 │ 8 │ 9 │ ✗ Stop, exit 1
(making progress but hit 3 attempt limit)
```
### CLI Interface
```bash
# Default: auto-retry enabled
node dist/index.js index /path/to/repo
# Disable auto-retry for manual control
node dist/index.js index /path/to/repo --no-auto-retry
```
## Implementation
### Code Changes
| File | Change |
|------|--------|
| `indexer_worker.ts` | Add `waitForElasticsearch()` health check before processing |
| `index_command.ts` | Add auto-retry loop after worker completes |
| `index_command.ts` | Add `--no-auto-retry` CLI option |
| `sqlite_queue.ts` | Add `getFailedCount()` method if not exists |
## Acceptance Criteria
- [ ] Indexer waits for ES health check before processing (up to 10 attempts, 5s apart)
- [ ] Auto-retry runs automatically after main indexing completes
- [ ] Auto-retry stops when: all recovered, insufficient progress (<10% or <3), or 3 attempts exhausted
- [ ] `--no-auto-retry` flag available to disable
- [ ] Clear logging shows retry progress and decisions
- [ ] Final message points to `queue:retry-failed` command if failures remain
- [ ] Exit code 0 = no failures, exit code 1 = failures persist
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading indexer_worker.ts and index_command.ts, then inspect sqlite_queue.ts and the existing queue:retry-failed command. Trace how indexing completes, failures are counted, and CLI options and exit codes are handled. Done means the listed health-check, retry, logging, flag, and failure-reporting acceptance criteria are covered by tests or observable command behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, typescript
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100