elastic / elastic/semantic-code-search-indexer

feat: add startup health check and intelligent auto-retry for failed documents

Open
#120 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
19
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Problem

Documents can fail during indexing due to transient issues (Elasticsearch not ready, network blips, inference service warming up). The current retry mechanism exhausts all 3 attempts in under a second - before transient issues resolve.

```
Current Behavior
════════════════

Indexer starts immediately, ES might not be ready:

┌─────────────────┐ ┌─────────────────┐
│ Indexer Start │────►│ Process Batch │──── ES not ready ────► FAIL
└─────────────────┘ └─────────────────┘ │

Retry 1 (immediate) ──► FAIL


Retry 2 (immediate) ──► FAIL


Retry 3 (immediate) ──► FAIL


Mark as FAILED permanently
(ES became ready 2 seconds later)

All 3 retries exhaust in < 1 second
═══════════════════════════════════
Transient issues typically resolve in 5-30 seconds
```

While `queue:retry-failed` exists for manual recovery, this requires human intervention - not practical for CI, automated pipelines, or single-run workflows.

## Proposed Solution

### 1. Startup Health Check

Wait for Elasticsearch before processing:

```
Proposed: Health Check Gate
═══════════════════════════

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Indexer Start │────►│ Health Check │────►│ Process Batch │
└─────────────────┘ └────────┬────────┘ └─────────────────┘


┌────────────────────────┐
│ ES cluster.health() │
└────────────┬───────────┘

┌────────────┴───────────┐
│ │
▼ ▼
✓ Ready ✗ Not Ready
│ │
│ ▼
│ Wait 5s, retry (up to 10x)
│ │
▼ ▼
Continue Fail after 50s total
```

### 2. Intelligent Auto-Retry

After main indexing completes, automatically retry failed documents:

```
Auto-Retry Flow
═══════════════

┌──────────────────┐
│ Main Index Done │
│ Failed: 1000 │
└────────┬─────────┘


┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 1 │────►│ Result: 1000 → 100 failed │
└──────────────────┘ │ Recovered: 900 (90%) │
│ Threshold: 100 (10% of 1000) │
│ 900 ≥ 100? YES ──► Continue │
└─────────────────────────────────────┘


┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 2 │────►│ Result: 100 → 8 failed │
└──────────────────┘ │ Recovered: 92 (92%) │
│ Threshold: 10 (10% of 100) │
│ 92 ≥ 10? YES ──► Continue │
└─────────────────────────────────────┘


┌──────────────────┐ ┌─────────────────────────────────────┐
│ Retry Attempt 3 │────►│ Result: 8 → 0 failed │
└──────────────────┘ │ All recovered ──► Exit 0 │
└─────────────────────────────────────┘
```

### Termination Conditions

```
When does auto-retry STOP?
══════════════════════════

┌─────────────────────────────────────────────────────────────────┐
│ │
│ 1. ALL RECOVERED ─────────────────────────────► Exit 0 ✓ │
│ Failed count reaches 0 │
│ │
│ 2. INSUFFICIENT PROGRESS ─────────────────────► Exit 1 ✗ │
│ Recovered < max(10% of previous, 3 docs) │
│ │
│ Example: 50 failed → 48 failed │
│ Recovered: 2 │
│ Threshold: max(5, 3) = 5 │
│ 2 < 5 → STOP (not making progress) │
│ │
│ 3. MAX ATTEMPTS ──────────────────────────────► Exit 1 ✗ │
│ 3 consecutive retry rounds exhausted │
│ (regardless of progress) │
│ │
└─────────────────────────────────────────────────────────────────┘
```

### Example Scenarios

```
Scenario A: Full Recovery
═════════════════════════

Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 1000→100 │ 900 │ 100 │ ✓ Continue
2 │ 100→8 │ 92 │ 10 │ ✓ Continue
3 │ 8→0 │ 8 │ 3 │ ✓ Done, exit 0

Scenario B: Stuck Failures (persistent errors)
══════════════════════════════════════════════

Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 50→45 │ 5 │ 5 │ ✓ Continue
2 │ 45→44 │ 1 │ 5 │ ✗ Stop, exit 1

(only 1 recovered, need 5 - these are likely permanent failures)

Scenario C: Slow Progress, Hit Max Attempts
═══════════════════════════════════════════

Attempt │ Failed │ Recovered │ Threshold │ Action
────────┼────────┼───────────┼───────────┼─────────────
1 │ 100→90 │ 10 │ 10 │ ✓ Continue
2 │ 90→81 │ 9 │ 9 │ ✓ Continue
3 │ 81→73 │ 8 │ 9 │ ✗ Stop, exit 1

(making progress but hit 3 attempt limit)
```

### CLI Interface

```bash
# Default: auto-retry enabled
node dist/index.js index /path/to/repo

# Disable auto-retry for manual control
node dist/index.js index /path/to/repo --no-auto-retry
```

## Implementation

### Code Changes

| File | Change |
|------|--------|
| `indexer_worker.ts` | Add `waitForElasticsearch()` health check before processing |
| `index_command.ts` | Add auto-retry loop after worker completes |
| `index_command.ts` | Add `--no-auto-retry` CLI option |
| `sqlite_queue.ts` | Add `getFailedCount()` method if not exists |

## Acceptance Criteria

- [ ] Indexer waits for ES health check before processing (up to 10 attempts, 5s apart)
- [ ] Auto-retry runs automatically after main indexing completes
- [ ] Auto-retry stops when: all recovered, insufficient progress (<10% or <3), or 3 attempts exhausted
- [ ] `--no-auto-retry` flag available to disable
- [ ] Clear logging shows retry progress and decisions
- [ ] Final message points to `queue:retry-failed` command if failures remain
- [ ] Exit code 0 = no failures, exit code 1 = failures persist

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading indexer_worker.ts and index_command.ts, then inspect sqlite_queue.ts and the existing queue:retry-failed command. Trace how indexing completes, failures are counted, and CLI options and exit codes are handled. Done means the listed health-check, retry, logging, flag, and failure-reporting acceptance criteria are covered by tests or observable command behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, typescript
Domain
backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.