elastic / elastic/semantic-code-search-indexer
feat: add detailed timing metrics for bottleneck identification
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 19
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Current metrics capture **counts and totals**, but not **per-phase timing breakdowns**. When indexing takes 21 hours, we can't identify which phase is the bottleneck.
```
Current Metrics: We Know WHAT, Not WHERE
════════════════════════════════════════
┌─────────────────────────────────────────────────────────────────┐
│ batchDuration = 5000ms │
│ (total only) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ [Dequeue] [ES Bulk Request] [Commit] │
│ SQLite Network + Infer SQLite │
│ │
│ ??? ??? ??? │
│ │
│ Is this Is inference slow? Is this │
│ slow? Network latency? slow? │
│ │
└─────────────────────────────────────────────────────────────────┘
We see: "batch took 5000ms"
We don't see: "4500ms was ES inference, 300ms was dequeue, 200ms was commit"
```
## Why Not Use ES/Kibana Monitoring?
ES Stack Monitoring shows **server-side** metrics:
- Bulk indexing rate
- Shard-level latency
- Cluster health
But it **cannot** show **client-side** timing:
- SQLite queue operations (enqueue/dequeue/commit)
- File parsing time
- Network latency vs ES processing
- Time waiting for inference
We need client-side instrumentation to identify whether bottlenecks are in:
- Our code (parsing, SQLite)
- Network (latency to ES)
- ES processing (inference, indexing)
## What Exists Today
```typescript
// metrics.ts - Current metrics
ParserMetrics:
✓ filesProcessed (counter)
✓ filesFailed (counter)
✓ chunksCreated (counter)
✓ chunkSize (histogram)
✗ parseDuration // MISSING
✗ enqueueDuration // MISSING
QueueMetrics:
✓ documentsEnqueued (counter)
✓ documentsDequeued (counter)
✗ enqueueDuration // MISSING
✗ dequeueDuration // MISSING
IndexerMetrics:
✓ batchDuration (histogram) // Total only
✓ batchSize (histogram)
✗ esRequestDuration // MISSING - ES call breakdown
✗ commitDuration // MISSING
```
## Proposed Solution
Add timing histograms for each phase:
```
Proposed: Per-Phase Timing
══════════════════════════
Producer Phase:
┌─────────────────────────────────────────────────────────────────┐
│ │
│ [Read File] → [Parse AST] → [Extract] → [Enqueue] │
│ │
│ fileReadDuration parseDuration extractDuration enqueueDuration
│ (histogram) (histogram) (histogram) (histogram) │
│ │
└─────────────────────────────────────────────────────────────────┘
Worker Phase:
┌─────────────────────────────────────────────────────────────────┐
│ │
│ [Dequeue] → [ES Bulk Request] → [Commit] │
│ │
│ dequeueDuration esRequestDuration commitDuration │
│ (histogram) (histogram) (histogram) │
│ │
│ │ │
│ ▼ │
│ (includes network + inference) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## Implementation
### New Metrics
```typescript
// Producer timing
producer.fileReadDuration // Time to read file from disk
producer.parseDuration // Time for tree-sitter/parser
producer.extractDuration // Time for symbol extraction
producer.enqueueDuration // Time for SQLite INSERT
// Worker timing
worker.dequeueDuration // Time for SQLite SELECT + UPDATE
worker.esRequestDuration // Time for ES bulk API call
worker.commitDuration // Time for SQLite DELETE
```
### Instrumentation Points
| File | Add Timing For |
|------|----------------|
| `producer_worker.ts` | File read, parse, extract |
| `sqlite_queue.ts` | Enqueue, dequeue, commit operations |
| `indexer_worker.ts` | ES request (wrap `indexCodeChunks` call) |
| `elasticsearch.ts` | Optionally: internal bulk timing |
### Example Output
After implementation, logs/metrics would show:
```
Bottleneck Analysis (P95 timings):
──────────────────────────────────
Phase P95 % of Total
─────────────────────────────────────────
File Read 5ms 0.1%
Parse 15ms 0.3%
Extract 10ms 0.2%
Enqueue 20ms 0.4%
Dequeue 25ms 0.5%
ES Request 4800ms 96.0% ◄── BOTTLENECK
Commit 25ms 0.5%
─────────────────────────────────────────
Recommendation: ES inference is the bottleneck.
Consider: Batch size tuning, ES scaling, or async inference.
```
## Acceptance Criteria
- [ ] Producer timing metrics: `fileReadDuration`, `parseDuration`, `extractDuration`, `enqueueDuration`
- [ ] Worker timing metrics: `dequeueDuration`, `esRequestDuration`, `commitDuration`
- [ ] All timing metrics are histograms (for percentile analysis)
- [ ] Metrics include repo/branch labels for filtering
- [ ] Existing `batchDuration` preserved for backward compatibility
- [ ] Documentation updated with new metric names
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading metrics.ts, then inspect the instrumentation points in producer_worker.ts, sqlite_queue.ts, indexer_worker.ts, and elasticsearch.ts. Confirm how existing histograms and repo/branch labels are defined, then add the listed phase metrics while preserving batchDuration. Done means all acceptance criteria are met and the documentation names the new metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, sqlite, typescript
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100