elastic / elastic/semantic-code-search-indexer

feat: ES bulk indexing optimizations (compression, refresh, replicas)

Open
#124 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
19
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Problem

Current ES bulk indexing doesn't use available optimizations. For large repos (21h+ indexing), these add up significantly.

```
Current: Unoptimized Bulk Indexing
══════════════════════════════════

┌─────────────────────────────────────────────────────────────────┐
│ │
│ Bulk Request │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ 500 docs × 5KB = 2.5MB uncompressed │ │
│ │ │ │
│ │ ──────────────► Network ──────────────► │ │
│ │ (2.5MB payload each request) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ Meanwhile, ES is also doing: │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ • Auto-refresh every 1 second (I/O + CPU) │ │
│ │ • Replicating to replica shards (network + I/O) │ │
│ │ • Competing for resources with bulk indexing │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```

## Current State

```typescript
// elasticsearch.ts
const baseOptions: Partial = {
requestTimeout: 90000,
// compression: NOT SET (defaults to none)
};

const bulkOptions = {
refresh: false, // ✓ Good: don't wait for refresh
operations,
// But ES still auto-refreshes every 1 second in background
};
```

**Note:** `refresh: false` only means "don't wait synchronously." ES still refreshes automatically every second, consuming resources during bulk indexing.

## Proposed Optimizations

### 1. Enable Gzip Compression (Easy Win)

```
Before: 2.5MB per request
════════════════════════

Client ────── 2.5MB ──────► ES Server
(uncompressed)

After: ~250KB per request
═════════════════════════

Client ────── 250KB ──────► ES Server
(gzipped)

~10x smaller payloads = faster network transfer
```

**Implementation:**
```typescript
const client = new Client({
node: elasticsearchConfig.endpoint,
compression: 'gzip', // ← One line change
});
```

### 2. Disable Auto-Refresh During Bulk Indexing

```
Current: Background refreshes compete with indexing
═══════════════════════════════════════════════════

Time ─────────────────────────────────────────────────────────►

Bulk 1 ──► Bulk 2 ──► Bulk 3 ──► Bulk 4 ──► Bulk 5
│ │ │
▼ ▼ ▼
[refresh] [refresh] [refresh]
(1 sec) (1 sec) (1 sec)
│ │ │
└─────── CPU/IO work ───┘
competes with
bulk indexing

Proposed: Single refresh at end
════════════════════════════════

Time ─────────────────────────────────────────────────────────►

[Set refresh=-1] ──► Bulk 1 ──► Bulk 2 ──► ... ──► Done ──► [Refresh]


Single refresh
All data searchable
```

**Implementation:**
```typescript
// Before bulk indexing
await client.indices.putSettings({
index: indexName,
body: { 'index.refresh_interval': '-1' }
});

// ... bulk indexing ...

// After bulk indexing
await client.indices.putSettings({
index: indexName,
body: { 'index.refresh_interval': '1s' }
});
await client.indices.refresh({ index: indexName });
```

### 3. Disable Replicas During Bulk Indexing

```
Current: Every write replicated immediately
═══════════════════════════════════════════

Primary ──► Write ──► Replicate ──► Wait for ACK


Replica shard
(network + I/O)

Proposed: Replicate once at end
════════════════════════════════

[Set replicas=0] ──► Bulk writes (primary only) ──► [Set replicas=1]


ES replicates
everything at once
```

**Implementation:**
```typescript
// Before bulk indexing
const originalSettings = await client.indices.getSettings({ index: indexName });
await client.indices.putSettings({
index: indexName,
body: { 'index.number_of_replicas': 0 }
});

// ... bulk indexing ...

// After bulk indexing
await client.indices.putSettings({
index: indexName,
body: { 'index.number_of_replicas': originalReplicas }
});
```

## Expected Impact

| Optimization | Effort | Impact |
|--------------|--------|--------|
| Gzip compression | One line | 5-10x smaller payloads |
| Disable refresh | ~20 lines | 2-5x faster bulk writes |
| Disable replicas | ~20 lines | ~2x faster (if replicas > 0) |

**Combined:** Potentially 2-10x faster ES indexing phase.

## Implementation Plan

### Phase 1: Compression (Low Risk)
- Add `compression: 'gzip'` to ES client options
- No lifecycle changes needed
- Can be toggled via env var if needed

### Phase 2: Index Settings Lifecycle (Medium Risk)
- Add `startBulkIndexing(indexName)` - saves settings, optimizes for bulk
- Add `finishBulkIndexing(indexName)` - restores settings, refreshes
- Wrap producer+worker in `index_command.ts`
- Handle errors (restore settings on failure)

### Code Changes

| File | Change |
|------|--------|
| `elasticsearch.ts` | Add `compression: 'gzip'` to client |
| `elasticsearch.ts` | Add `startBulkIndexing()`, `finishBulkIndexing()` |
| `index_command.ts` | Wrap indexing in lifecycle calls |
| `config.ts` | Add `ENABLE_BULK_OPTIMIZATIONS` flag (default: true) |

## Interaction with Other Issues

| Issue | Interaction |
|-------|-------------|
| #121 (filePaths aggregation) | If #121 changes from `bulk` to `update` API, verify settings optimizations still apply. Both APIs benefit from `refresh_interval=-1` and `replicas=0`. |
| #122 (parallel enqueue/dequeue) | Settings lifecycle must handle producer failure mid-way. Use try/finally to restore settings even on error. |
| #120 (auto-retry) | No conflict. Auto-retry happens after bulk indexing, settings already restored. |

## Acceptance Criteria

- [ ] Gzip compression enabled on ES client
- [ ] `refresh_interval` set to `-1` during bulk indexing
- [ ] `number_of_replicas` set to `0` during bulk indexing (configurable)
- [ ] Original settings restored after bulk indexing completes
- [ ] Settings restored even on error (try/finally)
- [ ] Final refresh ensures all data is searchable
- [ ] Can be disabled via environment variable
- [ ] Works correctly with `--clean` flag
- [ ] Works correctly with incremental indexing
- [ ] Works with both `bulk` and `update` APIs (future-proof for #121)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.