feat: pool worker threads in full indexing (avoid 1 Worker per file)

Context

Incremental indexing already uses a reusable worker pool (src/commands/incremental_index_command.ts) to avoid per-file worker startup/teardown overhead.

Full indexing (src/commands/full_index_producer.ts) still spawns one Worker per file, which can be a major perf and stability hit on large repos.

Historical context / likely rationale

I searched the repo's merged PR history for explicit rationale around producer worker lifecycle (worker pools vs per-file workers) and did not find a PR/issue that directly explains this choice.

The closest related production-history signals are about consumer-side overload when Elasticsearch is slow/timeouting:

• Backpressure to avoid OOM / unlimited dequeuing: https://github.com/elastic/semantic-code-search-indexer/pull/21
• Follow-up hardening with detailed prod symptoms when ES is slow (timeout death spirals): https://github.com/elastic/semantic-code-search-indexer/pull/113

Those fixes are about not overwhelming memory when ES is slow. They don’t directly require per-file producer workers; producer concurrency is already bounded by p-queue.

Plausible reasons for per-file workers (no historical ticket found):

• Simplicity: spawn a worker, parse one file, terminate.
• Defensive memory reset: if tree-sitter/native parsing accumulates memory over time, terminating per file forces a reset of native allocations.

Why this matters

• Worker startup/teardown overhead dominates when indexing large codebases.
• High worker churn increases memory pressure and can trigger OS limits / slowdowns.

Where in code

• src/commands/full_index_producer.ts: creates a new Worker inside the per-file loop.

Suggested fix

Pool worker threads for full indexing while keeping safety under load:

• Create N workers upfront (N = min(CPU_CORES, configured pool size, file count)).
• Maintain an idle-worker queue and assign jobs.
• Ensure event listeners are cleaned up per job.
• Terminate workers only once after queue drain.

Optional safety guardrail (to preserve the likely “memory reset” benefit of per-file workers):

• Add a worker recycle policy (terminate/recreate a worker after N files or after a memory threshold).

Config knob:

• If PR #135 lands, reuse PRODUCER_WORKER_POOL_SIZE and keep the same clamp-to-CPU behavior.
• If #135 does not land, introduce an equivalent pool-size config (env var or CLI flag).

Test plan

• Unit test asserting Worker is instantiated at most poolSize times during full indexing for many files.

Read src/commands/full_index_producer.ts and compare its per-file Worker lifecycle with the reusable pool in src/commands/incremental_index_command.ts. Implement bounded worker reuse with the stated cleanup and termination behavior, then add the proposed unit test showing that Worker is instantiated at most poolSize times during full indexing for many files.