ADORSYS-GIS / ADORSYS-GIS/lightbridge-code-intelligence

[Ticket]: Batch upsert_code_chunks the same way UNWIND batches upsert_graph

Offen
#630 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @leghadjeu-christian Auf GitHub ansehen
ticket
Vorherrschende Sprache
Rust
Sterne
0
Forks
0
Ø Merge
14 Std. 13 Min.
Gemergte PRs (30 T.)
16

Beschreibung

### Type
Performance

### Summary
We need to batch `upsert_code_chunks`'s writes into Postgres because it currently issues one
`INSERT ... ON CONFLICT DO UPDATE` per chunk, sequentially, inside a single transaction — the same
per-row-round-trip anti-pattern `upsert_graph` had against Neo4j before #626 batched it with
`UNWIND`.

Expected result: a repo's chunk batch is written to `code_chunks` in a small, bounded number of
round trips instead of one per chunk.

### Intent
`services/control-plane/src/db/code_chunks.rs::upsert_code_chunks` loops over every chunk and
executes its own `INSERT` inside one transaction. For a large repo (thousands of chunks — the same
class of repo that motivated #626's Neo4j fix), this is thousands of sequential round trips held
open in one transaction, inside one HTTP request the runner's `ControlPlaneClient` has no timeout
on. It's reachable from the exact same `index_checkout` flow as the already-fixed graph write.

### Source of truth (links)
Follow-up item from #626's review — see that PR's discussion and the implementation plan shared
alongside it (item 2, "Batch upsert_code_chunks the same way").

### Current Behavior
```rust
for chunk in chunks {
sqlx::query("INSERT INTO code_chunks (...) VALUES (...) ON CONFLICT (...) DO UPDATE SET ...")
.bind(...)
.execute(&mut *tx)
.await?;
}
```
One statement per chunk, sequentially, inside one transaction.

### Expected Behavior
A multi-row `INSERT ... VALUES (...), (...), ... ON CONFLICT (...) DO UPDATE SET ...` (built via
`sqlx::query_builder::QueryBuilder`, not currently used elsewhere in this codebase but already part
of the `sqlx` dependency already in use), issued in a small, bounded number of statements rather than
one per row. Same `ON CONFLICT` semantics, same public signature.

### Acceptance Criteria
- [ ] Given a batch of chunks, when `upsert_code_chunks` runs, then the same rows land in
`code_chunks` as today (insert-or-update semantics unchanged).
- [ ] Given a very large chunk count, when `upsert_code_chunks` runs, then rows are still batched in
bounded-size groups (not one unbounded statement) — mirrors why `upsert_graph`'s `UNWIND` payload
size is a known, accepted tradeoff only at today's observed graph sizes.
- [ ] Existing chunk-upsert tests (idempotency / `ON CONFLICT` behavior) pass unchanged.
- [ ] The `::vector` cast + `ON CONFLICT DO UPDATE` interaction is specifically covered by a test.

### Out of Scope
- Any change to `upsert_graph`/Neo4j (already done in #626).
- A request timeout on `ControlPlaneClient` — tracked separately.
- Running the live-Neo4j tests in CI — not part of this ticket.

### Technical Context
`services/control-plane/src/db/code_chunks.rs::upsert_code_chunks`, called from
`services/control-plane/src/http/internal.rs` (chunk-ingest endpoint), reachable from the same
per-task `index_checkout` flow as `upsert_graph`.

### Risks
Low-medium — the `ON CONFLICT` + `::vector` cast interaction is the one part worth testing
carefully; otherwise follows the same shape as #626's Neo4j batching.

### Test Plan
Existing `code_chunks` test suite, plus a new test with a chunk count large enough to exercise any
batch-size cap.

### Verification evidence
Test output: see the eventual PR.

### Human accountable owner
@leghadjeu-christian

### AI Usage Declaration
Drafting the ticket, Understanding code, Proposing implementation

### Human verification completed
- [x] I am the accountable owner and accept responsibility for this ticket.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.