fix: wait for transient SQLite locks in ExternalIngestIndex (fixes #328227)
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
### Summary
The copilot-chat extension throws an unhandled `Error: database is locked` (SQLITE_BUSY) from `ExternalIngestIndex`. The crash surfaces in the file-watcher event handler `onFileChanged` → `tryAddOrUpdateFile` → `get`, when a synchronous `SELECT` runs against the per-workspace `codebase-external.sqlite` file while another connection holds a write lock.
The root cause is a connection-setup gap: the SQLite connections are never given a busy timeout, so any transient lock contention immediately raises `SQLITE_BUSY` instead of briefly waiting. Because the same per-workspace database file can be opened by more than one VS Code window (and by more than one embedding-type `CodeSearchChunkSearch` in a window), brief write overlaps are expected and should be tolerated. Additionally, the existing-database open path applied none of the connection PRAGMAs that the fresh-database path applied, so an existing DB got an even more default (fail-fast) locking configuration.
The error escapes to telemetry because the reads happen inside event-driven handlers (`onFileChanged`/`onFileAdded`) that are not — and should not be — wrapped in error suppression. The correct fix is at the data producer: the connection configuration.
Fixes microsoft/vscode\#328227
Recommended reviewer: `@mjbvz`
### Culprit Commit
Not precisely bisected within the time budget. The `ExternalIngestIndex` SQLite layer is authored/owned by `@mjbvz` (all GDPR `owner` tags in the file are `mjbvz`); the bug is inherent to the connection-setup code (`openOrCreateDatabase` / `createFreshDatabase`) rather than a single regressing line. Marked `recent-regression` on the issue, consistent with the external-ingest feature being recently enabled for more resources (see nearby commits "Enable external ingest for non-file resources too").
### Code Flow
```mermaid
flowchart TD
A[Another window / connection holds write lock on codebase-external.sqlite] --> B[File watcher fires onDidChange]
B --> C[onFileChanged uri]
C --> D[tryAddOrUpdateFile uri]
D --> E[get uri : synchronous SELECT via _db.prepare.get]
E --> F{Connection has busy_timeout?}
F -- No default 0 --> G[SQLITE_BUSY: database is locked thrown to telemetry]
F -- Yes 5000ms --> H[Waits for lock to clear, query succeeds]
```
### Affected Files
- `extensions/copilot/src/platform/workspaceChunkSearch/node/codeSearch/externalIngestIndex.ts` — add a `busy_timeout` PRAGMA to every connection opened by the index (both the existing-database open path and the fresh-database creation path).
### Repro Steps
1. Open the same workspace/folder in two VS Code windows with the copilot-chat extension active and workspace chunk search / external ingest enabled.
2. Let both windows initialize `ExternalIngestIndex` against the shared `codebase-external.sqlite`.
3. Modify files in the workspace so the file watchers fire `onDidChange` in both windows concurrently while one connection is mid-write.
4. Observe `Error: database is locked` from `ExternalIngestIndex.get` in telemetry / the extension log.
### How the Fix Works
**Chosen approach** (`externalIngestIndex.ts`): Introduce a single `configureBusyTimeout(db)` helper that runs `PRAGMA busy_timeout = 5000;` and call it on every connection the index opens — both in the existing-database branch of `openOrCreateDatabase` (which previously applied no PRAGMAs at all) and in `createFreshDatabase`. This fixes the problem at the data producer (the connection configuration) rather than at the crash site: with a busy timeout set, SQLite blocks briefly and retries when it encounters a lock instead of immediately throwing `SQLITE_BUSY`, so the transient contention that the shared per-workspace file inevitably produces no longer escapes as an unhandled error. After this change, `get`'s `SELECT` at the watcher-driven call site cannot raise `database is locked` for a lock that clears within the timeout window, because the connection now waits for it instead of failing fast.
The existing constructor already falls back to an in-memory database if the initial open throws, and no `logService.error`/telemetry calls were removed — the error pipeline is untouched; only the fail-fast locking behavior of healthy connections is corrected. No `try/catch` was added around the crash site.
**Alternatives considered**:
- Wrapping `get`/`tryAddOrUpdateFile` in `try/catch` to swallow the error — rejected because it hides the contention at the consumer crash site instead of fixing the producer connection configuration, and would silently drop legitimate index updates.
- Serializing all DB access behind a mutex — rejected as heavier than needed; the contention is cross-connection/cross-window, which an in-process lock cannot solve, whereas `busy_timeout` addresses exactly the cross-connection case.
### Recommended Owner
`@mjbvz` — owner of all GDPR telemetry annotations in `externalIngestIndex.ts` and author of the external-ingest SQLite layer.
> Generated by [errors-fix](https://github.com/microsoft/vscode-engineering/actions/runs/30556504838) · opus48 · 320.1 AIC · ⌖ 11.7 AIC · ⊞ 18.1K · [◷](https://github.com/search?q=repo%3Amicrosoft%2Fvscode+%22gh-aw-workflow-id%3A+errors-fix%22&type=pullrequests)
---
> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 128
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/microsoft/vscode-engineering/actions/runs/30556504838)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.
To create a pull request with the changes:
```sh
# Download the artifact from the workflow run
gh run download 30556504838 -n agent -D /tmp/agent-30556504838
# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-30556504838/aw-microsoft-vscode-fix-external-ingest-db-locked-328227.bundle refs/heads/fix/external-ingest-db-locked-328227:refs/bundles/create-pr-fix-external-ingest-db-locked-328227-b9e99f5226bde0eb-8226ff46
git update-ref refs/heads/fix/external-ingest-db-locked-328227-b9e99f5226bde0eb refs/bundles/create-pr-fix-external-ingest-db-locked-328227-b9e99f5226bde0eb-8226ff46
git checkout fix/external-ingest-db-locked-328227-b9e99f5226bde0eb
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-fix-external-ingest-db-locked-328227-b9e99f5226bde0eb-8226ff46
# Push the branch to origin
git push https://github.com/bryanchen-d/vscode.git fix/external-ingest-db-locked-328227-b9e99f5226bde0eb
# Create the pull request
gh pr create --title 'fix: wait for transient SQLite locks in ExternalIngestIndex (fixes #328227)' --base main --head bryanchen-d:fix/external-ingest-db-locked-328227-b9e99f5226bde0eb --repo microsoft/vscode
```
Contributor guide
Assessment
This issue has not been assessed yet.