GitBlobstore: push to git remote fails with 'refCheckAllSources: Blob not found: <hash>.darc' — post-flush cache eviction races in-flight deferred table-file writes on long-lived servers
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 120
Description
## Summary
`dolt push` to a git-backed remote (`git+ssh://`) from a **long-lived sql-server process** fails with:
```
Error: dolt push failed: ... Error 1105 (HY000): unknown push error; addTableFiles,
refCheckAllSources: error reading archive file: Blob not found: 5rfe4ul2d8mgls535l8bqsu68ue6s2r8.darc
```
The missing blob is the **freshly built table file the puller just uploaded for that push** — a different hash on every attempt — and the failure recurs daily until the server is restarted, after which push succeeds immediately. We believe the root cause is the post-flush cache eviction in `GitBlobstore.remoteManagedWrite` racing in-flight deferred table-file writes.
Observed on the module at `v0.40.5-0.20260605230755-1bf533220ab0`; the relevant code is unchanged at v2.2.0 (`store/blobstore/git_blobstore.go:925-928`).
## Environment
- Long-lived `dolt sql-server` (up ~5 days), git-backed remote `git+ssh://git@github.com/...`
- ~1,930-issue database (beads workspace), remote `refs/dolt/data` holding 27 `.darc` table files
- Client: beads (`bd`) 1.1.0-rc.2, driving push via SQL `CALL DOLT_PUSH(...)`
## Key evidence
1. **The missing blob hash differs on every attempt** (server log):
- Jul 4 17:27: `gloviocdr3hfsu7ueom1bkq6vemaf0hh.darc`
- Jul 6 17:56: `qkvgi2penbma79q3velcsa5b535trh2p.darc`
- Jul 8 12:40: `5rfe4ul2d8mgls535l8bqsu68ue6s2r8.darc`
None of these hashes exist in the local noms dir, either local manifest (newgen/oldgen), the remote tree, or the remote manifest. Each is the fresh table file uploaded for that day's push (fresh content each day → fresh hash).
2. **All three failures happened inside one server process** (log timestamps `m=+105k`, `+274k`, `+425k` seconds — same process since ~Jul 3). The `GitBlobstore` instance for the remote persists across attempts, holding its append-only cache and any unflushed `pendingWrites` from prior failed pushes.
3. **The failing push mutated the remote mid-flight**: after the Jul 8 failure, `refs/dolt/data` on the remote is a single **orphan** commit `"gitblobstore: checkandput manifest"` committed *during* the failing 12:38–12:40 push. Orphan means `buildCommitForKeyWrite` had `prunedEntries > 0` (prune forces a parentless commit). The manifest in that commit does **not** reference the file (`5rfe…`) whose read subsequently failed.
4. **Restarting the server clears it**: `stop` + `start` + retry pushed successfully on the first attempt after three consecutive daily failures. The successful push produced exactly two manifest commits (add-files flush + root flush), matching the expected two-`CheckAndPut` choreography.
## Analysis
Push flow to a git-backed remote:
- `WriteTableFile` → `singleBlobBSPersister.CopyTableFile` → `GitBlobstore.Put(.darc)`. For non-manifest keys this is a **deferred** write (`pendingWrites`), visible only through the in-memory cache (`cacheUpdateForPlan`).
- `AddTableFilesToManifest` → `openChunkSourcesForManifestUpdateAndRebase` → `Open` → `newBSArchiveChunkSource` reads `.darc` from the cache (footer read succeeds), then `refCheckAllSources` → `iterateAllChunks` streams the data span via `bsTableReaderAt` → `GitBlobstore.Get(.darc, range)`. Mid-iteration this `Get` returns `NotFound{Key: .darc}` — which can only come from a `cacheObjects` miss or a `cacheChildren` miss. The cache is documented append-only, so something evicted the entry between the footer read and the failing span read.
The one code path that deletes cache entries is the post-flush eviction in `remoteManagedWrite` (`store/blobstore/git_blobstore.go:925-928` at v2.2.0):
```go
for _, p := range gbs.pendingCacheEvictions {
delete(gbs.cacheObjects, p)
parent, _ := splitGitPathParentBase(p)
delete(gbs.cacheChildren, parent) // deletes the ENTIRE parent children list
}
```
Two problems:
a) `delete(gbs.cacheChildren, parent)` removes the whole children list of the parent directory, not just the pruned child. For top-level pruned paths, `parent` is `""` (root); for chunked-part paths (`.darc/0001`) it removes `.darc`'s children. Any **sibling** entry that relied on that children list is collateral damage.
b) More fundamentally, the prune/evict design races in-flight table-file additions: `CheckAndPut("manifest")` flushes **all** `pendingWrites` with whichever manifest update comes first, and `buildCommitForKeyWrite` prunes/evicts every tree path not referenced by *that* manifest. A table file that has been uploaded (deferred `Put`) but not yet added to the manifest (`AddTableFilesToManifest` hasn't CAS'd yet) sits exactly in that window: a manifest flush that doesn't reference it prunes it from the tree and/or evicts it from the cache, and the subsequent `refCheckAllSources` (or the `addTableFiles` retry loop after a `gcGenMismatch`/CAS-conflict rebase) can no longer read it → `Blob not found`.
The mid-push orphan commit whose manifest doesn't reference the failing file (evidence item 3) is direct evidence of such a flush. Stale `pendingWrites` accumulated from earlier failed attempts in the same server process widen the window: each new push re-flushes old junk files no manifest references, guaranteeing `prunedEntries > 0` and eviction churn on every attempt — which is why the failure is sticky per-process and cleared by a restart.
## Repro conditions
- Long-lived `dolt sql-server` + git-backed remote.
- At least one failed/interrupted push earlier in the same server process (leaves `pendingWrites` queued), **or** a manifest CAS retry (concurrent pusher) within one push.
## Workaround
Restart the sql-server; the poisoned in-memory blobstore state is cleared and the next push succeeds.
## Suggested fixes
1. Never evict a path present in the current `pendingWrites` set (or written by an `extraWrite` in the same commit) — prune should only target *previously committed* garbage.
2. In the eviction loop, remove the single child entry from `cacheChildren[parent]` instead of deleting the whole list.
3. Drop `pendingWrites` when the operation that enqueued them fails permanently (scope them to a push/session), so a failed push doesn't poison the next one in server mode.
4. Consider having `refCheckAllSources` read through a pinned snapshot of the store rather than the live cache, so concurrent manifest flushes can't invalidate reads mid-check.
---
Full internal writeup: beads issue `bd-5ljnr` (gastownhall/beads workspace). Related earlier report from the same deployment: #11196. Happy to provide the full server logs and `refs/dolt/data` forensics.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.