View indexer advances `_local/lastSeq` past failed batch writes — permanent silent view index corruption
- Dominant language
- JavaScript
- Stars
- 17.6k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
## Issue
The map/reduce view indexer (`pouchdb-abstract-mapreduce`) can permanently and
silently drop documents from a view index. If a batch's row write to the
mrview database fails while a later batch's write succeeds, the later batch
stamps `_local/lastSeq` past the failed batch. The failed batch's documents
then have no view rows, no `_local/doc_` tracking record, and a checkpoint
that says they were processed — nothing ever revisits them. `db.query()`
reports success and simply returns fewer rows than it should, indefinitely,
across restarts.
Any transient write failure suffices: fd exhaustion, disk pressure / quota, a
process kill mid-index. We root-caused a production incident (medical records
system, LevelDB adapter) to this: 219 documents invisible in the application
across 134 patient records, with `_local/lastSeq` sitting above their update
seqs.
Four stacked causes in `updateViewInQueue` / `saveKeyValues`:
1. **Write is queued, never awaited** — `queue.add(processChange(...))` is
fire-and-forget; `processBatch` fetches the next changes batch while the
previous batch's write is still pending (and possibly failing). The
rejection surfaces as an `unhandledRejection`, not as a query error.
2. **The read pointer advances in memory regardless** — `currentSeq =
change.seq` per change, so the next batch reads `since: currentSeq` whether
or not anything was persisted.
3. **Failed writes are discarded** — `TaskQueue.add` chains
`this.promise.catch(function () { /* just recover */ })`, so a rejected
batch write is swallowed and the next queued write runs anyway, stamping
its own (higher) `lastSeq`.
4. **`bulkDocs` per-row results are unchecked** — rows and `_local/lastSeq`
go into one `bulkDocs` whose response is ignored, so per-document failures
(e.g. 409s — which do occur in practice, cf. #8525 with
`auto_compaction`) are invisible too.
## Info
- Environment: Node.js (also affects browsers — the code is shared)
- Adapter: leveldb (adapter-independent; the defect is in abstract-mapreduce)
- Version: reproduced on pouchdb@9.0.0; present in master and at least back
through 7.3.1
## Reproduce
Standalone script (attached / below): 30 docs, view batch size 10, reject the
second batch's mrview `bulkDocs` once (simulating a transient failure), then
reopen the database with storage healthy again.
Observed output on pouchdb@9.0.0:
```
unhandledRejection (fire-and-forget write): simulated transient write failure
1st query error surfaced: NO (swallowed)
1st query rows: 20 of 30
after reopen rows: 20 of 30 (PERMANENT SILENT HOLE)
```
Expected: either the query rejects (view stays stale and recovers on the next
query), or it succeeds with all 30 rows. A view must never be *holed* —
missing rows below its checkpoint.
```js
// PouchDB 9.0.0 — view indexer advances checkpoint past failed writes
// One transient batch-write failure => permanent silent view holes.
const fs = require('fs')
const os = require('os')
const path = require('path')
const PouchDB = require('pouchdb')
const BATCH = 10
const DOCS = 30
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'pouch-ckpt-repro-'))
const dbPath = path.join(dir, 'db')
let mrviewWrites = 0
let failAtWrite = 0 // 0 = disarmed
// the fire-and-forget write escapes as unhandledRejection
process.on('unhandledRejection', (e) => {
console.log('unhandledRejection (fire-and-forget write):', e.message)
})
// mrview db is instance-bound; hook its creation
function hook (db) {
const realReg = db.registerDependentDatabase.bind(db)
db.registerDependentDatabase = (...args) =>
realReg(...args).then((res) => {
const real = res.db.bulkDocs.bind(res.db)
res.db.bulkDocs = (...a) => {
const docs = a[0] && a[0].docs
// count only checkpoint-stamping batch writes
if (Array.isArray(docs) && docs.some((d) => d._id === '_local/lastSeq')) {
mrviewWrites++
if (failAtWrite && mrviewWrites === failAtWrite) {
// simulates fd exhaustion / disk pressure / crash
return Promise.reject(new Error('simulated transient write failure'))
}
}
return real(...a)
}
return res
})
return db
}
async function main () {
let db = hook(new PouchDB(dbPath, { view_update_changes_batch_size: BATCH }))
await db.put({
_id: '_design/test',
views: { byX: { map: 'function (doc) { emit(doc.x) }' } },
})
const docs = []
for (let i = 0; i < DOCS; i++) {
docs.push({ _id: 'doc-' + String(i).padStart(4, '0'), x: i })
}
await db.bulkDocs(docs)
failAtWrite = 2 // second batch write fails
let err = null
const r1 = await db.query('test/byX').catch((e) => { err = e })
console.log('1st query error surfaced:', err ? err.message : 'NO (swallowed)')
if (r1) console.log('1st query rows:', r1.rows.length, 'of', DOCS)
failAtWrite = 0 // storage healthy again
await db.close() // = process restart
db = new PouchDB(dbPath, { view_update_changes_batch_size: BATCH })
const r2 = await db.query('test/byX')
console.log('after reopen rows:', r2.rows.length, 'of', DOCS,
r2.rows.length === DOCS ? '(OK)' : '(PERMANENT SILENT HOLE)')
await db.close()
fs.rmSync(dir, { recursive: true, force: true })
}
main().catch((e) => { console.error('FATAL', e); process.exit(1) })
```
## Suggested fix
We patched our vendored copies (8.0.1 inline + abstract-mapreduce 7.3.1) as
follows, validated by regression tests; happy to turn this into a PR:
1. Await each batch's write before fetching the next batch (replace the
fire-and-forget `queue.add(processChange(...))` with awaiting
`processChange(...)()`), so a failure aborts the run with `_local/lastSeq`
still at the last persisted batch — the view goes stale, not holed.
2. Propagate the failure to the query caller (in v8+/master,
`updateViewInQueue`'s final `catch` must rethrow after
`activeTasks.remove(taskId, error)`).
3. Check `bulkDocs` per-row results. Note: naively failing on every row error
breaks recovery, because `getDocsToPersist`'s `isGenOne` shortcut writes
rev-less kv/meta docs when re-indexing an already-indexed generation-1 doc
— producing 409s that today are silently swallowed by the same unchecked
`bulkDocs`. We repair conflicts with a single `_rev`-refresh retry and fail
the run on anything else.
Cost: indexing loses read-ahead pipelining (writes are on the critical path),
and previously silent errors become visible — which is the point.
Related: #9177 refactors these exact functions (async/await) without changing
this behavior; if that lands first, the fix becomes a few lines simpler.
Contributor guide
Assessment
This issue has not been assessed yet.