apache / apache/pouchdb

Very large update requests trigger failure to sync due to revsDiff failure

Open
#9,111 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
17.6k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

I've just diagnosed a problem we were having where sometimes our local pouchDB database did not sync to the remote CouchDB. The source of the problem turned out to be due to our use of a very large batch size (1000) combined with a number of large attachments on documents.

The problem occurs in the implementation of revsDiff:

```javascript
api.revsDiff = adapterFun('revsDiff', async function (req, opts, callback) {
// If no options were given, set the callback to be the second parameter
if (typeof opts === 'function') {
callback = opts;
opts = {};
}

try {
// Get the missing document/revision IDs
const result = await fetchJSON(genDBUrl(host, '_revs_diff'), {
method: 'POST',
body: JSON.stringify(req)
});
callback(null, result.data);
} catch (error) {
callback(error);
}
});
```

We were seeing the failure of JSON.stringify with our large payload (RangeError: Invalid string length). This meant that the POST request was not sent. PouchDB then went on to try again with the same payload and failed again.

There is most likely an error reported via the appropriate handler but we missed it.

The solution was to reduce the batch size so that the payload of one request was much smaller.

However, I feel that this is something that could be worked around inside PouchDB - perhaps backing off to smaller batches in the case of a RangeError in this function.

Posting here for reference in case others see this issue and in case this can be prioritised for a fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.