queryPoll can emit diffs that temporarily cause results to have same doc id twice
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 152
- Forks
- 68
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 3
Description
Occasionally, with subscribed queries using a sort+limit and query polling, we're seeing the ShareDB server emit a diff list that temporarily causes the ShareDB client results to contain the same id twice, for an instant in time. Immediately afterwards, the Share server emits a diff to clean up the duplicate id, but this behavior causes the Racer query's idMap to become incorrect and unload docs from its model that are still subscribed to by Share. (I'll send a patch to make Racer defensive against this issue.)
This appears to only happen when the same client that's subscribed to the query goes and issues an op that would change the query's sort order. An example of the diff lists received by the Share client - note than an item is inserted at index 3 and then immediately removed:
2017-07-27 21:01:14Z Backend -> Server Client bf1a5cbee2621f20eedabb61071cfeac unauthenticated
[ RemoveDiff { index: 0, howMany: 1 },
InsertDiff { index: 3, values: [ [Object] ] } ]
2017-07-27 21:01:14Z Backend -> Server Client bf1a5cbee2621f20eedabb61071cfeac unauthenticated
[ RemoveDiff { index: 3, howMany: 1 },
MoveDiff { from: 1, to: 0, howMany: 2 } ]
After the first diff list is processed by the Share client, here's what the Share query results' ids are, as well as the last diff event emitted in the Share client's _handleDiff - note that the inserted item at index 3 has the same id as the item at index 0:
ids of shareQuery.results: [ 'ef...',
'c6...',
'2c...',
'ef...', ]
Last Share query diff event and args: insert [ { collection: 'myCollection',
id: 'ef...',
version: 21,
data:
{ ...,
id: 'ef...' } } ] 3
Index 0 is the new "correct" place for the item, based on the sort order. The move after the remove shuffles the results into the correct order.
I'm not sure if this is sharedb or sharedb-mongo, but I'm filing it here in sharedb-mongo for now as the Share code appears fine at first glance.
Example of a Racer query that we see will eventually trigger the issue:
var q = model.query('myCollection', {
some: 'field'
$sort: {foo: 1, bar: 1, _id: 1}
$limit: 20
}, {pollInterval: 30000});
model.subscribe(q, cb);
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the subscribed sort+limit query with polling from the Racer query example, then trace the Share client’s _handleDiff and Racer’s lib/Model/Query.js idMap behavior. Compare the sharedb-mongo query polling diffs with the two logged diff lists; done means a sort-changing operation no longer produces duplicate document ids or leaves subscriptions and model state inconsistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mongodb
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100