share / share/sharedb-mongo

queryPoll can emit diffs that temporarily cause results to have same doc id twice

Open
#55 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
152
Forks
68
Avg merge
1d 5h
Merged PRs (30d)
3

Description

Occasionally, with subscribed queries using a sort+limit and query polling, we're seeing the ShareDB server emit a diff list that temporarily causes the ShareDB client results to contain the same id twice, for an instant in time. Immediately afterwards, the Share server emits a diff to clean up the duplicate id, but this behavior causes the Racer query's idMap to become incorrect and unload docs from its model that are still subscribed to by Share. (I'll send a patch to make Racer defensive against this issue.)

This appears to only happen when the same client that's subscribed to the query goes and issues an op that would change the query's sort order. An example of the diff lists received by the Share client - note than an item is inserted at index 3 and then immediately removed:

2017-07-27 21:01:14Z Backend -> Server Client bf1a5cbee2621f20eedabb61071cfeac unauthenticated
[ RemoveDiff { index: 0, howMany: 1 },
  InsertDiff { index: 3, values: [ [Object] ] } ] 
2017-07-27 21:01:14Z Backend -> Server Client bf1a5cbee2621f20eedabb61071cfeac unauthenticated
[ RemoveDiff { index: 3, howMany: 1 },
  MoveDiff { from: 1, to: 0, howMany: 2 } ] 

After the first diff list is processed by the Share client, here's what the Share query results' ids are, as well as the last diff event emitted in the Share client's _handleDiff - note that the inserted item at index 3 has the same id as the item at index 0:

ids of shareQuery.results: [ 'ef...',
  'c6...',
  '2c...',
  'ef...', ]
Last Share query diff event and args: insert [ { collection: 'myCollection',
    id: 'ef...',
    version: 21,
    data: 
     { ...,
       id: 'ef...' } } ] 3

Index 0 is the new "correct" place for the item, based on the sort order. The move after the remove shuffles the results into the correct order.

I'm not sure if this is sharedb or sharedb-mongo, but I'm filing it here in sharedb-mongo for now as the Share code appears fine at first glance.

Example of a Racer query that we see will eventually trigger the issue:

var q = model.query('myCollection', {
  some: 'field'
  $sort: {foo: 1, bar: 1, _id: 1}
  $limit: 20
}, {pollInterval: 30000});
model.subscribe(q, cb);

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the subscribed sort+limit query with polling from the Racer query example, then trace the Share client’s _handleDiff and Racer’s lib/Model/Query.js idMap behavior. Compare the sharedb-mongo query polling diffs with the two logged diff lists; done means a sort-changing operation no longer produces duplicate document ids or leaves subscriptions and model state inconsistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.