share / share/sharedb

Wrapper for elasticsearch

Open
#49 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
JavaScript
Stars
6.5k
Forks
456
Avg merge
1d 5h
Merged PRs (30d)
5

Description

We are working on a wrapper for elasticsearch. We have not yet implemented the query API, but everything else seems to be passing the adaptations of tests from the mongo, memory, and postgres implementations that we found.

When we try to use this as the backend for shareJS, though, we are running into the following error:

ERROR: CORRUPT DATA DETECTED in document jibe.The Dark Side
If you're using redis, delete data for document. Please file an issue if you can recreate this state reliably.
Op error: Internal data corruption - cannot submit jibe The Dark Side { op: [ 78, 'a' ],
  v: 692,
  src: 'f28b32f7c9038fa2dd1b46defb992932',
  seq: 238,
  preValidate: undefined,
  validate: undefined,
  m: { ts: 1429532663042 } } { channelPrefix: null, dirtyData: undefined }

I inserted another console output at line 285 of liveDB's lib/index.js, and this is the state right before it crashes:

// index.js#285
console.error("index.js 285", submitData, "from = ", from, "to = ", to, "result", ops);

// console output:
index.js 285 { cName: 'jibe',
  docName: 'The Dark Side',
  opData: 
   { op: [ 78, 'a' ],
     v: 692,
     src: 'f28b32f7c9038fa2dd1b46defb992932',
     seq: 238,
     preValidate: undefined,
     validate: undefined,
     m: { ts: 1429532663042 } },
  submitOptions: { channelPrefix: null, dirtyData: undefined },
  callback: [Function],
  start: 1429532663042,
  projection: undefined,
  transformedOps: [],
  expectTransform: true } from =  692 to =  null result []

From what I can gather from looking at the source, the issue is that the current document version that liveDB expects to be able to find in the database doesn't actually exist there yet?

// There's an awful state that should never happen (but did happen to us
// recently) where a driver knows there are more ops for a document, but the
// ops don't seem to exist anywhere. In this case, we need to make sure we
// don't end up in an infinite retry loop.
this.expectTransform = false;

Everything works fine for a seemingly random amount of time, and then crashes horribly. I am unable to recreate the error own my own as well, even with multiple browsers open; I have to have someone else modifying the same document at the same time that I am.

Do you have any experience with errors like this, and how to resolve them? Thanks in advance for any help!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at lib/index.js around line 285 and review the Elasticsearch wrapper linked in the issue, along with the adapted Mongo, memory, and Postgres tests. Reproduce the corruption during concurrent document edits if possible, then identify why the expected document version or operations are missing and add coverage showing the backend no longer reaches the corrupt-data failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, javascript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.