Modifying a document during a fetch

Following on from the meeting on Wednesday, I thought I'd write up some notes and propose a documentation update, or maybe even an API change.

There are a couple of cases in our ShareDB usage where we would like to modify a document before being fetched by a client. For example:

• fetch or create: when the client performs a fetch, they will always get a sensible document back, and won't need to know anything about how to initialise a document)
• lazy migration of documents: when a client fetches a document with an old schema version, the document is migrated to the newer version

Below is a proposal for how to deal with this case. If this is a sensible proposal, I think we should add it to the documentation, or perhaps add a helper method.

backend.use('readSnapshots', (request, callback) => {
  const connection = backend.connect();
  let pendingModifications = 0;

  for (let i = 0; i < request.snapshots.length; i++) {
    const snapshot = request.snapshots[i];
    if (shouldModify(snapshot)) {
      pendingModifications++;
      const modificationOp = modify(snapshot);
      const doc = connection.get(request.collection, snapshot.id);

      // Ingest rather than fetch to avoid an infinite loop
      // of readSnapshots middleware
      doc.ingestSnapshot(snapshot, (error) => {
        if (error) return callback(error);
        doc.submitOp(modificationOp, (error) => {
          if (error) return callback(error);

          // Update the snapshot so the client gets the modified
          // version of the document
          snapshot.type = doc.type.uri;
          snapshot.v = doc.version;
          snapshot.data = doc.data;

          // Check if all snapshots have updated, and finally
          // unblock the fetch
          pendingModifications--;
          if (!pendingModifications) callback();
        });
      });
    }
  }
});

If this approach is acceptable, then we could potentially put together some sort of helper function on the readSnapshots request object. The client code might look something like:

backend.use('readSnapshots, (request, callback) => {
  const modifier = (snapshot, request, modifierCallback) => {
    // Do something to our snapshot and return an op to update it.
    // We pass in the request in case we want other contextual
    // information, like the collection, or agent.custom
    if (snapshot.schemaVersion < 2) {
      return modifierCallback(null, {p: ['schemaVersion'], oi: 2});
    }

    if (somethingBadHappened()) {
      return modifierCallback(new Error('Uh-oh!'));
    }

    // Returning a falsy op means we won't bother 
    // updating the snapshot
    return modifierCallback();
  };

  // This method will take care of:
  // - iterating over all snapshots in the request
  // - calling our modifier function
  // - submitting the resulting op (if appropriate)
  // - early returns for errors
  // - updating the snapshots in the request, so the client gets the updated snapshot
  request.submitOpAndUpdateSnapshots(callback);
});

What are people's thoughts on:

• the general approach for updating on fetch; are there any weird edge cases to consider? For example, I don't think this will play nicely with a client document with a non-null version. Is there a good way of handling that case? (Doesn't matter for fetch-or-create, but does matter if the server migrates up without forcing a client refresh, for example). Are there any other cases to think about? Like submitting the same modification twice (we guard against that in the apply hook - checking if we're trying to bump the schema version when it's already been bumped), or is that all left as an exercise for the reader?
• adding a helper method for the middleware

Start by reviewing the readSnapshots middleware and the proposed request.submitOpAndUpdateSnapshots helper in the issue. Investigate how fetches with non-null client versions, repeated modifications, errors, and concurrent snapshots are handled. Done means the project has a decided approach, documented edge cases, and either an accepted documentation update or a specified helper API.