Asynchronous Sources in DataScript

Currently, DataScript queries are synchronous. When a query is executed, each pattern has search called in sequential order against a record that implements datascript.db.ISearch. The search calls are synchronous, as they are currently operations against an in-memory immutable BTSet record. This makes sense for a single-threaded environment such as JavaScript / ClojureScript, and functions in Clojure as well.

This proposes to add the capability for asynchronous data sources to DataScript. This can be done as a branch or fork of DataScript in the interim, but I would like to see this merged into DataScript if possible.

This opens the following possibilities:

• Leveraging Clojure parallelism; each pattern queried for could be performed separately and asynchronously.
• LevelDB support in a Node environment; requires callbacks.
• IndexedDB support in a browser environment; requires callbacks.

In the case of LevelDB and IndexedDB, there are complications when it comes to guaranteeing immutability, but it is a separate subject from asynchronous queries and out of scope of this proposal.

At a high level, we would implement an IAsyncSearch protocol, where the return values are expected to be promises or channnels. The intermediate query functions would check for the implementation of the IAsyncSearch methods on the source object provided. On a call to datascript.core/q the sources would be checked. If any sources were provided that implement IAsyncSearch, we would synchronously block on the call if the platform supports it, otherwise, we would return a promise or channel.

It would be useful to note the requirements, implied or explicit:

• Asynchronous
• Minimal impact as possible on existing DataScript code.
• As little platform-specific code as possible.
• Synchronous calls should work as expected, with little performance degradation.
• JavaScript integration, returning a JavaScript standard promise, or accept a callback function.

There are a few ways that asyncronity can be gotten:

• Native - Clojure promises, and JavaScript promises (ECMAScript 6)
• core.async channels
• cljs-promises - ClojureScript promises built on core.async
• redlobster - ClojureScript promises for Node.
• promesa - Promises for Clojure and ClojureScript

Native

Clojure already has support for promises via promise and synchronously blocking on dereferencing. ECMAScript 6 also supports promises, but would require the JavaScript environment to support ECMAScript 6.

Implementing native support at this level would require a lot of platform specific code, but would work in a JavaScript-targeted environment.

core.async

core.async is well-supported on both Clojure and ClojureScript, but:

• Integrating with JavaScript would require a callback function to be provided on query call.
• While Clojure has the synchronously blocking <!! operator, ClojureScript only has the asynchronous <! operator and requires all channel operations to be wrapped in a go block. This would contaminate the Datascript code with core.async.
• core.async is primarily for message passing and synchronization.

core.async does have the advantage in that it is supported on earlier JVM and JavaScript platforms.

cljs-promises

cljs-promises is built on core.async and provides a promise facility that addresses one of the concerns above. It still does not solve the issue where it has to be asynchronous all the way up. It is also ClojureScript-specific, and does not satisfy the cross-platform requirement.

redlobster

redlobster is a ClojureScript promise facility with strong ties to NodeJS, but it has been shown to work in browsers as well if one ignores some of the Node-specific functionality. However, the query call itself would need to be asynchronous, returning a promise. It is also ClojureScript-specific.

promesa

promesa provides a cross-platform abstraction layer for both Clojure and ClojureScript.

• On Clojure, it is built on JDK8 completable futures.
• On ClojureScript, it is built on bluebird. bluebird is well-accepted in the JavaScript community, so a bluebird promise could be returned if invoked from JavaScript on an asynchronous data source.

A possibly big negative is the requirement for JDK8, transitively imposing a dependency onto DataScript. It however fulfills much of the other requirements, and would be my choice.

Start with datascript.core/q and the datascript.db.ISearch protocol mentioned in the proposal. Trace how query patterns invoke search, then determine how an IAsyncSearch implementation and asynchronous result handling would fit across Clojure, ClojureScript, and JavaScript. Done requires a settled cross-platform design, preserved synchronous behavior, and tests for asynchronous sources.