Questions about delta-queries

I started learning about delta queries and was hoping @frankmcsherry or someone else from the team could help to clarify a few questions.

Meta

• What is the status of this feature? It appears that code in dogsdogsdogs has not been released on crates.io. Is the plan to release it eventually or is this repository meant as an example that people should copy and modify to their needs (which is what I am planning to do, as I think I need a slightly different API, see below).

• Is there any additional documentation besides this README?

API questions

To clarify where I am coming from with the following questions, our applications often have rules like this:

X(x, u) :- A(x,y), B(y,z),C(z,q),D(q,u).

where columns joined on are typically unique keys in the corresponding tables, and so intermediate joins do not blow up; however I want to use delta queries to avoid maintaining arrangements for all prefixes of the rule. Trouble is, the API for delta queries in dogsdogsdogs/src/lib.rs appears to create a whole bunch of extra arrangements of A, B, C, D. I realize this this does not affect the worst-case asymptotic memory footprint, but it certainly will affect my workloads in practice :)

• First, it seems that the CollectionIndex type could be made more economical. CollectionIndex::index() computes three different arrangements and a distinct. However, this is not always required. For instance, if I only use the given collection in propose(), there is no reason I should maintain count_trace and validate_trace. Furthermore, one or more of the arrangements in CollectionIndex may exist elsewhere in the program and could be reused rather than computed from scratch. Does this make sense?

• Second, we need to create two instances of CollectionIndex per collection: with neu and alt timestamps. If I understand correctly, in the current implementation neu and alt indexes use separate arrangements. I wonder if there is a way to reuse some of this state, since these arrangements will mostly contain the same data (different in only the last timestamp).

• The extend_using function clones arrangements in CollectionIndex, and later modifies these clones in PrefixExtender::count(), propose(), validate(). My understanding is that these clones share most of the state with the original arrangements, and so I should not be worried about wasting more memory there. Is this correct?

Start with the dogsdogsdogs README and dogsdogsdogs/src/lib.rs, especially CollectionIndex::index(), extend_using(), and PrefixExtender::{count,propose,validate}. Review the 17-comment discussion to determine whether the questions have an agreed direction; done would require a maintainer-approved design for reducing or reusing arrangements, not just an implementation choice.