Collections interactions and performances

I have been in some discussions lately and I thought I’d share a bit of my initial thoughts. There are not fully formed ideas, more of a starting point.

Performance is an important part of what we do every day. Something that is slow will often not be acceptable. I have been in a few discussions lately where we were discussing how to add more drops while keeping performance in mind.

I believe we should do our best to offer a simple language that represents concepts at a high level and push runtime complexity under the hood.

What I’m exploring here might be folded into liquid directly or to be kept an implementation specific pattern, to be determined.

Collection access

A straightforward example is the where filter.

If you have a collection of hundreds of items, a common pattern is object.has_many_relation | where “color”, “green”. Under the hood this resolves as object.has_many_relation.select { |item| item[“color”] == “green”}.

When coupling this with data access (eg.: loading the collection of has_many_relation from the database), it can quickly become a performance nightmare where a lot of time is spent loading and filtering the collections elements in Ruby when alternatives might be more performant (eg.: filtering in MySQL directly).

I have been wondering how to turn object.has_many_relation from returning an array of drops ([Drop]) but instead be allowed to return some kind of scope which can be lazy evaluated only when needed (eg.: LazyCollectionResolver).

I understand that different use cases of liquid might require different patterns when it comes to execution plans, still I hope we can come up with some kind of primitive to forge an api around “collections” in Liquid that would be stronger than what Array currently offers us.

Filter dependencies

I see drops as the primitive here where domain specific logic lives which is specific to each. Filters on the other hand to be shareable logic helpers we can exchange between projects.

I am wondering if part of this exploration is to look at how to iterate and instead of having filters inquire as to the shape of inputs (eg.: input.respond_to?(:where)) to figure out a pattern for filters to inquire as to a more rich dependency integration (eg.: input.implements?(AFilterInterface)).

Should a filter be allowed to call any methods of an input? Since drops expose public methods in their public template api, this creates a weird dynamic where a filter currently needs a method to be public on the drop, then we need to override invokable_methods to keep it internal.

Should filters rely on specific namespaced methods specialized for them instead of reusing shared endpoints?

From my experience, the goal should be about tuning up drops so they work as best as they can for every filter available in the context on an individual basis, not finding the easiest path for the drop coder which would risk yielding average performance.

No files, tests, or entry points are named. Start by locating the collection access, filter, and drop implementations, then review how where evaluates collections and how filters access drop methods. The issue does not define an agreed API or concrete completion criteria; done would require a settled design and corresponding implementation plan.