google / google/go-jsonnet

Fancy array representation for quick concatenation

Open
#187 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
1.8k
Forks
263
PR merge metrics
No merged PRs in 30d

Description

Probably a low priority now, there are many easier ways to improve performance, but I want to dump it somewhere so it doesn't get lost.

The problem
==============

Concatenating arrays `a`, `b` takes time proportional to *sum* of the lengths. This means that in particular `[1] + [2] + [3] + ... [n]` takes quadratic time. And allocates a lot of memory, putting a lot of strain on GC.

The primary solution to that is to avoid constructing arrays from elements using `+`, but instead use library functions that construct it efficiently in one go (makeArray, map, flatMap). This is not always easy.

It would be better if people wouldn't need to care about it at all in most cases, it would just work reasonably either way. And this is very doable, IMO.

Desirable properties
==============

(1) Fast indexing. O(log n) at worst.
(2) Fast concatenation, O(log n)
(3) Fast slicing, O(log n), independent from the actual slice size.
(4) Fast updating of elements O(log n)
(5) Extremely fast in some common circumstances, for example:
(5a) Constructing arrays element by element and using it only after it's built.
(5b) Indexing an array constructed in one go in O(1).
(what else?)

Ideas/observations
==============
- Balanced trees can do 1-4. AVL trees and Treaps would definitely work, most other types probably too. (5) requires additional hacking. In particular it may cause a lot of overhead on simple indexing.
- Alternatively, if we dropped (2) in favor of really fast (5a) we could keep arrays represented as pairs (left, right) and flatten them when they are indexed. This is very easy and should help in most cases.
- There's naive cheap slicing (just keep the underlying array and indexes), but it causes memory problems (keeping the whole array) and makes other operations harder.
- Because of non-interactive nature of Jsonnet, we only ever care about amortized running time (as long as it doesn't confuse the users too much).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.