Optimize entities
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 5.8k
- Forks
- 318
- PR merge metrics
- No merged PRs in 30d
Description
Also partially dicussed on Slack:
Currently an attribute lookup on an entity does a search into the btset and then caches the result. This is inefficient since the all the attribute values are right next to each other in the eavt index.
Getting them all at once and storing them in a cache makes sense, but has huge problem:
- What if the entity has a many ref with many many references?
- What if the entity has MANY attributes
I think 2) is unlikely a use-case and can be ignored. However 1) is an issue.
Ideas:
- Save an
Iterinstance that represents(Datom. eid nil nil nil nil), ie, all Datoms belonging to an entity. This is fast to get. - Enhance
Iterto allow fast searching within anIter. This would mean we can avoid thecacheof an entity and just lookup in theIter. - For avoiding performance problems with 1) we could add a heuristic to fall back to the current implementation when the
countof anIteris "too large" (> 20??). For this implementbounded-countforIter. See #226
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the current entity attribute lookup, the eavt and btset indexes, and the Iter implementation; also review the bounded-count work in #226. Compare caching all entity datoms with retaining an Iter and define a fallback for large reference sets. Done means lookups are faster without problematic memory use for entities with many references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100