Replace id prefix text searches with binary index as for dataclips
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
User story
There are some searches in the app that allows filtering by id prefixes (history page and now on the dataclips selection with the new manual run).
We can save some db cycles filtering by binary id intervals instead of using like operator converting the id to text.
Details
Postgres performs a Parallel Sequential Scan when using the like operator while it can fully make use of the index when using binary intervals. This happens not only for uuid prefixes with odd number of chars but for any id prefix.
Implementation notes
Here is the dynamic filter: https://github.com/OpenFn/lightning/blob/9e255a660f7f3c6df36677bc073c955f9d47e36d/lib/lightning/invocation.ex#L72-L79
The case of prefixes of odd number of chars (1, 3, 5...) requires an additional filtering once 2 hex chars compose 1 byte https://github.com/OpenFn/lightning/pull/3196/commits/9e255a660f7f3c6df36677bc073c955f9d47e36d.
Release notes
User acceptance criteria
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dynamic filter in lib/lightning/invocation.ex at lines 72-79, then compare the existing dataclips implementation and the approach referenced in pull request 3196. Trace the history-page and dataclips prefix searches, including odd-length prefixes, and verify that matching behavior and indexed query plans are preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, postgresql
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100