OpenFn / OpenFn/lightning

Replace id prefix text searches with binary index as for dataclips

Open
#3,225 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

User story

There are some searches in the app that allows filtering by id prefixes (history page and now on the dataclips selection with the new manual run).

We can save some db cycles filtering by binary id intervals instead of using like operator converting the id to text.

Details

Postgres performs a Parallel Sequential Scan when using the like operator while it can fully make use of the index when using binary intervals. This happens not only for uuid prefixes with odd number of chars but for any id prefix.

Image

Implementation notes

Here is the dynamic filter: https://github.com/OpenFn/lightning/blob/9e255a660f7f3c6df36677bc073c955f9d47e36d/lib/lightning/invocation.ex#L72-L79

The case of prefixes of odd number of chars (1, 3, 5...) requires an additional filtering once 2 hex chars compose 1 byte https://github.com/OpenFn/lightning/pull/3196/commits/9e255a660f7f3c6df36677bc073c955f9d47e36d.

Release notes
User acceptance criteria

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dynamic filter in lib/lightning/invocation.ex at lines 72-79, then compare the existing dataclips implementation and the approach referenced in pull request 3196. Trace the history-page and dataclips prefix searches, including odd-length prefixes, and verify that matching behavior and indexed query plans are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.