OpenFn / OpenFn/kit

sandbox: lazy-load Project data

Open
#1,015 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DevX
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

A Project can be pretty large on disk - up to several megabytes of data in a single file.

This can make all sorts of related operations slow.

A great idea from @doc-han is to provide some sort of streaming of data from a project file. Because, for example, openfn project only needs to top metadata of a project - it doesn't need to load everything into memory.

I'd suggest we implement a mixture of static and lazy properties on a project:

  • All basic project metadata (name, timestamps, parent) is always available
  • The list of workflows is always available
  • The steps of a workflow should be lazy loaded
  • The expression of a step should be lazy loaded

What this basically means is that some functions of the Project and Workflow class should become async, and should only load data when it's actually needed. For example, when doing a partial merge of two projects, I only need to load the relevant workflows - not all of them.

I guess basically anything that gets checked out onto disk should be made lazy. Or should it just be step expressions? Thinking about it, it's probably only the expressions that need to be lazy. The actual workflow structure should be small in most cases?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Project and Workflow classes and the openfn project entry point. Review how project files are read and how partial merges access workflows. Done means basic metadata and the workflow list remain available without loading all step data, while required step expressions load only when needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.