OpenFn / OpenFn/lightning

Make `handle_delayed_response` to not need `finalState` from worker

Open
#4,559 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

architecture needs technical design question
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

In newer versions of the worker, we'd like to not send final_state if final_dataclip_id is present in the run:complete payload. This would save us some network traffic.

Right now, final_state is still required for sync-mode workflows because we prefer taking it directly from the worker over fetching a dataclip from the DB before responding:

See: https://github.com/OpenFn/lightning/blob/main/lib/lightning_web/channels/run_channel.ex#L322
And: https://github.com/OpenFn/lightning/blob/main/lib/lightning_web/controllers/webhooks_controller.ex#L130

Question: Should we keep sending finalState from the worker, even if it's a bit redundant? Or should we find a way to only send final_state from the worker when it hasn't already been sent (and referenced via final_dataclip_id) and provide that earlier copy of final_state to the HTTP requester for sync-mode workflows?

Deeper question: How big a deal is it to send a second copy of the final_state over the wire? Like, this is a trivial fix on lightning and worker, but if sending that extra JSON object isn't a big deal then I almost feel like the worker should always send it. Cause from the worker's perspective... who knows whether the consumer wants it or not?

Super deep question: I think this is really the deepest @stuartc question: If optimizing the data over the wire requires that the worker "knows" that Lightning doesn't need final_state then should we skip it anyway? It might be nice to know that no matter what the worker emits final_state and let the controlling system simply drop it on the floor if it wants.

cc @josephjclark here. this is not urgent for either of you

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the handling at lib/lightning_web/channels/run_channel.ex around line 322 and lib/lightning_web/controllers/webhooks_controller.ex around line 130. Trace how final_state and final_dataclip_id reach sync-mode HTTP responses, then define and verify a consistent behavior for when the worker omits the redundant state.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.