Make `handle_delayed_response` to not need `finalState` from worker
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
In newer versions of the worker, we'd like to not send final_state if final_dataclip_id is present in the run:complete payload. This would save us some network traffic.
Right now, final_state is still required for sync-mode workflows because we prefer taking it directly from the worker over fetching a dataclip from the DB before responding:
See: https://github.com/OpenFn/lightning/blob/main/lib/lightning_web/channels/run_channel.ex#L322
And: https://github.com/OpenFn/lightning/blob/main/lib/lightning_web/controllers/webhooks_controller.ex#L130
Question: Should we keep sending finalState from the worker, even if it's a bit redundant? Or should we find a way to only send final_state from the worker when it hasn't already been sent (and referenced via final_dataclip_id) and provide that earlier copy of final_state to the HTTP requester for sync-mode workflows?
Deeper question: How big a deal is it to send a second copy of the final_state over the wire? Like, this is a trivial fix on lightning and worker, but if sending that extra JSON object isn't a big deal then I almost feel like the worker should always send it. Cause from the worker's perspective... who knows whether the consumer wants it or not?
Super deep question: I think this is really the deepest @stuartc question: If optimizing the data over the wire requires that the worker "knows" that Lightning doesn't need final_state then should we skip it anyway? It might be nice to know that no matter what the worker emits final_state and let the controlling system simply drop it on the floor if it wants.
cc @josephjclark here. this is not urgent for either of you
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the handling at lib/lightning_web/channels/run_channel.ex around line 322 and lib/lightning_web/controllers/webhooks_controller.ex around line 130. Trace how final_state and final_dataclip_id reach sync-mode HTTP responses, then define and verify a consistent behavior for when the worker omits the redundant state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100