OpenFn / OpenFn/kit

Engine: asynchronous code in the worker's publish causes events to be lost

Open
#1,394 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DevX
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

Inside the engine, we run job code in a worker thread. The worker thread sends events out to the main engine process through parentPort.postMessage

In production right now this is synchronous. When the message is posted the payload is processed and sent out of the child worker. This ensures that messages leave the worker_thread in the same order that they were emitted from the runtime.

In #1366 we use an async loop to read the size of a payload before sending it. This async loop is causing two problems:

  1. It no longer guarantees that events will leave the worker thread in the same order as they're emitted, because of the async yield. If message a takes longer to process than message b, than message b might be piped through the port first.
  2. Some messages don't get emitted at all because the worker thread is shut down while the events are being processed. When we complete the task and exit the run, I don't think there's anyway to know that messages are still processing and waiting to be emitted.

I don't think this is affecting production, but it's a possible cause of #1072. What I'm worried about is that a message emitted with a large-ish payload, but the worker thread gets killed before that message is completed.

I'd sort of expect node.js to handle that for me and not exit the worker until the message has finished posting, but it's possible that we're exiting too soon (because of code or config in the engine).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the asynchronous payload-size loop introduced in #1366 and trace parentPort.postMessage through worker shutdown. Reproduce the ordering and lost-message cases described here, then verify that emitted events leave the worker in order and are not dropped when the task exits.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.