Engine: asynchronous code in the worker's publish causes events to be lost
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 23
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 17
Description
Inside the engine, we run job code in a worker thread. The worker thread sends events out to the main engine process through parentPort.postMessage
In production right now this is synchronous. When the message is posted the payload is processed and sent out of the child worker. This ensures that messages leave the worker_thread in the same order that they were emitted from the runtime.
In #1366 we use an async loop to read the size of a payload before sending it. This async loop is causing two problems:
- It no longer guarantees that events will leave the worker thread in the same order as they're emitted, because of the async yield. If message a takes longer to process than message b, than message b might be piped through the port first.
- Some messages don't get emitted at all because the worker thread is shut down while the events are being processed. When we complete the task and exit the run, I don't think there's anyway to know that messages are still processing and waiting to be emitted.
I don't think this is affecting production, but it's a possible cause of #1072. What I'm worried about is that a message emitted with a large-ish payload, but the worker thread gets killed before that message is completed.
I'd sort of expect node.js to handle that for me and not exit the worker until the message has finished posting, but it's possible that we're exiting too soon (because of code or config in the engine).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the asynchronous payload-size loop introduced in #1366 and trace parentPort.postMessage through worker shutdown. Reproduce the ordering and lost-message cases described here, then verify that emitted events leave the worker in order and are not dropped when the task exits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100