temporalio / temporalio/sdk-typescript

[Bug] Worker crashes with "async hook stack has become corrupted" on Workflow Task timeout

Open
#939 8 comments 1 reaction 1 assignee View on GitHub

@mjameswh is already working on this.

Since Jan 24, 2023.

bug
Dominant language
TypeScript
Stars
917
Forks
224
Avg merge
3d 16h
Merged PRs (30d)
43

Description

What are you really trying to do?

Tell us at a high level what you're doing, to avoid XY problem (https://en.wikipedia.org/wiki/XY_problem)
We had a workflow in development where we were still developing on it. Older version of workflow is still running when we change the code of workflow and deployed it and we see a crash in the system

Describe the bug

Error: async hook stack has become corrupted (actual: 112970, expected: 3)
1: 0x10229dab8 node::AsyncHooks::FailWithCorruptedAsyncStack(double) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
2: 0x102232164 node::AsyncHooks::pop_async_context(double) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
3: 0x102231d8c node::InternalCallbackScope::Close() [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
4: 0x10223266e node::InternalMakeCallback(node::Environment*, v8::Localv8::Object, v8::Localv8::Object, v8::Localv8::Function, int, v8::Localv8::Value, node::async_context) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
5: 0x10224a238 node::AsyncWrap::MakeCallback(v8::Localv8::Function, int, v8::Localv8::Value
) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
6: 0x102346af6 node::worker::MessagePort::OnMessage(node::worker::MessagePort::MessageProcessingMode) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
7: 0x102cd275b uv__async_io [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
8: 0x102ce654b uv__io_poll [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
9: 0x102cd2ce1 uv_run [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
10: 0x102232f9f node::SpinEventLoop(node::Environment*) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
11: 0x1023a9cae node::worker::Worker::Run() [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
12: 0x1023ad6a2 node::worker::Worker::StartThread(v8::FunctionCallbackInfov8::Value const&)::$_3::__invoke(void*) [/Users/random/.nvm/versions/node/v16.16.0/bin/node]
13: 0x7ff81c0e84e1 _pthread_start [/usr/lib/system/libsystem_pthread.dylib]
14: 0x7ff81c0e3f6b thread_start [/usr/lib/system/libsystem_pthread.dylib]
[nodemon] app crashed - waiting for file changes before starting...
^C

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.