coinbase / coinbase/temporal-ruby
Activity/Workflow TaskProcessor opens a brand-new gRPC channel per task, causing native grpc-core crashes (SIGABRT) under load
- Dominant language
- Ruby
- Stars
- 287
- Forks
- 113
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 1
Description
**Environment**
- temporal-ruby: current `master` (`b5efd2cef8`)
- grpc gem: 1.66.0
- Ruby 3.3.6, Rails 7
**What happened**
Our Rails-based Temporal workers (`Temporal::Worker`, `activity_thread_pool_size: 10`, `workflow_thread_pool_size: 6`) crash intermittently with **exit code 134 (SIGABRT)** — a native abort inside the `grpc` gem's C++ core, uncatchable from Ruby. Two different internal failure signatures observed on different occasions:
```
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct null not valid
terminate called recursively
Aborted (core dumped)
```
and, separately:
```
F0000 ... work_stealing_thread_pool.cc:186] Check failed: pool_->IsQuiesced()
*** Check failure stack trace: ***
Aborted (core dumped)
```
**Root cause**
`Activity::Poller#process(task)` / `Workflow::Poller#process(task)` construct a brand-new `TaskProcessor` for every polled task:
https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/poller.rb#L110-L114
`TaskProcessor#connection` memoizes its own `Temporal::Connection::GRPC`:
https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/task_processor.rb#L71-L73
So every single activity/workflow task opens a brand-new gRPC channel (fresh DNS resolution, fresh TLS handshake, fresh subchannels/LB policy) and tears it down again right after finishing. Under load (several tasks/sec with a non-trivial `activity_thread_pool_size`), this produces heavy concurrent gRPC channel churn. We confirmed this directly via `GRPC_TRACE=call_error,client_channel`: **48 distinct channel handles**, 33 `creating client_channel` / 89 `destroying subchannel wrapper` lines in a single ~3 minute window on one worker pod.
That churn races grpc-core's internal C++ lifecycle bookkeeping (subchannel refcounting, LB policy teardown, and the EventEngine thread pool's shutdown/quiescence accounting) and trips different fatal internal assertions depending on timing — which is why we saw two different crash signatures for what appears to be the same underlying stressor.
This may also explain, or be related to, #291 ("Unable to poll" / `GRPC::Unavailable: Socket closed` errors happening frequently under similar thread-pool concurrency), and possibly #280.
**Proposed fix**
`TaskProcessor` should reuse the `Poller`'s own long-lived connection instead of building its own per task. gRPC channels are explicitly designed to be shared across concurrent calls, so this is safe even with several `TaskProcessor`s running concurrently on the poller's thread pool — the connection's only `Mutex` (`poll_mutex`) guards solely the long-poll bookkeeping (`poll_activity_task_queue`/`poll_workflow_task_queue`/`cancel_polling_request`), not the `respond_*_task_completed`/`respond_*_task_failed` calls concurrent task processors make, so no new lock contention is introduced by sharing.
We've deployed this exact fix downstream (as a monkeypatch, since we can't modify the gem source directly in our app) and confirmed **0 crashes over a multi-day soak** in an environment that was previously crash-looping every ~10 minutes to a few hours.
Happy to open a PR with this fix — backward-compatible, adds an optional `connection:` keyword arg to `TaskProcessor#initialize` defaulting to `nil`, so existing behavior is unchanged for anyone not passing it. Let me know if that's welcome.
Contributor guide
Assessment
This issue has not been assessed yet.