coinbase / coinbase/temporal-ruby

Activity/Workflow TaskProcessor opens a brand-new gRPC channel per task, causing native grpc-core crashes (SIGABRT) under load

Open
#361 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
287
Forks
113
Avg merge
6d 11h
Merged PRs (30d)
1

Description

**Environment**
- temporal-ruby: current `master` (`b5efd2cef8`)
- grpc gem: 1.66.0
- Ruby 3.3.6, Rails 7

**What happened**

Our Rails-based Temporal workers (`Temporal::Worker`, `activity_thread_pool_size: 10`, `workflow_thread_pool_size: 6`) crash intermittently with **exit code 134 (SIGABRT)** — a native abort inside the `grpc` gem's C++ core, uncatchable from Ruby. Two different internal failure signatures observed on different occasions:

```
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct null not valid
terminate called recursively
Aborted (core dumped)
```

and, separately:

```
F0000 ... work_stealing_thread_pool.cc:186] Check failed: pool_->IsQuiesced()
*** Check failure stack trace: ***
Aborted (core dumped)
```

**Root cause**

`Activity::Poller#process(task)` / `Workflow::Poller#process(task)` construct a brand-new `TaskProcessor` for every polled task:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/poller.rb#L110-L114

`TaskProcessor#connection` memoizes its own `Temporal::Connection::GRPC`:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/task_processor.rb#L71-L73

So every single activity/workflow task opens a brand-new gRPC channel (fresh DNS resolution, fresh TLS handshake, fresh subchannels/LB policy) and tears it down again right after finishing. Under load (several tasks/sec with a non-trivial `activity_thread_pool_size`), this produces heavy concurrent gRPC channel churn. We confirmed this directly via `GRPC_TRACE=call_error,client_channel`: **48 distinct channel handles**, 33 `creating client_channel` / 89 `destroying subchannel wrapper` lines in a single ~3 minute window on one worker pod.

That churn races grpc-core's internal C++ lifecycle bookkeeping (subchannel refcounting, LB policy teardown, and the EventEngine thread pool's shutdown/quiescence accounting) and trips different fatal internal assertions depending on timing — which is why we saw two different crash signatures for what appears to be the same underlying stressor.

This may also explain, or be related to, #291 ("Unable to poll" / `GRPC::Unavailable: Socket closed` errors happening frequently under similar thread-pool concurrency), and possibly #280.

**Proposed fix**

`TaskProcessor` should reuse the `Poller`'s own long-lived connection instead of building its own per task. gRPC channels are explicitly designed to be shared across concurrent calls, so this is safe even with several `TaskProcessor`s running concurrently on the poller's thread pool — the connection's only `Mutex` (`poll_mutex`) guards solely the long-poll bookkeeping (`poll_activity_task_queue`/`poll_workflow_task_queue`/`cancel_polling_request`), not the `respond_*_task_completed`/`respond_*_task_failed` calls concurrent task processors make, so no new lock contention is introduced by sharing.

We've deployed this exact fix downstream (as a monkeypatch, since we can't modify the gem source directly in our app) and confirmed **0 crashes over a multi-day soak** in an environment that was previously crash-looping every ~10 minutes to a few hours.

Happy to open a PR with this fix — backward-compatible, adds an optional `connection:` keyword arg to `TaskProcessor#initialize` defaulting to `nil`, so existing behavior is unchanged for anyone not passing it. Let me know if that's welcome.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.