fullstorydev / fullstorydev/grpcurl

grpcurl not closing threads when exiting due to network issues (transport is closing)

Open
#382 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
12.8k
Forks
580
Avg merge
1h 8m
Merged PRs (30d)
5

Description

We have a grpcurl client which is connecting to a gRPC server. Due to some network issue in our environment, our clients are terminated with error "transport is closing"
When we start grpcurl with -vv option, then we receive this message during termination
```
Response trailers received:
(empty)
Sent 1 request and received 19 responses
ERROR:
Code: Unavailable
Message: error reading from server: read tcp 192.168.219.193:39110->100.XX.XX.XXX:7055: read: connection reset by peer
```
In such scenarios the threads which were opened by grpcurl are not closed and it keeps pilling up on the node:
```
$ lsof | grep grpcurl| wc -l
420
```

Sample lsof output
```
grpcurl 686141 686159 1025 NOFD /proc/686141/task/686159/fd (opendir: Permission denied)
grpcurl 686141 686175 1025 cwd unknown /proc/686141/task/686175/cwd (readlink: Permission denied)
grpcurl 686141 686175 1025 rtd unknown /proc/686141/task/686175/root (readlink: Permission denied)
grpcurl 686141 686175 1025 txt unknown /proc/686141/task/686175/exe (readlink: Permission denied)
grpcurl 686141 686175 1025 NOFD /proc/686141/task/686175/fd (opendir: Permission denied)
grpcurl 686141 686176 1025 cwd unknown /proc/686141/task/686176/cwd (readlink: Permission denied)
grpcurl 686141 686176 1025 rtd unknown /proc/686141/task/686176/root (readlink: Permission denied)
grpcurl 686141 686176 1025 txt unknown /proc/686141/task/686176/exe (readlink: Permission denied)
grpcurl 686141 686176 1025 NOFD /proc/686141/task/686176/fd (opendir: Permission denied)
grpcurl 686141 686177 1025 cwd unknown /proc/686141/task/686177/cwd (readlink: Permission denied)
grpcurl 686141 686177 1025 rtd unknown /proc/686141/task/686177/root (readlink: Permission denied)
grpcurl 686141 686177 1025 txt unknown /proc/686141/task/686177/exe (readlink: Permission denied)
grpcurl 686141 686177 1025 NOFD /proc/686141/task/686177/fd (opendir: Permission denied)
grpcurl 686141 686178 1025 cwd unknown /proc/686141/task/686178/cwd (readlink: Permission denied)
grpcurl 686141 686178 1025 rtd unknown /proc/686141/task/686178/root (readlink: Permission denied)
grpcurl 686141 686178 1025 txt unknown /proc/686141/task/686178/exe (readlink: Permission denied)
grpcurl 686141 686178 1025 NOFD /proc/686141/task/686178/fd (opendir: Permission denied)
grpcurl 762963 1025 cwd unknown /proc/762963/cwd (readlink: Permission denied)
grpcurl 762963 1025 rtd unknown /proc/762963/root (readlink: Permission denied)
grpcurl 762963 1025 txt unknown /proc/762963/exe (readlink: Permission denied)
grpcurl 762963 1025 NOFD /proc/762963/fd (opendir: Permission denied)
grpcurl 762963 762975 1025 cwd unknown /proc/762963/task/762975/cwd (readlink: Permission denied)
...
```

Is there any way to make sure grpcurl connections are gracefully terminated and all threads are closed before exiting?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the network-reset scenario using grpcurl with -vv and observe whether the process leaves worker threads behind. Trace the shutdown behavior after the "transport is closing" error; done means the command exits cleanly without accumulating threads or open resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.