fullstorydev / fullstorydev/grpcurl
grpcurl not closing threads when exiting due to network issues (transport is closing)
- Dominant language
- Go
- Stars
- 12.8k
- Forks
- 580
- Avg merge
- 1h 8m
- Merged PRs (30d)
- 5
Description
We have a grpcurl client which is connecting to a gRPC server. Due to some network issue in our environment, our clients are terminated with error "transport is closing"
When we start grpcurl with -vv option, then we receive this message during termination
```
Response trailers received:
(empty)
Sent 1 request and received 19 responses
ERROR:
Code: Unavailable
Message: error reading from server: read tcp 192.168.219.193:39110->100.XX.XX.XXX:7055: read: connection reset by peer
```
In such scenarios the threads which were opened by grpcurl are not closed and it keeps pilling up on the node:
```
$ lsof | grep grpcurl| wc -l
420
```
Sample lsof output
```
grpcurl 686141 686159 1025 NOFD /proc/686141/task/686159/fd (opendir: Permission denied)
grpcurl 686141 686175 1025 cwd unknown /proc/686141/task/686175/cwd (readlink: Permission denied)
grpcurl 686141 686175 1025 rtd unknown /proc/686141/task/686175/root (readlink: Permission denied)
grpcurl 686141 686175 1025 txt unknown /proc/686141/task/686175/exe (readlink: Permission denied)
grpcurl 686141 686175 1025 NOFD /proc/686141/task/686175/fd (opendir: Permission denied)
grpcurl 686141 686176 1025 cwd unknown /proc/686141/task/686176/cwd (readlink: Permission denied)
grpcurl 686141 686176 1025 rtd unknown /proc/686141/task/686176/root (readlink: Permission denied)
grpcurl 686141 686176 1025 txt unknown /proc/686141/task/686176/exe (readlink: Permission denied)
grpcurl 686141 686176 1025 NOFD /proc/686141/task/686176/fd (opendir: Permission denied)
grpcurl 686141 686177 1025 cwd unknown /proc/686141/task/686177/cwd (readlink: Permission denied)
grpcurl 686141 686177 1025 rtd unknown /proc/686141/task/686177/root (readlink: Permission denied)
grpcurl 686141 686177 1025 txt unknown /proc/686141/task/686177/exe (readlink: Permission denied)
grpcurl 686141 686177 1025 NOFD /proc/686141/task/686177/fd (opendir: Permission denied)
grpcurl 686141 686178 1025 cwd unknown /proc/686141/task/686178/cwd (readlink: Permission denied)
grpcurl 686141 686178 1025 rtd unknown /proc/686141/task/686178/root (readlink: Permission denied)
grpcurl 686141 686178 1025 txt unknown /proc/686141/task/686178/exe (readlink: Permission denied)
grpcurl 686141 686178 1025 NOFD /proc/686141/task/686178/fd (opendir: Permission denied)
grpcurl 762963 1025 cwd unknown /proc/762963/cwd (readlink: Permission denied)
grpcurl 762963 1025 rtd unknown /proc/762963/root (readlink: Permission denied)
grpcurl 762963 1025 txt unknown /proc/762963/exe (readlink: Permission denied)
grpcurl 762963 1025 NOFD /proc/762963/fd (opendir: Permission denied)
grpcurl 762963 762975 1025 cwd unknown /proc/762963/task/762975/cwd (readlink: Permission denied)
...
```
Is there any way to make sure grpcurl connections are gracefully terminated and all threads are closed before exiting?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the network-reset scenario using grpcurl with -vv and observe whether the process leaves worker threads behind. Trace the shutdown behavior after the "transport is closing" error; done means the command exits cleanly without accumulating threads or open resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100