fullstorydev / fullstorydev/grpcurl
grpcurl not able to connect to secondary IPs in FQDN if primary is not reachable
- Dominant language
- Go
- Stars
- 12.8k
- Forks
- 580
- Avg merge
- 1h 8m
- Merged PRs (30d)
- 5
Description
Hi,
We have an FQDN which resolves to two IPs for High Availability. The client needs to check the reachable IP and attempt to connect with the same.
In our script we first use `nc` command to make sure at least 1 IP in the FQDN is reachable
```
$ nc some.fqdn.svc.cluster.local 7055 -v -z -w2
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to 192.106.167.129 failed: Connection timed out.
Ncat: Trying next address...
Ncat: Connected to 192.106.226.1:7055.
Ncat: 0 bytes sent, 0 bytes received in 2.30 seconds.
```
and then execute grpcurl with the FQDN. But it fails to establish connection sporadically which I am assuming is because it tries to connect to an IP which is not reachable and when we execute grpcurl again then it works.
I see there are two observations (not sure if these are issues or expected behavior):
1. `-v` or `-vv` is not printing the IP to which grpcurl is connecting to when using FQDN
2. Automatic lookup and attempting connection of all available IPs inside FQDN is not happenening
When using -v or -vv we get output similar to below (which doesn't contain IP used)
```
$ ./grpcurl -vv -cacert /certs/cacert.pem -cert /certs/cert.pem -key /certs/key.pem -proto secretstore.proto -d '{"vnfid": "DEMO", "relver":"1","mulappid": "secrets","netype":"ABC"}' [some.fqdn.svc.cluster.local]:7055
Resolved method descriptor:
rpc Watch ( ) returns ( stream );
Request metadata to send:
(empty)
Response headers received:
content-type: application/grpc
date: Tue, 04 Apr 2023 07:07:00 GMT
server: envoy-ingress-ss
x-envoy-upstream-service-time: 99
Estimated response size: 2698 bytes
Response contents:
...
```
and the connection fails with this error message (when I assume the connection is attempted using non-reachable IP)
`Failed to dial target host "some.fqdn.svc.cluster.local:7055": context deadline exceeded`
Platform Details
```
$ ./grpcurl -version
grpcurl v1.8.7
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"
```
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files or tests are named. Reproduce the failure with the grpcurl command using an FQDN that resolves to reachable and unreachable IPs, then trace the target dialing and -v/-vv output paths. Done means the requested multi-address connection behavior and connection details are covered by tests or the issue's expected behavior is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- cli, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100