fullstorydev / fullstorydev/grpcurl

grpcurl not able to connect to secondary IPs in FQDN if primary is not reachable

Open
#384 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
12.8k
Forks
580
Avg merge
1h 8m
Merged PRs (30d)
5

Description

Hi,

We have an FQDN which resolves to two IPs for High Availability. The client needs to check the reachable IP and attempt to connect with the same.

In our script we first use `nc` command to make sure at least 1 IP in the FQDN is reachable
```
$ nc some.fqdn.svc.cluster.local 7055 -v -z -w2
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to 192.106.167.129 failed: Connection timed out.
Ncat: Trying next address...
Ncat: Connected to 192.106.226.1:7055.
Ncat: 0 bytes sent, 0 bytes received in 2.30 seconds.
```

and then execute grpcurl with the FQDN. But it fails to establish connection sporadically which I am assuming is because it tries to connect to an IP which is not reachable and when we execute grpcurl again then it works.

I see there are two observations (not sure if these are issues or expected behavior):
1. `-v` or `-vv` is not printing the IP to which grpcurl is connecting to when using FQDN
2. Automatic lookup and attempting connection of all available IPs inside FQDN is not happenening

When using -v or -vv we get output similar to below (which doesn't contain IP used)
```
$ ./grpcurl -vv -cacert /certs/cacert.pem -cert /certs/cert.pem -key /certs/key.pem -proto secretstore.proto -d '{"vnfid": "DEMO", "relver":"1","mulappid": "secrets","netype":"ABC"}' [some.fqdn.svc.cluster.local]:7055

Resolved method descriptor:
rpc Watch ( ) returns ( stream );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
date: Tue, 04 Apr 2023 07:07:00 GMT
server: envoy-ingress-ss
x-envoy-upstream-service-time: 99

Estimated response size: 2698 bytes

Response contents:
...
```

and the connection fails with this error message (when I assume the connection is attempted using non-reachable IP)
`Failed to dial target host "some.fqdn.svc.cluster.local:7055": context deadline exceeded`

Platform Details
```
$ ./grpcurl -version
grpcurl v1.8.7

$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

```

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Reproduce the failure with the grpcurl command using an FQDN that resolves to reachable and unreachable IPs, then trace the target dialing and -v/-vv output paths. Done means the requested multi-address connection behavior and connection details are covered by tests or the issue's expected behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
cli, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.