envoyproxy / envoyproxy/envoy

Crash on cluster initialization/healthcheck when ipv6 disabled

Open
#19,992 2 comments 3 reactions 0 assignees View on GitHub
area/cluster bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

_Crash already cleared w/ envoy-security@, they OKed this being filed as an issue_

**Title: Crash on cluster initialization/healthcheck when ipv6 disabled**

### Description
- **Current Behavior**: Envoy crash on cluster initialization/healthcheck when probing a domain w/ an IPv6 address, if IPv6 is disabled at the OS level
- **Expected Behavior**: Hopefully, Envoy could detect IPv6 unavailability and not attempt the IPv6 check, or should cleanly handle the failure and not crash.
- **OS**: Reproduced in both RHEL-7.9.21 and RHEL-8.5.1
- **Envoy version**: 1.20.1 (also reproduced with similar configs in 1.9.1)

```
[seanm514@286987d3 ~]$ /opt/vz/bin/envoy --version
/opt/vz/bin/envoy version: ea23f47b27464794980c05ab290a3b73d801405e/1.20.1/Yahoo/RELEASE/BoringSSL
```

### Repro steps

1. Stand up RHEL instance (tested against 7.9.21 and 8.5.1)
2. Install envoy (tested against 1.20.1 and reproduced with similar configs in 1.9.1)
3. Create test envoy configuration expressing a LOGICAL_DNS cluster pointing to an IPv6-enabled endpoint ([yahoo.com](http://yahoo.com/) in this case, but any domain expressing an IPv6 address should work)
4. Disable IPv6 via GRUB (along [these lines](https://www.techrepublic.com/article/how-to-disable-ipv6-through-grub-in-linux/))
5. Reboot instance
6. Reproduce crash by starting envoy with the test configuration

#### Detailed Steps
```

[seanm514@286987d3 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 (Ootpa)

[seanm514@286987d3 ~]$ sudo yum install envoy-1.20.1
Last metadata expiration check: 0:00:06 ago on Wed 09 Feb 2022 06:05:39 PM UTC.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
envoy x86_64 1.20.1-1.el8 media_edge-release 17 M
...
Installed:
envoy-1.20.1-1.el8.x86_64
...

[seanm514@286987d3 ~]$ /opt/vz/bin/envoy --version
/opt/vz/bin/envoy version: ea23f47b27464794980c05ab290a3b73d801405e/1.20.1/Yahoo/RELEASE/BoringSSL

[seanm514@286987d3 ~]$ cat /tmp/ipv6_test.yaml
static_resources:
clusters:
- name: service_test
connect_timeout: 1s
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_test
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
protocol: TCP
address: [yahoo.com](http://yahoo.com/)
port_value: 443

[seanm514@286987d3 ~]$ ping -6 [yahoo.com](http://yahoo.com/)
PING [yahoo.com](http://yahoo.com/)([media-router-fp74.prod.media.vip.bf1.yahoo.com](http://media-router-fp74.prod.media.vip.bf1.yahoo.com/) (2001:4998:124:1507::f001)) 56 data bytes
64 bytes from [media-router-fp74.prod.media.vip.bf1.yahoo.com](http://media-router-fp74.prod.media.vip.bf1.yahoo.com/) (2001:4998:124:1507::f001): icmp_seq=1 ttl=44 time=90.3 ms

[seanm514@286987d3 ~]$ sudo vi /etc/default/grub

[seanm514@286987d3 ~]$ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="crashkernel=auto console=tty0 console=ttyS0,115200n8 net.ifnames=0 ipv6.disable=1"

[seanm514@286987d3 ~]$ sudo /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-348.12.2.el8_5.x86_64
Found initrd image: /boot/initramfs-4.18.0-348.12.2.el8_5.x86_64.img
done

[seanm514@286987d3 ~]$ ping -6 [yahoo.com](http://yahoo.com/)
ping: socket: Address family not supported by protocol

[seanm514@286987d3 ~]$ /opt/vz/bin/envoy -c /tmp/ipv6_test.yaml
[2022-02-09 18:14:34.326][1469][info][main] [source/server/server.cc:368] initializing epoch 0 (base id=0, hot restart version=11.104)
...

Aborted (core dumped)
```

### Config
```
static_resources:
clusters:
- name: service_test
connect_timeout: 1s
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_test
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
protocol: TCP
address: yahoo.com
port_value: 443
```

### Logs
```
[seanm514@286987d3 ~]$ /opt/vz/bin/envoy -c /tmp/ipv6_test.yaml
[2022-02-09 18:14:34.326][1469][info][main] [source/server/server.cc:368] initializing epoch 0 (base id=0, hot restart version=11.104)
...
[2022-02-09 18:14:34.332][1469][info][main] [source/server/server.cc:740] runtime: {}
[2022-02-09 18:14:34.332][1469][warning][main] [source/server/server.cc:585] No admin address given, so no admin HTTP server started.
[2022-02-09 18:14:34.332][1469][info][config] [source/server/configuration_impl.cc:127] loading tracing configuration
[2022-02-09 18:14:34.332][1469][info][config] [source/server/configuration_impl.cc:87] loading 0 static secret(s)
[2022-02-09 18:14:34.332][1469][info][config] [source/server/configuration_impl.cc:93] loading 1 cluster(s)
[2022-02-09 18:14:34.333][1469][info][config] [source/server/configuration_impl.cc:97] loading 0 listener(s)
[2022-02-09 18:14:34.333][1469][info][config] [source/server/configuration_impl.cc:109] loading stats configuration
[2022-02-09 18:14:34.333][1469][info][main] [source/server/server.cc:836] starting main dispatch loop
[2022-02-09 18:14:34.335][1469][critical][main] [source/exe/terminate_handler.cc:12] std::terminate called! (possible uncaught exception, see trace)
[2022-02-09 18:14:34.335][1469][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-02-09 18:14:34.335][1469][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: ea23f47b27464794980c05ab290a3b73d801405e/1.20.1/Yahoo/RELEASE/BoringSSL
[2022-02-09 18:14:34.343][1469][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x555fb19d491f]
[2022-02-09 18:14:34.350][1469][critical][backtrace] [./source/server/backtrace.h:98] #1: [0x555fb19d46a9]
[2022-02-09 18:14:34.356][1469][critical][backtrace] [./source/server/backtrace.h:96] #2: __cxxabiv1::__terminate() [0x555fb204ac7c]
[2022-02-09 18:14:34.356][1469][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x5327000005bd
[2022-02-09 18:14:34.356][1469][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-02-09 18:14:34.356][1469][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: ea23f47b27464794980c05ab290a3b73d801405e/1.20.1/Yahoo/RELEASE/BoringSSL
[2022-02-09 18:14:34.357][1469][critical][backtrace] [./source/server/backtrace.h:96] #0: __restore_rt [0x7f7911650c20]
[2022-02-09 18:14:34.363][1469][critical][backtrace] [./source/server/backtrace.h:98] #1: [0x555fb19d46a9]
[2022-02-09 18:14:34.363][1469][critical][backtrace] [./source/server/backtrace.h:96] #2: __cxxabiv1::__terminate() [0x555fb204ac7c]
Aborted (core dumped)
```

### Call Stack
See Logs above

### Mitigations
The crash can be avoided by:

- Re-enabling IPv6 for the OS
- Marking the cluster with `dns_lookup_family: V4_ONLY`

However, it is somewhat brittle to rely on manually configuring `V4_ONLY` for each new cluster we add, so this issue seemed worth reporting.

Contributor guide

Open the contributing guide

Research direction

Reproduce the crash with the supplied LOGICAL_DNS configuration on RHEL with IPv6 disabled, then use the logged startup path and the Envoy cluster DNS/socket handling to locate the failing operation. Compare behavior with dns_lookup_family: V4_ONLY and verify that an IPv6-unavailable environment no longer aborts during cluster initialization or health checking.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.