envoyproxy / envoyproxy/envoy

Postgres proxy: Envoy Crash during pgbench test

Open
#36,471 17 comments 0 reactions 1 assignee Claimed by @cpakulski View on GitHub
area/postgres bug help wanted stale
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Title*: *Envoy Crash during pgbench test*

*Description*:
We have run pgbench to test postgres server proxied by Envoy. During the test Envoy crashed with the following log:

```bash
[2024-09-19 14:22:51.227][42][critical][backtrace] [./source/server/backtrace.h:127] Caught Segmentation fault, suspect faulting address 0x0
[2024-09-19 14:22:51.227][42][critical][backtrace] [./source/server/backtrace.h:111] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-09-19 14:22:51.227][42][critical][backtrace] [./source/server/backtrace.h:112] Envoy version: ca7ce529646b1f7c8c760e0e249e285cee4549ed/1.30.2/Modified/RELEASE/BoringSSL
[2024-09-19 14:22:51.227][42][critical][backtrace] [./source/server/backtrace.h:114] Address mapping: 560479564000-56047cf63000 /usr/local/bin/envoy
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #0: [0x7fa621681520]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #1: [0x56047b34bfc4]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #2: [0x56047b34949a]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #3: [0x56047b348d88]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #4: [0x56047b345b6d]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #5: [0x56047b3304fa]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #6: [0x56047b32fd00]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #7: [0x56047b3443e7]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #8: [0x56047bb7aec5]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #9: [0x56047bb72663]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #10: [0x56047bb6f2f8]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #11: [0x56047bb4d2f1]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #12: [0x56047bb4e89d]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #13: [0x56047ca79480]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #14: [0x56047ca77dc1]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #15: [0x56047ad6332f]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #16: [0x56047cafac63]
[2024-09-19 14:22:51.228][42][critical][backtrace] [./source/server/backtrace.h:121] #17: [0x7fa6216d3ac3]
ConnectionImpl 0x26537f7fe640, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 32768
socket_:
ListenSocketImpl 0x26537f0aeab0, transport_protocol_: raw_buffer
connection_info_provider_:
ConnectionInfoSetterImpl 0x26537f09e518, remote_address_: 100.126.0.213:54396, direct_remote_address_: 100.126.0.213:54396, local_address_: 100.126.5.153:50815, server_name_:
```
*Testing scope:*
Envoy configuration: https://gist.github.com/shiponcs/600fff61cddba5911bc3b9e538a4fc8c
pgbench command: `pgbench -h localhost -p 50815 -U postgres -c 50 -j 1 -T 00 example`

*Relevant*:
We have tried to find the reason; by running git bisect we discovered that, [this commit](https://github.com/envoyproxy/envoy/commit/61c9a47b8f68c4e66b584fbe86b046e082fec2d9) is causing the crash. If we get rid of the changes introduced in the commit, the issue gets resolved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.