GoogleCloudPlatform / GoogleCloudPlatform/esp-v2

Shutdown error logging behaviour

Open
#867 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
307
Forks
185
Avg merge
14h 45m
Merged PRs (30d)
6

Description

Hi,

we are seeing a lot of error logging messages on v2.46.0 shutdown.

The most common messages,

> 54][filter]Failed to call check, error: CANCELLED: Request cancelled, str body:
> Failed in Check call: Request cancelled

disappear if we remove

```
- --health_check_grpc_backend
- --healthz=healthz
```

however the rest of the logging is a confusing mix of JSON / text / info / warn / error / critical logging. Surely this is not the intended behaviour?

Log extracts:

![Skjermbilde fra 2023-10-31 14-51-41](https://github.com/GoogleCloudPlatform/esp-v2/assets/1031478/6c838f4a-c6d9-4680-8bc6-6d5bd7731b82)

then a lot of repeated lines

![Skjermbilde fra 2023-10-31 14-51-17](https://github.com/GoogleCloudPlatform/esp-v2/assets/1031478/51ab88e7-365b-453a-929e-8b9435eabd88)

Note about the health checks: the downstream container is still alive when this happens (delayed by a `preStop` sleep for 30 seconds).

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Reproduce the v2.46.0 shutdown with --health_check_grpc_backend and --healthz=healthz, then compare the logs with those flags removed while the downstream remains alive. Done should mean shutdown produces consistent, non-repeated logging at appropriate levels.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.