canonical / canonical/postgresql-operator

Excessive logging pattern when relating to otel-collector causing very high CPU utilization.

Open
#1,758 6 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

1. deploy psql
2. deploy otel-collector
3. relate psql with otel-collector (that is related to COS)

## Expected behavior
Otel collector receivers are not configured with `/**`. However, that still doesn't resolve the issue completely as per psql log rotation [#736](https://github.com/canonical/postgresql-operator/issues/736)

```
include:
- /snap/opentelemetry-collector/25/shared-logs/patroni/*.log

include:
- /snap/opentelemetry-collector/25/shared-logs/pgbackrest/*.log

include:
- /snap/opentelemetry-collector/25/shared-logs/pgbouncer/*.log

include:
- /snap/opentelemetry-collector/25/shared-logs/postgresql/*.log
```

## Actual behavior
As of right now the receivers have `/**`

```
include:
- /snap/opentelemetry-collector/25/shared-logs/patroni/**

include:
- /snap/opentelemetry-collector/25/shared-logs/pgbackrest/**

include:
- /snap/opentelemetry-collector/25/shared-logs/pgbouncer/**

include:
- /snap/opentelemetry-collector/25/shared-logs/postgresql/**
```

Causing quite some pressure on CPU

```
:/var/lib/juju/storage/rootfs/9/2/var/log/postgresql$ ll *.log | wc -l
10080
```
```
sudo ls -l /proc/$(pgrep -x otelcol)/fd | wc -l
1143
```

Image

## Versions

Operating system: Ubuntu 22.04.5 LTS

Juju CLI: 3.6.23-genericlinux-amd64

Juju agent: 3.6.21

Charm revision: 468 /14.12

LXD:

## Log output

## Additional context

I find the following issues to be relevant.

[grafana-agent-#153](https://github.com/canonical/grafana-agent-operator/issues/153)
[#736](https://github.com/canonical/postgresql-operator/issues/736)
Also [path_exclude](https://charmhub.io/opentelemetry-collector/configurations#path_exclude) config option doesn't apply here as it only works on `/var/log/`

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by deploying psql and otel-collector, relating them, and checking the generated receiver includes and open file descriptors. Start at the charm code that handles the otel-collector relation and log receiver configuration, then trace the effect of PostgreSQL log rotation. Done means receivers avoid the excessive /** pattern without losing the intended PostgreSQL logs, with coverage for the resulting configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.