canonical / canonical/postgresql-operator
Excessive logging pattern when relating to otel-collector causing very high CPU utilization.
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
1. deploy psql
2. deploy otel-collector
3. relate psql with otel-collector (that is related to COS)
## Expected behavior
Otel collector receivers are not configured with `/**`. However, that still doesn't resolve the issue completely as per psql log rotation [#736](https://github.com/canonical/postgresql-operator/issues/736)
```
include:
- /snap/opentelemetry-collector/25/shared-logs/patroni/*.log
include:
- /snap/opentelemetry-collector/25/shared-logs/pgbackrest/*.log
include:
- /snap/opentelemetry-collector/25/shared-logs/pgbouncer/*.log
include:
- /snap/opentelemetry-collector/25/shared-logs/postgresql/*.log
```
## Actual behavior
As of right now the receivers have `/**`
```
include:
- /snap/opentelemetry-collector/25/shared-logs/patroni/**
include:
- /snap/opentelemetry-collector/25/shared-logs/pgbackrest/**
include:
- /snap/opentelemetry-collector/25/shared-logs/pgbouncer/**
include:
- /snap/opentelemetry-collector/25/shared-logs/postgresql/**
```
Causing quite some pressure on CPU
```
:/var/lib/juju/storage/rootfs/9/2/var/log/postgresql$ ll *.log | wc -l
10080
```
```
sudo ls -l /proc/$(pgrep -x otelcol)/fd | wc -l
1143
```
## Versions
Operating system: Ubuntu 22.04.5 LTS
Juju CLI: 3.6.23-genericlinux-amd64
Juju agent: 3.6.21
Charm revision: 468 /14.12
LXD:
## Log output
## Additional context
I find the following issues to be relevant.
[grafana-agent-#153](https://github.com/canonical/grafana-agent-operator/issues/153)
[#736](https://github.com/canonical/postgresql-operator/issues/736)
Also [path_exclude](https://charmhub.io/opentelemetry-collector/configurations#path_exclude) config option doesn't apply here as it only works on `/var/log/`
Contributor guide
Research direction
Reproduce the issue by deploying psql and otel-collector, relating them, and checking the generated receiver includes and open file descriptors. Start at the charm code that handles the otel-collector relation and log receiver configuration, then trace the effect of PostgreSQL log rotation. Done means receivers avoid the excessive /** pattern without losing the intended PostgreSQL logs, with coverage for the resulting configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100