canonical / canonical/postgresql-operator

Patroni logs with grafana-agent causing a lot of unrelease file and huge syslog

Open
#524 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce

1. install postgresql charm `14/edge`, revision `425`
2. integrate with COS
3. grafana-agent will keep track of all the logs files under `/var/snap/charmed-postgresql/common/var/log/patroni`, which causing `syslog` become several GB huge, spamming with logs like below:
```
Jun 30 00:00:03 landscapesql-new-1 grafana-agent.grafana-agent[416696]: ts=2024-06-30T00:00:03.940798064Z caller=filetargetmanager.go:181 level=info component=logs logs_config=log_file_scraper msg="received file watcher event" name=/snap/grafana-agent/16/shared-logs/patroni/patroni.log.6545 op=CREATE
```
4. running `lsof -nP | grep '(deleted)'` on the postgresql with grafana-agent, we can see if the patroni logs are getting rotated, grafana-agent couldn't handle it correctly and causing a lot of unrelease files.
```
agent 1740025 1740154 agent root *510r REG 252,2 488 568380 /snap/grafana-agent/16/shared-logs/patroni/patroni.log.10080 (deleted)
agent 1740025 1740154 agent root *511r REG 252,2 488 561238 /snap/grafana-agent/16/shared-logs/patroni/patroni.log.10080 (deleted)
```

## Expected behavior
1. A smaller number of logs files, either
a. set by default in `/var/snap/charmed-postgresql/115/etc/patroni/patroni.yaml`
b. have an option in charm to configure the log.file_num
2. Logs files are cleaned correctly

## Actual behavior

You may find the logs in the summary above.

## Versions

Operating system:
Ubuntu 22.04.4 LTS

Juju CLI:
3.1.8-genericlinux-amd64

Juju agent:
3.1.8

Charm revision: 425

LXD: 4.0.9

## Log output

Juju debug log:

## Additional context

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with PostgreSQL charm 14/edge revision 425 integrated with COS, then inspect `/var/snap/charmed-postgresql/115/etc/patroni/patroni.yaml` and the Grafana Agent file-watcher behavior during Patroni log rotation. Verify the configured `log.file_num` and use `lsof -nP | grep '(deleted)'` to track open rotated files. Done means rotated Patroni logs are cleaned correctly without excessive files or syslog messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, postgresql, python
Domain
databases, devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.