canonical / canonical/postgresql-operator
Patroni logs with grafana-agent causing a lot of unrelease file and huge syslog
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
## Steps to reproduce
1. install postgresql charm `14/edge`, revision `425`
2. integrate with COS
3. grafana-agent will keep track of all the logs files under `/var/snap/charmed-postgresql/common/var/log/patroni`, which causing `syslog` become several GB huge, spamming with logs like below:
```
Jun 30 00:00:03 landscapesql-new-1 grafana-agent.grafana-agent[416696]: ts=2024-06-30T00:00:03.940798064Z caller=filetargetmanager.go:181 level=info component=logs logs_config=log_file_scraper msg="received file watcher event" name=/snap/grafana-agent/16/shared-logs/patroni/patroni.log.6545 op=CREATE
```
4. running `lsof -nP | grep '(deleted)'` on the postgresql with grafana-agent, we can see if the patroni logs are getting rotated, grafana-agent couldn't handle it correctly and causing a lot of unrelease files.
```
agent 1740025 1740154 agent root *510r REG 252,2 488 568380 /snap/grafana-agent/16/shared-logs/patroni/patroni.log.10080 (deleted)
agent 1740025 1740154 agent root *511r REG 252,2 488 561238 /snap/grafana-agent/16/shared-logs/patroni/patroni.log.10080 (deleted)
```
## Expected behavior
1. A smaller number of logs files, either
a. set by default in `/var/snap/charmed-postgresql/115/etc/patroni/patroni.yaml`
b. have an option in charm to configure the log.file_num
2. Logs files are cleaned correctly
## Actual behavior
You may find the logs in the summary above.
## Versions
Operating system:
Ubuntu 22.04.4 LTS
Juju CLI:
3.1.8-genericlinux-amd64
Juju agent:
3.1.8
Charm revision: 425
LXD: 4.0.9
## Log output
Juju debug log:
## Additional context
Contributor guide
Research direction
Start by reproducing the issue with PostgreSQL charm 14/edge revision 425 integrated with COS, then inspect `/var/snap/charmed-postgresql/115/etc/patroni/patroni.yaml` and the Grafana Agent file-watcher behavior during Patroni log rotation. Verify the configured `log.file_num` and use `lsof -nP | grep '(deleted)'` to track open rotated files. Done means rotated Patroni logs are cleaned correctly without excessive files or syslog messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, postgresql, python
- Domain
- databases, devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100