canonical / canonical/haproxy-operator

[Bug]: Lack of logrotate configuration causes disk exhaustion and unit crashes for high-traffic ingresses

Open
#638 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4
Forks
12
Avg merge
1d 8h
Merged PRs (30d)
22

Description

### Bug Description

We are experiencing disk exhaustion issues on units running the haproxy charm when handling high-traffic ingresses. Because the charm currently does not expose any configuration for logrotate, it falls back to the default log rotation policy, which only rotates logs once a week.

For high-volume environments, this weekly rotation is far too slow, causing the HAProxy logs to fill up the entire disk before the rotation is ever triggered. When the disk is filled, the charm goes into error and stops serving.

We already expanded the disk size but this is not a sustainable solution, we have and application that stops every 2 days. We need the ability to configure log rotation (e.g., size-based rotation, daily/hourly frequency, or capping retention) directly through the charm.

### Impact

Medium (functionality degraded, workaround exists)

### Impact Rationale

_No response_

### To Reproduce

1. Deploy the haproxy charm and relate it to workloads as an ingress.
2. Direct a high and continuous volume of traffic to the ingress.
3. Monitor the disk usage on the HAProxy unit.
4. Observe that the HAProxy log files grow unbounded over the course of the week because log rotation only runs weekly.
5. Once the disk fills up, observe the unit going into error.

### Environment

channel=2.8/edge
revision=537
running on canonical-k8s

### Relevant log output

```shell
2026-08-14 05:44:13 INFO juju.worker.uniter.operation runhook.go:186 ran "haproxy-route-relation-created" hook (via hook dispatching script: dispatch)
2026-08-14 05:44:14 ERROR unit.ingress-ps7-certification/15.juju-log server.go:405 haproxy-route:1220: 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ingress-ps7-certification-15/charm/lib/charms/haproxy/v2/haproxy_route.py", line 272, in load
return cls.model_validate_json(json.dumps(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-ingress-ps7-certification-15/charm/venv/lib/python3.12/site-packages/pydantic/main.py", line 782, in model_validate_json
return cls.__pydantic_validator__.validate_json(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
2026-08-14 05:44:14 ERROR unit.ingress-ps7-certification/15.juju-log server.go:405 haproxy-route:1220: Invalid requirer application data for remote-dcefa3f1833d4dc18e0cd06dcb274f63
2026-08-14 05:44:14 ERROR unit.ingress-ps7-certification/15.juju-log server.go:405 haproxy-route:1220: 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ingress-ps7-certification-15/charm/lib/charms/haproxy/v2/haproxy_route.py", line 272, in load
return cls.model_validate_json(json.dumps(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-ingress-ps7-certification-15/charm/venv/lib/python3.12/site-packages/pydantic/main.py", line 782, in model_validate_json
return cls.__pydantic_validator__.validate_json(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/missing
2026-08-14 05:44:14 ERROR unit.ingress-ps7-certification/15.juju-log server.go:405 haproxy-route:1220: Invalid requirer application data for remote-dcefa3f1833d4dc18e0cd06dcb274f63
2026-08-14 05:44:16 INFO unit.ingress-ps7-certification/15.juju-log server.go:405 haproxy-route:1220: ha integration is not ready, skipping.
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the charm entry point that configures HAProxy logging and the current logrotate setup. Review the issue's reproduction steps and determine how size, frequency, and retention should be exposed as charm configuration. Done means high-volume logs rotate before exhausting disk space, the unit remains healthy, and the behavior is covered by the repository's relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.