canonical / canonical/cos-proxy-operator
Vector runs out of memory when loki rejects data
- Dominant language
- Python
- Stars
- 3
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
Loki wasn't accepting new data (for whatever reason), so vector's buffers got full to the point of OOM.
- May need to tell vector to drop new data: https://vector.dev/docs/about/under-the-hood/architecture/buffering-model/#drop-the-event-drop_newest
- Should fire an alert if vector is dropping data. (Use pebble checks/notices?)
### To Reproduce
Potentially:
1. Be sure to turn off `large_dir` (disabled by default anyway?) on the filesystem where the chunks folder for Loki is, e.g. `tune2fs -O large_dir /dev/vda2`. You can verify if `large_dir` is enabled or disabled by running `sudo tune2fs -l /dev/LOKI | grep -i large_dir`.
2. Fill up Loki chunks folder (originally with ceph, but could probably use flog too)
3. Wait :)
### Environment
```
cos-proxy n/a active 1 cos-proxy latest/edge 46
```
### Relevant log output
```shell
2023-11-04T23:34:58.143549Z WARN sink{component_kind="sink" component_id=loki-0 component_type=loki component_name=loki-0}:request{request_id=643645}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true
```
### Additional context
Observed by @dnegreira.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the Vector Loki sink configuration and the existing pebble checks/notices entry points. Reproduce Loki rejection or a full chunks folder, then determine how dropped data should be surfaced. Done means Vector no longer reaches OOM and an alert is emitted when data is dropped.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100