canonical / canonical/cos-proxy-operator

Vector runs out of memory when loki rejects data

Open
#130 6 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

Loki wasn't accepting new data (for whatever reason), so vector's buffers got full to the point of OOM.

- May need to tell vector to drop new data: https://vector.dev/docs/about/under-the-hood/architecture/buffering-model/#drop-the-event-drop_newest
- Should fire an alert if vector is dropping data. (Use pebble checks/notices?)

### To Reproduce

Potentially:
1. Be sure to turn off `large_dir` (disabled by default anyway?) on the filesystem where the chunks folder for Loki is, e.g. `tune2fs -O large_dir /dev/vda2`. You can verify if `large_dir` is enabled or disabled by running `sudo tune2fs -l /dev/LOKI | grep -i large_dir`.
2. Fill up Loki chunks folder (originally with ceph, but could probably use flog too)
3. Wait :)

### Environment

```
cos-proxy n/a active 1 cos-proxy latest/edge 46
```

### Relevant log output

```shell
2023-11-04T23:34:58.143549Z WARN sink{component_kind="sink" component_id=loki-0 component_type=loki component_name=loki-0}:request{request_id=643645}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true
```

### Additional context

Observed by @dnegreira.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the Vector Loki sink configuration and the existing pebble checks/notices entry points. Reproduce Loki rejection or a full chunks folder, then determine how dropped data should be surfaced. Done means Vector no longer reaches OOM and an alert is emitted when data is dropped.

Written by the indexing model from the issue text.

Assessment

Domain
devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.