fluent / fluent/fluent-bit

in_kubernetes_events: add SQLite indexes for event deduplication and cleanup

Open Beginner friendly
#12,327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**

With `db` enabled, `in_kubernetes_events` performs per-event deduplication by `uid` and cleanup by `created`. Neither column is indexed, so both queries scan the SQLite table.

With about 91,000 Kubernetes Event objects, this caused sustained CPU usage of one core during relisting. After the watch stabilized, CPU remained around 200m.

**Describe the solution you'd like**

Create indexes for new and existing databases:

```sql
CREATE INDEX IF NOT EXISTS idx_in_kubernetes_events_uid
ON in_kubernetes_events(uid);

CREATE INDEX IF NOT EXISTS idx_in_kubernetes_events_created
ON in_kubernetes_events(created);
```

**Describe alternatives you've considered**

- Removing `db` avoids the scans, but loses persistent deduplication across restarts.
- Creating the indexes manually works, but must be repeated if the database is recreated.

**Additional context**

Creating both indexes manually reduced steady CPU from about 200m to 20m without restarting Fluent Bit or changing its configuration.

The missing `uid` index was mentioned in #9787, but #9894 only fixed the cleanup timestamp calculation. The indexes are still absent from the current schema.

Contributor guide

Open the contributing guide

Research direction

Start at the in_kubernetes_events schema and its database initialization or migration entry points, then inspect the uid deduplication and created cleanup queries. Verify the requested indexes are created for both new and existing databases and confirm the relevant queries use them.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, kubernetes, sqlite
Domain
databases
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.