guibranco / guibranco/logstream-server

[FEATURE] Log sampling — configurable entry drop rate

Open
#61 0 comments 0 reactions 0 assignees View on GitHub
✨ feature enhancement good first issue hacktoberfest help wanted
Dominant language
PHP
Stars
1
Forks
0
Avg merge
1m
Merged PRs (30d)
4

Description

## Summary

Drop a configurable percentage of high-volume, low-priority log entries before they reach storage, reducing costs without requiring application code changes.

## Motivation

High-throughput applications (especially via UDP) can generate millions of `debug` entries daily. Sampling allows retaining statistical representativeness while dramatically reducing storage volume.

## Proposed configuration

```json
{
"sampling": [
{ "name": "sample-billing-debug", "app_key": "billing-api", "level": "debug", "rate": 0.10 },
{ "name": "sample-all-debug", "level": "debug", "rate": 0.05 },
{ "name": "sample-info", "level": "info", "rate": 0.50 }
]
}
```

`rate: 0.10` means accept 10% of matching entries at random. Entries not accepted are permanently discarded.

## Behaviour

- Applied **before** the ingestion queue — at the HTTP router and UDP listener
- `error` and `critical` levels are **never sampled** regardless of config
- Accepted entries get `_sampled: true` and `_sample_rate: 0.10` added to `context`
- Loaded from `SAMPLING_CONFIG=./sampling.json`

## Acceptance criteria

- [ ] Sampling config loaded from file at startup
- [ ] Correct percentage of entries accepted (statistically, over large N)
- [ ] `error` and `critical` entries always pass through regardless of config
- [ ] Applied at both HTTP and UDP ingestion paths
- [ ] Accepted entries tagged with `_sampled` and `_sample_rate` in context
- [ ] `GET /api/info` includes `sampling: true` in features
- [ ] Unit tests for sampling decision logic
- [ ] Documentation in `docs/user-guide/sampling.md`

Contributor guide

Open the contributing guide

Research direction

Start by tracing the HTTP router and UDP listener, then inspect startup configuration loading and the GET /api/info feature response. Implement the stated sampling behavior at both ingestion paths, including the level exemptions and context tags, and verify it with unit tests plus the documentation in docs/user-guide/sampling.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.