guibranco / guibranco/logstream-server
[FEATURE] Log sampling — configurable entry drop rate
- Dominant language
- PHP
- Stars
- 1
- Forks
- 0
- Avg merge
- 1m
- Merged PRs (30d)
- 4
Description
## Summary
Drop a configurable percentage of high-volume, low-priority log entries before they reach storage, reducing costs without requiring application code changes.
## Motivation
High-throughput applications (especially via UDP) can generate millions of `debug` entries daily. Sampling allows retaining statistical representativeness while dramatically reducing storage volume.
## Proposed configuration
```json
{
"sampling": [
{ "name": "sample-billing-debug", "app_key": "billing-api", "level": "debug", "rate": 0.10 },
{ "name": "sample-all-debug", "level": "debug", "rate": 0.05 },
{ "name": "sample-info", "level": "info", "rate": 0.50 }
]
}
```
`rate: 0.10` means accept 10% of matching entries at random. Entries not accepted are permanently discarded.
## Behaviour
- Applied **before** the ingestion queue — at the HTTP router and UDP listener
- `error` and `critical` levels are **never sampled** regardless of config
- Accepted entries get `_sampled: true` and `_sample_rate: 0.10` added to `context`
- Loaded from `SAMPLING_CONFIG=./sampling.json`
## Acceptance criteria
- [ ] Sampling config loaded from file at startup
- [ ] Correct percentage of entries accepted (statistically, over large N)
- [ ] `error` and `critical` entries always pass through regardless of config
- [ ] Applied at both HTTP and UDP ingestion paths
- [ ] Accepted entries tagged with `_sampled` and `_sample_rate` in context
- [ ] `GET /api/info` includes `sampling: true` in features
- [ ] Unit tests for sampling decision logic
- [ ] Documentation in `docs/user-guide/sampling.md`
Contributor guide
Research direction
Start by tracing the HTTP router and UDP listener, then inspect startup configuration loading and the GET /api/info feature response. Implement the stated sampling behavior at both ingestion paths, including the level exemptions and context tags, and verify it with unit tests plus the documentation in docs/user-guide/sampling.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100