guibranco / guibranco/logstream-server

[FEATURE] Log forwarding to external systems (Loki, Seq, webhook)

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
✨ feature enhancement good first issue hacktoberfest help wanted integration
Dominant language
PHP
Stars
1
Forks
0
Avg merge
1m
Merged PRs (30d)
4

Description

## Summary

Forward log entries to external systems as they arrive, enabling logstream-server to act as an aggregation and fan-out layer in front of existing observability tooling.

## Motivation

Many teams already have Grafana/Loki, Seq, or custom log pipelines. Rather than replacing them, logstream-server should be able to forward entries so both can coexist, or so logstream-server provides the lightweight UDP-capable ingest layer while a heavier system handles long-term storage.

## Proposed forwarders

### Loki

```json
{
"type": "loki",
"url": "https://loki.example.com/loki/api/v1/push",
"labels": { "job": "logstream", "env": "production" },
"app_key": "billing-api"
}
```

### Seq

```json
{
"type": "seq",
"url": "https://seq.example.com",
"api_key": "your-seq-api-key"
}
```

### Generic webhook

```json
{
"type": "webhook",
"url": "https://example.com/logs",
"headers": { "X-My-Key": "secret" },
"method": "POST",
"format": "json_array"
}
```

## Configuration

Loaded from `FORWARD_CONFIG=./forwarding.json`. Forwarders can be scoped by `app_key`, `level`, or applied globally.

## Implementation notes

- Forwarding is **async and non-blocking** — entries queued and forwarded in batches on a timer
- Failed forwards retried up to 3 times with exponential backoff
- Local storage always succeeds regardless of forward outcome
- `GET /api/forwarders/status` shows queue depth and recent error rate per forwarder

## Acceptance criteria

- [ ] Loki forwarder sends entries in the Loki push API format
- [ ] Seq forwarder sends entries as CLEF JSON
- [ ] Generic webhook forwarder sends configurable JSON payload
- [ ] Forwarding is async — does not block the event loop
- [ ] Failed forwards retried with exponential backoff (max 3 attempts)
- [ ] Local storage always succeeds regardless of forward outcome
- [ ] `GET /api/forwarders/status` returns per-forwarder queue depth and error rate
- [ ] Unit tests for each forwarder's serialisation logic
- [ ] Documentation in `docs/user-guide/forwarding.md`

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing ingestion and local-storage flow, then inspect how FORWARD_CONFIG is loaded; the issue names no source files or tests. Use the listed Loki, Seq, and webhook payloads as serialization targets. Done means asynchronous forwarding, retries, status reporting, unit tests, and docs/user-guide/forwarding.md satisfy every acceptance criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
api, backend, documentation, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.