apache / apache/hertzbeat

[Feature] <Support pulling logs from Grafana Loki as log source>

Open
#4,090 2 comments 0 reactions 0 assignees View on GitHub
new feature
Dominant language
Java
Stars
7.4k
Forks
1.3k
Avg merge
1d 20h
Merged PRs (30d)
32

Description

### Feature Request

Support pulling logs from Grafana Loki as log source

### Is your feature request related to a problem? Please describe

Currently, HertzBeat's log integration module supports receiving logs via OTLP protocol, but lacks native integration with Grafana Loki as a log source. Many users have deployed Loki for log storage and want HertzBeat to be able to pull logs from Loki for unified monitoring and analysis.

### Describe the solution you'd like

Add native Loki log source integration to enable HertzBeat to pull logs from Loki:

### Describe alternatives you've considered

1. **Loki Log Pulling**: Configure Loki datasource URL and query parameters
2. **LogQL Support**: Support LogQL query language for filtering and querying logs
3. **Real-time Streaming**: Implement real-time log streaming from Loki using Loki's tail API
4. **Label Filtering**: Filter logs by Loki labels (app, env, namespace, etc.)
5. **Time Range Query**: Support configurable time range for log queries

### Additional context

### Benefits:
- **Unified Observability**: Combine metrics from HertzBeat with logs from Loki
- **Cost Effective**: Leverage existing Loki deployment
- **Powerful Queries**: Use LogQL for precise log filtering
- **Grafana Ecosystem**: Integrate with existing Grafana/Loki dashboards

### Technical References:
- Loki HTTP API: https://grafana.com/docs/loki/latest/api/
- LogQL Documentation: https://grafana.com/docs/loki/latest/logql/
- Loki Tail API: https://grafana.com/docs/loki/latest/api/#tail-logs

### Implementation Suggestions:
1. Add Loki client library dependency (e.g., `com.grafana:loki-client`)
2. Create `LokiLogSource` class implementing the log source interface
3. Implement LogQL query parser and executor
4. Add periodic polling or WebSocket streaming support
5. Convert Loki log entries to HertzBeat's internal log format

Contributor guide

Open the contributing guide

Research direction

Start with HertzBeat's log integration module and its existing OTLP log path, then compare the requested behavior with the Loki HTTP API and LogQL references. Done would require a defined Loki source, query and label handling, time-range behavior, streaming or polling approach, and conversion to HertzBeat's internal log format.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.