Add monitoring and metrics for batched logging
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Objective
Add comprehensive monitoring and metrics for the batched logging system to track performance and detect issues.
## Metrics to Add
### Queue Metrics
1. **Queue Depth**
- Gauge: Current number of items in queue
- Per log type (audit, error, event)
- Alert if approaching capacity
1. **Enqueue Rate**
- Counter: Total enqueue operations
- Histogram: Enqueue latency
### Batch Writer Metrics
1. **Batch Size**
- Histogram: Actual batch sizes written
- Track flush trigger (time vs threshold)
1. **Flush Operations**
- Counter: Total flush operations
- Histogram: Flush duration
- Counter: Failed flush attempts
1. **DB Write Performance**
- Histogram: Bulk insert duration
- Counter: Items written per flush
### Redis Backup Metrics
1. **Backup Operations**
- Counter: Successful/failed backup writes
- Histogram: Backup write latency
1. **Recovery Operations**
- Counter: Logs recovered on startup
- Histogram: Recovery duration
## Implementation
Use existing `ai.backend.common.metrics` infrastructure:
```python
class BatchLoggingMetrics:
queue_depth: Gauge
enqueue_total: Counter
enqueue_duration: Histogram
flush_total: Counter
flush_duration: Histogram
batch_size: Histogram
```
## Dashboards
- Add Grafana dashboard for batched logging
- Include queue depth, flush rate, batch size trends
- Alert configuration for queue capacity
## Testing
- Verify metrics collection
- Test metric accuracy under load
## Acceptance Criteria
- All metrics implemented and exposed
- Grafana dashboard created
- Alerts configured
- Documentation updated
JIRA Issue: BA-4238
Contributor guide
Assessment
This issue has not been assessed yet.