lablup / lablup/backend.ai

Add monitoring and metrics for batched logging

Open
#8,546 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Add comprehensive monitoring and metrics for the batched logging system to track performance and detect issues.

## Metrics to Add

### Queue Metrics

1. **Queue Depth**
- Gauge: Current number of items in queue
- Per log type (audit, error, event)
- Alert if approaching capacity
1. **Enqueue Rate**
- Counter: Total enqueue operations
- Histogram: Enqueue latency

### Batch Writer Metrics

1. **Batch Size**
- Histogram: Actual batch sizes written
- Track flush trigger (time vs threshold)
1. **Flush Operations**
- Counter: Total flush operations
- Histogram: Flush duration
- Counter: Failed flush attempts
1. **DB Write Performance**
- Histogram: Bulk insert duration
- Counter: Items written per flush

### Redis Backup Metrics

1. **Backup Operations**
- Counter: Successful/failed backup writes
- Histogram: Backup write latency
1. **Recovery Operations**
- Counter: Logs recovered on startup
- Histogram: Recovery duration

## Implementation

Use existing `ai.backend.common.metrics` infrastructure:

```python
class BatchLoggingMetrics:
queue_depth: Gauge
enqueue_total: Counter
enqueue_duration: Histogram
flush_total: Counter
flush_duration: Histogram
batch_size: Histogram
```

## Dashboards

- Add Grafana dashboard for batched logging
- Include queue depth, flush rate, batch size trends
- Alert configuration for queue capacity

## Testing

- Verify metrics collection
- Test metric accuracy under load

## Acceptance Criteria

- All metrics implemented and exposed
- Grafana dashboard created
- Alerts configured
- Documentation updated

JIRA Issue: BA-4238

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.