lablup / lablup/backend.ai

Implement Batched Logging System with Queue

Open
#8,539 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Implement a batched logging system to reduce database load by queuing logs in memory and periodically batch-writing them to the database.

## Current Problem

- Audit logs, error logs, and event logs are written to DB immediately on each event
- Each log creation opens a new database transaction
- High DB load during peak logging periods

## Proposed Solution

**Hybrid Approach: Memory Queue + Batch Write + Redis Backup**

### Architecture

```
Log Source → Memory Queue (asyncio.Queue) → Batch Writer → DB

Redis (for critical logs)
```

### Components

1. **Memory Queue (asyncio.Queue)**
- Fast, non-blocking log ingestion
- Minimal latency impact
1. **Batch Writer**
- Periodic flush (e.g., every 5 seconds)
- Threshold-based flush (e.g., 100 items)
- Uses BulkCreator pattern for efficient batch inserts
1. **Redis Backup (Optional)**
- For critical logs (audit logs)
- Provides persistence across process restarts

### Benefits

- **Performance**: Reduced DB transactions (bulk insert)
- **Responsiveness**: Non-blocking log operations
- **Reliability**: Optional Redis backup for critical logs
- **Scalability**: Handles high log volume efficiently

## Implementation Strategy

1. Design and implement core batched logging infrastructure
1. Migrate logs incrementally (audit → error → event)
1. Add Redis backup for critical logs
1. Monitor and optimize performance

## Success Criteria

- Reduced DB load for log writes (target: 90% reduction in transactions)
- No log data loss during normal operation
- Minimal latency impact (< 10ms for log calls)

JIRA Issue: BA-4231

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.