Implement Batched Logging System with Queue
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Implement a batched logging system to reduce database load by queuing logs in memory and periodically batch-writing them to the database.
## Current Problem
- Audit logs, error logs, and event logs are written to DB immediately on each event
- Each log creation opens a new database transaction
- High DB load during peak logging periods
## Proposed Solution
**Hybrid Approach: Memory Queue + Batch Write + Redis Backup**
### Architecture
```
Log Source → Memory Queue (asyncio.Queue) → Batch Writer → DB
↓
Redis (for critical logs)
```
### Components
1. **Memory Queue (asyncio.Queue)**
- Fast, non-blocking log ingestion
- Minimal latency impact
1. **Batch Writer**
- Periodic flush (e.g., every 5 seconds)
- Threshold-based flush (e.g., 100 items)
- Uses BulkCreator pattern for efficient batch inserts
1. **Redis Backup (Optional)**
- For critical logs (audit logs)
- Provides persistence across process restarts
### Benefits
- **Performance**: Reduced DB transactions (bulk insert)
- **Responsiveness**: Non-blocking log operations
- **Reliability**: Optional Redis backup for critical logs
- **Scalability**: Handles high log volume efficiently
## Implementation Strategy
1. Design and implement core batched logging infrastructure
1. Migrate logs incrementally (audit → error → event)
1. Add Redis backup for critical logs
1. Monitor and optimize performance
## Success Criteria
- Reduced DB load for log writes (target: 90% reduction in transactions)
- No log data loss during normal operation
- Minimal latency impact (< 10ms for log calls)
JIRA Issue: BA-4231
Contributor guide
Assessment
This issue has not been assessed yet.