lablup / lablup/backend.ai

Migrate event log to batched logging system

Open
#8,544 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Migrate event log creation from direct DB writes to the batched logging system.

## Implementation Details

### Event Reporter Changes

1. **Update** `EventLogger` (`src/ai/backend/manager/event_dispatcher/reporters.py`)
- Replace direct DB write with queue enqueue
- Batch multiple events together

```python
class EventLogger(AbstractEventReporter):
def __init__(self, db: ExtendedAsyncSAEngine, queue_manager: LogQueueManager) -> None:
self._db = db
self._queue_manager = queue_manager

async def prepare_event_report(
self, event: AbstractEvent, arg: PrepareEventReportArgs
) -> None:
event_log = EventLogRow.from_event(event)
await self._queue_manager.enqueue_event_log(event_log)
```

1. **Add Batch Writer for Events**
- Create `EventLogBatchWriter`
- Handle bulk insert to `event_logs` table

### Configuration

- Batch size: 200 (higher than other logs, events are frequent)
- Flush interval: 10 seconds (can tolerate delay)
- Redis backup: disabled

## Testing

- Test event log enqueueing
- Test high-volume event scenarios
- Integration tests with event dispatcher

## Acceptance Criteria

- Event logs written in batches
- No event data loss
- Reduced DB load during high event volume
- All tests passing

JIRA Issue: BA-4236

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.