lablup / lablup/backend.ai

Migrate audit log to batched logging system

Open
#8,542 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Objective

Migrate audit log creation from direct DB writes to the batched logging system.

## Implementation Details

### Repository Changes

1. **Update** `AuditLogDBSource`
- Add `bulk_create()` method using `BulkCreator`
- Keep existing `create()` for backward compatibility

```python
async def bulk_create(
self,
creators: Sequence[Creator[AuditLogRow]]
) -> list[AuditLogData]:
async with self._db.begin_session() as db_sess:
result = await execute_bulk_creator(db_sess, BulkCreator(specs=[c.spec for c in creators]))
return [row.to_dataclass() for row in result.rows]
```

1. **Update** `AuditLogRepository`
- Route `create()` calls to queue manager
- Add `flush()` method for manual flush operations

### Service Changes

1. **Update** `AuditLogService`
- Change `create()` to enqueue instead of direct DB write
- Return immediately without waiting for DB write

### Integration

- Wire LogQueueManager into AuditLogService
- Configure batch writer for audit logs
- Set Redis backup enabled for audit logs (critical)

## Testing

- Test audit log enqueueing
- Test batch creation with multiple entries
- Test flush on interval and threshold
- Integration tests with real DB

## Migration Strategy

- Feature flag for gradual rollout
- Monitor queue depth and flush latency
- Rollback plan if issues detected

## Acceptance Criteria

- Audit logs written in batches
- No functional regression
- Reduced DB transaction count (target: 90%)
- All tests passing

JIRA Issue: BA-4234

Contributor guide

Open the contributing guide

Research direction

Start by locating AuditLogDBSource, AuditLogRepository, AuditLogService, BulkCreator, and LogQueueManager, then trace the existing audit-log creation path. Review the queue and batch-writer configuration, including Redis backup and the feature flag. Done means batched writes, enqueue and flush behavior, integration coverage, and the stated transaction-reduction target without regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend, databases, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.