lablup / lablup/backend.ai

Add Redis backup for critical audit logs

Open
#8,545 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Add Redis persistence for audit logs to prevent data loss during process restarts or crashes.

## Implementation Details

### Redis Backup Writer

1. **Create** `RedisLogBackup` (`src/ai/backend/manager/logging/redis_backup.py`)
- Use existing `ValkeyStreamClient` infrastructure
- Write audit logs to Redis streams
- Async write to avoid blocking

```python
class RedisLogBackup:
async def backup_audit_log(self, log: AuditLogRow) -> None:
"""Write audit log to Redis stream for persistence."""
await self._client.xadd(
stream_key="audit_logs:backup",
fields={"data": log.to_json()},
)
```

1. **Recovery Mechanism**
- On startup, check Redis for pending logs
- Replay backed-up logs to DB
- Clear Redis after successful write

### Configuration

```python
@dataclass
class AuditLogBackupConfig:
redis_backup_enabled: bool = True
redis_stream_key: str = "audit_logs:backup"
recovery_on_startup: bool = True
```

### Integration

- Add to audit log batch writer
- Write to both queue and Redis
- Implement recovery on manager startup

## Testing

- Test Redis write operations
- Test recovery after simulated crash
- Test Redis unavailability handling

## Acceptance Criteria

- Audit logs persisted to Redis
- Recovery works correctly on startup
- Graceful handling of Redis failures
- All tests passing

JIRA Issue: BA-4237

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.