lablup / lablup/backend.ai

Design batched logging architecture and interfaces

Open
#8,540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Design the architecture and interfaces for the batched logging system before implementation.

## Tasks

1. **Define Core Interfaces**
- `LogQueueManager` - Manages memory queues for different log types
- `BatchWriter` - Handles periodic/threshold-based batch writes
- `LogBuffer` - Memory buffer with configurable capacity
1. **Define Configuration**
- Batch size threshold (default: 100 items)
- Flush interval (default: 5 seconds)
- Queue capacity limits
- Redis backup settings (per log type)
1. **Design Data Flow**
- Log ingestion path (non-blocking)
- Batch assembly logic
- Flush trigger conditions (time-based, size-based)
- Error handling and retry logic
1. **Integration Points**
- Repository layer changes (add bulk insert methods)
- Service layer changes (replace direct DB calls)
- Configuration management (unified config)

## Deliverables

- Architecture document or design diagrams
- Interface definitions (Protocol classes)
- Configuration schema
- Integration plan

## Acceptance Criteria

- Clear separation between log ingestion and DB writing
- Support for different log types with different policies
- Graceful handling of backpressure scenarios
- Design reviewed and approved

JIRA Issue: BA-4232

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.