fireproof-storage / fireproof-storage/fireproof

Add warning logs for high parallelism database access patterns

Open
#1,053 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
973
Forks
58
PR merge metrics
No merged PRs in 30d

Description

## Problem

When applications hit the database with too much parallelism, it can lead to performance degradation and potential issues. Currently, there's no visibility into when queue sizes are growing due to high concurrent access.

## Solution

Add logging to detect and warn about high parallelism patterns by monitoring:

1. **ApplyHeadQueue size** - logs warning when > 5 items queued
2. **WriteQueue size** - logs warning when > 10 items queued

## Implementation

Added logging in two key locations:

### ApplyHeadQueue Monitoring (`crdt-clock.ts`)
- Debug logs show queue size on every `int_applyHead` operation
- Warning logs when queue size exceeds 5 items
- Includes context about whether updates are local

### WriteQueue Monitoring (`write-queue.ts`)
- Debug logs show queue size and bulk task count on every operation
- Warning logs when write queue exceeds 10 items
- Helps identify when bulk operations are backing up

## Benefits

- **Visibility**: Developers can see when their access patterns cause queue buildup
- **Performance tuning**: Helps identify bottlenecks in high-concurrency scenarios
- **Debugging**: Easier to correlate performance issues with queue sizes
- **Stress testing**: Can monitor queue sizes during load testing

## Testing

The logging can be tested by creating concurrent database operations:

```typescript
// Multiple concurrent puts
const promises = Array.from({ length: 20 }, (_, i) =>
db.put({ id: i, data: `test-${i}` })
);

// Multiple concurrent bulk operations
const bulkPromises = Array.from({ length: 5 }, (_, i) => {
const docs = Array.from({ length: 10 }, (_, j) => ({ batch: i, item: j }));
return db.bulk(docs);
});

await Promise.all([...promises, ...bulkPromises]);
```

This should trigger warning logs when queue sizes exceed thresholds.

## Future Enhancements

- Make warning thresholds configurable
- Add metrics/telemetry for queue sizes
- Consider adaptive backpressure mechanisms
- Add guidance on optimal concurrency patterns

This helps developers identify and optimize high-parallelism access patterns before they become performance problems.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.