fireproof-storage / fireproof-storage/fireproof
Add warning logs for high parallelism database access patterns
- Dominant language
- TypeScript
- Stars
- 973
- Forks
- 58
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
When applications hit the database with too much parallelism, it can lead to performance degradation and potential issues. Currently, there's no visibility into when queue sizes are growing due to high concurrent access.
## Solution
Add logging to detect and warn about high parallelism patterns by monitoring:
1. **ApplyHeadQueue size** - logs warning when > 5 items queued
2. **WriteQueue size** - logs warning when > 10 items queued
## Implementation
Added logging in two key locations:
### ApplyHeadQueue Monitoring (`crdt-clock.ts`)
- Debug logs show queue size on every `int_applyHead` operation
- Warning logs when queue size exceeds 5 items
- Includes context about whether updates are local
### WriteQueue Monitoring (`write-queue.ts`)
- Debug logs show queue size and bulk task count on every operation
- Warning logs when write queue exceeds 10 items
- Helps identify when bulk operations are backing up
## Benefits
- **Visibility**: Developers can see when their access patterns cause queue buildup
- **Performance tuning**: Helps identify bottlenecks in high-concurrency scenarios
- **Debugging**: Easier to correlate performance issues with queue sizes
- **Stress testing**: Can monitor queue sizes during load testing
## Testing
The logging can be tested by creating concurrent database operations:
```typescript
// Multiple concurrent puts
const promises = Array.from({ length: 20 }, (_, i) =>
db.put({ id: i, data: `test-${i}` })
);
// Multiple concurrent bulk operations
const bulkPromises = Array.from({ length: 5 }, (_, i) => {
const docs = Array.from({ length: 10 }, (_, j) => ({ batch: i, item: j }));
return db.bulk(docs);
});
await Promise.all([...promises, ...bulkPromises]);
```
This should trigger warning logs when queue sizes exceed thresholds.
## Future Enhancements
- Make warning thresholds configurable
- Add metrics/telemetry for queue sizes
- Consider adaptive backpressure mechanisms
- Add guidance on optimal concurrency patterns
This helps developers identify and optimize high-parallelism access patterns before they become performance problems.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.