eclipse-vertx / eclipse-vertx/vertx-sql-client

Connection Storm Prevention with Jitter in Connection Pool

Open
#1,495 6 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
912
Forks
212
Avg merge
17h 30m
Merged PRs (30d)
3

Description

## Problem
When running multiple instances of an application that uses PgPool, we observed a connection storm issue that leads to uneven load distribution across database readers.

### Scenario
- 10 application instances
- Each instance: 2x large with 8 verticles
- Database setup: Multiple read replicas behind PgPool
- Issue timing: Application startup

### Current Behavior
When multiple application instances start simultaneously:
1. All verticles attempt to establish database connections at nearly the same time
2. This creates a "connection storm" where most connections are directed to the same reader
3. Results in skewed connection distribution across available readers
4. Some readers become overloaded, while others are underutilized

### Example from Load Test
In our load test with 10 instances (8 verticles each):
- Most connections were established to Reader-1
- Created uneven load distribution
- Impacted application performance
- Reduced the effectiveness of having multiple readers

Attaching a screenshot of connections made on readers. One of the readers has 227 connections, and the other has 72.

![Image](https://github.com/user-attachments/assets/d4c10b29-79c0-489d-82cf-9f0edaf3579d)

## Proposed Solution
Introduce connection jitter to randomize connection timing:

1. Add `jitter` parameter to PoolOptions
2. When setting `maxLifetime`, apply random jitter within the specified range
3. This spreads out connection creation/renewal across a time window

### Implementation

#### Contribution

https://github.com/eclipse-vertx/vertx-sql-client/pull/1496

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.