microsoft / microsoft/mssql-python

Connection pool can briefly exceed max_size when pooling is disabled under load

Open
#746 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: connectivity-auth bug triage needed
Dominant language
Python
Stars
472
Forks
60
Avg merge
2d 11h
Merged PRs (30d)
36

Description

Describe the bug

The connection pool can transiently exceed its configured max_size when pooling is disabled (or the process begins shutting down) while new physical connections are being opened concurrently. The pool's internal reserved-capacity counter (_current_size) can drift below the true number of live connections, after which the pool opens more physical connections than max_size allows. It is transient and self-corrects as connections close.

Root cause: _current_size is a bare counter with no record of which reservation a given decrement belongs to. The failed-open cleanup runs if (_current_size > 0) --_current_size, while a pool disable/close runs _current_size = 0. If a reset lands between a thread reserving a slot and that same thread's open failing, the thread's decrement cancels a different thread's live reservation instead of its own.

Exception message: none. This is a silent _current_size accounting drift, no exception or stack trace.
To reproduce

This is a rare timing race, so it is not deterministically reproducible from a plain script. It requires a new connection to fail to open at the exact moment pooling is disabled or the process is shutting down. The interleave that produces it, with max_size = 2:

import mssql_python
from concurrent.futures import ThreadPoolExecutor

mssql_python.pooling(max_size=2)

# 1. Thread A calls connect(), reserves a slot -> _current_size = 1,
#    then starts opening a new physical connection (outside the pool lock).
# 2. Another thread disables pooling: closePools() sets _current_size = 0.
# 3. Thread B calls connect(), reserves the freed slot -> _current_size = 1.
# 4. Thread A's open fails (handle alloc / network error): its cleanup runs
#    --_current_size -> 0, cancelling B's slot instead of A's.
# 5. Thread B's open succeeds, but _current_size now reads 0.
# 6. Further connect() calls see room and open past max_size.

Observable symptom while the count is drifted: more live sessions than the configured cap.

SELECT COUNT(*) FROM sys.dm_exec_sessions WHERE login_name = 'yourapp';
-- transiently returns 3 with max_size = 2
Expected behavior

The number of live physical connections should never exceed max_size, regardless of a connection-open failure racing a pooling disable or process shutdown.

Further technical details

Python version: any
SQL Server version: any
Operating system: any. The race is in the cross-platform C++ pool, not platform specific.

Additional context

Area: mssql_python/pybind/connection/connection_pool.cpp, the Phase 3 failure catch in ConnectionPool::acquire, plus closePools() and ConnectionPool::close.

Regression status: pre-existing for the connect()-failure path. PR #678 moved Connection construction out of _mutex (required to fix the #671 deadlock), which widens the same race to also cover constructor failures such as ODBC env init, handle allocation, and OOM. Raised during review of #678.

Proposed fix: give the pool a generation counter. close()/closePools() bumps it under the lock, a reserver snapshots it at ++_current_size, and the failure cleanup only decrements if the generation still matches. This makes the decrement attributable and closes both the constructor and connect paths. Roughly 6 to 8 lines plus a regression test that forces the interleave.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read mssql_python/pybind/connection/connection_pool.cpp, focusing on ConnectionPool::acquire's Phase 3 failure catch, closePools(), and ConnectionPool::close. Trace the reservation and cleanup paths, then inspect how PR #678 changed Connection construction. Done means a regression test covers the failure/disable interleave and live physical connections never exceed max_size.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.