MagicStack / MagicStack/asyncpg
SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
`SCRAMAuthentication._generate_salted_password()` implements PBKDF2-HMAC-SHA256 as a Python-level loop. `hashlib.pbkdf2_hmac('sha256', ...)` computes the identical value in C and is **~27–31× faster**. Since this runs on the event loop on *every* new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).
### Where
[`asyncpg/protocol/scram.pyx`](https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/scram.pyx) (current master):
```python
ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = self._bytes_xor(u, ui.digest())
return u
```
That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with `dkLen = hLen` — which the stdlib already provides.
`_bytes_xor` is a Python generator over `zip()`, so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.
### Measurements
PostgreSQL's default `scram_iterations` is 4096.
| Environment | Python loop | `hashlib.pbkdf2_hmac` | Factor |
|---|---|---|---|
| Debian container, x86_64, CPython 3.11 | 28.6 ms | 0.915 ms | 31× |
| macOS 26, arm64, CPython 3.12 | 9.28 ms | 0.337 ms | 27× |
Output is bit-identical:
```python
import hashlib, hmac, time
def asyncpg_way(p, s, it):
ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
u = ui.digest()
for _ in range(it - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = bytes(x ^ y for x, y in zip(u, ui.digest()))
return u
p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)
for name, fn in (("loop", asyncpg_way),
("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
fn(p, s, iters)
t = time.perf_counter(); n = 0
while time.perf_counter() - t < 2.0:
fn(p, s, iters); n += 1
print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")
```
### Why it matters in practice
We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with `password_encryption = scram-sha-256`. `py-spy` attributed **~16 % of total event-loop CPU** to `hmac.py` with no application frame above it — the caller is invisible because `scram.pyx` is Cython-compiled, so only the Python `hmac.new` frames show up. It took a while to identify.
The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 **synchronously on the event loop** — roughly 7 seconds of blocked loop per minute.
Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.
### Suggested change
```python
cdef _generate_salted_password(self, str password, bytes salt, int iterations):
"""This follows the "Hi" algorithm specified in RFC5802"""
return hashlib.pbkdf2_hmac(
'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
)
```
Note the loop already hardcodes `hashlib.sha256` (while the first `hmac.new` uses `self.DIGEST`), so the function is SHA-256-only as it stands — no digest agility is lost by naming `'sha256'` explicitly. If `DIGEST` should ever become configurable, `pbkdf2_hmac` takes the algorithm name as its first argument, so the change doesn't stand in the way.
`_bytes_xor` would become unused unless it's referenced elsewhere.
Happy to open a PR if the direction looks right.
### Related
#378 (*Support using pre-hashed passwords*) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in asyncpg/protocol/scram.pyx at SCRAMAuthentication._generate_salted_password() and review the existing PBKDF2 loop and _bytes_xor helper. Replace the Python-level derivation with the standard-library equivalent, then verify its output remains bit-identical for the existing SHA-256 SCRAM inputs and confirm the unused helper has no other references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- authentication, backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100