MagicStack / MagicStack/asyncpg

SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical

Open Beginner friendly
#1,357 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

### Summary

`SCRAMAuthentication._generate_salted_password()` implements PBKDF2-HMAC-SHA256 as a Python-level loop. `hashlib.pbkdf2_hmac('sha256', ...)` computes the identical value in C and is **~27–31× faster**. Since this runs on the event loop on *every* new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).

### Where

[`asyncpg/protocol/scram.pyx`](https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/scram.pyx) (current master):

```python
ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = self._bytes_xor(u, ui.digest())
return u
```

That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with `dkLen = hLen` — which the stdlib already provides.

`_bytes_xor` is a Python generator over `zip()`, so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.

### Measurements

PostgreSQL's default `scram_iterations` is 4096.

| Environment | Python loop | `hashlib.pbkdf2_hmac` | Factor |
|---|---|---|---|
| Debian container, x86_64, CPython 3.11 | 28.6 ms | 0.915 ms | 31× |
| macOS 26, arm64, CPython 3.12 | 9.28 ms | 0.337 ms | 27× |

Output is bit-identical:

```python
import hashlib, hmac, time

def asyncpg_way(p, s, it):
ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
u = ui.digest()
for _ in range(it - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = bytes(x ^ y for x, y in zip(u, ui.digest()))
return u

p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)

for name, fn in (("loop", asyncpg_way),
("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
fn(p, s, iters)
t = time.perf_counter(); n = 0
while time.perf_counter() - t < 2.0:
fn(p, s, iters); n += 1
print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")
```

### Why it matters in practice

We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with `password_encryption = scram-sha-256`. `py-spy` attributed **~16 % of total event-loop CPU** to `hmac.py` with no application frame above it — the caller is invisible because `scram.pyx` is Cython-compiled, so only the Python `hmac.new` frames show up. It took a while to identify.

The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 **synchronously on the event loop** — roughly 7 seconds of blocked loop per minute.

Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.

### Suggested change

```python
cdef _generate_salted_password(self, str password, bytes salt, int iterations):
"""This follows the "Hi" algorithm specified in RFC5802"""
return hashlib.pbkdf2_hmac(
'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
)
```

Note the loop already hardcodes `hashlib.sha256` (while the first `hmac.new` uses `self.DIGEST`), so the function is SHA-256-only as it stands — no digest agility is lost by naming `'sha256'` explicitly. If `DIGEST` should ever become configurable, `pbkdf2_hmac` takes the algorithm name as its first argument, so the change doesn't stand in the way.

`_bytes_xor` would become unused unless it's referenced elsewhere.

Happy to open a PR if the direction looks right.

### Related

#378 (*Support using pre-hashed passwords*) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in asyncpg/protocol/scram.pyx at SCRAMAuthentication._generate_salted_password() and review the existing PBKDF2 loop and _bytes_xor helper. Replace the Python-level derivation with the standard-library equivalent, then verify its output remains bit-identical for the existing SHA-256 SCRAM inputs and confirm the unused helper has no other references.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
authentication, backend, databases, performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.