MagicStack / MagicStack/asyncpg
SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical
Personne n'a encore pris cette issue.
- Langage dominant
- Python
- Étoiles
- 8.1k
- Forks
- 468
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
### Summary
`SCRAMAuthentication._generate_salted_password()` implements PBKDF2-HMAC-SHA256 as a Python-level loop. `hashlib.pbkdf2_hmac('sha256', ...)` computes the identical value in C and is **~27–31× faster**. Since this runs on the event loop on *every* new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).
### Where
[`asyncpg/protocol/scram.pyx`](https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/scram.pyx) (current master):
```python
ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = self._bytes_xor(u, ui.digest())
return u
```
That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with `dkLen = hLen` — which the stdlib already provides.
`_bytes_xor` is a Python generator over `zip()`, so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.
### Measurements
PostgreSQL's default `scram_iterations` is 4096.
| Environment | Python loop | `hashlib.pbkdf2_hmac` | Factor |
|---|---|---|---|
| Debian container, x86_64, CPython 3.11 | 28.6 ms | 0.915 ms | 31× |
| macOS 26, arm64, CPython 3.12 | 9.28 ms | 0.337 ms | 27× |
Output is bit-identical:
```python
import hashlib, hmac, time
def asyncpg_way(p, s, it):
ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
u = ui.digest()
for _ in range(it - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = bytes(x ^ y for x, y in zip(u, ui.digest()))
return u
p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)
for name, fn in (("loop", asyncpg_way),
("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
fn(p, s, iters)
t = time.perf_counter(); n = 0
while time.perf_counter() - t < 2.0:
fn(p, s, iters); n += 1
print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")
```
### Why it matters in practice
We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with `password_encryption = scram-sha-256`. `py-spy` attributed **~16 % of total event-loop CPU** to `hmac.py` with no application frame above it — the caller is invisible because `scram.pyx` is Cython-compiled, so only the Python `hmac.new` frames show up. It took a while to identify.
The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 **synchronously on the event loop** — roughly 7 seconds of blocked loop per minute.
Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.
### Suggested change
```python
cdef _generate_salted_password(self, str password, bytes salt, int iterations):
"""This follows the "Hi" algorithm specified in RFC5802"""
return hashlib.pbkdf2_hmac(
'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
)
```
Note the loop already hardcodes `hashlib.sha256` (while the first `hmac.new` uses `self.DIGEST`), so the function is SHA-256-only as it stands — no digest agility is lost by naming `'sha256'` explicitly. If `DIGEST` should ever become configurable, `pbkdf2_hmac` takes the algorithm name as its first argument, so the change doesn't stand in the way.
`_bytes_xor` would become unused unless it's referenced elsewhere.
Happy to open a PR if the direction looks right.
### Related
#378 (*Support using pre-hashed passwords*) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Piste de recherche
Commencez dans asyncpg/protocol/scram.pyx, au niveau de SCRAMAuthentication._generate_salted_password(), et examinez la boucle PBKDF2 existante ainsi que l’helper _bytes_xor. Remplacez la dérivation au niveau Python par l’équivalent de la bibliothèque standard, puis vérifiez que sa sortie reste identique bit à bit pour les entrées SCRAM SHA-256 existantes et confirmez que l’helper inutilisé n’a aucune autre référence.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- postgresql, python
- Domaine
- authentication, backend, databases, performance
- Type d'issue
- Refactorisation
- Difficulté
- 2/5
- Temps estimé
- 1-3 heures
- Activité
- Active
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 78/100