MagicStack / MagicStack/asyncpg

SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical

Ouverte Adaptée aux débutants
#1,357 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Langage dominant
Python
Étoiles
8.1k
Forks
468
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

### Summary

`SCRAMAuthentication._generate_salted_password()` implements PBKDF2-HMAC-SHA256 as a Python-level loop. `hashlib.pbkdf2_hmac('sha256', ...)` computes the identical value in C and is **~27–31× faster**. Since this runs on the event loop on *every* new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).

### Where

[`asyncpg/protocol/scram.pyx`](https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/scram.pyx) (current master):

```python
ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = self._bytes_xor(u, ui.digest())
return u
```

That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with `dkLen = hLen` — which the stdlib already provides.

`_bytes_xor` is a Python generator over `zip()`, so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.

### Measurements

PostgreSQL's default `scram_iterations` is 4096.

| Environment | Python loop | `hashlib.pbkdf2_hmac` | Factor |
|---|---|---|---|
| Debian container, x86_64, CPython 3.11 | 28.6 ms | 0.915 ms | 31× |
| macOS 26, arm64, CPython 3.12 | 9.28 ms | 0.337 ms | 27× |

Output is bit-identical:

```python
import hashlib, hmac, time

def asyncpg_way(p, s, it):
ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
u = ui.digest()
for _ in range(it - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = bytes(x ^ y for x, y in zip(u, ui.digest()))
return u

p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)

for name, fn in (("loop", asyncpg_way),
("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
fn(p, s, iters)
t = time.perf_counter(); n = 0
while time.perf_counter() - t < 2.0:
fn(p, s, iters); n += 1
print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")
```

### Why it matters in practice

We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with `password_encryption = scram-sha-256`. `py-spy` attributed **~16 % of total event-loop CPU** to `hmac.py` with no application frame above it — the caller is invisible because `scram.pyx` is Cython-compiled, so only the Python `hmac.new` frames show up. It took a while to identify.

The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 **synchronously on the event loop** — roughly 7 seconds of blocked loop per minute.

Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.

### Suggested change

```python
cdef _generate_salted_password(self, str password, bytes salt, int iterations):
"""This follows the "Hi" algorithm specified in RFC5802"""
return hashlib.pbkdf2_hmac(
'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
)
```

Note the loop already hardcodes `hashlib.sha256` (while the first `hmac.new` uses `self.DIGEST`), so the function is SHA-256-only as it stands — no digest agility is lost by naming `'sha256'` explicitly. If `DIGEST` should ever become configurable, `pbkdf2_hmac` takes the algorithm name as its first argument, so the change doesn't stand in the way.

`_bytes_xor` would become unused unless it's referenced elsewhere.

Happy to open a PR if the direction looks right.

### Related

#378 (*Support using pre-hashed passwords*) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez dans asyncpg/protocol/scram.pyx, au niveau de SCRAMAuthentication._generate_salted_password(), et examinez la boucle PBKDF2 existante ainsi que l’helper _bytes_xor. Remplacez la dérivation au niveau Python par l’équivalent de la bibliothèque standard, puis vérifiez que sa sortie reste identique bit à bit pour les entrées SCRAM SHA-256 existantes et confirmez que l’helper inutilisé n’a aucune autre référence.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
postgresql, python
Domaine
authentication, backend, databases, performance
Type d'issue
Refactorisation
Difficulté
2/5
Temps estimé
1-3 heures
Activité
Active
Clarté
Clairement spécifiée
Accessibilité débutants
78/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.