MagicStack / MagicStack/asyncpg
SCRAM: _generate_salted_password reimplements PBKDF2 in Python; hashlib.pbkdf2_hmac is ~30x faster and bit-identical
Nadie ha tomado este issue todavía.
- Lenguaje dominante
- Python
- Estrellas
- 8.1k
- Forks
- 468
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
### Summary
`SCRAMAuthentication._generate_salted_password()` implements PBKDF2-HMAC-SHA256 as a Python-level loop. `hashlib.pbkdf2_hmac('sha256', ...)` computes the identical value in C and is **~27–31× faster**. Since this runs on the event loop on *every* new connection, it shows up as a measurable stall for workloads that open connections often (pool overflow, bursty traffic, short-lived tasks).
### Where
[`asyncpg/protocol/scram.pyx`](https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/scram.pyx) (current master):
```python
ui = hmac.new(p, s + b'\x00\x00\x00\x01', self.DIGEST)
u = ui.digest()
for x in range(iterations - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = self._bytes_xor(u, ui.digest())
return u
```
That is exactly the "Hi" function from RFC 5802, i.e. PBKDF2-HMAC-SHA256 with `dkLen = hLen` — which the stdlib already provides.
`_bytes_xor` is a Python generator over `zip()`, so each of the 4095 iterations allocates two digests and XORs 32 bytes one byte at a time.
### Measurements
PostgreSQL's default `scram_iterations` is 4096.
| Environment | Python loop | `hashlib.pbkdf2_hmac` | Factor |
|---|---|---|---|
| Debian container, x86_64, CPython 3.11 | 28.6 ms | 0.915 ms | 31× |
| macOS 26, arm64, CPython 3.12 | 9.28 ms | 0.337 ms | 27× |
Output is bit-identical:
```python
import hashlib, hmac, time
def asyncpg_way(p, s, it):
ui = hmac.new(p, s + b'\x00\x00\x00\x01', hashlib.sha256)
u = ui.digest()
for _ in range(it - 1):
ui = hmac.new(p, ui.digest(), hashlib.sha256)
u = bytes(x ^ y for x, y in zip(u, ui.digest()))
return u
p, s, iters = b"correct horse battery staple", b"0123456789abcdef", 4096
assert asyncpg_way(p, s, iters) == hashlib.pbkdf2_hmac("sha256", p, s, iters)
for name, fn in (("loop", asyncpg_way),
("pbkdf2_hmac", lambda p, s, i: hashlib.pbkdf2_hmac("sha256", p, s, i))):
fn(p, s, iters)
t = time.perf_counter(); n = 0
while time.perf_counter() - t < 2.0:
fn(p, s, iters); n += 1
print(f"{name:12} {(time.perf_counter()-t)/n*1000:7.3f} ms")
```
### Why it matters in practice
We hit this while profiling event-loop stalls in a FastAPI/SQLAlchemy service with `password_encryption = scram-sha-256`. `py-spy` attributed **~16 % of total event-loop CPU** to `hmac.py` with no application frame above it — the caller is invisible because `scram.pyx` is Cython-compiled, so only the Python `hmac.new` frames show up. It took a while to identify.
The trigger was connection churn: our SQLAlchemy pool was discarding overflow connections, so ~255 connections/minute were being established, each paying ~28.6 ms of PBKDF2 **synchronously on the event loop** — roughly 7 seconds of blocked loop per minute.
Fixing the churn on our side was the main remedy, and I'm not suggesting asyncpg is responsible for that. But a connection establishment costing 28.6 ms of CPU rather than 0.9 ms makes any such situation ~30× worse than it needs to be, and it's on the loop.
### Suggested change
```python
cdef _generate_salted_password(self, str password, bytes salt, int iterations):
"""This follows the "Hi" algorithm specified in RFC5802"""
return hashlib.pbkdf2_hmac(
'sha256', password.encode('utf8'), base64.b64decode(salt), iterations
)
```
Note the loop already hardcodes `hashlib.sha256` (while the first `hmac.new` uses `self.DIGEST`), so the function is SHA-256-only as it stands — no digest agility is lost by naming `'sha256'` explicitly. If `DIGEST` should ever become configurable, `pbkdf2_hmac` takes the algorithm name as its first argument, so the change doesn't stand in the way.
`_bytes_xor` would become unused unless it's referenced elsewhere.
Happy to open a PR if the direction looks right.
### Related
#378 (*Support using pre-hashed passwords*) would sidestep the derivation entirely, which is a broader change; this one is a drop-in replacement with identical output.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Línea de trabajo
Comienza en asyncpg/protocol/scram.pyx, en SCRAMAuthentication._generate_salted_password(), y revisa el bucle PBKDF2 existente y el helper _bytes_xor. Sustituye la derivación a nivel de Python por el equivalente de la biblioteca estándar; después, verifica que su salida siga siendo bit a bit idéntica para las entradas SCRAM existentes con SHA-256 y confirma que el helper no utilizado no tiene otras referencias.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- postgresql, python
- Área
- authentication, backend, databases, performance
- Tipo de issue
- Refactorización
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Estado de actividad
- Activo
- Claridad
- Bien especificado
- Aptitud para principiantes
- 78/100