aws / aws/aws-lc

transient RNDR failure on aarch64 aborts the process (no retry, unlike rdrand)

Closed
#3,453 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Assembly
Stars
830
Forks
212
Avg merge
2d 22h
Merged PRs (30d)
61

Description

### Security issue notifications

n/a - availability bug, no security impact

### Problem:

On aarch64 CPUs with `FEAT_RNG`, a single failed `RNDR` read aborts the whole process.

`CRYPTO_rndr_multiple8` (crypto/fipsmodule/rand/asm/rndr-armv8.pl) does one mrs per 8 bytes and bails on the first failure. RNDR is allowed to fail transiently per the Arm ARM. The rdrand path retries 10 times (`CALL_RDRAND_WITH_RETRY`), the rndr path doesn't retry at all.

On failure `entropy_get_prediction_resistance` returns 0 and `rand_maybe_get_ctr_drbg_pred_resistance` in `rand.c calls abort()`. So any `RAND_bytes` caller can take down the process.

There is production issue reported in Istio community slack [here](https://istio.slack.com/archives/C0AL7LKQG7Q/p1779992994722599) with istio ztunnel (rustls -> aws-lc-rs, aws-lc-sys 0.33.0, code unchanged on main). On GKE n4a (Axion / Neoverse V2) nodes ztunnel crashloops with exit 133 (SIGTRAP), sometimes for hours until whatever node condition makes RNDR flaky clears up. Core dumps all show the same thing:

```
#0 abort ()
#1 rand_bytes_private.part ()
#2 aws_lc_0_33_0_RAND_bytes ()
#3 rustls SecureRandom::fill (also seen via EC_KEY_generate_key and X25519_keypair)
```

Core state rules out the other abort paths in rand_bytes_core: UBE generation numbers match, entropy_source is valid, jitter seeding worked (multiple successful reseeds earlier in the same process), `pred_resistance` buffer still zeroed. The same process had done successful `RNDR` reads seconds before, so this is intermittent hw failure, not a missing feature.

Ampere t2a (no `FEAT_RNG`) and x86 nodes never hit it, which matches the analysis. Graviton3/4 expose `FEAT_RNG` too so this isn't specific to Google hardware.

### Solution:

- retry loop in `CRYPTO_rndr_multiple8`, same idea as `CALL_RDRAND_WITH_RETRY`
- consider not aborting when the prediction resistance source fails after retries. `get_prediction_resistance == NULL` is already a supported config, so degrading to that for the failing call seems more reasonable than killing the process

* **Does this change any public APIs?** no
* **Which algorithm(s) will this impact?** `RAND_bytes` on aarch64 with `FEAT_RNG`

### Requirements / Acceptance Criteria:

a transient RNDR failure doesn't abort the process

* **RFC links:** n/a. RNDR spec: https://developer.arm.com/documentation/ddi0601/latest/AArch64-Registers/RNDR--Random-Number
* **Related Issues:** #2029 added RNDR support
* **Will the Usage Guide or other documentation need to be updated?** no
* **Testing:** existing rndr tests; failure injection would need have_hw_rng_aarch64_for_testing style hooks
* **Will this change trigger AWS LibCrypto Formal Verification changes?** don't think so
* **Should this change be fuzz tested?** no untrusted input

### Out of scope:
why Axion RNDR fails intermittently in the first place - that's between Google and Arm

Contributor guide

Open the contributing guide

Research direction

Start with crypto/fipsmodule/rand/asm/rndr-armv8.pl and compare CRYPTO_rndr_multiple8 with CALL_RDRAND_WITH_RETRY. Then trace entropy_get_prediction_resistance into rand_maybe_get_ctr_drbg_pred_resistance in rand.c and run the existing RNDR tests, using the mentioned testing hooks if adding failure coverage. Done means a transient RNDR failure no longer aborts a RAND_bytes caller.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl
Domain
cryptography
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.