Go: Data race triggered by Encrypt/Decrypt (trips Go race detector)
- Dominant language
- Rust
- Stars
- 66
- Forks
- 30
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 1
Description
While running tests using the AWS Encryption SDK for Go, we are seeing failed tests due to Go's race detector.
The relevant bits of the stack trace are:
```
WARNING: DATA RACE
Write at 0x00c000010d38 by goroutine 20:
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:937 +0x30
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:202 +0x18
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:948 +0x2c
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:272 +0x48
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:290 +0x4c
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:361 +0x94
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:224 +0x10a4
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:246 +0xb58
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:353 +0xc24
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:281 +0x50
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/ESDK/ESDK.go:435 +0x110
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
/encryption.(*awsEncryptionClient).Encrypt()
....
/encryption.TestHandler_Encrypt_And_Decrypt.func3()
....
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
testing.(*T).Run.gowrap1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c
Previous write at 0x00c000010d38 by goroutine 19:
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:937 +0x30
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:202 +0x18
github.com/dafny-lang/DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
/home/runner/go/pkg/mod/github.com/dafny-lang/!dafny!runtime!go/v4@v4.11.2/dafny/dafnyFromDafny.go:948 +0x2c
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:272 +0x48
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:290 +0x4c
github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library@v0.3.0/Base64/Base64.go:361 +0x94
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:224 +0x10a4
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/Materials/Materials.go:246 +0xb58
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials_k()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:353 +0xc24
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/AwsCryptographyMaterialProvidersTypes.(*CompanionStruct_ICryptographicMaterialsManager_).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/AwsCryptographyMaterialProvidersTypes/AwsCryptographyMaterialProvidersTypes.go:3643 +0x50
github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/DefaultCMM.(*DefaultCMM).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl@v0.3.0/DefaultCMM/DefaultCMM.go:281 +0x50
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/EncryptDecryptHelpers.(*CompanionStruct_Default___).GetEncryptionMaterials()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/EncryptDecryptHelpers/EncryptDecryptHelpers.go:691 +0x144
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/AwsEncryptionSdkOperations.(*CompanionStruct_Default___).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/AwsEncryptionSdkOperations/AwsEncryptionSdkOperations.go:380 +0x1184
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/ESDK.(*ESDKClient).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/ESDK/ESDK.go:435 +0x110
github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated.(*Client).Encrypt()
/home/runner/go/pkg/mod/github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk@v0.3.0/awscryptographyencryptionsdksmithygenerated/api_client.go:38 +0x958
/encryption.(*awsEncryptionClient).Encrypt()
....
/encryption.TestHandler_DecryptAndVerify.func1()
...
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
testing.(*T).Run.gowrap1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c
Goroutine 20 (running) created at:
testing.(*T).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x6e0
/encryption.TestHandler_Encrypt_And_Decrypt()
....
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
testing.(*T).Run.gowrap1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c
Goroutine 19 (running) created at:
testing.(*T).Run()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x6e0
/encryption.TestHandler_DecryptAndVerify()
....
testing.tRunner()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1934 +0x164
testing.(*T).Run.gowrap1()
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.8.linux-arm64/src/testing/testing.go:1997 +0x3c
==================
--- FAIL: TestHandler_Encrypt_And_Decrypt (0.00s)
--- FAIL: TestHandler_Encrypt_And_Decrypt/Decrypts_ciphertext_produced_in_a_separate_Encrypt_call (0.05s)
testing.go:1617: race detected during execution of test
--- FAIL: TestHandler_Encrypt_And_Decrypt/Successfully_encrypts_and_decrypts (0.07s)
testing.go:1617: race detected during execution of test
--- FAIL: TestHandler_DecryptAndVerify (0.00s)
--- FAIL: TestHandler_DecryptAndVerify/Returns_plaintext_when_expected_context_matches (0.05s)
testing.go:1617: race detected during execution of test
--- FAIL: TestHandler_DecryptAndVerify/Empty_expected_context_behaves_like_Decrypt (0.05s)
testing.go:1617: race detected during execution of test
```
It appears that the race is actually within Dafny, but filing the issue here as it's affecting the encryption SDK.
I used Claude to help investigate the root cause, the full detailed analysis provides much more detail.
AI Root Cause Analysis
## Affected versions
- `github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk` v0.3.0
- `github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl` v0.3.0
- `github.com/aws/aws-cryptographic-material-providers-library/releases/go/smithy-dafny-standard-library` v0.3.0
- `github.com/dafny-lang/DafnyRuntimeGo/v4` v4.11.2
- Go 1.25.x
## Summary
The Go release of the AWS Encryption SDK is not goroutine-safe for concurrent use within a single process. Two goroutines that each call `Client.Encrypt` (or `Decrypt`) on independent `ESDKClient` instances — i.e. with no shared client state of their own — still race on a process-global `ArraySequence` inside the Dafny runtime. The Go `-race` detector flags this on every concurrent invocation.
## Reproducer
```go
package main
import (
"context"
"sync"
"testing"
mpl "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygenerated"
mpltypes "github.com/aws/aws-cryptographic-material-providers-library/releases/go/mpl/awscryptographymaterialproviderssmithygeneratedtypes"
esdkclient "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygenerated"
esdktypes "github.com/aws/aws-encryption-sdk/releases/go/encryption-sdk/awscryptographyencryptionsdksmithygeneratedtypes"
)
func TestConcurrentEncryptRaces(t *testing.T) {
ctx := context.Background()
var wg sync.WaitGroup
for i := 0; i < 2; i++ {
wg.Add(1)
go func() {
defer wg.Done()
matProv, _ := mpl.NewClient(mpltypes.MaterialProvidersConfig{})
keyring, _ := matProv.CreateRawAesKeyring(ctx, mpltypes.CreateRawAesKeyringInput{
KeyName: "k",
KeyNamespace: "ns",
WrappingKey: []byte("superSecureKeySecureKey32bytes32"),
WrappingAlg: mpltypes.AesWrappingAlgAlgAes256GcmIv12Tag16,
})
c, _ := esdkclient.NewClient(esdktypes.AwsEncryptionSdkConfig{})
_, _ = c.Encrypt(ctx, esdktypes.EncryptInput{
Plaintext: []byte("hello"),
EncryptionContext: map[string]string{"k": "v"},
Keyring: keyring,
})
}()
}
wg.Wait()
}
```
Run with `go test -race`. The race detector reports the issue reliably.
## Race trace (excerpt)
```
WARNING: DATA RACE
Write at 0x... by goroutine A:
DafnyRuntimeGo/v4/dafny.(*ArraySequence).IsString_set_()
dafny/dafnyFromDafny.go:937 +0x30
DafnyRuntimeGo/v4/dafny.(*CompanionStruct_Sequence_).SetString()
dafny/dafnyFromDafny.go:202 +0x18
DafnyRuntimeGo/v4/dafny.(*ArraySequence).SetString()
dafny/dafnyFromDafny.go:948 +0x2c
smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).FromIndicesToChars()
Base64/Base64.go:272 +0x48
smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).EncodeUnpadded()
Base64/Base64.go:290 +0x4c
smithy-dafny-standard-library/Base64.(*CompanionStruct_Default___).Encode()
Base64/Base64.go:361 +0x94
mpl/Materials.(*CompanionStruct_Default___).InitializeEncryptionMaterials.func2()
Materials/Materials.go:224 +0x10a4
...
encryption-sdk/ESDK.(*ESDKClient).Encrypt()
Previous write at 0x... by goroutine B:
[identical stack]
```
## Root cause
`Base64.FromIndicesToChars()` returns a process-global `ArraySequence` (the base64 alphabet). `Base64.Encode` then calls `SetString(true)` on it, which dispatches to `ArraySequence.IsString_set_` — an unsynchronized field write on the receiver (`dafnyFromDafny.go:937`). Because the alphabet sequence is shared, every concurrent `Encrypt`/`Decrypt` walks this code path and races on the same `isString` field.
The writes happen to be idempotent (always `true`), so on x86/ARM the field's final value is still correct in practice. But under the Go memory model, unsynchronized concurrent writes are undefined behavior, and `-race` flags them on every run, which makes the SDK unusable in any Go test suite that runs with `-race` (including the default for any project that values race-free CI).
## Expected
Concurrent calls to `Encrypt`/`Decrypt` from independent `ESDKClient` instances complete without a race detector warning.
## Suggested fix
Either:
1. Make `ArraySequence.IsString_set_` race-safe — e.g. back `isString` with `sync/atomic` or set it once via `sync.Once` since the value never transitions back.
2. Avoid mutating the shared alphabet sequence in `Base64.Encode`; treat the alphabet as an immutable value and produce new sequences for the encoded output.
Option 2 is preferable because it eliminates a class of similar latent races in the broader Dafny stdlib and matches the immutable-sequence semantics Dafny otherwise advertises.
## Workarounds
- Run encryption tests serially (`-parallel=1` or no `t.Parallel()`).
- Add a process-wide mutex around every SDK call. This serializes all encryption in the process, which is unacceptable for servers that issue per-request KMS-backed encryption.
Neither is satisfying for a server runtime that does concurrent encryption per request.
## Impact
Any Go service that calls `Client.Encrypt`/`Decrypt` from per-request goroutines (i.e. the typical pattern for encrypting PII on incoming HTTP requests) hits this race continuously. Production behavior is likely benign in practice because the racing write is idempotent, but the SDK cannot be cleanly validated under `go test -race`, and there is no upstream guidance on whether the SDK is intended to be goroutine-safe in this configuration.
Contributor guide
Research direction
Start by running TestHandler_Encrypt_And_Decrypt and TestHandler_DecryptAndVerify with Go's race detector. Trace the reported calls through EncryptDecryptHelpers, AwsEncryptionSdkOperations, ESDK, and the generated Dafny Base64 and Materials paths named in the stack trace. Done means the affected encryption and decryption tests pass under -race without data-race reports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cryptography, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100