Always Encrypted modernisation/optimisation
- Dominant language
- C#
- Stars
- 989
- Forks
- 340
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 72
Description
This is a broad design document tracking ongoing modernization and performance optimization of SqlClient's Always Encrypted (AE) implementation. I'll keep it up to date as the work progresses.
I'm aiming to make use of the improvements in .NET 5.0's cryptography APIs to reduce memory allocations, and take a closer look to try to eliminate some questionable buffer-to-buffer copying on the hot path. Separately, I'm performing a more general cleanup: a secondary goal is improve code maintainability, ensure correctness with respect to the TDS specification, and simplify some of the implementation's internal architecture.
### Goals
1. Performance: use newer .NET cryptography oneshot APIs to reduce unnecessary allocations in hot paths; reduce unnecessary buffer copies.
2. Refactor: centralise duplicated encryption/decryption logic into reusable primitives.
3. Namespace organization: move internal AE types into a dedicated `AlwaysEncrypted` namespace to group them together, away from the root `Microsoft.Data.SqlClient` namespace.
4. Code hygiene: remove redundant parameters and internal type prefixes; introduce nullability annotations.
5. Cross-platform support: enable a column encryption provider to work on multiple platforms if its underlying mechanism is supported.
6. TDS specification compliance: verify that Always Encrypted implementation matches the [TDS specification](https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-tds/b46a581a-39de-4745-b076-ec4dbb7d13ec) for data types, structures, and protocols.
### Approach
This has been progressing from the public-facing API inward toward the core cryptographic algorithms.
#### Public API: cross-platform support
PR: #3014 - Cross-platform `SqlColumnEncryptionCertificateStoreProvider` support
* Remove Unix-specific guards from certificate store provider.
* Enables column encryption support on Linux and macOS, enabling cross-platform AE support in the core library.
#### TDS specification compliance: metadata alignment
PR: #4240 - Align reads of `CekMdVersion` and `EkValueCount` with TDS specification
* Change `cekMdVersion` from byte array to `ulong`, aligning with TDS specification.
* Change `tableSize` from `short` to `ushort` for correctness.
* Eliminate one allocation per metadata read.
#### Public API implementation: lower-allocation primitives, consolidation
PRs: #3554, #3612, #3660
Introduce reusable lower-allocation primitives for CEK encryption/decryption and refactored all three column encryption key store providers to use them. This eliminates duplicated code across providers, removes some unnecessary buffer-to-buffer copies and achieves 15-64% allocation reduction (depending upon the key store provider). This changeset also introduces nullability annotations to the touched files.
* Centralize CEK encryption/decryption serialization logic to eliminate duplication across providers.
* Create reusable lower-allocation CEK encryption/decryption primitives using .NET 5.0+ oneshot APIs and refactor `SqlColumnEncryptionCspProvider`, `SqlColumnEncryptionCngProvider` and `SqlColumnEncryptionCertificateStoreProvider` to use them.
* Remove 1300+ lines of duplicated encryption/decryption code and fixed resource disposal issue in CNG provider
* Introduce nullability annotations to all affected files.
#### Algorithm factory layer: design simplification, namespace & organization
PR: #4138 - Cleanup Always Encrypted cryptographic algorithm factories
* Move `SqlClientEncryptionAlgorithmFactoryList`, `SqlClientEncryptionAlgorithmFactory`, and `SqlAeadAes256CbcHmac256Factory` to `AlwaysEncrypted` namespace.
* Simplify factory list by replacing a statically-constructed `ConcurrentDictionary` with static `switch` block.
* Seal algorithm factory implementation class.
* Move `AlgorithmVersion` constant into algorithm class.
#### Encryption key generation and caching
PR: #4256 - Reduce allocations in Always Encrypted key handling, reorganise
This results in a modest reduction in memory usage for cold-cache scenarios - from 3136 B to 216 B.
* Fast-path cache lookup before lock acquisition in `SymmetricKeyCache`.
* Removed redundant algorithm name parameter from key constructor, and cache Unicode-encoded byte array of this.
* Replace three `SymmetricKey` wrapper allocations with direct byte arrays.
* Introduce nullability annotations to touched files.
### Future work
#### Performance improvements to `SqlAeadAes256CbcHmac256Algorithm`
Once #4256 is reviewed and merged, I'll move deeper and seek to optimize the cryptographic operations on the hot path. The updated cryptographic one-shots can make a repeat appearance, but we also perform a fair amount of copying between buffers which can be safely removed. I'll provide benchmarks to verify this.
#### Always Encrypted with enclave support
It's worth noting that there's an open issue to remove some of the Azure-adjacent dependencies. This issue is likely to require the movement of the Azure enclave attestation provider, and the public exposure of some of the enclave-specific types. I don't want to do that work under this issue, but I'll keep that requirement in mind so that it doesn't generate unnecessary problems for the type movement later.
There's a fair amount of room to simplify the enclave provider type hierarchy, enable nullability annotations and to remove redundant copies between buffers. I'll start work on this after the release with the `SqlAeadAes256CbcHmac256Algorithm` performance improvements has been published.
### Security considerations
All changes in this work maintain existing security guarantees:
* Column encryption key derivation logic remains unchanged.
* Key validation and key usage ordering preserved.
* CEK serialization format remains compatible with TDS specification.
* All provider implementations continue to serialize CEK encryption/decryption calls.
* There'll be no changes to the underlying AE algorithms; at this layer, the goal is simply to ensure that it's using modern cryptographic APIs for the same algorithm and to avoid unnecessary buffer allocations.
### Testing strategy
* Existing automated AE test suite validates correctness across all changes on multiple platforms.
* Manual validation with SSMS-created AE-enabled tables.
* Add unit testing of each layer as it's modernised.
* Benchmarks attached to subsequent PRs.
Contributor guide
Assessment
This issue has not been assessed yet.