Self-host SMTP TLS to Microsoft 365 broken by DigiCert G1 root distrust (Alpine CA bundle); bwdata/ca-certificates silently ignores DER certs
- Dominant language
- C#
- Stars
- 20.1k
- Forks
- 1.8k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 75
Description
### Steps To Reproduce
1. Self-host Bitwarden 2026.7.0 (standard `bitwarden.sh` Docker deployment) with SMTP configured against Microsoft 365 / Exchange Online Protection (`.mail.protection.outlook.com`, port 25, `startTls=true`).
2. Attempt any action that sends mail (e.g. admin portal passwordless login).
### Expected Result
Email is sent; admin portal login link arrives.
### Actual Result
`MailKit.Security.SslHandshakeException` on every send:
```
The server's SSL certificate could not be validated for the following reasons:
• The server certificate has the following errors:
• unable to get certificate CRL
• An intermediate certificate has the following errors:
• unable to get certificate CRL
• The root certificate has the following errors:
• self-signed certificate in certificate chain
```
Additionally, on the admin portal the failed login POST redirects to `/login` (missing the `/admin` path base) which returns 404, masking the underlying SMTP error entirely (see "Related cosmetic bug" below).
### Root cause
Not a Bitwarden code bug, but a trust-store regression in the shipped images plus two hardening gaps that made it very hard to diagnose:
1. On **15 April 2026** Mozilla and Chrome removed the **DigiCert Global Root CA (G1)** from their trust stores (industry-wide G1 distrust). Alpine inherits Mozilla's bundle, so the Alpine-based images introduced in newer releases ship a CA bundle **without** this root (verified: 119 roots in the bundle, G1 absent).
2. Microsoft's EOP MX endpoints (`*.mail.protection.outlook.com`) **still serve a chain anchored to that distrusted G1 root** (leaf → `DigiCert SHA2 Secure Server CA` → `DigiCert Global Root CA`). See the mozilla dev-security-policy thread "Microsoft MX servers use certificate chaining to obsolete DigiCert Global Root CA" and Microsoft advisory MC1282565.
3. Result: every Alpine-image self-host instance relaying via M365 fails outbound SMTP TLS validation with `UntrustedRoot`. Reproduced outside Bitwarden entirely: a minimal .NET 10 `SslStream` client on `mcr.microsoft.com/dotnet/sdk:10.0-alpine` fails identically against the same relay; adding the G1 root to the trust store makes the same handshake succeed with `SslPolicyErrors.None` (revocation Online).
### Workaround (works, documented here for other affected users)
Place the DigiCert Global Root CA in **PEM** form into `bwdata/ca-certificates` and restart:
```bash
curl -s http://cacerts.digicert.com/DigiCertGlobalRootCA.crt \
| openssl x509 -inform DER -out ./bwdata/ca-certificates/digicert-global-root-ca.crt
./bitwarden.sh restart
```
Note the DER→PEM conversion is **required** — see gap 2 below.
### Suggested actions for the Bitwarden team
1. **Release notes / awareness**: when a base-image bump changes the shipped CA bundle (especially removals such as the April 2026 G1 distrust), call it out in release notes. This change silently broke outbound SMTP for M365 relays with no configuration change on the user's side.
2. **`bwdata/ca-certificates` silently ignores non-PEM files.** The .NET certificate-directory loader (`OpenSslCachedSystemStoreProvider`) reads PEM only. DigiCert (and other CAs) serve roots from their AIA URLs in DER, so the obvious `curl` of the cert into that directory produces a file that is silently skipped — no log line, no error. The previous root-entrypoint flow (`cp ... && update-ca-certificates`) tolerated this; the current non-root entrypoint performs no processing at all. Suggestion: at container start, attempt to parse each file in `/etc/bitwarden/ca-certificates` and log a clear warning for any file that yields no certificates (or auto-convert DER). This would have reduced a multi-hour diagnosis to a one-line log read.
3. **Docs**: the [Certificate Options](https://bitwarden.com/help/certificates/) page only covers inbound NGINX TLS. The outbound-SMTP trust mechanism (`bwdata/ca-certificates`, PEM requirement, which containers consume it) is undocumented.
4. **Related cosmetic bug**: when `LoginController.Index` throws (any unhandled mail exception), the admin portal ends up redirecting to `/login` without the `/admin` path base, producing a 404 that hides the real error. The 404 sent this investigation down a reverse-proxy rabbit hole before the SMTP exception was found in container logs.
### Environment
- Self-hosted 2026.7.0 (standard Docker / `bitwarden.sh` deployment)
- SMTP: `.mail.protection.outlook.com:25`, `startTls=true` (M365/EOP direct-send)
- Reproduced independently on `mcr.microsoft.com/dotnet/sdk:10.0-alpine` with a minimal `SslStream` client
### References
- Mozilla dev-security-policy: [Microsoft MX servers use certificate chaining to obsolete DigiCert Global Root CA](https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/kJ5qysLlo2I)
- Microsoft 365 advisory [MC1282565](https://mc.merill.net/message/MC1282565) (April 2026 industry-wide DigiCert Global Root CA (G1) distrust)
- Parallel downstream report: [home-assistant/operating-system#4775](https://github.com/home-assistant/operating-system/issues/4775)
- DigiCert: [G1 root removal advisory](https://docs.digicert.com/en/certcentral/order-and-manage-certificates/reports-and-advisories/review-the-g1-root-removal-advisory.html)
---
*Troubleshooting and issue write-up assisted by Claude Code (Anthropic). Root cause isolated by reproducing the failure on a clean `mcr.microsoft.com/dotnet/sdk:10.0-alpine` image with a minimal `SslStream` client, then bisecting trust-store configurations against the live EOP relay.*
Contributor guide
Research direction
Start by tracing container startup handling for /etc/bitwarden/ca-certificates and the referenced LoginController.Index redirect behavior. Review the Certificate Options documentation and the CA directory consumption described in the issue. Done means the relevant certificate handling or diagnostics and documentation cover PEM requirements and affected containers, with the admin path-base behavior addressed if included.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, docker
- Domain
- backend, documentation, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100