aws / aws/aws-iot-device-sdk-cpp

"Server Certificate Verification failed" with Custom Domain

Open
#213 4 comments 1 reaction 0 assignees View on GitHub
bug needs-triage p3
Dominant language
C++
Stars
124
Forks
108
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

Hi,

I wanted to report this issue we have been experiencing while trying to switch the IoT Core to custom domain.

Understood that V2 is the solution and should have resolved most issues, we do plan to gradually migrate to V2.

However the thing is we still do have a large number of legacy hardware (Ubuntu 18.04) deployed in the field that we need to support, which makes it a major risk & high cost for us to bring those to V2.

Therefore, we would really appreciate if anyone could provide some insights on the issues with the IoT Core custom domain work with V1, so we can save our legacy hardware.

# Issue
We are experiencing `Server Certificate Verification failed.` during the SSL handshake when connecting to our custom domain, which is also reproducible with the PubSub example.

However, the SDK works perfectly when simply connects to the ATS endpoint.

The only thing we swapped is the endpoint URL in the config file of the SDK, from ATS endpoint to our custom domain. Everything else is properly configured including the ACM SSL certs, VPC endpoint to the iot:data plane, etc.

# What we have tried
1. Tried both IoT CPP SDK V2 and IoT Python SDK to connect to the same custom domain: PubSub example works without problem for both SDKs, and MQTT test client can pub&sub normally
2. Connect to ATS endpoint using CPP SDK V1: works
3. Reinstall both OpenSSL 1.1.1 and 1.1.0g and recompile CPP SDK V1: issue persists
4. Test with `openssl s_client -connect :443 -CAfile certs/rootCA.crt` as well as `openssl s_client -connect :443 -CAfile certs/rootCA.crt`: Both showed verification `OK`.

However we did notice without specifying the SNI, the s_client always uses the amazonaws SSL cert over our custom domain cert. The results are the following:
```
subject=CN = *.iot.us-west-2.amazonaws.com

issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M01

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5502 bytes and written 442 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 5C520F92903515F3208ED38EF9710B67D08C79298A107727B0859CEF2620F9C0
Session-ID-ctx:
Master-Key: 83E247145B24F9654DDE25C4F5088AEBAF771C8FF8083D0FFBB7D7883ED9331480BE2729531008AEC6800E66DAD6DD0B
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1706304653
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: yes
```
5. Tried both the oldest Domain security policy (IoTSecurityPolicy_TLS12_1_0_2015_01) on AWS console as well as the latest for the custom domain: issue persists
6. Following the [IoT server auth doc](https://docs.aws.amazon.com/iot/latest/developerguide/server-authentication.html), added other cross-signed CA (G2-RootCA1.pem & SFSRootCAG2.pem) to the trusted store of the Linux environment of the hardware: issue persists

Detailed logs are (The `Error resolving hostname: -5` and `SSL Error Code: 2` are fine because they also exist when we connect to the ATS endpoint which works. However the `SSL Error Code: 1` seems to be the cause, which maps to `SSL_ERROR_SSL` in our environment):
```
./pub-sub-sample
[DEBUG] Thu Jan 25 18:57:38 2024
:450 [Thread Task] [548328968208] Creating Thread Outbound Action Processing!!
[DEBUG] Thu Jan 25 18:57:38 2024
:451 [Thread Task] [548328968208] Creating Thread TLS Read Action Runner!!
[DEBUG] Thu Jan 25 18:57:38 2024
:452 [Thread Task] [548328968208] Creating Thread MQTT Keep alive Action!!
[TRACE] Thu Jan 25 18:57:38 2024
:452 [Network Read] [548313076176] [PerformAction:L128] : Network Read Thread, TLS Status : 0
[DEBUG] Thu Jan 25 18:57:38 2024
:452 [OpenSSL Wrapper] [548328968208] [LoadCerts:L369] : Root CA : /home/pi/aws-iot-device-sdk-cpp/build/bin/certs/rootCA.pem
[DEBUG] Thu Jan 25 18:57:38 2024
:453 [OpenSSL Wrapper] [548328968208] [LoadCerts:L377] : Device crt : /home/pi/aws-iot-device-sdk-cpp/build/bin/certs/cert.pem
[DEBUG] Thu Jan 25 18:57:38 2024
:453 [OpenSSL Wrapper] [548328968208] [LoadCerts:L382] : Device privkey : /home/pi/aws-iot-device-sdk-cpp/build/bin/certs/privkey.pem
[ERROR] Thu Jan 25 18:57:38 2024
:482 [OpenSSL Wrapper] [548328968208] [ConnectTCPSocket:L244] : Error resolving hostname: -5
[ERROR] Thu Jan 25 18:57:38 2024
:483 [OpenSSL Wrapper] [548328968208] [PerformSSLConnect:L408] : TCP Connection error
[INFO] Thu Jan 25 18:57:38 2024
:518 [OpenSSL Wrapper] [548328968208] [ConnectTCPSocket:L271] : resolved to xxx.xxx.xxx.xxx
[INFO] Thu Jan 25 18:57:38 2024
:546 [OpenSSL Wrapper] [548328968208] [AttemptConnect:L336] : SSL Error Code: 2
[INFO] Thu Jan 25 18:57:38 2024
:601 [OpenSSL Wrapper] [548328968208] [AttemptConnect:L336] : SSL Error Code: 2
[INFO] Thu Jan 25 18:57:38 2024
:606 [OpenSSL Wrapper] [548328968208] [AttemptConnect:L336] : SSL Error Code: 1
[INFO] Thu Jan 25 18:57:38 2024
:607 [OpenSSL Wrapper] [548328968208] [PerformSSLConnect:L424] : AttemptConnect Code: -405
[ERROR] Thu Jan 25 18:57:38 2024
:607 [OpenSSL Wrapper] [548328968208] [PerformSSLConnect:L426] : Server Certificate Verification failed.
```

# Summary
Sorry if I didn't make this concise enough but at this point seemingly we have run out of what to try and we are hammering a wall right now... All points to be an SDK V1 specific issue that may be lack of support of custom domain (which would be surprising if it is)?

Could you suggest anything else to check or try?

### Expected Behavior

After switching the endpoint in the config file from ATS to custom domain, it should connect to IoT Core data plane with no issue

### Current Behavior

SDK gives `Server Certificate Verification failed.` which seems to be related to SSL

### Reproduction Steps

1. Configure a custom domain on IoT Core settings with SSL cert from ACM
2. Connect VPC endpoint to iot:data plane
3. Put an Network Load Balancer across each AZ in the VPC
4. Use a global accelerator in front of the NLB to get a static IP
5. Link the custom domain (same as configured in the IoT Core settings) to the global accelerator using A record in the DNS.
6. (optional) Put the same domain in the PubSub example in the V2 SDK, it connects and PubSub as expected.
7. (optional) Put the ATS endpoint in the config file and run the V1 PubSub, it connects and PubSub as expected.
8. Put the custom domain in the config file `endpoint` property and run the PubSub example, and it gets `Server Certificate Verification failed.`

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### SDK version used

Compiled from the latest master branch

### Environment details (OS name and version, etc.)

Ubuntu 18.04, OpenSSL 1.1.0g

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.