Some keybase.io servers returning incomplete TLS cert chain

Hi folks,

While calling the Keybase.io API, we encountered an issue in which sometimes the certificate chain that is being returned is missing one of the intermediates. The error which we get intermittently is:

`bad: Get "https://keybase.io/_/api/1.0/user/lookup.json?usernames=jefferai,hashicorp&fields=public_keys": x509: certificate signed by unknown authority`

Running `openssl s_client -host keybase.io -port 443 -prexit |head -n 20` repeatedly has different results -- one with a complete chain:

`Certificate chain
0 s:CN = keybase.io
i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
3 s:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services`

and one with an incomplete chain (the Sectigo cert is missing in the output below):

`---
Certificate chain
0 s:CN = keybase.io
i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
1 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
2 s:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
---`

Below is a minimal Golang repro for this issue, which, when run (with go 1..16.3) should fail with the error message described above:

`package main

import (
"fmt"
"net/http"
"net/url"
"strings"
"github.com/kr/pretty"
)

func main() {
for i := 1; i <= 10000; i++ {
keytest()
}
}

func keytest() string {
input := []string{"keybase:jefferai", "keybase:hashicorp"}
client := &http.Client{}
// client := cleanhttp.DefaultClient()
if client == nil {
return "unable to create an http client"
}
usernames := make([]string, 0, len(input))
u := fmt.Sprintf("https://keybase.io/_/api/1.0/user/lookup.json?usernames=%s&fields=public_keys", strings.Join(usernames, ","))
resp, err := client.Get(u)
if err != nil {
if ue, ok := err.(*url.Error); ok {
pretty.Print(ue.Err)
}
return err.Error()
}
defer resp.Body.Close()
return ""
}
`
Thanks!

PS: Sorry for the formatting in the go code -- I can't seem to get it to properly format on GitHub.

The issue names no repository file or test. Start by running the provided Go 1.16.3 reproducer and repeatedly comparing the keybase.io certificate output with the supplied openssl command. Done means establishing whether the intermittent incomplete chain is caused by the service or the client and documenting a verified resolution path.

Written by the indexing model from the issue text.