cloudnative-pg / cloudnative-pg/plugin-barman-cloud
s3 Certificate validation problem
- Dominant language
- Go
- Stars
- 191
- Forks
- 72
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 21
Description
Hi,
I'm having some problems with getting SSL validation to work for our backup jobs to our on-prem s3 solution hosted on our NetApp ONTAP s3. We keep getting the following errors in our plugin-barman-cloud containers:
`
ERROR: Barman cloud WAL archiver exception: SSL validation failed for https:// [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate
`
We have signed our certificate using ACME and Let's Encrypt E6 CA. I'm not sure if this is where the problem originates from, since the E6 CA is an intermediate CA which in turn is signed by ISRG ROOT X2 CA. But I Can't see why this would be a problem since the ISRG ROOT X2 CA certificate exists in most lists of trusted CAs?
Anyway, I tried to use the endpointCA configuration option in the ObjectStore configuration to try to manually set the certificate chain for the CA as trusted. Our configuration looks like this:
```
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
name: s3-postgress
namespace: postgres
spec:
configuration:
data:
additionalCommandArgs:
- '--min-chunk-size=5MB'
- '--read-timeout=60'
- '-vv'
destinationPath: s3://s3-postgres/postgres/customer
endpointCA:
key: ca.crt
name: pg-s3-ca
endpointURL: https://
s3Credentials:
accessKeyId:
key: ACCESS_KEY_ID
name: pg-s3-creds
secretAccessKey:
key: ACCESS_SECRET_KEY
name: pg-s3-creds
wal:
compression: gzip
instanceSidecarConfiguration:
env:
- name: AWS_CA_BUNDLE
value: /projected/ca.crt
- name: REQUESTS_CA_BUNDLE
value: /projected/ca.crt
- name: WALG_S3_CA_CERT_FILE
value: /projected/ca.crt
- name: SSL_CERT_FILE
value: /projected/ca.crt
```
As you can see I have tried a lot of different options for trying to specify the trusted CA(s), but none of them have worked. I have mounted the ca certificate at /projected/ca.crt using the following configuration in my Cluster definition:
```
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: customer-db
namespace: postgres
spec:
...
projectedVolumeTemplate:
sources:
- secret:
items:
- key: ca.crt
path: ca.crt
name: pg-s3-ca
```
Which seems to mount the secret into the same path for the sidecar container for the barman plugin as well.
Just a list of things I have already tried:
- Not using the endpointCA configuration snippet
- Not injecting the ca.crt file manually and setting envvars to point to this file
- Running the commands locally from my postgres container
- Changing the name of the secret key passed to endpointCA to tls.crt (as its named in the code example here: [minio-sotre.yaml](https://github.com/cloudnative-pg/plugin-barman-cloud/blob/54e3562e44fa117c74fb15318f96acdde5bad6a9/docs/examples/minio-store.yaml) )
- Changing the name of the secret key passed to endpointCA to barman-ca.crt to match constant name specified in the source code here: [constants.go](https://github.com/cloudnative-pg/plugin-barman-cloud/blob/main/internal/cnpgi/metadata/constants.go)
- Using openssl to extract the certificate from the s3 endpoint and ensuring that it is valid
I'm running out of things to try and as far as I can see there isn't anything in the documentation or any open issues that I have missed. So could someone maybe point me in the right direction here?
As you can see above I enabled verbose logging, so I will add the logs from the plugin-barman-cloud container when it tries to archive the WAL in the submitted file.
[logs.txt](https://github.com/user-attachments/files/21186830/logs.txt)
Contributor guide
Assessment
This issue has not been assessed yet.