cloudnative-pg / cloudnative-pg/plugin-barman-cloud

s3 Certificate validation problem

Offen
#446 2 Kommentare 3 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
191
Forks
72
Ø Merge
2 T. 21 Std.
Gemergte PRs (30 T.)
21

Beschreibung

Hi,

I'm having some problems with getting SSL validation to work for our backup jobs to our on-prem s3 solution hosted on our NetApp ONTAP s3. We keep getting the following errors in our plugin-barman-cloud containers:

`
ERROR: Barman cloud WAL archiver exception: SSL validation failed for https:// [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate
`

We have signed our certificate using ACME and Let's Encrypt E6 CA. I'm not sure if this is where the problem originates from, since the E6 CA is an intermediate CA which in turn is signed by ISRG ROOT X2 CA. But I Can't see why this would be a problem since the ISRG ROOT X2 CA certificate exists in most lists of trusted CAs?

Anyway, I tried to use the endpointCA configuration option in the ObjectStore configuration to try to manually set the certificate chain for the CA as trusted. Our configuration looks like this:

```
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
name: s3-postgress
namespace: postgres
spec:
configuration:
data:
additionalCommandArgs:
- '--min-chunk-size=5MB'
- '--read-timeout=60'
- '-vv'
destinationPath: s3://s3-postgres/postgres/customer
endpointCA:
key: ca.crt
name: pg-s3-ca
endpointURL: https://
s3Credentials:
accessKeyId:
key: ACCESS_KEY_ID
name: pg-s3-creds
secretAccessKey:
key: ACCESS_SECRET_KEY
name: pg-s3-creds
wal:
compression: gzip
instanceSidecarConfiguration:
env:
- name: AWS_CA_BUNDLE
value: /projected/ca.crt
- name: REQUESTS_CA_BUNDLE
value: /projected/ca.crt
- name: WALG_S3_CA_CERT_FILE
value: /projected/ca.crt
- name: SSL_CERT_FILE
value: /projected/ca.crt
```

As you can see I have tried a lot of different options for trying to specify the trusted CA(s), but none of them have worked. I have mounted the ca certificate at /projected/ca.crt using the following configuration in my Cluster definition:

```
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: customer-db
namespace: postgres
spec:
...
projectedVolumeTemplate:
sources:
- secret:
items:
- key: ca.crt
path: ca.crt
name: pg-s3-ca
```
Which seems to mount the secret into the same path for the sidecar container for the barman plugin as well.

Just a list of things I have already tried:
- Not using the endpointCA configuration snippet
- Not injecting the ca.crt file manually and setting envvars to point to this file
- Running the commands locally from my postgres container
- Changing the name of the secret key passed to endpointCA to tls.crt (as its named in the code example here: [minio-sotre.yaml](https://github.com/cloudnative-pg/plugin-barman-cloud/blob/54e3562e44fa117c74fb15318f96acdde5bad6a9/docs/examples/minio-store.yaml) )
- Changing the name of the secret key passed to endpointCA to barman-ca.crt to match constant name specified in the source code here: [constants.go](https://github.com/cloudnative-pg/plugin-barman-cloud/blob/main/internal/cnpgi/metadata/constants.go)
- Using openssl to extract the certificate from the s3 endpoint and ensuring that it is valid

I'm running out of things to try and as far as I can see there isn't anything in the documentation or any open issues that I have missed. So could someone maybe point me in the right direction here?

As you can see above I enabled verbose logging, so I will add the logs from the plugin-barman-cloud container when it tries to archive the WAL in the submitted file.

[logs.txt](https://github.com/user-attachments/files/21186830/logs.txt)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with internal/cnpgi/metadata/constants.go and the endpointCA configuration shown in docs/examples/minio-store.yaml, then inspect the attached logs.txt from the plugin-barman-cloud container. Reproduce the certificate validation failure with the provided ObjectStore and Cluster configuration; done means the configured CA chain is accepted for the on-premises S3 endpoint.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
go, kubernetes, postgresql
Bereich
cloud, devops, security
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.