cloudnative-pg / cloudnative-pg/plugin-barman-cloud
Changes to ObjectStore spec do not get applied until after the cluster is healthy
- Lenguaje dominante
- Go
- Estrellas
- 191
- Forks
- 72
- Merge medio
- 1 d 16 h
- PR fusionados (30 d)
- 18
Descripción
We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:
```
ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest)
when calling the PutObject operation: The Content-MD5 or checksum value that
you specified is not valid.
```
The fix is already mentioned in the documentation, where we add the following to the object store:
```yaml
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
spec:
configuration:
...
instanceSidecarConfiguration:
env:
- name: AWS_REQUEST_CHECKSUM_CALCULATION
value: when_required
- name: AWS_RESPONSE_CHECKSUM_VALIDATION
value: when_required
```
and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).
I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.
Guía de contribución
Línea de trabajo
El issue menciona ObjectStore, instanceSidecarConfiguration, el operador barman-cloud-plugin y una réplica atascada en 1/2 Ready; empieza rastreando cómo se aplica esa configuración cuando un pod no está saludable. Reproduce el error de checksum de AWS con las variables de entorno mostradas y una réplica bloqueada, y verifica después que la configuración pendiente se aplique sin eliminar manualmente el pod.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- aws, go, postgresql
- Área
- backend, cloud
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 42/100