cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Changes to ObjectStore spec do not get applied until after the cluster is healthy

Abierto
#918 2 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Go
Estrellas
191
Forks
72
Merge medio
1 d 16 h
PR fusionados (30 d)
18

Descripción

We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:

```
ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest)
when calling the PutObject operation: The Content-MD5 or checksum value that
you specified is not valid.
```

The fix is already mentioned in the documentation, where we add the following to the object store:

```yaml
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
spec:
configuration:
...
instanceSidecarConfiguration:
env:
- name: AWS_REQUEST_CHECKSUM_CALCULATION
value: when_required
- name: AWS_RESPONSE_CHECKSUM_VALIDATION
value: when_required
```

and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).

I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

El issue menciona ObjectStore, instanceSidecarConfiguration, el operador barman-cloud-plugin y una réplica atascada en 1/2 Ready; empieza rastreando cómo se aplica esa configuración cuando un pod no está saludable. Reproduce el error de checksum de AWS con las variables de entorno mostradas y una réplica bloqueada, y verifica después que la configuración pendiente se aplique sin eliminar manualmente el pod.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, go, postgresql
Área
backend, cloud
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Necesita aclaración
Aptitud para principiantes
42/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.