Proxy Service not shutting down gracefully
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 274
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 103
Description
# **Bug Report**
---
## **Describe the bug**
The `proxy` service does not shut down gracefully when a termination signal (e.g., `SIGTERM`) is received. This results in ongoing requests being abruptly terminated, leading to **502 Bad Gateway** errors. The service does not provide sufficient time for in-flight requests to complete before shutting down.
---
## **Steps to reproduce**
1. Deploy the `proxy` service in a Kubernetes environment.
2. Send a long-running request to the `proxy` service.
3. Trigger a pod termination (e.g., by scaling down the Deployment or deleting the pod).
4. Observe that the request is terminated prematurely, resulting in a **502 Bad Gateway** error.
---
## **Expected behavior**
The `proxy` service should:
- Stop accepting new requests when a termination signal is received.
- Allow ongoing requests to complete before shutting down.
- Mark the pod as "not ready" to prevent Kubernetes from routing new traffic to the terminating pod.
---
## **Actual behavior**
- The `proxy` service does not handle termination signals gracefully.
- Ongoing requests are terminated abruptly, resulting in **502 Bad Gateway** errors.
- The service does not provide sufficient time for in-flight requests to complete.
---
## **Setup**
The `proxy` service was started with the following environment variables:
Environment Variables
The server was started in Kubernetes via the Helm Chart https://github.com/owncloud/ocis-charts
```yaml
containers:
- args:
- proxy
- server
command:
- ocis
env:
- name: MICRO_REGISTRY
value: nats-js-kv
- name: MICRO_REGISTRY_ADDRESS
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_CACHE_STORE
value: nats-js-kv
- name: OCIS_CACHE_STORE_NODES
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_CACHE_DISABLE_PERSISTENCE
value: 'true'
- name: OCIS_EVENTS_ENDPOINT
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_EVENTS_CLUSTER
value: ocis-cluster
- name: OCIS_EVENTS_ENABLE_TLS
value: 'false'
- name: OCIS_EVENTS_TLS_INSECURE
value: 'true'
- name: OCIS_EVENTS_TLS_ROOT_CA_CERTIFICATE
- name: PROXY_LOG_COLOR
value: 'true'
- name: PROXY_LOG_LEVEL
value: debug
- name: PROXY_LOG_PRETTY
value: 'true'
- name: PROXY_TRACING_ENABLED
value: 'false'
- name: PROXY_TRACING_TYPE
value: jaeger
- name: PROXY_TRACING_ENDPOINT
value: jaeger.jaeger.svc.cluster.local:6831
- name: PROXY_TRACING_COLLECTOR
value: jaeger.jaeger.svc.cluster.local:14268/api/traces
- name: PROXY_DEBUG_PPROF
value: 'false'
- name: PROXY_HTTP_ADDR
value: 0.0.0.0:9200
- name: PROXY_DEBUG_ADDR
value: 0.0.0.0:9205
- name: PROXY_OIDC_ISSUER
value: https://keycloak.kube.domain.io/realms/oCIS
- name: PROXY_OIDC_REWRITE_WELLKNOWN
value: 'true'
- name: PROXY_USER_OIDC_CLAIM
value: sub
- name: PROXY_USER_CS3_CLAIM
value: userid
- name: PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD
value: jwt
- name: PROXY_OIDC_SKIP_USER_INFO
value: 'false'
- name: PROXY_TLS
value: 'false'
- name: PROXY_OIDC_INSECURE
value: 'true'
- name: PROXY_JWT_SECRET
valueFrom:
secretKeyRef:
key: jwt-secret
name: jwt-secret
- name: PROXY_MACHINE_AUTH_API_KEY
valueFrom:
secretKeyRef:
key: machine-auth-api-key
name: machine-auth-api-key
- name: PROXY_SERVICE_ACCOUNT_ID
valueFrom:
configMapKeyRef:
key: service-account-id
name: auth-service
- name: PROXY_SERVICE_ACCOUNT_SECRET
valueFrom:
secretKeyRef:
key: service-account-secret
name: service-account-secret
- name: PROXY_CSP_CONFIG_FILE_LOCATION
value: /etc/ocis/csp.yaml
- name: PROXY_AUTOPROVISION_ACCOUNTS
value: 'true'
- name: PROXY_AUTOPROVISION_CLAIM_EMAIL
value: email
- name: PROXY_AUTOPROVISION_CLAIM_DISPLAYNAME
value: name
- name: PROXY_AUTOPROVISION_CLAIM_GROUPS
value: groups
- name: PROXY_AUTOPROVISION_CLAIM_USERNAME
value: preferred_username
image: owncloud/ocis:7.1.1
```
## **Additional context**
The issue occurs during pod termination in a Kubernetes environment.
The pod has a `terminationGracePeriodSeconds` set, but the service does not utilize this time to complete ongoing requests.
Contributor guide
Assessment
This issue has not been assessed yet.