owncloud / owncloud/ocis

Proxy Service not shutting down gracefully

Open
#11,170 1 comment 0 reactions 0 assignees View on GitHub
Type:Bug
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

# **Bug Report**

---

## **Describe the bug**
The `proxy` service does not shut down gracefully when a termination signal (e.g., `SIGTERM`) is received. This results in ongoing requests being abruptly terminated, leading to **502 Bad Gateway** errors. The service does not provide sufficient time for in-flight requests to complete before shutting down.

---

## **Steps to reproduce**
1. Deploy the `proxy` service in a Kubernetes environment.
2. Send a long-running request to the `proxy` service.
3. Trigger a pod termination (e.g., by scaling down the Deployment or deleting the pod).
4. Observe that the request is terminated prematurely, resulting in a **502 Bad Gateway** error.

---

## **Expected behavior**
The `proxy` service should:
- Stop accepting new requests when a termination signal is received.
- Allow ongoing requests to complete before shutting down.
- Mark the pod as "not ready" to prevent Kubernetes from routing new traffic to the terminating pod.

---

## **Actual behavior**
- The `proxy` service does not handle termination signals gracefully.
- Ongoing requests are terminated abruptly, resulting in **502 Bad Gateway** errors.
- The service does not provide sufficient time for in-flight requests to complete.

---

## **Setup**
The `proxy` service was started with the following environment variables:

Environment Variables
The server was started in Kubernetes via the Helm Chart https://github.com/owncloud/ocis-charts

```yaml
containers:
- args:
- proxy
- server
command:
- ocis
env:
- name: MICRO_REGISTRY
value: nats-js-kv
- name: MICRO_REGISTRY_ADDRESS
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_CACHE_STORE
value: nats-js-kv
- name: OCIS_CACHE_STORE_NODES
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_CACHE_DISABLE_PERSISTENCE
value: 'true'
- name: OCIS_EVENTS_ENDPOINT
value: nats.ocis-nats.svc.cluster.local:4222
- name: OCIS_EVENTS_CLUSTER
value: ocis-cluster
- name: OCIS_EVENTS_ENABLE_TLS
value: 'false'
- name: OCIS_EVENTS_TLS_INSECURE
value: 'true'
- name: OCIS_EVENTS_TLS_ROOT_CA_CERTIFICATE
- name: PROXY_LOG_COLOR
value: 'true'
- name: PROXY_LOG_LEVEL
value: debug
- name: PROXY_LOG_PRETTY
value: 'true'
- name: PROXY_TRACING_ENABLED
value: 'false'
- name: PROXY_TRACING_TYPE
value: jaeger
- name: PROXY_TRACING_ENDPOINT
value: jaeger.jaeger.svc.cluster.local:6831
- name: PROXY_TRACING_COLLECTOR
value: jaeger.jaeger.svc.cluster.local:14268/api/traces
- name: PROXY_DEBUG_PPROF
value: 'false'
- name: PROXY_HTTP_ADDR
value: 0.0.0.0:9200
- name: PROXY_DEBUG_ADDR
value: 0.0.0.0:9205
- name: PROXY_OIDC_ISSUER
value: https://keycloak.kube.domain.io/realms/oCIS
- name: PROXY_OIDC_REWRITE_WELLKNOWN
value: 'true'
- name: PROXY_USER_OIDC_CLAIM
value: sub
- name: PROXY_USER_CS3_CLAIM
value: userid
- name: PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD
value: jwt
- name: PROXY_OIDC_SKIP_USER_INFO
value: 'false'
- name: PROXY_TLS
value: 'false'
- name: PROXY_OIDC_INSECURE
value: 'true'
- name: PROXY_JWT_SECRET
valueFrom:
secretKeyRef:
key: jwt-secret
name: jwt-secret
- name: PROXY_MACHINE_AUTH_API_KEY
valueFrom:
secretKeyRef:
key: machine-auth-api-key
name: machine-auth-api-key
- name: PROXY_SERVICE_ACCOUNT_ID
valueFrom:
configMapKeyRef:
key: service-account-id
name: auth-service
- name: PROXY_SERVICE_ACCOUNT_SECRET
valueFrom:
secretKeyRef:
key: service-account-secret
name: service-account-secret
- name: PROXY_CSP_CONFIG_FILE_LOCATION
value: /etc/ocis/csp.yaml
- name: PROXY_AUTOPROVISION_ACCOUNTS
value: 'true'
- name: PROXY_AUTOPROVISION_CLAIM_EMAIL
value: email
- name: PROXY_AUTOPROVISION_CLAIM_DISPLAYNAME
value: name
- name: PROXY_AUTOPROVISION_CLAIM_GROUPS
value: groups
- name: PROXY_AUTOPROVISION_CLAIM_USERNAME
value: preferred_username
image: owncloud/ocis:7.1.1
```

## **Additional context**
The issue occurs during pod termination in a Kubernetes environment.
The pod has a `terminationGracePeriodSeconds` set, but the service does not utilize this time to complete ongoing requests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.