kakao / kakao/network-node-manager

failed to destroy rule externalIP to clusterIP {"error": "exit status 2"}

Open
#19 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
107
Forks
19
PR merge metrics
No merged PRs in 30d

Description

안녕하세요. RKE2 기반의 쿠버네티스 배포판을 온프레미스 환경에서 운영 중입니다.
yatai라고 하고 BentoML 모델을 쿠버네티스에 쉽게 배포해주는 오픈소스를 쓰는 와중에 100MB 정도되는 모델을 yatai에 올릴려고 하니까 40~50MB 정도 업로드 되었을 때 항상 connection reset by peer에러가 발생했고, 검색하다 보니 감사하게도 공유해주신 이 오픈소스를 찾게 되었습니다.

그런데

Enable Drop Invalid Packet Rule in INPUT chain 옵션은 true,
Enable External-IP to Cluster-IP DNAT Rule 옵션은 false

으로 설치하고 설정했을 때 모든 Pod들에서 Crashloopbackoff 에러가 발생하고 있습니다.
env 설정 값은 정상적으로 들어간 것 같습니다.
혹시 어디서부터 손대면 좋을지 조언을 여쭤볼 수 있을까요?

```
Wed, Mar 1 2023 1:27:50 pm | I0301 04:27:50.915845 1 request.go:655] Throttling request took 1.019118868s, request: GET:https://10.43.0.1:443/apis/coordination.k8s.io/v1?timeout=32s
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.096Z INFO setup starting manager
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.096Z INFO controller-runtime.manager.controller.service Starting EventSource {"reconciler group": "", "reconciler kind": "Service", "source": "kind source: /, Kind="}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controller-runtime.manager.controller.service Starting Controller {"reconciler group": "", "reconciler kind": "Service"}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controller-runtime.manager.controller.service Starting workers {"reconciler group": "", "reconciler kind": "Service", "worker count": 1}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize initalize service contoller
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config IPv4 pod CIDR {"IPv4 pod cIDR": "10.42.0.0/16"}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config IPv6 pod CIDR {"IPv6 pod cIDR": ""}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config for drop invalid packet in INPUT chain {"enabled": true}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config for externalIP to clusterIP {"enabled": false}
Wed, Mar 1 2023 1:27:52 pm | 2023-03-01T04:27:52.698Z ERROR controllers.Service.initalize {"error": "exit status 2"}
Wed, Mar 1 2023 1:27:52 pm | github.com/go-logr/zapr.(*zapLogger).Error
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/pkg/rules.DestoryRulesExternalCluster
Wed, Mar 1 2023 1:27:52 pm | /workspace/pkg/rules/rule_external_cluster.go:87
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
Wed, Mar 1 2023 1:27:52 pm | /workspace/controllers/service_controller.go:133
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
Wed, Mar 1 2023 1:27:52 pm | 2023-03-01T04:27:52.699Z ERROR controllers.Service.initalize failed to destroy rule externalIP to clusterIP {"error": "exit status 2"}
Wed, Mar 1 2023 1:27:52 pm | github.com/go-logr/zapr.(*zapLogger).Error
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
Wed, Mar 1 2023 1:27:52 pm | /workspace/controllers/service_controller.go:134
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.