aws / aws/aws-cloud-map-mcs-controller-for-k8s
Panic in reconciler causing segmentation violation
- Dominant language
- Go
- Stars
- 98
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
During benchmark testing, controller completely halted with a panic due to the attempt of accessing an invalid memory address or nil pointer dereference causing a segmentation violation.
Here are the last logs from the manager:
```
{"level":"info","ts":1655941251.2284162,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":9}
{"level":"info","ts":1655941251.228578,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":10}
{"level":"info","ts":1655941252.7608504,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.7608776,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.76091,"logger":"cloudmap","msg":"creating a new service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8043206,"logger":"cloudmap","msg":"service created","namespace":"demov2","name":"nginx-benchmark-service-61","id":"srv-vvx3saukvkvkytng"}
{"level":"info","ts":1655941252.8043518,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8277564,"msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"serviceexport","controllerGroup":"multicluster.x-k8s.io","controllerKind":"ServiceExport","serviceExport":{"name":"nginx-benchmark-service-61","namespace":"demov2"},"namespace":"demov2","name":"nginx-benchmark-service-61","reconcileID":"f1c0f919-fdb1-4e63-be72-6aa2f4df35e7"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x15204f7]
goroutine 221 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x16819e0, 0x271b0e0})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).handleUpdate(0xc000461e40, {0x1aad4b0, 0xc0007644c0}, 0xc00032be00, 0xc000d08000)
/workspace/pkg/controllers/serviceexport_controller.go:136 +0x597
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).Reconcile(0xc000461e40, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x18}, {0xc000b68780, 0x40d3a7}}})
/workspace/pkg/controllers/serviceexport_controller.go:93 +0x7d5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1aad4b0, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x17a4d00}, {0xc000b68780, 0x40ee1d}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121 +0xd1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00031b7c0, {0x1aad4b0, 0xc000461c40}, {0x16e0800, 0xc00098b8c0})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00031b7c0, {0x1aad4b0, 0xc000461c40})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:230 +0x36f
```
Note that the code lines do not line up exactly with the Controller's code.
`serviceexport_controller.go:136` points to the `Current: cmService.Endpoints` line
and `serviceexport_controller.go:93` to the `return r.handleUpdate(ctx, &serviceExport, &service)`
Contributor guide
Research direction
Start in pkg/controllers/serviceexport_controller.go at the reported lines 93 and 136, then trace handleUpdate and the Current: cmService.Endpoints assignment. Reproduce the benchmark scenario if available and inspect which value is nil when the ServiceExport reconciles. Done means the reconciler no longer panics on this flow and the benchmark completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100