aws / aws/containers-roadmap

[EKS] [CoreDNS]: Recovered from panic in server: "dns://:53" runtime error: invalid memory address or nil pointer dereference

Open
#2,146 1 comment 3 reactions 0 assignees View on GitHub
EKS EKS Networking Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
CoreDNS 1.9.3 for EKS 1.24 appears to be experiencing potential memory leaks after recovering from a panic.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When CoreDNS recovers from a panic, it stops responding to requests, and memory starts to steadily increase. I observed a memory increase, and just before this occurred, I received this error:
`[ERROR] Recovered from panic in server: "dns://:53" runtime error: invalid memory address or nil pointer dereference. `

**Are you currently working around this issue?**
Yes, it's a production environment, and I am actively seeking a solution to this error.

**Additional context**
I found some merged PRs in newer CoreDNS versions.
https://github.com/coredns/coredns/pull/6120 (I'm not using DoH, but could be a relation.)
https://github.com/coredns/coredns/pull/6179

Just one plugin has stopped responding to queries.

**Attachments**
Memory increase graph
![memory](https://github.com/aws/containers-roadmap/assets/5014087/a6c3c6f5-f430-46cb-84a1-a1ac67d127d7)

Panic and coredns_dns_requests_total (Pink represents panic, while other colors represent coredns_dns_requests_total for each type AAAA, CNAME, and A)
![coredns_dns_requests_total](https://github.com/aws/containers-roadmap/assets/5014087/adbb2e7a-2443-47a7-bb66-8743fa751c01)

Contributor guide

Open the contributing guide

Research direction

Start by comparing the reported CoreDNS 1.9.3 behavior on EKS 1.24 with the newer versions and merged PRs 6120 and 6179 mentioned in the issue. Use the panic message, memory graph, and coredns_dns_requests_total observations to investigate; done means a recovered panic does not leave the plugin unresponsive or cause steadily increasing memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.