aws / aws/aws-cloud-map-mcs-controller-for-k8s
Connection refused error when calling service endpoint
- Dominant language
- Go
- Stars
- 98
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
**Problem**
Calling `Clusterset` service endpoint after deleting a pod for that service will result in connection refused error.
**Step to reproduce connection refused error**→
1. Have pods in 2 clusters → (for example stag-eks , stag-eks-2)
2. Create a ServiceExport for the service you are trying to expose in stag-eks
3. Step 2 will automatically create a ServiceImport on both clusters
4. Create a dummy pod in stag-eks-2, and exec into it. Run a curl command to the ClusterSet endpoint that was exported in step 2 (The curl command will successfully obtain a response)
5. Delete the service pod u created in step 2 in stag-eks
6. Wait for the pod to get recreated, and run the curl command again (which will get a connection refused error)
**Steps to resolve the issue**
1. Delete ServiceImport in stag-eks-2 (where the caller is from)
2. Rerun the curl command in the dummy pod in stag-eks-2, and u will get a successful response
Based on my current observation, it seems like coreDNS is not getting the latest pod IP and is still resolving to the old pod ip.
When the `ServiceImport` is recreated, it started to work fine again probably because the coreDNS record is updated due to the recreation.
Its also worth to add, that removing readinessProbe from the deployment manifest fixes the issue mentioned above (which i dont really understand how that fixes it)
Contributor guide
Research direction
No files or tests are identified. First reproduce the ServiceExport/ServiceImport scenario across two clusters, then trace how the controller handles endpoint updates after pod replacement and how CoreDNS resolves the ClusterSet endpoint, including the effect of readinessProbe. Done means the curl succeeds after the service pod is recreated without deleting ServiceImport.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- devops, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100