aws / aws/aws-cloud-map-mcs-controller-for-k8s

Connection refused error when calling service endpoint

Open
#273 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
98
Forks
32
PR merge metrics
No merged PRs in 30d

Description

**Problem**

Calling `Clusterset` service endpoint after deleting a pod for that service will result in connection refused error.

**Step to reproduce connection refused error**→

1. Have pods in 2 clusters → (for example stag-eks , stag-eks-2)
2. Create a ServiceExport for the service you are trying to expose in stag-eks
3. Step 2 will automatically create a ServiceImport on both clusters
4. Create a dummy pod in stag-eks-2, and exec into it. Run a curl command to the ClusterSet endpoint that was exported in step 2 (The curl command will successfully obtain a response)
5. Delete the service pod u created in step 2 in stag-eks
6. Wait for the pod to get recreated, and run the curl command again (which will get a connection refused error)

**Steps to resolve the issue**
1. Delete ServiceImport in stag-eks-2 (where the caller is from)
2. Rerun the curl command in the dummy pod in stag-eks-2, and u will get a successful response

Based on my current observation, it seems like coreDNS is not getting the latest pod IP and is still resolving to the old pod ip.
When the `ServiceImport` is recreated, it started to work fine again probably because the coreDNS record is updated due to the recreation.

Its also worth to add, that removing readinessProbe from the deployment manifest fixes the issue mentioned above (which i dont really understand how that fixes it)

Contributor guide

Open the contributing guide

Research direction

No files or tests are identified. First reproduce the ServiceExport/ServiceImport scenario across two clusters, then trace how the controller handles endpoint updates after pod replacement and how CoreDNS resolves the ClusterSet endpoint, including the effect of readinessProbe. Done means the curl succeeds after the service pod is recreated without deleting ServiceImport.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
devops, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.