Blizzard / Blizzard/node-rdkafka
Broker Transport Failure over VPN
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
I am using this Kafka helm chart provided by Bitnami: https://github.com/bitnami/charts/tree/main/bitnami/kafka/
I am accessing Kafka in Kubernetes over a VPN connection using an SSH tunnel.
I am having a problem when Kafka redirects my connection to an internal IP address that I cannot reach over the VPN.
The external listener for Kafka works fine as that first connection is made to localhost:9094 to a port I have opened up in the SSH tunnel.
```
ℹ kafka-consumer-log {"severity":7,"fac":"CONTROLLERID","message":"[thrd:main]: localhost:9094/bootstrap: ControllerId update -1 -> 0"} 16:40:51
```
Then my connection gets redirected to the Kafka broker, and is using an Internal IP address that I cannot access 100.64.0.15:30896 and results in a timeout error. I would like it to redirect to port 30896 over localhost like above.
```
ℹ kafka-consumer-log {"severity":7,"fac":"BRKMAIN","message":"[thrd:100.64.0.15:30896/0]: 100.64.0.15:30896/0: Enter main broker thread"}
```
I'm not sure how to adjust this redirect to the broker using the helm chart. Here's my helm chart configuration:
```
externalAccess:
## @param externalAccess.enabled Enable Kubernetes external cluster access to Kafka brokers
##
enabled: true
## External IPs auto-discovery configuration
## An init container is used to auto-detect LB IPs or node ports by querying the K8s API
## Note: RBAC might be required
##
autoDiscovery:
## @param externalAccess.autoDiscovery.enabled Enable using an init container to auto-detect external IPs/ports by querying the K8s API
##
enabled: true
## Bitnami Kubectl image
## ref: https://hub.docker.com/r/bitnami/kubectl/tags/
## @param externalAccess.autoDiscovery.image.registry Init container auto-discovery image registry
## @param externalAccess.autoDiscovery.image.repository Init container auto-discovery image repository
## @param externalAccess.autoDiscovery.image.tag Init container auto-discovery image tag (immutable tags are recommended)
## @param externalAccess.autoDiscovery.image.digest Kubectl image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
## @param externalAccess.autoDiscovery.image.pullPolicy Init container auto-discovery image pull policy
## @param externalAccess.autoDiscovery.image.pullSecrets Init container auto-discovery image pull secrets
##
image:
registry: docker.io
repository: bitnami/kubectl
tag: 1.25.8-debian-11-r2
digest: ""
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets (secrets must be manually created in the namespace)
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
## e.g:
## pullSecrets:
## - myRegistryKeySecretName
##
pullSecrets: []
## Init Container resource requests and limits
## ref: https://kubernetes.io/docs/user-guide/compute-resources/
## @param externalAccess.autoDiscovery.resources.limits The resources limits for the auto-discovery init container
## @param externalAccess.autoDiscovery.resources.requests The requested resources for the auto-discovery init container
##
resources:
limits: {}
requests: {}
## Parameters to configure K8s service(s) used to externally access Kafka brokers
## Note: A new service per broker will be created
##
service:
## @param externalAccess.service.type Kubernetes Service type for external access. It can be NodePort, LoadBalancer or ClusterIP
##
type: NodePort
## @param externalAccess.service.ports.external Kafka port used for external access when service type is LoadBalancer
##
ports:
external: 9094
## @param externalAccess.service.loadBalancerIPs Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount
## e.g:
## loadBalancerIPs:
## - X.X.X.X
## - Y.Y.Y.Y
##
loadBalancerIPs: []
## @param externalAccess.service.loadBalancerNames Array of load balancer Names for each Kafka broker. Length must be the same as replicaCount
## e.g:
## loadBalancerNames:
## - broker1.external.example.com
## - broker2.external.example.com
##
loadBalancerNames: []
## @param externalAccess.service.loadBalancerAnnotations Array of load balancer annotations for each Kafka broker. Length must be the same as replicaCount
## e.g:
## loadBalancerAnnotations:
## - external-dns.alpha.kubernetes.io/hostname: broker1.external.example.com.
## - external-dns.alpha.kubernetes.io/hostname: broker2.external.example.com.
##
loadBalancerAnnotations: []
## @param externalAccess.service.loadBalancerSourceRanges Address(es) that are allowed when service is LoadBalancer
## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
## e.g:
## loadBalancerSourceRanges:
## - 10.10.10.0/24
##
loadBalancerSourceRanges: []
## @param externalAccess.service.nodePorts Array of node ports used for each Kafka broker. Length must be the same as replicaCount
## e.g:
## nodePorts:
## - 30001
## - 30002
##
nodePorts: [32487]
## @param externalAccess.service.externalIPs Use distinct service host IPs to configure Kafka external listener when service type is NodePort. Length must be the same as replicaCount
## e.g:
## externalIPs:
## - X.X.X.X
## - Y.Y.Y.Y
##
externalIPs: ["100.64.0.15"]
## @param externalAccess.service.useHostIPs Use service host IPs to configure Kafka external listener when service type is NodePort
##
useHostIPs: false
## @param externalAccess.service.usePodIPs using the MY_POD_IP address for external access.
##
usePodIPs: false
## @param externalAccess.service.domain Domain or external ip used to configure Kafka external listener when service type is NodePort or ClusterIP
## NodePort: If not specified, the container will try to get the kubernetes node external IP
## ClusterIP: Must be specified, ingress IP or domain where tcp for external ports is configured
##
domain: "kafka.kube"
## @param externalAccess.service.publishNotReadyAddresses Indicates that any agent which deals with endpoints for this Service should disregard any indications of ready/not-ready
## ref: https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/
##
publishNotReadyAddresses: false
## @param externalAccess.service.labels Service labels for external access
##
labels: {}
## @param externalAccess.service.annotations Service annotations for external access
##
annotations: {}
## @param externalAccess.service.extraPorts Extra ports to expose in the Kafka external service
##
extraPorts: []
```
Contributor guide
Research direction
No repository file or test is named. Start by determining whether node-rdkafka or the Bitnami Kafka Helm chart owns the advertised broker address, then compare the reported externalAccess settings with the broker connection behavior. Done means identifying a repository-scoped change or confirming that the problem is configuration outside node-rdkafka.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100