Unable to start Kafka server on Azure VMs using Terraform
- Dominant language
- HCL
- Stars
- 925
- Forks
- 930
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 5
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Where would you like this feature added?
Other/Unknown
### Is this feature ask due to a problem that you are encountering?
While working to deploy Kafka cluster on Azure VMs using Terraform and Ansible, I am facing issues while creating a topic, I have checked and found that all three Zookeeper servers are running in cluster but one of the Kafka servers is not getting started in cluster.
VMs used: Dpsv5 series
OS: Ubuntu 20.04
Architecture: amd64 and arm64
The error which I am getting is:
```
[2023-05-30 08:35:45,623] INFO [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2023-05-30 08:35:45,623] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (myvm.internal.cloudapp.net/10.1.0.9:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-05-30 08:35:45,623] INFO [Controller id=1, targetBrokerId=1] Client requested connection close from node 1 (org.apache.kafka.clients.NetworkClient)
```
The code written to create the resources through terraform in main.tf file:
[main.txt](https://github.com/Azure/terraform/files/11612111/main.txt)
[providers.txt](https://github.com/Azure/terraform/files/11630935/providers.txt)
[variables.txt](https://github.com/Azure/terraform/files/11630937/variables.txt)
[zookeeper_cluster.txt](https://github.com/Azure/terraform/files/11630938/zookeeper_cluster.txt)
[kafka_cluster.txt](https://github.com/Azure/terraform/files/11630942/kafka_cluster.txt)
Steps followed to deploy the kafka cluster.
Run the following commands to create resources through terraform:
```
terraform init
terraform apply
```
To know the ip of kafka servers:
```
cat /tmp/inventery
```
To install the Zookeeper and the required dependencies and start zookeeper server:
```
ansible-playbook zookeeper_cluster.yaml -i /tmp/inventory
```
To install the Kafka and the required dependencies:
```
ansible-playbook kafka_cluster.yaml -i /tmp/inventory
```
To start kafka on all three server first do ssh on all three machines on which kafka is installed and run below command:
```
bin/kafka-server-start.sh config/server.properties
```
However, When I have created virtual machines manually(not through Terraform) and installed zookeeper and kafka in them with the same configuration and tried running Kafka cluster and it is running fine.
It will be really helpful if you share pointers on the above mentioned issue.
### Describe potential solutions.
When I have created virtual machines manually(not through Terraform) and installed zookeeper and kafka in them with the same configuration and tried running Kafka cluster and it is running fine.
### Anything else?
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the attached main.txt, providers.txt, variables.txt, zookeeper_cluster.txt, and kafka_cluster.txt files, then compare the Terraform/Ansible deployment with the manually created VMs. Reproduce the deployment with terraform init, terraform apply, and the listed ansible-playbook commands, then inspect the Kafka startup output around the broker connection failure. Done means identifying the configuration or infrastructure difference that prevents one broker from joining the cluster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, azure, kafka, terraform, ubuntu
- Domain
- cloud, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100