Problems with new helm chart (adding table and minion stateless)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
I'm currently deploying a new pinot cluster and I was facing some issues while adding tables using [this guide](https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/pinot-realtime-quickstart.yml).
After debugging it a lot, I guess I discovered the solution. Below are the problems that I found:
- The new helm package is missing on the repo. It's not possible to deploy minion stateless (or any update) using this repo as a helm repo.
- Using stateless minion I got a connection refused on the controller while trying to add tables.
- The combination of stateful minion and `latest` image, results in the errors below:
Job logs:
`2022/07/08 14:22:23.062 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}`
Controller logs:
`Caused by: org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.scram.ScramLoginModule`
- Readiness and Liveness probes are not working on minion stateless, so it should be disabled on values file.
**Resolution:**
We have another cluster and everything works fine, so I checked the differences and I realized the image used on this cluster was `latest:jdk-11`. Changing to this image I was able to create tables using `pinot-admin.sh` as described [here](https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/pinot-realtime-quickstart.yml).
But, this solution only worked using stateful minion, stateless is still returning connection refused after changing the image.
Contributor guide
Research direction
Start with kubernetes/helm/pinot/pinot-realtime-quickstart.yml and the chart values file, then compare stateful and stateless minion deployments. Reproduce table creation with pinot-admin.sh using the reported image tags and inspect the controller connection, Kafka authentication, and minion readiness and liveness probes. Done means the chart is available as a Helm repository and stateless minion table creation and probes work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, java, kafka, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100