apache / apache/pinot

Problems with new helm chart (adding table and minion stateless)

Open
#9,032 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I'm currently deploying a new pinot cluster and I was facing some issues while adding tables using [this guide](https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/pinot-realtime-quickstart.yml).

After debugging it a lot, I guess I discovered the solution. Below are the problems that I found:

- The new helm package is missing on the repo. It's not possible to deploy minion stateless (or any update) using this repo as a helm repo.

- Using stateless minion I got a connection refused on the controller while trying to add tables.

- The combination of stateful minion and `latest` image, results in the errors below:
Job logs:
`2022/07/08 14:22:23.062 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}`
Controller logs:
`Caused by: org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.scram.ScramLoginModule`

- Readiness and Liveness probes are not working on minion stateless, so it should be disabled on values file.

**Resolution:**
We have another cluster and everything works fine, so I checked the differences and I realized the image used on this cluster was `latest:jdk-11`. Changing to this image I was able to create tables using `pinot-admin.sh` as described [here](https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/pinot-realtime-quickstart.yml).

But, this solution only worked using stateful minion, stateless is still returning connection refused after changing the image.

Contributor guide

Open the contributing guide

Research direction

Start with kubernetes/helm/pinot/pinot-realtime-quickstart.yml and the chart values file, then compare stateful and stateless minion deployments. Reproduce table creation with pinot-admin.sh using the reported image tags and inspect the controller connection, Kafka authentication, and minion readiness and liveness probes. Done means the chart is available as a Helm repository and stateless minion table creation and probes work.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, java, kafka, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.