apache / apache/openwhisk

Cannot run openwhisk-standalone.jar with Kafka

Open
#5,184 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

## Environment details:

* local deployment, native ubuntu, Mac OS, Kubernetes, ...
* version of docker, ubuntu, ...

## Steps to reproduce the issue:

1. `./gradlew :core:standalone:build`
2. `java -jar bin/openwhisk-standalone.jar --all --no-ui`

## Provide the expected results and outputs:

(runs without errors)

## Provide the actual results and outputs:

Kafka cannot start

```
================================================================================
Running pre flight checks ...

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
59bf1c3509f3: Pulling fs layer
59bf1c3509f3: Verifying Checksum
59bf1c3509f3: Download complete
59bf1c3509f3: Pull complete
Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
Status: Downloaded newer image for alpine:latest
Local Host Name: 172.17.0.1
Local Internal Name: 172.17.0.1

[ OK ] 'docker' cli found. (Docker version 19.03.5, build 633a0ea838)
[ OK ] 'docker' version 19.3.5 is newer than minimum supported 18.3.0
[ OK ] 'docker' is running.
[ OK ] 'wsk' cli found. (2021-04-01T23:49:54.523+0000)
[ OK ] 'wsk' configured for namespace [guest].
[ OK ] 'wsk' configured to connect to http://172.17.0.1:3233.
[ OK ] Server port [3233] is free
[ OK ] Api gateway port [3234] is free

================================================================================
[2021-12-13T17:56:14.656Z] [INFO] Starting OpenWhisk standalone on port 3233
[2021-12-13T17:56:14.974Z] [INFO] Slf4jLogger started
[2021-12-13T17:56:15.167Z] [INFO] Using [/u/hwen5/.openwhisk/standalone/server-3233] as data directory
[2021-12-13T17:56:15.470Z] [INFO] [#tid_sid_unknown] [StandaloneDockerClient] Detected docker client version 19.03.5
[2021-12-13T17:56:15.556Z] [INFO] [#tid_sid_standalone] [StandaloneDockerClient] running /usr/bin/docker ps --quiet --no-trunc --all --filter name=whisk- (timeout: 1 minute) [marker:invoker_docker.ps_start:8]
[2021-12-13T17:56:15.729Z] [INFO] [#tid_sid_standalone] [StandaloneDockerClient] [marker:invoker_docker.ps_finish:182:43]
[2021-12-13T17:56:15.763Z] [INFO] [#tid_sid_standalone] [StandaloneDockerSupport] removing 0 containers launched for Standalone server support.
[2021-12-13T17:56:15.833Z] [INFO] [#tid_sid_apiMgmt] [ApiGwLauncher] Starting Redis at 6379
[2021-12-13T17:56:15.834Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker pull redis:4.0 (timeout: 10 minutes) [marker:invoker_docker.pull_start:69]
[2021-12-13T17:56:20.055Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:4289:4220]
[2021-12-13T17:56:20.061Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-redis --network bridge -p 6379:6379 redis:4.0 (timeout: 1 minute) [marker:invoker_docker.run_start:4296]
[2021-12-13T17:56:23.779Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.run_finish:8014:3718]
[2021-12-13T17:56:23.782Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} c3b41b816096812d0efd1f200495c4c9a9df959df129b15fe8eb5c3ce2a87ee6 (timeout: 1 minute) [marker:invoker_docker.inspect_start:8016]
[2021-12-13T17:56:23.829Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:8064:47]
[2021-12-13T17:56:23.836Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker run --rm --name whisk-redis-test redis:4.0 redis-cli -h 172.17.0.2 -p 6379 ping (timeout: 1 minute) [marker:invoker_docker.run_start:8071]
[2021-12-13T17:56:25.833Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.run_finish:10068:1997]
[2021-12-13T17:56:25.835Z] [INFO] [#tid_sid_apiMgmt] [ApiGwLauncher] Starting Api Gateway at api port: 9000, management port: 3234
[2021-12-13T17:56:25.836Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker pull openwhisk/apigateway:0.11.0 (timeout: 10 minutes) [marker:invoker_docker.pull_start:10071]
[2021-12-13T17:56:26.182Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:10416:345]
[2021-12-13T17:56:26.183Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-apigw --network bridge -e REDIS_HOST=172.17.0.2 -e REDIS_PORT=6379 -e PUBLIC_MANAGEDURL_PORT=3234 -e PUBLIC_MANAGEDURL_HOST=172.17.0.1 -e BACKEND_HOST=http://172.17.0.1:3233 -p 9000:9000 -p 3234:8080 openwhisk/apigateway:0.11.0 (timeout: 1 minute) [marker:invoker_docker.run_start:10418]
[2021-12-13T17:56:27.552Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.run_finish:11786:1368]
[2021-12-13T17:56:27.552Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} 814cc06c32f349be6df37778bf138df1eeadb575a7cb82cd2f88c4a577558181 (timeout: 1 minute) [marker:invoker_docker.inspect_start:11787]
[2021-12-13T17:56:27.598Z] [INFO] [#tid_sid_apiMgmt] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:11832:45]
Waiting for ApiGateway server at http://172.17.0.1:9000/v1/apis to start since 4.297 ms
[2021-12-13T17:56:27.738Z] [INFO] [#tid_sid_apiMgmt] [StandaloneOpenWhisk] Api Gateway started successfully at http://172.17.0.1:3234
[2021-12-13T17:56:31.479Z] [INFO] [#tid_sid_kafka] [KafkaLauncher] Starting Kafka Drop UI port: 34631
[2021-12-13T17:56:31.479Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] running /usr/bin/docker pull obsidiandynamics/kafdrop (timeout: 10 minutes) [marker:invoker_docker.pull_start:3741]
[2021-12-13T17:56:49.509Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:21771:18030]
[2021-12-13T17:56:49.510Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-kafka-drop-ui --network bridge -e ZOOKEEPER_CONNECT=172.17.0.1:2181 -e KAFKA_BROKERCONNECT=172.17.0.1:9091 -e JVM_OPTS=-Xms32M -Xmx64M -e SERVER_SERVLET_CONTEXTPATH=/ -p 34631:9000 obsidiandynamics/kafdrop (timeout: 1 minute) [marker:invoker_docker.run_start:21772]
[2021-12-13T17:57:00.037Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] [marker:invoker_docker.run_finish:32299:10527]
[2021-12-13T17:57:00.038Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} 9989ca98b5a46d6cba11a1585bac9919b63dac57779e12900eac61d2bfc7ec9c (timeout: 1 minute) [marker:invoker_docker.inspect_start:32300]
[2021-12-13T17:57:00.085Z] [INFO] [#tid_sid_kafka] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:32347:47]
[2021-12-13T17:57:00.086Z] [INFO] [#tid_sid_kafka] [StandaloneOpenWhisk] Kafka started successfully at http://172.17.0.1:9092
[2021-12-13T17:57:00.483Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Using pre-existing database from /u/hwen5/.openwhisk/standalone/server-3233/couchdb
[2021-12-13T17:57:00.483Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Starting CouchDB at 5984
[2021-12-13T17:57:00.484Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] running /usr/bin/docker pull apache/couchdb:2.3 (timeout: 10 minutes) [marker:invoker_docker.pull_start:397]
[2021-12-13T17:57:08.059Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:7972:7575]
[2021-12-13T17:57:08.060Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-couch --network bridge -e COUCHDB_USER=whisk_admin -e COUCHDB_PASSWORD=some_passw0rd -p 5984:5984 -v /u/hwen5/.openwhisk/standalone/server-3233/couchdb:/opt/couchdb/data apache/couchdb:2.3 (timeout: 1 minute) [marker:invoker_docker.run_start:7973]
[2021-12-13T17:57:13.418Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] [marker:invoker_docker.run_finish:13331:5358]
[2021-12-13T17:57:13.419Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} 7df6abb655358625f261ec88efd7fb870e3be7688e20d47f6b58ac9efd6e93ad (timeout: 1 minute) [marker:invoker_docker.inspect_start:13331]
[2021-12-13T17:57:13.467Z] [INFO] [#tid_sid_couchDB] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:13380:48]
Waiting for CouchDB server at http://172.17.0.1:5984/_utils/ to start since 206.6 μs
Waiting for CouchDB server at http://172.17.0.1:5984/_utils/ to start since 1.005 s
[2021-12-13T17:57:14.753Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file auth_design_document_for_subjects_db_v2.0.0.json for db whisk_local_subjects
[2021-12-13T17:57:14.764Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file filter_design_document.json for db whisk_local_subjects
[2021-12-13T17:57:14.765Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file namespace_throttlings_design_document_for_subjects_db.json for db whisk_local_subjects
[2021-12-13T17:57:14.817Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file whisks_design_document_for_entities_db_v2.1.0.json for db whisk_local_whisks
[2021-12-13T17:57:14.818Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file filter_design_document.json for db whisk_local_whisks
[2021-12-13T17:57:14.850Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file whisks_design_document_for_activations_db_v2.1.0.json for db whisk_local_activations
[2021-12-13T17:57:14.851Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file whisks_design_document_for_activations_db_filters_v2.1.1.json for db whisk_local_activations
[2021-12-13T17:57:14.851Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file filter_design_document.json for db whisk_local_activations
[2021-12-13T17:57:14.852Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file activations_design_document_for_activations_db.json for db whisk_local_activations
[2021-12-13T17:57:14.853Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Creating view doc from file logCleanup_design_document_for_activations_db.json for db whisk_local_activations
[2021-12-13T17:57:14.875Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] Invalidated config cached
[2021-12-13T17:57:14.875Z] [INFO] [#tid_sid_couchDB] [CouchDBLauncher] CouchDB started successfully at http://172.17.0.1:5984/_utils . Username: [whisk_admin], Password: [some_passw0rd]
[2021-12-13T17:57:14.908Z] [INFO] [#tid_sid_userevents] [UserEventLauncher] Starting User Events: 3235
[2021-12-13T17:57:14.908Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker pull openwhisk/user-events:nightly (timeout: 10 minutes) [marker:invoker_docker.pull_start:32]
[2021-12-13T17:57:40.962Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:26086:26054]
[2021-12-13T17:57:40.963Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-user-events --network bridge -e KAFKA_HOSTS=172.17.0.1:9091 -p 3235:9095 openwhisk/user-events:nightly (timeout: 1 minute) [marker:invoker_docker.run_start:26087]
[2021-12-13T17:57:44.052Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.run_finish:29176:3089]
[2021-12-13T17:57:44.052Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} 5c255d42cc7684a6c0d8d347b6d66f4741842ec75269feabd89ff2f714aaf6ae (timeout: 1 minute) [marker:invoker_docker.inspect_start:29176]
[2021-12-13T17:57:44.105Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:29229:53]
[2021-12-13T17:57:44.106Z] [INFO] [#tid_sid_userevents] [UserEventLauncher] Starting Prometheus at 9090
[2021-12-13T17:57:44.108Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker pull prom/prometheus:v2.5.0 (timeout: 10 minutes) [marker:invoker_docker.pull_start:29231]
[2021-12-13T17:57:44.436Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.pull_finish:29560:328]
[2021-12-13T17:57:44.437Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker run -d --name whisk-prometheus --network bridge -p 9090:9090 -v /u/hwen5/.openwhisk/standalone/server-3233/prometheus:/prometheus -v /u/hwen5/.openwhisk/standalone/server-3233/tmp/prometheus:/etc/prometheus/ prom/prometheus:v2.5.0 (timeout: 1 minute) [marker:invoker_docker.run_start:29561]
[2021-12-13T17:57:46.035Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.run_finish:31158:1597]
[2021-12-13T17:57:46.035Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker inspect --format {{.NetworkSettings.Networks.bridge.IPAddress}} 5b922a3af570affee928dc3e4f55e61aa41f3d3c9b587a1367a0041fe790fa02 (timeout: 1 minute) [marker:invoker_docker.inspect_start:31159]
[2021-12-13T17:57:46.553Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] [marker:invoker_docker.inspect_finish:31677:518]
[2021-12-13T17:57:46.555Z] [INFO] [#tid_sid_userevents] [StandaloneDockerClient] running /usr/bin/docker rm -f 5b922a3af570affee928dc3e4f55e61aa41f3d3c9b587a1367a0041fe790fa02 (timeout: 1 minute) [marker:invoker_docker.rm_start:31679]
[2021-12-13T17:57:46.565Z] [ERROR] [#tid_sid_userevents] [StandaloneOpenWhisk] Error starting Kafkajava.lang.IllegalArgumentException: requirement failed: ContainerIp must not be empty
[2021-12-13T17:57:46.569Z] [ERROR] Thread Thread[main,5,main] died
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:47)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:86)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.IllegalArgumentException: requirement failed: ContainerIp must not be empty
at scala.Predef$.require(Predef.scala:281)
at org.apache.openwhisk.core.containerpool.ContainerAddress.(Container.scala:50)
at org.apache.openwhisk.core.containerpool.docker.DockerClient.$anonfun$inspectIPAddress$1(DockerClient.scala:165)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
```

## Additional information you deem important:
I tried this in both a RedHat Linux server and a Fedora virtual machine. Both report the same issue.

Contributor guide

Open the contributing guide

Research direction

Start by running `./gradlew :core:standalone:build` and `java -jar bin/openwhisk-standalone.jar --all --no-ui`. Compare the reported Kafka failure with the `KafkaLauncher` output and the stated Docker environment; done means the standalone JAR starts with Kafka without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kafka, scala
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.