apache / apache/iotdb

[Bug] Apache IoTDB confignode - unable to start pod on OpenShift

Open
#13,484 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

1.3.2-confignode, 1.3.2-standalone, OS: Windows and OpenShift 4.16

### Describe the bug and provide the minimal reproduce step

ApacheIoTDB is unable to start in OpenShift due to error on creation of folder

```
----------------------------
Starting IoTDB ConfigNode
----------------------------
Notice: in some systems, ConfigNode must run in sudo mode to write data. The process may fail.
Checking whether the ports are already occupied...
Warning: If you do not use sudo, the checking may not detect all the occupied ports.
WARN:
WARN: the value of net.core.somaxconn (=4096) is too small, please set it to a larger value using the following command.
WARN: sudo sysctl -w net.core.somaxconn=65535
WARN: The original net.core.somaxconn value will be set back when the os reboots.
WARN:
setting local JMX...
ConfigNode on heap memory size = 6552MB, off heap memory size = 1640MB
If you want to change this configuration, please check conf/confignode-env.sh.
2024-09-11 12:56:25,088 [main] INFO o.a.i.c.c.ConfigNodeDescriptor:107 - Start to read config file file:/iotdb/sbin/../conf/iotdb-common.properties
2024-09-11 12:56:25,094 [main] INFO o.a.i.c.c.ConfigNodeDescriptor:129 - start reading ConfigNode conf file: file:/iotdb/sbin/../conf/iotdb-confignode.properties
2024-09-11 12:56:25,166 [main] INFO o.a.i.c.s.ConfigNode:106 - IoTDB-ConfigNode environment variables:
CONFIGNODE_HOME=/iotdb/sbin/..;
CONFIGNODE_CONF=/iotdb/sbin/../conf;
2024-09-11 12:56:25,167 [main] INFO o.a.i.c.s.ConfigNode:110 - IoTDB-ConfigNode default charset is: UTF-8
2024-09-11 12:56:25,169 [main] INFO o.a.i.c.s.ConfigNodeCommandLine:68 - Running mode -s
2024-09-11 12:56:25,170 [main] INFO o.a.i.c.s.ConfigNodeCommandLine:72 - Starting IoTDB 1.3.1 (Build: 214695d)
2024-09-11 12:56:25,173 [main] INFO o.a.i.c.c.ConfigNodeStartupCheck:65 - configNode port check successful.
2024-09-11 12:56:25,174 [main] INFO o.a.i.c.s.StartupChecks:59 - Start JMX locally.
2024-09-11 12:56:25,175 [main] INFO o.a.i.c.s.StartupChecks:45 - JDK version is 11.
2024-09-11 12:56:25,175 [main] INFO o.a.i.c.c.ConfigNodeStartupCheck:201 - Make dirs: /iotdb/sbin/../data/confignode/system
2024-09-11 12:56:25,176 [main] INFO o.a.i.c.c.ConfigNodeStartupCheck:201 - Make dirs: /iotdb/sbin/../data/confignode/consensus
2024-09-11 12:56:25,183 [main] INFO o.a.i.c.s.ConfigNode:118 - Activating IoTDB-ConfigNode...
2024-09-11 12:56:25,184 [main] INFO o.a.i.c.s.RegisterManager:54 - The JMX ServerService service is started successfully, which takes 0 ms.
2024-09-11 12:56:25,496 [main] INFO o.a.i.c.s.ConfigNode:251 - Successfully setup internal services.
2024-09-11 12:56:25,588 [main] INFO o.a.i.c.a.a.BasicAuthorizer$InstanceHolder:104 - Authorizer provider class: org.apache.iotdb.commons.auth.authorizer.LocalFileAuthorizer
2024-09-11 12:56:25,592 [main] INFO o.a.i.c.a.u.LocalFileUserAccessor:399 - user info dir /iotdb/data/confignode/system/users is created
2024-09-11 12:56:25,688 [main] INFO o.a.i.c.a.u.BasicUserManager:129 - Admin initialized
2024-09-11 12:56:25,690 [main] INFO o.a.i.c.a.r.LocalFileRoleAccessor:270 - role info dir /iotdb/data/confignode/system/roles is created
2024-09-11 12:56:25,723 [main] ERROR o.a.i.c.s.ConfigNode:299 - Can't start ConfigNode consensus group!
java.io.IOException: Cannot create directory '/iotdb/sbin/../ext/udf/tmp'.
at org.apache.commons.io.FileUtils.mkdirs(FileUtils.java:2287)
at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1380)
at org.apache.iotdb.commons.file.SystemFileFactory.makeDirIfNecessary(SystemFileFactory.java:82)
at org.apache.iotdb.commons.udf.service.UDFExecutableManager.setupAndGetInstance(UDFExecutableManager.java:48)
at org.apache.iotdb.confignode.persistence.UDFInfo.(UDFInfo.java:78)
at org.apache.iotdb.confignode.manager.ConfigManager.(ConfigManager.java:261)
at org.apache.iotdb.confignode.service.ConfigNode.initConfigManager(ConfigNode.java:297)
at org.apache.iotdb.confignode.service.ConfigNode.active(ConfigNode.java:127)
at org.apache.iotdb.confignode.service.ConfigNodeCommandLine.activeConfigNodeInstance(ConfigNodeCommandLine.java:97)
at org.apache.iotdb.confignode.service.ConfigNodeCommandLine.run(ConfigNodeCommandLine.java:79)
at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
at org.apache.iotdb.confignode.service.ConfigNode.main(ConfigNode.java:114)
2024-09-11 12:56:25,724 [main] INFO o.a.i.c.s.ConfigNode:409 - Deactivating IoTDB-ConfigNode...
2024-09-11 12:56:25,724 [main] INFO o.a.i.c.s.RegisterManager:73 - deregister all service.
2024-09-11 12:56:25,724 [main] INFO o.a.i.c.s.ConfigNode:415 - IoTDB-ConfigNode is deactivated.
2024-09-11 12:56:25,759 [Thread-0] INFO o.a.i.c.s.ConfigNodeShutdownHook:49 - ConfigNode exiting...
Exception in thread "Thread-0" java.lang.NullPointerException
at org.apache.iotdb.confignode.service.ConfigNodeShutdownHook.run(ConfigNodeShutdownHook.java:51)
```

### What did you expect to see?

Pod starting

### What did you see instead?

Pod crashing with error "CrashLoopBackOff"

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the OpenShift reproduction and the stack trace at UDFExecutableManager.setupAndGetInstance, which fails while creating /iotdb/sbin/../ext/udf/tmp. Inspect the ConfigNode startup path around ConfigNodeStartupCheck and ConfigNode, then verify that the pod can start and no longer enters CrashLoopBackOff.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.