[Bug] IotDB pods crash with OOM because we calculate the memory based on the node not pod resources
- Dominant language
- Java
- Stars
- 6.4k
- Forks
- 1.2k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 115
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
latest
### Describe the bug and provide the minimal reproduce step
Start IOTDB datanode and confignode pods with memory limits, for example 8 GB, and allocate 8 GB of resources to each pod.
Pods keep crashing with OOM errors because the JVM is trying to allocate 16 GB of memory.
### What did you expect to see?
```
# When running in a container/pod, use cgroup memory limit instead of host memory
if [ -f /sys/fs/cgroup/memory.max ]; then
# cgroup v2
cgroup_mem=`cat /sys/fs/cgroup/memory.max`
if [ "$cgroup_mem" != "max" ]; then
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
elif [ -f /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
# cgroup v1
cgroup_mem=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes`
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
```
8GB
I would expect the memory to be auto-calculated based on the pod resources (8 GB), not the node resources (32 GB).
```
# scripts\conf\datanode-env.sh
system_memory_in_mb=`free -m | sed -n '2p' | awk '{print \$2}'` returns 32 GB.
```
### What did you see instead?
32 GB and a lot of pod restarts
### Anything else?
```
# iotdb\WORKING_CONFIGS.md
## 2) JVM Memory (Linux)
Edit these files:
- conf/confignode-env.sh
- conf/datanode-env.sh
Set MEMORY_SIZE explicitly to avoid auto-sizing surprises.
### ConfigNode memory
```bash
# conf/confignode-env.sh
MEMORY_SIZE=2G
```
### DataNode memory
```bash
# conf/datanode-env.sh
MEMORY_SIZE=8G
``````
Why are there no env varibales for this setting?
Do you expect the clouad env to manualy go and change this limit?
This is a hack, and we should not have to do this in a pod
```
- IOTDB_JMX_OPTS=-Xmx4G
```
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Research direction
Start with conf/confignode-env.sh and conf/datanode-env.sh, especially the system_memory_in_mb calculation using free -m. Compare the cgroup v2 and v1 memory-limit cases described in the issue, then verify that a pod limited to 8 GB sizes the JVM from that limit rather than the host's memory and avoids OOM restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, shell
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100