apache / apache/iotdb

[Bug] IotDB pods crash with OOM because we calculate the memory based on the node not pod resources

Open
#17,764 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.

### Version

latest

### Describe the bug and provide the minimal reproduce step

Start IOTDB datanode and confignode pods with memory limits, for example 8 GB, and allocate 8 GB of resources to each pod.

Image

Pods keep crashing with OOM errors because the JVM is trying to allocate 16 GB of memory.

### What did you expect to see?

```
# When running in a container/pod, use cgroup memory limit instead of host memory
if [ -f /sys/fs/cgroup/memory.max ]; then
# cgroup v2
cgroup_mem=`cat /sys/fs/cgroup/memory.max`
if [ "$cgroup_mem" != "max" ]; then
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
elif [ -f /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
# cgroup v1
cgroup_mem=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes`
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
```
8GB

I would expect the memory to be auto-calculated based on the pod resources (8 GB), not the node resources (32 GB).
```
# scripts\conf\datanode-env.sh
system_memory_in_mb=`free -m | sed -n '2p' | awk '{print \$2}'` returns 32 GB.
```

### What did you see instead?

32 GB and a lot of pod restarts

### Anything else?

```
# iotdb\WORKING_CONFIGS.md

## 2) JVM Memory (Linux)

Edit these files:
- conf/confignode-env.sh
- conf/datanode-env.sh

Set MEMORY_SIZE explicitly to avoid auto-sizing surprises.

### ConfigNode memory

```bash
# conf/confignode-env.sh
MEMORY_SIZE=2G
```

### DataNode memory

```bash
# conf/datanode-env.sh
MEMORY_SIZE=8G
``````

Why are there no env varibales for this setting?
Do you expect the clouad env to manualy go and change this limit?

This is a hack, and we should not have to do this in a pod
```
- IOTDB_JMX_OPTS=-Xmx4G
```

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with conf/confignode-env.sh and conf/datanode-env.sh, especially the system_memory_in_mb calculation using free -m. Compare the cgroup v2 and v1 memory-limit cases described in the issue, then verify that a pod limited to 8 GB sizes the JVM from that limit rather than the host's memory and avoids OOM restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
databases, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.