java.lang.OutOfMemoryError: Failed to create a thread
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
在16C32G配置的虚机服务器上,通过java应用维护了并行度为9的线程池,线程池中总共有63个任务,并行度为9代表同时会启动9个datax脚本,脚本为:`python /app/datax/datax/bin/datax.py ...`。
当线程池中的任务执行完一半时,剩下的正在执行的任务抛出日志中抛出如下错误,但是此时服务器的内存剩余充足(如下所示),datax.py的JVM参数设置:`DEFAULT_JVM = "-Xms256M -Xmx512M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=%s/log" % (DATAX_HOME)`,通过各自方式排查,均找不到解决方案,请大佬分析一下该问题原因所在,多谢。
```shell
$ free -m
total used free shared buffers cached
Mem: 32097 12850 19247 1 806 7877
-/+ buffers/cache: 4166 27930
Swap: 8199 49 8150
```
json文件的setting属性配置参数如下:
```json
"setting": {
"speed": {
"channel": "10",
"record":-1,
"byte":-1,
"batchSize":4000
}
}
```
日志错误信息:
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2023-03-17 09:54:25.128 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-03-17 09:54:25.129 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-03-17 09:54:25.130 [job-0] INFO JobContainer - Job set Channel-Number to 10 channels.
2023-03-17 09:54:25.134 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2023-03-17 09:54:25.135 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [1] tasks.
2023-03-17 09:54:25.165 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-03-17 09:54:25.176 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-03-17 09:54:25.178 [job-0] INFO JobContainer - Running by standalone Mode.
2023-03-17 09:54:25.381 [job-0] ERROR JobContainer - Exception when job run
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
at java.lang.Thread.start(Thread.java:988) ~[na:2.9 (11-06-2019)]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:968) ~[na:1.8.0]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) ~[na:1.8.0]
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.startAllTaskGroup(ProcessInnerScheduler.java:30) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:55) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-03-17 09:54:25.404 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.startAllTaskGroup and the ThreadPoolExecutor/Thread.start frames in the supplied stack trace. Reproduce the failure with the reported parallel job setup, then determine the relevant thread-creation constraint and document a scoped fix or configuration change with evidence that the remaining tasks start successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100