alibaba / alibaba/DataX

java.lang.OutOfMemoryError: Failed to create a thread

Open
#1,721 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

在16C32G配置的虚机服务器上,通过java应用维护了并行度为9的线程池,线程池中总共有63个任务,并行度为9代表同时会启动9个datax脚本,脚本为:`python /app/datax/datax/bin/datax.py ...`。
当线程池中的任务执行完一半时,剩下的正在执行的任务抛出日志中抛出如下错误,但是此时服务器的内存剩余充足(如下所示),datax.py的JVM参数设置:`DEFAULT_JVM = "-Xms256M -Xmx512M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=%s/log" % (DATAX_HOME)`,通过各自方式排查,均找不到解决方案,请大佬分析一下该问题原因所在,多谢。
```shell
$ free -m
total used free shared buffers cached
Mem: 32097 12850 19247 1 806 7877
-/+ buffers/cache: 4166 27930
Swap: 8199 49 8150
```
json文件的setting属性配置参数如下:
```json
"setting": {
"speed": {
"channel": "10",
"record":-1,
"byte":-1,
"batchSize":4000
}
}
```

日志错误信息:
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2023-03-17 09:54:25.128 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-03-17 09:54:25.129 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-03-17 09:54:25.130 [job-0] INFO JobContainer - Job set Channel-Number to 10 channels.
2023-03-17 09:54:25.134 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2023-03-17 09:54:25.135 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [1] tasks.
2023-03-17 09:54:25.165 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-03-17 09:54:25.176 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-03-17 09:54:25.178 [job-0] INFO JobContainer - Running by standalone Mode.
2023-03-17 09:54:25.381 [job-0] ERROR JobContainer - Exception when job run
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
at java.lang.Thread.start(Thread.java:988) ~[na:2.9 (11-06-2019)]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:968) ~[na:1.8.0]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) ~[na:1.8.0]
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.startAllTaskGroup(ProcessInnerScheduler.java:30) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:55) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-03-17 09:54:25.404 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.startAllTaskGroup and the ThreadPoolExecutor/Thread.start frames in the supplied stack trace. Reproduce the failure with the reported parallel job setup, then determine the relevant thread-creation constraint and document a scoped fix or configuration change with evidence that the remaining tasks start successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.