fastjson2 java.lang.OutOfMemoryError 序列化超过64Mb的数据直接异常退出,能不能抛出异常但是不退出程序
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
2023-10-01 16:20:35.438 [job-0] ERROR Engine -
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)
- java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39)
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99)
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
at com.alibaba.datax.core.Engine.start(Engine.java:86)
at com.alibaba.datax.core.Engine.entry(Engine.java:168)
at com.alibaba.datax.core.Engine.main(Engine.java:201)
Caused by: java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with com.alibaba.datax.core.transport.record.DefaultRecord.toString and the OtsWriterSlaveProxyOld.write call shown in the stack trace, then trace how the OutOfMemoryError reaches Engine and JobContainer. Determine the expected failure boundary and verify that oversized serialization reports an exception without terminating the process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100