alibaba / alibaba/DataX

fastjson2 java.lang.OutOfMemoryError 序列化超过64Mb的数据直接异常退出,能不能抛出异常但是不退出程序

Open
#1,939 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

2023-10-01 16:20:35.438 [job-0] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)
- java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)

at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39)
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99)
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
at com.alibaba.datax.core.Engine.start(Engine.java:86)
at com.alibaba.datax.core.Engine.entry(Engine.java:168)
at com.alibaba.datax.core.Engine.main(Engine.java:201)
Caused by: java.lang.OutOfMemoryError
at com.alibaba.fastjson2.JSONWriterUTF16.ensureCapacity(JSONWriterUTF16.java:1059)
at com.alibaba.fastjson2.JSONWriterUTF16JDK8.writeString(JSONWriterUTF16JDK8.java:176)
at com.alibaba.fastjson2.writer.ObjectWriterImplString.write(ObjectWriterImplString.java:18)
at com.alibaba.fastjson2.writer.OWG_3_3_StringColumn.write(Unknown Source)
at com.alibaba.fastjson2.writer.ObjectWriterImplList.write(ObjectWriterImplList.java:373)
at com.alibaba.fastjson2.writer.ObjectWriterImplMap.write(ObjectWriterImplMap.java:548)
at com.alibaba.fastjson2.JSON.toJSONString(JSON.java:2261)
at com.alibaba.datax.core.transport.record.DefaultRecord.toString(DefaultRecord.java:71)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriterSlaveProxyOld.write(OtsWriterSlaveProxyOld.java:107)
at com.alibaba.datax.plugin.writer.otswriter.OtsWriter$Task.startWrite(OtsWriter.java:105)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:750)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with com.alibaba.datax.core.transport.record.DefaultRecord.toString and the OtsWriterSlaveProxyOld.write call shown in the stack trace, then trace how the OutOfMemoryError reaches Engine and JobContainer. Determine the expected failure boundary and verify that oversized serialization reports an exception without terminating the process.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.