[SUPPORT]Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1}

操作步骤:

1.Flink 批处理将Hive历史数据迁移到Hudi表中；
2.Flink 接入Kafka数据写入HDFS；
3.Hudi 采用的是离线合并；
4.Spark每小时读取Hudi全量数据。

**出现的问题**

Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1}

**运行环境**

* Hudi version : 0.14.1

* Spark version : 3.3.2

* Hive version : 11.0-cdh5.13.2

* Hadoop version : 2.6.0-cdh5.13.2

* Storage (HDFS/S3/GCS..) : HDFS

* Running on Docker? (yes/no) : no

**具体的报错**
2024-12-28 16:30:28,806 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Failed to trigger checkpoint for job dbb88271f16bc32329acd07b5da75321 because Some tasks of the job have already finished and checkpointing with finished tasks is not enabled. Failure reason: Not all required tasks are currently running.
2024-12-28 16:30:37,983 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task (38/60) (8d6d4c0f31ae7ce0af767c5dd919b407) switched from RUNNING to FAILED on container_1724338424009_17149855_01_011191 @ bdc-dn3077.bigdata.jxtelecom.cn (dataPort=38642).
org.apache.hudi.exception.HoodieException: Exception when reading log file
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:414) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:220) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:201) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:117) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:76) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:466) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:210) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:147) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.sink.compact.CompactOperator.doCompaction(CompactOperator.java:142) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.sink.compact.CompactOperator.processElement(CompactOperator.java:130) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://bdccluster-1/telecom_data/CORE_MODEL/IP/BSS/DAY/INF_SS_OFFER_INST_D_HUDI_ALL_FLINK2/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1}
at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:486) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.(HoodieLogFileReader.java:121) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFormatReader.(HoodieLogFormatReader.java:64) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
Caused by: org.apache.hadoop.ipc.RemoteException: java.lang.ArrayIndexOutOfBoundsException

at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1413) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy35.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy36.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1235) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1222) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:264) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:203) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:484) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.(HoodieLogFileReader.java:121) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFormatReader.(HoodieLogFormatReader.java:64) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job dbb88271f16bc32329acd07b5da75321: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=59}]
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 0d75f8675b58c9bef24c60fb5f5d0a16_37.
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 62 tasks should be restarted to recover the failed task 0d75f8675b58c9bef24c60fb5f5d0a16_37.

Start at HoodieLogFileReader.getFSDataInputStream and trace the reported failure through DistributedFileSystem.open, DFSInputStream, and getBlockLocations. Reproduce the Hudi 0.14.1 Flink job against the listed Spark, Hive, Hadoop, and HDFS environment, then verify that the log file can be opened without the remote ArrayIndexOutOfBoundsException.

Written by the indexing model from the issue text.