[SUPPORT] Failed to write with hudi 0.13.0

We were using emr version emr-6.11.0 , hudi version 0.13.0-amzn-0 , spark version = 3.3.2 , hive version = 3.1.3

We recently migrated from hudi 0.12.2 to 0.13.0. The main reason for migration was too many files in /metadata/.hoodie/ folder i.e. archival for metadata table was not triggering and hence more s3 listing cost.
We tried with migration for few tables to 0.13.0, its working fine for majority of tables but for few, we are getting below exception.
Because of below error we gfaced 2 issues.
1. Streaming query is failing and getting succeeded after multiple reties.
2. For the batches where we faced this issue, we observe data loss.

[2023-09-01 04:33:38,761] ERROR Got exception when reading log file (org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader)
java.lang.IllegalStateException: Block has already been inflated
at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.block.HoodieLogBlock.inflate(HoodieLogBlock.java:276) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.block.HoodieLogBlock.inflate(HoodieLogBlock.java:287) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.block.HoodieDataBlock.readRecordsFromBlockPayload(HoodieDataBlock.java:166) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordIterator(HoodieDataBlock.java:128) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.getRecordsIterator(AbstractHoodieLogRecordReader.java:805) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processDataBlock(AbstractHoodieLogRecordReader.java:630) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:672) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:367) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:223) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:198) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:114) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:73) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:464) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieMetadataLogRecordReader$Builder.build(HoodieMetadataLogRecordReader.java:218) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:546) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:447) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getOrCreateReaders$13(HoodieBackedTableMetadata.java:430) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_382]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:430) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$3(HoodieBackedTableMetadata.java:239) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0_382]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:237) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:152) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:339) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:150) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:69) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$15(AbstractTableFileSystemView.java:424) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_382]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:415) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFile(AbstractTableFileSystemView.java:712) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.handlers.BaseFileHandler.getLatestDataFile(BaseFileHandler.java:51) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.RequestHandler.lambda$registerDataFilesAPI$7(RequestHandler.java:258) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:526) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) ~[?:1.8.0_382]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) ~[?:1.8.0_382]
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:516) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
[2023-09-01 04:33:38,809] ERROR Got runtime exception servicing request partition=dt%3D2023-08-03&basepath=s3a%3A%2F%2F%2F%2F%2Fcourier_orders_snapshot_nrt&fileid=ec3dc84e-8a62-4144-a43b-19b239d695d0-0&lastinstantts=20230901034840623&timelinehash=00919203bc476a822f1f4495614a5cb9f9903e53b9a14b9d71fa53c86fef60e7 (org.apache.hudi.timeline.service.RequestHandler)
org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files in partition s3a://///courier_orders_snapshot_nrt/dt=2023-08-03 from metadata
at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:152) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:69) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$15(AbstractTableFileSystemView.java:424) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_382]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:415) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFile(AbstractTableFileSystemView.java:712) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.handlers.BaseFileHandler.getLatestDataFile(BaseFileHandler.java:51) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.RequestHandler.lambda$registerDataFilesAPI$7(RequestHandler.java:258) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:526) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) ~[?:1.8.0_382]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) ~[?:1.8.0_382]
at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:516) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[hudi-spark3-bundle_2.12-0.13.0-amzn-0.jar:0.13.0-amzn-0]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

Start with HoodieLogBlock.inflate and HoodieDataBlock.readRecordsFromBlockPayload, then trace the metadata path through HoodieMetadataLogRecordReader and HoodieBackedTableMetadata using the listed EMR, Hudi, and Spark versions. Reproduce the migration scenario and verify that streaming batches complete without the exception or reported data loss.

Written by the indexing model from the issue text.