apache / apache/incubator-xtable
no rows in Trino when pyspark xtable source from delta and write into Hudi and Iceberg with S3 + HMS
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
I wrote 6 rows into delta on s3 and then used xtable to covert the data to iceberg and hudi. Next I used run_sync_tool.sh to register the hudi files in HMS and then query it into Trino. No data in Trino.
I don't see any errors.
----
xtable looks good
```
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadataWriter:451 - Initializing COLUMN_STATS index in metadata table took 145 in ms
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:395 - Updating at 20240604232400230 from Commit/null. #partitions_updated=6, #files_added=6
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:942 - Loading latest file slices for metadata table partition column_stats
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (column_stats)
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:942 - Loading latest file slices for metadata table partition files
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files)
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadataWriter:1103 - New commit at 20240604232400230 being applied to MDT.
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.CleanerUtils:221 - Cleaned failed attempts if any
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[00000000000000011__deltacommit__COMPLETED__20240604232709963]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.client.BaseHoodieWriteClient:945 - Generate a new instant time: 20240604232400230 action: deltacommit
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:202 - Creating a new instant [==>20240604232400230__deltacommit__REQUESTED]
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.requested
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.inflight
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[==>20240604232400230__deltacommit__INFLIGHT__20240604232710097]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.async.AsyncCleanerService:63 - The HoodieWriteClient is not configured to auto & async clean. Async clean service will not start.
2024-06-04 23:27:10 INFO org.apache.hudi.async.AsyncArchiveService:61 - The HoodieWriteClient is not configured to auto & async archive. Async archive service will not start.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor:74 - Total update fileIDs 3, total inserts 0 for commit 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:165 - Creating InMemory based view for basePath s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files)
# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/files/files-0000-0_0-0-0_00000000000000010.hfile.marker.APPEND
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/files/files-0000-0_0-0-0_00000000000000010.hfile.marker.APPEND in 17 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath files filePath files/.files-0000-0_00000000000000010.log.2_0-0-0, took 301 ms.
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (column_stats)
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0001-0_0-0-0_00000000000000011.hfile.marker.APPEND
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0001-0_0-0-0_00000000000000011.hfile.marker.APPEND in 12 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0001-0_00000000000000011.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0001-0_00000000000000011.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath column_stats filePath column_stats/.col-stats-0001-0_00000000000000011.log.2_0-0-0, took 43 ms.
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:197 - Creating Marker Path=s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0000-0_0-0-0_00000000000000011.hfile.marker.APPEND
2024-06-04 23:27:10 INFO org.apache.hudi.table.marker.DirectWriteMarkers:202 - [direct] Created marker file s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230/column_stats/col-stats-0000-0_0-0-0_00000000000000011.hfile.marker.APPEND in 8 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:215 - Building HoodieLogFormat Writer
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder:270 - HoodieLogFile on path s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0000-0_00000000000000011.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter:122 - Append not supported.. Rolling over to HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/column_stats/.col-stats-0000-0_00000000000000011.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.io.HoodieAppendHandle:438 - AppendHandle for partitionPath column_stats filePath column_stats/.col-stats-0000-0_00000000000000011.log.2_0-0-0, took 24 ms.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseCommitActionExecutor:182 - Auto commit enabled: Committing 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.CommitUtils:121 - Creating metadata for UPSERT_PREPPED numWriteStats:3 numReplaceFileIds:0
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:205 - Committing 20240604232400230, action Type deltacommit
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:211 - Committing 20240604232400230, action Type deltacommit
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:220 - Marking instant complete [==>20240604232400230__deltacommit__INFLIGHT]
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit.inflight
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/metadata/.hoodie/20240604232400230.deltacommit
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:224 - Completed [==>20240604232400230__deltacommit__INFLIGHT]
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor:219 - Committed 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.fs.FSUtils:716 - Removed directory at s3a://warehouse/people/.hoodie/metadata/.hoodie/.temp/20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:220 - Marking instant complete [==>20240604232400230__replacecommit__INFLIGHT]
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:615 - Checking for file exists ?s3a://warehouse/people/.hoodie/20240604232400230.replacecommit.inflight
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:623 - Create new file for toInstant ?s3a://warehouse/people/.hoodie/20240604232400230.replacecommit
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:224 - Completed [==>20240604232400230__replacecommit__INFLIGHT]
2024-06-04 23:27:10 INFO org.apache.hudi.client.BaseHoodieWriteClient:238 - Committed 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View Manager with storage type :MEMORY
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating in-memory based Table View
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.FileSystemViewManager:165 - Creating InMemory based view for basePath s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 9 ms to read 1 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:109 - Load all partitions and files into file system view in advance.
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:927 - Loading latest merged file slices for metadata table partition files
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files)
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 3 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 96 ms
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:334 - Listed partitions from metadata: #partitions=5
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 0 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 18 ms
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:397 - Listed files in 5 partitions from metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieTableMetadataUtil:927 - Loading latest merged file slices for metadata table partition files
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building file system view for partition (files)
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:451 - Opened metadata base file from s3a://warehouse/people/.hoodie/metadata/files/files-0000-0_0-0-0_00000000000000010.hfile at instant 00000000000000010 in 0 ms
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:293 - Reading a delete block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.1_0-0-0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieLogFormatReader:115 - Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:250 - Scanning log file HoodieLogFile{pathStr='s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0', fileLen=-1}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:286 - Reading a data block from file s3a://warehouse/people/.hoodie/metadata/files/.files-0000-0_00000000000000010.log.2_0-0-0 at instant 20240604232400230
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:402 - Merging the final data blocks
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader:818 - Number of remaining logblocks to merge 1
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:206 - Number of log files scanned => 2
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:207 - MaxMemoryInBytes allowed for compaction => 1073741824
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:208 - Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:209 - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 5568
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:210 - Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner:211 - Size of file spilled to disk => 0
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata:501 - Opened 2 metadata log files (dataset instant=20240604232400230, metadata instant=20240604232400230) in 19 ms
2024-06-04 23:27:10 INFO org.apache.hudi.metadata.BaseTableMetadata:334 - Listed partitions from metadata: #partitions=5
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=DFW, retaining latest 0 commits.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=DFW
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=NYC, retaining latest 0 commits.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=NYC
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=ORD, retaining latest 0 commits.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=ORD
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=SEA, retaining latest 0 commits.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=SEA
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:317 - Cleaning city=SFO, retaining latest 0 commits.
2024-06-04 23:27:10 INFO org.apache.hudi.table.action.clean.CleanPlanner:501 - 0 patterns used to delete in partition path:city=SFO
2024-06-04 23:27:10 WARN org.apache.hudi.client.HoodieTimelineArchiver:83 - The configured archival configs hoodie.keep.min.commits=0 is more aggressive than the cleaning configs as the earliest commit to retain is [20240604232400230__replacecommit__COMPLETED__20240604232710563]. Adjusted the archival configs to be hoodie.keep.min.commits=2 and hoodie.keep.max.commits=3
2024-06-04 23:27:10 WARN org.apache.hudi.client.HoodieTimelineArchiver:95 - Cleaning configs: hoodie.cleaner.policy=KEEP_LATEST_BY_HOURS hoodie.cleaner.hours.retained=168
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/metadata/.hoodie/hoodie.properties
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://warehouse/people/.hoodie/metadata
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__deltacommit__COMPLETED__20240604232710519]}
2024-06-04 23:27:10 INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 0 ms to read 0 instants, 0 replaced file groups
2024-06-04 23:27:10 INFO org.apache.hudi.common.util.ClusteringUtils:147 - Found 0 files in pending clustering operations
2024-06-04 23:27:10 INFO org.apache.hudi.client.HoodieTimelineArchiver:514 - Not archiving as there is no compaction yet on the metadata table
2024-06-04 23:27:10 INFO org.apache.hudi.client.HoodieTimelineArchiver:177 - No Instants to archive
2024-06-04 23:27:10 INFO org.apache.iceberg.hadoop.HadoopTableOperations:164 - Committed a new metadata file s3a://warehouse/people/metadata/v1.metadata.json
2024-06-04 23:27:10 INFO org.apache.iceberg.SnapshotScan:120 - Scanning empty table s3a://warehouse/people
2024-06-04 23:27:11 INFO org.apache.iceberg.SnapshotProducer:414 - Committed snapshot 369315014296903406 (BaseOverwriteFiles)
2024-06-04 23:27:11 INFO org.apache.iceberg.metrics.LoggingMetricsReporter:38 - Received metrics report: CommitReport{tableName=s3a://warehouse/people, snapshotId=369315014296903406, sequenceNumber=1, operation=overwrite, commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.085428125S, count=1}, attempts=CounterResult{unit=COUNT, value=1}, addedDataFiles=CounterResult{unit=COUNT, value=6}, removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=6}, addedDeleteFiles=null, addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null, removedDeleteFiles=null, removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null, totalDeleteFiles=CounterResult{unit=COUNT, value=0}, addedRecords=CounterResult{unit=COUNT, value=6}, removedRecords=null, totalRecords=CounterResult{unit=COUNT, value=6}, addedFilesSizeInBytes=CounterResult{unit=BYTES, value=7621}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{unit=BYTES, value=7621}, addedPositionalDeletes=null, removedPositionalDeletes=null, totalPositionalDeletes=CounterResult{unit=COUNT, value=0}, addedEqualityDeletes=null, removedEqualityDeletes=null, totalEqualityDeletes=CounterResult{unit=COUNT, value=0}}, metadata={iceberg-version=Apache Iceberg 1.4.2 (commit f6bb9173b13424d77e7ad8439b5ef9627e530cb2)}}
2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:124 - Expiring snapshots older than: 2024-05-28T23:27:11.069+00:00 (1716938831069)
2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:309 - Committed snapshot changes
2024-06-04 23:27:11 INFO org.apache.iceberg.RemoveSnapshots:328 - Cleaning up expired files (local, incremental)
2024-06-04 23:27:11 INFO org.apache.iceberg.hadoop.HadoopTableOperations:164 - Committed a new metadata file s3a://warehouse/people/metadata/v2.metadata.json
2024-06-04 23:27:11 INFO org.apache.xtable.conversion.ConversionController:133 - Sync is successful for the following formats HUDI,ICEBERG
```
hudi-hive-sync/run_sync_tool.sh looks good
```
root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh --metastore-uris 'thrift://hive-metastore:9083' --partitioned-by city --base-path 's3a://warehouse/people' --database hudi_db --table people --sync-mode hms
ls: cannot access '/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-*.jar': No such file or directory
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop: org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 23:30:49,554 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-04 23:30:49,765 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 23:30:49,799 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 23:30:50,319 INFO [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 23:30:50,362 INFO [main] table.HoodieTableConfig (HoodieTableConfig.java:(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:30:50,374 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:30:50,374 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 23:30:50,392 INFO [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:(172)) - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:30:50,427 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083
2024-06-04 23:30:50,435 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1
2024-06-04 23:30:50,445 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore.
2024-06-04 23:30:50,473 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-04 23:30:50,473 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-04 23:30:50,659 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=ORD/part-00003-6b980503-b6c2-42cb-aaa5-2d2feacda190.c000.snappy.parquet_20240604232400230_hudiext
2024-06-04 23:30:50,675 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:hasOperationField(470)) - Failed to read operation field from avro schema (Failed to read data schema)
2024-06-04 23:30:50,698 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncFirstTime(321)) - Sync table people for the first time.
2024-06-04 23:30:50,747 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(265)) - Last commit time synced was found to be null, last commit completion time is found to be null
2024-06-04 23:30:50,748 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(279)) - Sync all partitions given the last commit time synced is empty or before the start of the active timeline. Listing all partitions in s3a://warehouse/people, file system: S3AFileSystem{uri=s3a://warehouse, workingDir=s3a://warehouse/user/root, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='NONE', metastore=NullMetadataStore, authoritative=false, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25, available=25, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@66813e6e[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], DirectoryMarkerRetention{policy='delete'}, statistics {31810 bytes read, 0 bytes written, 7 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=3ce354be-808a-4240-813d-fc67413cd80a-warehouse} {fsURI=s3a://warehouse/people/.hoodie} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=6} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=1} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=4} {object_continue_list_requests=0} {object_metadata_requests=6} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {s3guard_metadatastore_put_path_request=5} {s3guard_metadatastore_initialization=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {S3guard_metadatastore_put_path_latencyNumOps=0} {S3guard_metadatastore_put_path_latency50thPercentileLatency=0} {S3guard_metadatastore_put_path_latency75thPercentileLatency=0} {S3guard_metadatastore_put_path_latency90thPercentileLatency=0} {S3guard_metadatastore_put_path_latency95thPercentileLatency=0} {S3guard_metadatastore_put_path_latency99thPercentileLatency=0} {stream_read_fully_operations=0} {stream_opened=3} {stream_bytes_skipped_on_seek=0} {stream_closed=3} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=31810} {stream_read_operations_incomplete=5} {stream_bytes_discarded_in_abort=0} {stream_close_operations=3} {stream_read_operations=33} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
2024-06-04 23:30:50,780 INFO [main] table.HoodieTableConfig (HoodieTableConfig.java:(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:30:50,906 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(298)) - Sync complete for people
2024-06-04 23:30:50,909 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0
```
trino should show rows
```
trino@trino:/$ trino
trino> select * from hudi.hudi_db.people;
-> select * from delta.delta_db.people;
-> select * from iceberg.iceberg_db.people;
_hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_part>
---------------------+----------------------+--------------------+------------->
(0 rows)
Query 20240604_233108_00000_ujp5k, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
1.36 [0 rows, 0B] [0 rows/s, 0B/s]
id | name | age | city | create_ts
----+---------+-----+------+---------------------
5 | Bob | 28 | SEA | 2023-09-23 00:00:00
6 | Charlie | 31 | DFW | 2023-08-29 00:00:00
4 | Andrew | 40 | NYC | 2023-10-28 00:00:00
1 | John | 25 | NYC | 2023-09-28 00:00:00
3 | Michael | 35 | ORD | 2023-09-28 00:00:00
2 | Emily | 30 | SFO | 2023-09-28 00:00:00
(6 rows)
Query 20240604_233113_00001_ujp5k, FINISHED, 1 node
Splits: 6 total, 6 done (100.00%)
0.83 [6 rows, 7.44KB] [7 rows/s, 8.96KB/s]
id | name | age | city | create_ts
----+---------+-----+------+---------------------
1 | John | 25 | NYC | 2023-09-28 00:00:00
4 | Andrew | 40 | NYC | 2023-10-28 00:00:00
2 | Emily | 30 | SFO | 2023-09-28 00:00:00
3 | Michael | 35 | ORD | 2023-09-28 00:00:00
6 | Charlie | 31 | DFW | 2023-08-29 00:00:00
5 | Bob | 28 | SEA | 2023-09-23 00:00:00
(6 rows)
Query 20240604_233114_00002_ujp5k, FINISHED, 1 node
Splits: 6 total, 6 done (100.00%)
0.95 [6 rows, 7.44KB] [6 rows/s, 7.8KB/s]
```
### Are you willing to submit PR?
- [X] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the S3/HMS workflow from the report: write rows to Delta, convert them to Hudi and Iceberg, run run_sync_tool.sh, and query the registered Hudi table in Trino. Compare the registered table and generated files with the six source rows; done means the rows are visible in Trino without errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100