apache / apache/hudi

[SUPPORT]Hudi-cli cleans show OOM

Open
#7,979 4 comments 0 reactions 0 assignees View on GitHub
area:table-service component:cli priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I use hudi-cli(0.11.1 version) to do cleans show command, and I get a OOM exception:
```
hudi:ds_segments->cleans show
2023-02-15 02:33:00,699 INFO timeline.HoodieActiveTimeline: Loaded instants upto : Option{val=[20230214035435843__clean__COMPLETED]}
Command failed java.lang.OutOfMemoryError: Java heap space
Exception in thread "Spring Shell" java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.(String.java:463)
at org.apache.avro.util.Utf8.toString(Utf8.java:158)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:322)
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:219)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:456)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
at org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:298)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:183)
at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:354)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:185)
at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:251)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:236)
at [org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:206](http://org.apache.hudi.common.table.timeline.timelinemetadatautils.deserializeavrometadata%28timelinemetadatautils.java:206/))
at [org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeHoodieCleanMetadata(TimelineMetadataUtils.java:170](http://org.apache.hudi.common.table.timeline.timelinemetadatautils.deserializehoodiecleanmetadata%28timelinemetadatautils.java:170/))
at [org.apache.hudi.cli.commands.CleansCommand.showCleans(CleansCommand.java:74](http://org.apache.hudi.cli.commands.cleanscommand.showcleans%28cleanscommand.java:74/))
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
2023-02-15 02:36:14,433 INFO support.GenericApplicationContext: Closing org.springframework.context.support.GenericApplicationContext@47ef968d: startup date [Wed Feb 15 02:32:39 UTC 2023]; root of context hierarchy
2023-02-15 02:36:14,435 INFO support.DefaultLifecycleProcessor: Stopping beans in phase 1
2023-02-15 02:36:14,441 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2023-02-15 02:36:14,441 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2023-02-15 02:36:14,442 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
```

Then I checked the code in CleansCommand.java and found that when I do cleans show, it will get all the clean first, and deserialize avro, which causes OOM.
```
HoodieActiveTimeline activeTimeline = HoodieCLI.getTableMetaClient().getActiveTimeline();
HoodieTimeline timeline = activeTimeline.getCleanerTimeline().filterCompletedInstants();
List cleans = timeline.getReverseOrderedInstants().collect(Collectors.toList());
List rows = new ArrayList<>();
for (HoodieInstant clean : cleans) {
HoodieCleanMetadata cleanMetadata = TimelineMetadataUtils.deserializeHoodieCleanMetadata(timeline.getInstantDetails(clean).get());
rows.add(new Comparable[]{clean.getTimestamp(), cleanMetadata.getEarliestCommitToRetain(),
cleanMetadata.getTotalFilesDeleted(), cleanMetadata.getTimeTakenInMillis()});
cleanMetadata = null;
}

TableHeader header =
new TableHeader().addTableHeaderField(HoodieTableHeaderFields.HEADER_CLEAN_TIME)
.addTableHeaderField(HoodieTableHeaderFields.HEADER_EARLIEST_COMMAND_RETAINED)
.addTableHeaderField(HoodieTableHeaderFields.HEADER_TOTAL_FILES_DELETED)
.addTableHeaderField(HoodieTableHeaderFields.HEADER_TOTAL_TIME_TAKEN);
return HoodiePrintHelper.print(header, new HashMap<>(), sortByField, descending, limit, headerOnly, rows);
```

can we do some optimization here?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in org.apache.hudi.cli.commands.CleansCommand.showCleans and trace its calls to TimelineMetadataUtils.deserializeHoodieCleanMetadata, using the reported `cleans show` failure as the reproduction. The change is done when the command can process a large cleaner timeline without the reported Java heap-space error while retaining the existing clean summary fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.