apache / apache/hudi

[BUG] hudi-trino connector cleanup: leaks, a MOR crash path, and a dead config knob

Open
#19,344 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Twelve code-level defects in the Trino connector, found while reviewing #18837. Locations use the `hudi-trino/` paths introduced by that PR; none of them is caused by it.

1. `TableStatisticsReader` is never closed. Its constructor creates a `TableMetadataReader` (a `HoodieBackedTableMetadata`) and a `HoodieTableFileSystemView`, but the class is not `Closeable`, and `HudiMetadata.triggerAsyncStatsRefresh` creates one on every stats refresh.

2. `HudiSplitSource.close()` only calls `queue.finish()`. `HudiSnapshotDirectoryLister.close()` is implemented but has no caller, so the metadata-backed file-system view is leaked once per query.

3. `HudiPageSourceProvider.createPageSource` dereferences `hudiBaseFileOpt.get()` unconditionally, while the `isEmpty()` guard above it only covers `COPY_ON_WRITE`. A `MERGE_ON_READ` split with no base file, which `HudiSplitFactory.createSplitForMergeOnRead` deliberately produces, throws `NoSuchElementException`.

4. `HudiPageSourceProvider.remapColumnIndicesToPhysical` unboxes a possibly-null `Integer` into `HiveColumnHandle`'s primitive `int` parameter, so a column missing from the Parquet file schema throws NPE. `TestHudiPageSourceProviderTest` currently pins that NPE as the expected behaviour.

5. `TableStatisticsReader` divides by `nullCount + valueCount` with no zero guard, so a column with no values produces `Estimate.of(NaN)`, which throws. The exception is swallowed by `catch (Throwable)` in `HudiMetadata.triggerAsyncStatsRefresh`, so the cache never fills and every planning attempt re-triggers a failing refresh.

6. `HudiMetadata.tableStatisticsCache` is a `static` `ConcurrentHashMap` keyed by table base path, with no eviction, no size bound, and no cleanup on connector shutdown.

7. `HudiSessionProperties.isIgnoreAbsentPartitions` has no production caller. `hudi.ignore-absent-partitions` and its `ignore_absent_partitions` session property are documented knobs that do nothing, and `HudiSplitManager.getPartitions` always throws `HUDI_PARTITION_NOT_FOUND`.

8. `IndexSupportFactory` instantiates each enabled index strategy before testing `canApply`, and those constructors submit metadata-table lookups to `ForkJoinPool.commonPool()`, so discarded strategies keep scanning. `createPartitionStatsIndexSupport` additionally calls `getStrategy()` twice.

9. `HudiPageSource.getNextSourcePage` drains the whole file group into a single `PageBuilder` with no `isFull()` break, and `getMemoryUsage()` does not include the builder's retained size.

10. `HudiUtil.buildTableMetaClient` drops the original exception: neither catch branch passes `e` as the `TrinoException` cause. Its `catch (Throwable)` also relabels `Error`s as meta-client failures.

11. `HudiPartitionStatsIndexSupport` builds its logger with `Logger.get(HudiColumnStatsIndexSupport.class)` and passes it to `super(...)`, so partition-stats messages are emitted under the column-stats logger name.

12. `HudiSplit.getRetainedSizeInBytes` uses two hard-coded `+ 10` terms in place of the sizes of `baseFile` and `logFiles`, so split memory accounting is short by roughly the length of every path string.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the hudi-trino/ classes named in the report, especially TableStatisticsReader, HudiSplitSource, HudiPageSourceProvider, HudiMetadata, and HudiUtil; inspect their callers and lifecycle paths. Run TestHudiPageSourceProviderTest first to understand the current missing-column expectation, then verify each of the twelve listed defects has a focused correction and appropriate regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.