[BUG] hudi-trino connector cleanup: leaks, a MOR crash path, and a dead config knob
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Twelve code-level defects in the Trino connector, found while reviewing #18837. Locations use the `hudi-trino/` paths introduced by that PR; none of them is caused by it.
1. `TableStatisticsReader` is never closed. Its constructor creates a `TableMetadataReader` (a `HoodieBackedTableMetadata`) and a `HoodieTableFileSystemView`, but the class is not `Closeable`, and `HudiMetadata.triggerAsyncStatsRefresh` creates one on every stats refresh.
2. `HudiSplitSource.close()` only calls `queue.finish()`. `HudiSnapshotDirectoryLister.close()` is implemented but has no caller, so the metadata-backed file-system view is leaked once per query.
3. `HudiPageSourceProvider.createPageSource` dereferences `hudiBaseFileOpt.get()` unconditionally, while the `isEmpty()` guard above it only covers `COPY_ON_WRITE`. A `MERGE_ON_READ` split with no base file, which `HudiSplitFactory.createSplitForMergeOnRead` deliberately produces, throws `NoSuchElementException`.
4. `HudiPageSourceProvider.remapColumnIndicesToPhysical` unboxes a possibly-null `Integer` into `HiveColumnHandle`'s primitive `int` parameter, so a column missing from the Parquet file schema throws NPE. `TestHudiPageSourceProviderTest` currently pins that NPE as the expected behaviour.
5. `TableStatisticsReader` divides by `nullCount + valueCount` with no zero guard, so a column with no values produces `Estimate.of(NaN)`, which throws. The exception is swallowed by `catch (Throwable)` in `HudiMetadata.triggerAsyncStatsRefresh`, so the cache never fills and every planning attempt re-triggers a failing refresh.
6. `HudiMetadata.tableStatisticsCache` is a `static` `ConcurrentHashMap` keyed by table base path, with no eviction, no size bound, and no cleanup on connector shutdown.
7. `HudiSessionProperties.isIgnoreAbsentPartitions` has no production caller. `hudi.ignore-absent-partitions` and its `ignore_absent_partitions` session property are documented knobs that do nothing, and `HudiSplitManager.getPartitions` always throws `HUDI_PARTITION_NOT_FOUND`.
8. `IndexSupportFactory` instantiates each enabled index strategy before testing `canApply`, and those constructors submit metadata-table lookups to `ForkJoinPool.commonPool()`, so discarded strategies keep scanning. `createPartitionStatsIndexSupport` additionally calls `getStrategy()` twice.
9. `HudiPageSource.getNextSourcePage` drains the whole file group into a single `PageBuilder` with no `isFull()` break, and `getMemoryUsage()` does not include the builder's retained size.
10. `HudiUtil.buildTableMetaClient` drops the original exception: neither catch branch passes `e` as the `TrinoException` cause. Its `catch (Throwable)` also relabels `Error`s as meta-client failures.
11. `HudiPartitionStatsIndexSupport` builds its logger with `Logger.get(HudiColumnStatsIndexSupport.class)` and passes it to `super(...)`, so partition-stats messages are emitted under the column-stats logger name.
12. `HudiSplit.getRetainedSizeInBytes` uses two hard-coded `+ 10` terms in place of the sizes of `baseFile` and `logFiles`, so split memory accounting is short by roughly the length of every path string.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the hudi-trino/ classes named in the report, especially TableStatisticsReader, HudiSplitSource, HudiPageSourceProvider, HudiMetadata, and HudiUtil; inspect their callers and lifecycle paths. Run TestHudiPageSourceProviderTest first to understand the current missing-column expectation, then verify each of the twelve listed defects has a focused correction and appropriate regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100