apache / apache/hudi

Spark show_* procedures apply limit before filter, so filter => ... only sees the first limit rows

Open
#19,862 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem**

Every `show_*` procedure that takes both `limit` and `filter` truncates its rows to `limit` first and evaluates `filter` on the truncated list (`BaseProcedure.applyFilter`), so `limit => 10, filter => ...` returns the matching subset of the first 10 rows rather than the first 10 matching rows. With a selective filter the result is empty even when matching rows exist past the cutoff, and every procedure applies a default `limit` (10, 20 or 100), so a plain `filter => ...` call is affected too. Introduced with the generic filter option in #13736 / #13790 (HUDI-9726 / HUDI-9746).

Affected on master `9903b6d83fbb` (limit -> applyFilter):

- `ShowFileSystemViewProcedure.scala:263 -> :264` (`rows.stream().limit(limit)`, default 10)
- `ShowTablePropertiesProcedure.scala:60 -> :61` (default 10)
- `ShowHoodieLogFileMetadataProcedure.scala:136 -> :137` (default 10)
- `ShowMetadataTableFilesProcedure.scala:86 -> :90` (default 100)
- `ShowInvalidParquetProcedure.scala:110 -> :114` (`parquetRdd.take(limit)`, default 100)
- `ShowBootstrapMappingProcedure.scala:100/:102 -> :104` (`df.orderBy(...).limit(limit).collect()`, default 10)
- `ShowFsPathDetailProcedure.scala:82/:88 -> :93` (`df.orderBy(...).limit(limit).collect()`, default 100)
- `ShowCleansProcedure.scala:175 -> :179` (`.take(limit)` after the limited timeline read, default 10)
- `ShowTimelineProcedure.scala:140 -> :142` (limit applied inside `getTimelineEntries`, default 20)
- `ShowHoodieLogFileRecordsProcedure.scala:104 -> :120` (limit bounds record collection, default 10)

**To reproduce**

On a table with more than 10 file slices, where only a slice past the 10th exceeds ``:

```sql
call show_fsview_all(table => 't', filter => 'data_file_size > ');
```

returns 0 rows, and `limit => 1000` makes the row appear. `TestFsViewProcedure` does not catch this because its tables have 2 files, below the default limit.

**Suggested fix**

For the seven procedures whose rows are materialized before the cut (`stream().limit`, `take`, `df.limit`), apply the filter first: `applyFilter(rows, filter, outputType).take(limit)`. For `show_logfile_records`, `show_timeline` and `show_cleans`, where `limit` bounds how much is read, either document that `filter` only sees the first `limit` entries or push the predicate into the read loop. Add one procedure-level test per procedure with more rows than `limit` and a filter that matches only rows past the cutoff.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with BaseProcedure.applyFilter and the affected implementations, including ShowFileSystemViewProcedure.scala, ShowTablePropertiesProcedure.scala, ShowInvalidParquetProcedure.scala, ShowBootstrapMappingProcedure.scala, ShowFsPathDetailProcedure.scala, ShowCleansProcedure.scala, ShowTimelineProcedure.scala, and ShowHoodieLogFileRecordsProcedure.scala. Review TestFsViewProcedure, then add procedure-level coverage with more rows than the limit and a filter matching only rows past the cutoff; done means each procedure returns the first matching rows consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.