Spark show_* procedures apply limit before filter, so filter => ... only sees the first limit rows
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem**
Every `show_*` procedure that takes both `limit` and `filter` truncates its rows to `limit` first and evaluates `filter` on the truncated list (`BaseProcedure.applyFilter`), so `limit => 10, filter => ...` returns the matching subset of the first 10 rows rather than the first 10 matching rows. With a selective filter the result is empty even when matching rows exist past the cutoff, and every procedure applies a default `limit` (10, 20 or 100), so a plain `filter => ...` call is affected too. Introduced with the generic filter option in #13736 / #13790 (HUDI-9726 / HUDI-9746).
Affected on master `9903b6d83fbb` (limit -> applyFilter):
- `ShowFileSystemViewProcedure.scala:263 -> :264` (`rows.stream().limit(limit)`, default 10)
- `ShowTablePropertiesProcedure.scala:60 -> :61` (default 10)
- `ShowHoodieLogFileMetadataProcedure.scala:136 -> :137` (default 10)
- `ShowMetadataTableFilesProcedure.scala:86 -> :90` (default 100)
- `ShowInvalidParquetProcedure.scala:110 -> :114` (`parquetRdd.take(limit)`, default 100)
- `ShowBootstrapMappingProcedure.scala:100/:102 -> :104` (`df.orderBy(...).limit(limit).collect()`, default 10)
- `ShowFsPathDetailProcedure.scala:82/:88 -> :93` (`df.orderBy(...).limit(limit).collect()`, default 100)
- `ShowCleansProcedure.scala:175 -> :179` (`.take(limit)` after the limited timeline read, default 10)
- `ShowTimelineProcedure.scala:140 -> :142` (limit applied inside `getTimelineEntries`, default 20)
- `ShowHoodieLogFileRecordsProcedure.scala:104 -> :120` (limit bounds record collection, default 10)
**To reproduce**
On a table with more than 10 file slices, where only a slice past the 10th exceeds ``:
```sql
call show_fsview_all(table => 't', filter => 'data_file_size > ');
```
returns 0 rows, and `limit => 1000` makes the row appear. `TestFsViewProcedure` does not catch this because its tables have 2 files, below the default limit.
**Suggested fix**
For the seven procedures whose rows are materialized before the cut (`stream().limit`, `take`, `df.limit`), apply the filter first: `applyFilter(rows, filter, outputType).take(limit)`. For `show_logfile_records`, `show_timeline` and `show_cleans`, where `limit` bounds how much is read, either document that `filter` only sees the first `limit` entries or push the predicate into the read loop. Add one procedure-level test per procedure with more rows than `limit` and a filter that matches only rows past the cutoff.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with BaseProcedure.applyFilter and the affected implementations, including ShowFileSystemViewProcedure.scala, ShowTablePropertiesProcedure.scala, ShowInvalidParquetProcedure.scala, ShowBootstrapMappingProcedure.scala, ShowFsPathDetailProcedure.scala, ShowCleansProcedure.scala, ShowTimelineProcedure.scala, and ShowHoodieLogFileRecordsProcedure.scala. Review TestFsViewProcedure, then add procedure-level coverage with more rows than the limit and a filter matching only rows past the cutoff; done means each procedure returns the first matching rows consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100