type:devtask empty projection (COUNT(*)) fails on Lance-backed tables
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
## TL;DR
`SELECT COUNT(*) FROM ` fails with:
```
Lance batch column count 14 does not match expected Spark schema size 0
for file: .../category=Abyssinian/....lance
at org.apache.hudi.io.storage.LanceRecordIterator.hasNext(LanceRecordIterator.java:124)
```
Any query shape that triggers Spark's "no columns needed, just count rows" optimization (`COUNT(*)`, `EXISTS`, `CREATE TABLE AS SELECT 1 FROM ...`) blows up on a Lance-backed Hudi table. Parquet-backed tables work fine.
## Why it happens
[`LanceRecordIterator.java:122-127`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java#L122-L127) has a strict equality check when building `ColumnVector[]`:
```java
StructField[] sparkFields = sparkSchema.fields();
if (sparkFields.length != fieldVectors.size()) {
throw new HoodieException("Lance batch column count " + fieldVectors.size()
+ " does not match expected Spark schema size " + sparkFields.length + ...);
}
```
When Spark's optimizer prunes all columns for an aggregate-only read (`COUNT`, `EXISTS`), the request arrives with `sparkSchema.fields().length == 0`, but the Lance file's batch always has the full column set. The reader sees `0 != 14` and throws.
The Parquet reader handles this naturally — `ParquetFileFormat` has a zero-column fast path where it just yields N empty rows (where N is the row count) so the aggregate can count them without reading any data. Lance needs the equivalent.
## Workaround
Use `COUNT()` instead of `COUNT(*)`. On a non-null primary key the two are semantically equivalent, but the former forces Spark to request one column, satisfying the check.
## Proposed fix
In `LanceRecordIterator.hasNext()`:
- If `sparkSchema.fields().length == 0`, skip the `ColumnVector[]` build entirely.
- Still call `arrowReader.loadNextBatch()` to advance, and yield empty rows matching the Arrow `VectorSchemaRoot.getRowCount()` so downstream count aggregators work.
- Add a test in [`TestLanceDataSource.scala`](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLanceDataSource.scala) exercising `spark.sql("SELECT COUNT(*) FROM …")` over a Lance-backed table and `df.count()` on the same.
## Related code paths
- [`LanceRecordIterator.java`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java)
- [`HoodieSparkLanceReader.java`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java)
- [`TestLanceDataSource.scala`](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLanceDataSource.scala)
## Environment
- Hudi `master` @ commit `4d0e9cd47f9e`
- Spark datasource path with Lance-backed base files
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with LanceRecordIterator.java, especially hasNext() around lines 122-127, and review HoodieSparkLanceReader.java for the reader flow. Add coverage in TestLanceDataSource.scala for COUNT(*) and df.count() on a Lance-backed table; done means empty projections yield the correct row counts without the schema-size exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100