apache / apache/spark

EXPLAIN COST gives wrong estimated input size when querying via VIEW instead of physical Delta path

Open
#56,990 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

[SQL] EXPLAIN COST gives wrong estimated input size when querying via VIEW instead of physical Delta path

### Component
Spark SQL (Catalyst statistics / EXPLAIN COST)

### Describe the problem
`EXPLAIN COST` reports the correct estimated input size when querying a Delta table by its physical path. But if a `VIEW` (or table) is created on top of that same path, `EXPLAIN COST` on the view returns a wrong estimated input size for the identical data. Query results are correct — only the cost/size estimate is wrong.

### Steps to reproduce
```sql
-- 1. Correct size
EXPLAIN COST SELECT * FROM delta.`/path/to/table_x`;

-- 2. Wrap in a view
CREATE OR REPLACE VIEW table_x AS
SELECT * FROM delta.`/path/to/table_x`;

-- 3. Wrong size
EXPLAIN COST SELECT * FROM table_x;
```

### Observed
Step 1: correct `sizeInBytes`.
Step 3: incorrect `sizeInBytes` for the same underlying data.

### Expected
`EXPLAIN COST` should report the same, accurate estimated input size whether queried via the physical path or via a view/logical table name wrapping it.

### Environment
* Spark: 3.5.6
* Delta Lake: 3.3.0

* Deployment: (k8s)

### Willingness to contribute
- [x] Yes, independently
- [ ] Yes, with guidance
- [] No

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied SQL reproducer with Spark SQL's EXPLAIN COST against the physical Delta path and then the view. Trace the Catalyst statistics and sizeInBytes calculation for both plans; done means the view and physical-path queries report the same accurate estimated input size while query results remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.