apache / apache/drill

Drill scan all the parquet file from query root for metadata if there is a "inner join " in query

Open
#2,402 2 comments 0 reactions 0 assignees View on GitHub
to-be-verified
Dominant language
Java
Stars
2k
Forks
990
Avg merge
1d 8h
Merged PRs (30d)
5

Description

**Describe the bug**
Drill scan all the parquet file from query root for metadata if there is a "inner join " in query.

**To Reproduce**
Steps to reproduce the behavior:

```
# /data/01/ is the query root in this case
# prepare the directory for drill to use in query
mkdir -p /data/01/2021/11/2021-11-23/
# download a parquet file for drill to query
curl https://raw.githubusercontent.com/apache/drill/master/sample-data/nation.parquet -o /data/01/2021/11/2021-11-23/data.parquet

# prepre inner join directory ,
mkdir -p /data/PRO/item
# we prepare a invalid parquet file, this file is not supposed to be scan in query
mkdir -p /data/01/2010/01/2010-01-01/
echo "abc" > /data/01/2010/01/2010-01-01/data.parquet

# query drill endpoint by curl
json="{\"queryType\":\"SQL\", \"query\": \"SELECT COUNT(*) FROM dfs.\`/data/01\` as t INNER JOIN dfs.\`/data/PRO/item\` item ON t.N_REGIONKEY = item.ID WHERE t.dir2 >='2021-11-23' AND t.dir2<='2021-11-30' AND (REPEATED_CONTAINS(item.CATEGORIES,1031) OR REPEATED_CONTAINS(item.CATEGORIES,1047))\", \"autoLimit\":1}"
drill_host="localhost:8047"
curl -XPOST -H "Content-Type: application/json" "$drill_host/query.json" -d "$json"

```
**Expected behavior**
As we only query t.dir2 >='2021-11-23' AND t.dir2<='2021-11-30' , and invalite file is under dir2="2010-01-01" ,
the expected behavior is drill perform query without any error, but it it return data.parquet is not a Parquet file, the result approve that drill scan all the parquet file from query root directory.

**Screenshots**
```
{
"errorMessage" : "SYSTEM ERROR: RuntimeException: file:/data/01/2010/01/2010-01-01/data.parquet is not a Parquet file (too small length: 4)\n\n\nPlease, refer to logs for more information.\n\n[Error Id: ce4e61af-5df8-440e-81d2-673c89106e5f on drill-0.drill:31010]"
}
```

**Additional context**
Drill return successfully if no inner join in query
```
# query drill endpoint by curl
json="{\"queryType\":\"SQL\", \"query\": \"SELECT COUNT(*) FROM dfs.\`/data/01\` as t WHERE t.dir2 >='2021-11-23' AND t.dir2<='2021-11-30'\", \"autoLimit\":1}"
drill_host="localhost:8047"
curl -XPOST -H "Content-Type: application/json" "$drill_host/query.json" -d "$json"
```

```
{
"queryId" : "1e4ce295-3052-a66c-b68f-96cf4a97806d",
"columns" : [ "EXPR$0" ],
"rows" : [ {
"EXPR$0" : "25"
} ],
"metadata" : [ "BIGINT" ],
"queryState" : "COMPLETED",
"attemptedAutoLimit" : 1
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue through Drill's /query.json endpoint using the supplied directory layout and compare the INNER JOIN query with the single-table query. Trace how the query root and dir2 filters are handled during Parquet metadata scanning, then verify that the invalid 2010 file is excluded and the filtered join completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.