apache / apache/datafusion

`FieldNotFound` but field is shown in `valid_fields`

Open
#6,699 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I'm doing a query with aliases, and when trying to execute it, it gives back a `SchemaError` saying `FieldNotFound`, however that error also contains a list of valid fields, and the field it's saying isn't found, is listed in there.
The error (formatted):
```
SchemaError(FieldNotFound {
field:
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "776ff32c7d7bfb1c1a49d5c0d94e0db8" },
valid_fields: [
Column { relation: Some(Bare { table: "75620d04a8b7b34c70c68f7af68953af" }), name: "qid" },
Column { relation: Some(Bare { table: "75620d04a8b7b34c70c68f7af68953af" }), name: "timestamp" },
Column { relation: Some(Bare { table: "75620d04a8b7b34c70c68f7af68953af" }), name: "createdDate" },
Column { relation: Some(Bare { table: "75620d04a8b7b34c70c68f7af68953af" }), name: "82ad7656c6b6be56e9abfac772c89b4c" },
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "dataScope" },
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "qid" },
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "timestamp" },
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "createdDate" },
Column { relation: Some(Bare { table: "linkedtwintag" }), name: "776ff32c7d7bfb1c1a49d5c0d94e0db8" }
] })
```

As you can see, the field that is not found is the last one in the `valid_fields` list.

### To Reproduce

Create two mysql tables, and use the following crate to pull them in: https://github.com/splitgraph/seafowl/tree/main/datafusion_remote_tables.

Execute the following (or similar query):
```sql
SELECT "75620d04a8b7b34c70c68f7af68953af".*
FROM "75620d04a8b7b34c70c68f7af68953af"
JOIN "f4be58656cd5e54b9bffa1706c3c49ff" AS "linkedTwintag" ON "75620d04a8b7b34c70c68f7af68953af"."82ad7656c6b6be56e9abfac772c89b4c" = "linkedTwintag"."776ff32c7d7bfb1c1a49d5c0d94e0db8"
WHERE 1 = 1 AND "linkedTwintag"."createdDate" = ? AND "linkedTwintag"."qid" = ? ORDER BY "75620d04a8b7b34c70c68f7af68953af"."qid" ASC"
```

### Expected behavior

I get back a normal resultbatch.

### Additional context

I've not used datafusion a lot, so I'm not 100% sure whether its a bug in the crate for pulling in mysql tables, or if its a bug in datafusion itself.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied SQL query with the datafusion_remote_tables crate and two MySQL tables. Trace whether the failure comes from alias resolution in DataFusion or from importing the MySQL table schema, using the FieldNotFound output as the comparison point. Done means the query returns a normal result batch or the responsible component and a focused regression case are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.