datafusion-contrib / datafusion-contrib/datafusion-flight-sql-server

service does not work with parquet file

Open
#15 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
33
Forks
15
Avg merge
1h 2m
Merged PRs (30d)
4

Description

i tried to modify the example to work with a parquet file but it seems doesn't work.

In the flight-sql.rs, I change one line code from:

remote_ctx
        .register_csv("test", "./examples/test.csv", CsvReadOptions::new())
        .await.expect("Register csv");

to

remote_ctx
        .register_parquet("test", "./examples/yellow_tripdata_2015-01.parquet", ParquetReadOptions::new())
        .await.expect("Register parquet");

however after the above change the client will report an error as following:

show failed: External("arrow error: IpcError(\"Status { code: Internal, message: \\\"SchemaError(FieldNotFound { field: Column { relation: Some(Bare {
table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"vendorid\\\\\\\" }, valid_fields: [Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"VendorID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tpep_pickup_datetime\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tpep_dropoff_datetime\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }),
name: \\\\\\\"passenger_count\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"trip_distance\\\\\\\" }, Column {
relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"RatecodeID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"store_and_fwd_flag\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"PULocationID\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"DOLocationID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"payment_type\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"fare_amount\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"extra\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"mta_tax\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tip_amount\\\\\\\" }, Column { relation: Some(Bare
table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tolls_amount\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"improvement_surcharge\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"total_amount\\\\\\\" }, Column
{relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"congestion_surcharge\\\\\\\" }, Column { relation: Some(Bare { table:
\\\\\\\"test\\\\\\\" }), name: \\\\\\\"airport_fee\\\\\\\" }] }, Some(\\\\\\\"\\\\\\\"))\\\", metadata: MetadataMap { headers: {\\\"content-type\\\":
\\\"application/grpc\\\", \\\"date\\\": \\\"Sat, 05 Apr 2025 18:27:17 GMT\\\", \\\"content-length\\\": \\\"0\\\"} }, source: None }\")")

Could you give me some advice on such error? Is there any thing I got wrong?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in flight-sql.rs with the change from register_csv to register_parquet and reproduce the reported show failure using the referenced Parquet file. Trace the reported vendorid versus VendorID field error and determine the expected behavior for this example. Done means the service can register and query the Parquet file without the schema error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.