datafusion-contrib / datafusion-contrib/datafusion-flight-sql-server
service does not work with parquet file
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 33
- Forks
- 15
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 4
Description
i tried to modify the example to work with a parquet file but it seems doesn't work.
In the flight-sql.rs, I change one line code from:
remote_ctx
.register_csv("test", "./examples/test.csv", CsvReadOptions::new())
.await.expect("Register csv");
to
remote_ctx
.register_parquet("test", "./examples/yellow_tripdata_2015-01.parquet", ParquetReadOptions::new())
.await.expect("Register parquet");
however after the above change the client will report an error as following:
show failed: External("arrow error: IpcError(\"Status { code: Internal, message: \\\"SchemaError(FieldNotFound { field: Column { relation: Some(Bare {
table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"vendorid\\\\\\\" }, valid_fields: [Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"VendorID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tpep_pickup_datetime\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tpep_dropoff_datetime\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }),
name: \\\\\\\"passenger_count\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"trip_distance\\\\\\\" }, Column {
relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"RatecodeID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"store_and_fwd_flag\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"PULocationID\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"DOLocationID\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"payment_type\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"fare_amount\\\\\\\" }, Column { relation:
Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"extra\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"mta_tax\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tip_amount\\\\\\\" }, Column { relation: Some(Bare
table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"tolls_amount\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name:
\\\\\\\"improvement_surcharge\\\\\\\" }, Column { relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"total_amount\\\\\\\" }, Column
{relation: Some(Bare { table: \\\\\\\"test\\\\\\\" }), name: \\\\\\\"congestion_surcharge\\\\\\\" }, Column { relation: Some(Bare { table:
\\\\\\\"test\\\\\\\" }), name: \\\\\\\"airport_fee\\\\\\\" }] }, Some(\\\\\\\"\\\\\\\"))\\\", metadata: MetadataMap { headers: {\\\"content-type\\\":
\\\"application/grpc\\\", \\\"date\\\": \\\"Sat, 05 Apr 2025 18:27:17 GMT\\\", \\\"content-length\\\": \\\"0\\\"} }, source: None }\")")
Could you give me some advice on such error? Is there any thing I got wrong?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in flight-sql.rs with the change from register_csv to register_parquet and reproduce the reported show failure using the referenced Parquet file. Trace the reported vendorid versus VendorID field error and determine the expected behavior for this example. Done means the service can register and query the Parquet file without the schema error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100