apache / apache/datafusion-ballista
TPCH benchmark fails with `Failed to open partition file at "/var/folders/..../data.arrow"`
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Describe the bug**
tpch.py prints an error after registering the tables for the Parquet files:
```
Registering table part at path tpch-data/part.parquet
Registering table supplier at path tpch-data/supplier.parquet
Registering table partsupp at path tpch-data/partsupp.parquet
Registering table customer at path tpch-data/customer.parquet
Registering table orders at path tpch-data/orders.parquet
Registering table lineitem at path tpch-data/lineitem.parquet
Registering table nation at path tpch-data/nation.parquet
Registering table region at path tpch-data/region.parquet
grpc client failed to fetch partition: PartitionId { job_id: "0381QN35XP000", stage_id: 3, partition_id: 0 } , message: "Status { code: Internal, message: \"Ballista Error: General(\\\"Failed to open partition file at \\\\\\\"/var/folders/0g/qbtl2t512f5bh4sp79cvfy_00000gn/T/.tmpIzsco9/0381QN35XP000/3/0/data.arrow\\\\\\\": Os { code: 2, kind: NotFound, message: \\\\\\\"No such file or directory\\\\\\\" }\\\")\", metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Tue, 25 Aug 2026 06:43:25 GMT\"} }, source: None }"
Traceback (most recent call last):
File "/Users/mgrigorov/git/apache/datafusion/datafusion-ballista/benchmarks/tpch.py", line 55, in
df.show()
File "/Users/mgrigorov/git/apache/datafusion/datafusion-ballista/python/python/ballista/extension.py", line 58, in method_wrapper
return getattr(df, func)(*argz, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: DataFusion error: Arrow error: External error: External error: Shuffle fetch partition error from Executor 9bbf0041-d0d6-43bc-a452-d35d92f0e627, map_stage 3, map_partition 0, error desc: Status { code: Internal, message: "Ballista Error: General(\"Failed to open partition file at \\\"/var/folders/0g/qbtl2t512f5bh4sp79cvfy_00000gn/T/.tmpIzsco9/0381QN35XP000/3/0/data.arrow\\\": Os { code: 2, kind: NotFound, message: \\\"No such file or directory\\\" }\")", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 25 Aug 2026 06:43:25 GMT"} }, source: None }
```
@andygrove [guesses](https://discord.com/channels/885562378132000778/1179822705525141605/1539686066909352067) that it might be related to the new DataFusion + old Ballista Python bindings state in current HEAD (50d732299c9129947aa41949a709c5aa1697cf3e)
**To Reproduce**
1. Start a Scheduler
2. Start at least one Executor
3. `cd datafusion-ballista/benchmarks`
4. `tpchgen-cli -s 1 --format parquet --output-dir tpch-data`
5. `python3 tpch.py --path tpch-data --query q1`
**Expected behavior**
Only the `Registering table ...` messages should be printed on the stdout.
**Additional context**
Ballista v54.0.0
DataFusion v55.0.0
Git HEAD: 50d732299c9129947aa41949a709c5aa1697cf3e perf(core): reuse shuffle partition-index buffers across batches
The job is properly executed and `Completed` according to the TUI app
Contributor guide
Assessment
This issue has not been assessed yet.