File IO on Tokio Workers
Open
bug
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Both CsvExec and NdJsonExec perform file IO on the tokio worker threads, this acts to starve the worker pool, potentially leaving CPUs idle. In cases involving FIFO files, this could even lead to deadlock
### To Reproduce
_No response_
### Expected behavior
We should probably just convert the files to streams and decode them that way
### Additional context
We probably want to wait for https://github.com/apache/arrow-datafusion/issues/7994 first, so that any changes don't negatively impact the streaming use-case
Contributor guide
Assessment
This issue has not been assessed yet.