Improve DataFusion error handling
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I'm trying to collect known issues related to error handling in datafusion.
So far I can see different categories which can be improved
1. DataFusion heavily relies on Rust `Result` and sometimes its hard to identify context where the actual error is happening, the situation becomes even worse for downstream projects.
- [x] https://github.com/apache/arrow-datafusion/issues/5283
- [x] https://github.com/apache/arrow-datafusion/issues/3410 (this related to upstream arrow-rs)
2. Overuse panics
- [x] https://github.com/apache/arrow-datafusion/issues/3313
- [x] https://github.com/apache/arrow-datafusion/issues/3317
- [ ] https://github.com/apache/arrow-datafusion/issues/3316
3. Overuse internal error type in situations where more specific error kind can be used
- [x] https://github.com/apache/arrow-datafusion/issues/6108
- [x] https://github.com/apache/arrow-datafusion/issues/9164
4. General
- [x] https://github.com/apache/arrow-datafusion/issues/7552
### Describe the solution you'd like
We can try to address some of the issues with implementing a generic error macros that will be used for handling DF errors. Having done that allows to process errors in standard unified way, experiment with stacktraces and other details needed to be exposed for the end user.
- [x] https://github.com/apache/arrow-datafusion/pull/7115
- [x] https://github.com/apache/arrow-datafusion/pull/7293
- [x] https://github.com/apache/arrow-datafusion/pull/7340
- [x] https://github.com/apache/arrow-datafusion/pull/7361
- [x] https://github.com/apache/arrow-datafusion/pull/8586
- [x] SQL Error macros
- [ ] Parquet Error macros
- [ ] ObjectStore Error macros
- [ ] IOError macros
- [x] https://github.com/apache/arrow-datafusion/pull/8620
- [x] https://github.com/apache/datafusion/pull/11374
- [ ] External Error macros
- [ ] Context Error macros
- [x] Substrait Error macros
- [x] https://github.com/apache/arrow-datafusion/pull/7434
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Review the checked and unchecked items in the issue, especially the linked issues and existing pull requests, to understand which error-handling work remains. Establish a narrower scope before starting; done should mean one agreed error category is addressed with the proposed generic macro approach and its related work is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100