apache / apache/datafusion-comet
[blog post] Spark compatibility testing
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
There have been some discussions around writing technical deep dive blog posts on the DataFusion blog. I would like to see if there is interest in a blog post focussing on Spark compatibility testing.
Possible topics:
- Challenges (different floating point implementation, regexp, unicode differences, date/time range differences)
- QueryPlanSerde overview - framework for context-specific tagging as Compatible, Incompatible, Unsupported
- Lessons learned from Fuzz testing approach
- Standalone fuzz testing vs fuzz testing in unit/integration tests
- Lessons learned from agentic coding
- Testing against multiple Spark versions
- Shims
- SQL-based testing, inspired by sqllogictest
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.