apache / apache/datafusion-comet

[blog post] Spark compatibility testing

Open
#3,679 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

There have been some discussions around writing technical deep dive blog posts on the DataFusion blog. I would like to see if there is interest in a blog post focussing on Spark compatibility testing.

Possible topics:

- Challenges (different floating point implementation, regexp, unicode differences, date/time range differences)
- QueryPlanSerde overview - framework for context-specific tagging as Compatible, Incompatible, Unsupported
- Lessons learned from Fuzz testing approach
- Standalone fuzz testing vs fuzz testing in unit/integration tests
- Lessons learned from agentic coding
- Testing against multiple Spark versions
- Shims
- SQL-based testing, inspired by sqllogictest

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.