apache / apache/datafusion

Implement SQLancer (a end-to-end SQL fuzz testing library)

Open
#11,030 9 comments 19 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I noticed an awesome SQL fuzzing framework [SQLancer](https://github.com/sqlancer/sqlancer/tree/main) can be implemented on DataFusion, and it is able to detect many bugs even in [PostgreSQL and SQLite](https://www.manuelrigger.at/dbms-bugs/)

## Update:
Implementation is now at [datafusion-sqlancer](https://github.com/datafusion-contrib/datafusion-sqlancer)
### Supported SQL Features
* `JOIN`s, `ORDER BY`, `WHERE`
* Numeric scalar functions/expression operators
* String scalar functions/expression operators
- [x] Aggregate functions, `HAVING` clause
- [ ] Time related data type functions
- [x] Window functions
- [ ] Subquery
- [ ] Queries from parquet, csv
- [ ] Exploit different configurations (change config knobs like `target_partition`, `prefer_hash_join` etc.
### Supported Test Oracles
Note: most oracles only apply to a subset of available query types, for advanced SQL features like window functions we can only generate random queries and report crashes.
More context for below test oracles at https://github.com/sqlancer/sqlancer/tree/main
* NoREC
* TLP
- [x] PQS
- [ ] DQP for logical bugs in joins
- [ ] [EET](https://www.usenix.org/conference/osdi24/presentation/jiang#:~:text=To%20find%20logic%20bugs%20in,is%20independent%20of%20query%20patterns.) for logic bugs in joins and subqueries

# How SQLancer works in short
1. It's a black box fuzzer, which will be implemented on [SQLancer](https://github.com/sqlancer/sqlancer/tree/main)'s starter code, and connect to DataFusion using `JDBC` to do SQL level testings
2. It will generate random chaotic SQL queries to stress the system, and make sure it won't crash
3. And do extra logical consistency checks using randomly generated SQLs, `SQLancer` has 5 logic check oracles, one of them works like:
```
NoREC consistency check oracle
Randomly generated query(Q1):
select * from t1 where v1 > 0;
Mutated query(Q2):
select v1 > 0 from t1;
Consistency check:
result size of Q1 should be equal to the number of `True` in Q2's output
```
Above showed consistency check generated Q1 (very likely to be optimized by predicate pushdown), and Q2(hard to be optimized), such test suit focus on correctness of the optimizer. There are 5 similar test oracles available to be implemented, those carefully designed checks make this testing framework really powerful.

### Describe the solution you'd like

I plan to implement `SQLancer` on `DataFusion`(starting with a specific test oralcle `NoREC` which requires less engineering effort).
For now, a minimal subset of SQL features is implemented: it hasn't detected any logical bug yet, just 2 bad-input bugs for some scalar functions showed up
(Will share the code once it is cleaned up)

If you have any features (SQL clauses / data types / specific functions) would like to be further tested, I can implement them first :)

### Describe alternatives you've considered

`SQLsmith` looks like another popular choice, I haven't looked into it carefully yet.
But if it's only generating random SQL to test if the system will crash, then `SQLancer` should be a more comprehensive tool.

### Additional context

[SQLancer](https://github.com/sqlancer/sqlancer)'s page have several papers/YouTube talk video recordings available

Contributor guide

Open the contributing guide

Research direction

The update says implementation is now in datafusion-sqlancer, so start by reviewing that project rather than this repository. Compare its supported SQL features and test oracles with the unchecked items in this issue; the issue does not name files or tests, and completion would require selecting and implementing additional scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.