apache / apache/datafusion

Proposal: DataFusion Cookbook

Open
#24,569 8 comments 2 reactions 0 assignees View on GitHub
documentation
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

A **cookbook** is a collection of short, self-contained, task-framed recipes ("I want to do X → here's how, here's why, here's the gotchas") — as opposed to reference docs (API surface) or a linear tutorial (one path through the basics). It's a proven format once a technology has enough surface area that no single tutorial covers it well.

**Why it'd help DataFusion**

DataFusion is powerful enough that the hard part for new adopters isn't "does this feature exist" but "how do I wire it together for my use case" — custom `TableProvider`, which UDF/UDAF/UDWF/UDTF variant to use, pushdown, catalog integration, remote object stores. A cookbook is the standard way projects close that gap.

**Examples of cookbooks elsewhere**

Published books:
- AWS Cookbook (O'Reilly) — https://www.amazon.com/AWS-Cookbook-Recipes-Success/dp/1492092606
- Terraform Cookbook (O'Reilly) — https://www.amazon.com/Terraform-Cookbook-Kerim-Satirli/dp/1804613477
- Django Web Development Cookbook (Packt, 4th ed.) — https://www.amazon.com/Django-Web-Development-Cookbook-development/dp/1838987428
- Polars Cookbook (Packt) — https://www.amazon.com/Polars-Cookbook-practical-transform-manipulate/dp/1805121154

Same-ecosystem, in-project precedents:
- Rust Cookbook — https://github.com/rust-lang-nursery/rust-cookbook
- ADBC Python Cookbook, embedded in the Arrow docs — https://arrow.apache.org/adbc/current/python/recipe/index.html

**Relationship with examples**

`datafusion-examples` (https://github.com/apache/datafusion/blob/main/datafusion-examples/README.md) already has ~50 runnable, tested examples across 11 groups: UDF/UDAF/UDWF/UDTF, custom `TableProvider`, custom file formats, Parquet encryption (incl. KMS), custom optimizer/analyzer rules, Flight SQL, querying S3, remote catalogs, and more.

These examples can be turned into recipe for a cookbook by rewriting each as a short step-by-step recipe — problem statement, code, explanation, pitfalls.

Probably 50 recipes are excessive, a good number would be in the 15-20, at least to start.

**Organization options**

Existing cookbooks adopt both the colocated and the separate repo approach, so we would need to decide between these two options:
- As a new section of the existing docs site, next to `datafusion-examples`, versioned with the code (closest precedent: ADBC's cookbook, which lives directly in the Sphinx docs).
- A separate repo, lower barrier for outside contributors, easier to restructure freely, but risks drifting from releases (precedent: Rust Cookbook, Spice.ai's cookbook repo).

Contributor guide

Open the contributing guide

Research direction

Start with datafusion-examples/README.md and review the existing runnable examples and their 11 groups. Decide whether the cookbook belongs in the versioned docs site or a separate repository, then identify 15–20 recipes and define their problem, steps, explanations, and pitfalls. Done means the organization is chosen and an initial cookbook scope is agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.