apache / apache/datafusion

[DISCUSS] publishing nightly / non-official releases to crates.io (even more frequent releases)

Open
#5,023 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
DataFusion's mission statement is to be "easy to embed" from https://arrow.apache.org/datafusion/user-guide/introduction.html#why-datafusion

Within the rust ecosystem, publishing to crates.io is the key way to make something easy to embed (other projects can then use it with a single line of `toml`). Thanks to @andygrove and other's work, we now are publishing official versions about once a month ❤️ (e.g. #4776 )

However, releasing even every month is still slower than optimal, as sometimes there are changes / bugs to the interfaces which aren't caught until after the release (e.g. most recently necessitating a patch release https://github.com/apache/arrow-datafusion/issues/4948)

Many active users of DataFusion today still use a fork or work off the master branch directly but this is not always possible for all projects as it is somewhat challenging.

Having more frequently published versions on crates.io would help others integrate datafusion more quickly between major official releases

**Describe the solution you'd like**
I propose publishing "nightly" releases of DataFusion to crates.io, much like the rust compiler itself.

These releases would be clearly labeled as "NON OFFICIAL RELEASES" perhaps with versions like `17.0.0.dev-2023-01-22

**Describe alternatives you've considered**
We could also try to run the official release process more frequently (every 2 weeks perhaps) -- given the voting process takes at least three days it will still be limited in how quickly it can be done

**Additional context**
See https://github.com/apache/arrow-datafusion/issues/2327 for prior discussion

Contributor guide

Open the contributing guide

Research direction

Start by reading the prior discussion in issue #2327 and the current official release process referenced through #4776. Determine whether crates.io supports the proposed nightly versioning and publication workflow, and document an agreed approach that clearly distinguishes non-official releases from official versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.