apache / apache/datafusion

[Meta-EPIC] Organizing DataFusion's Growth

Open
#21,036 0 comments 11 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

I feel that DataFusion's community (users, contributors, stakeholders) has grown to a point where we may need to devote some more time ("resources") to more explicit organization to support that growth.

There are several factors that are driving growth
- More users!
- More systems "in critical production" in the sense that even small changes in behavior can be perceived as regressions for existing users
- More contributors and parallel work, which increases the need for explicit roadmap and coordination.

This is my attempt to gather a meta summary of challenges related to growth in one place.

## Release Challenges

- [ ] https://github.com/apache/datafusion/issues/21034
- [x] https://github.com/apache/datafusion/issues/5269
- [ ] https://github.com/apache/datafusion/discussions/20230

## Review Queue / PR reviews challenges

I don't always think it is easy for people to find ways to help and the PR review queue backlog is growing. For example, if I have some time to review PRs, which one would be the most helpful?

Related PRs

- [ ] https://github.com/apache/datafusion/issues/21038
- [ ] https://github.com/apache/datafusion/issues/23839

## Communication channels and “where should this conversation happen?” are a source of friction.

Possibly related to above, As @neilconway recently told me, the current DataFusion communication system seems a bit "everything everywhere all the time" in the sense that there is discord, github discussions, github issues, github pull requests, slack the website, and the weekly sync call and it is not always clear how to make sure you see relevant conversations

- [ ] https://github.com/apache/datafusion/issues/14389
- [ ] https://github.com/apache/datafusion/discussions/2861
- [ ] https://github.com/apache/datafusion/discussions/7320
- [ ] https://github.com/apache/datafusion/issues/14389

## Not clear how to get consensus for large changes

I think in general it is not clear
1. What "large changes" are currently being considered
2. What criteria is used to decide what constitutes a large change
3. Who "decides" if we will make a large change

This is definitely related to roadmap discussions as well

- [ ] https://github.com/apache/datafusion/discussions/15365
- [x] https://github.com/apache/datafusion/discussions/21017
- [x] https://github.com/apache/datafusion/issues/18494
- [ ] https://github.com/apache/datafusion/issues/22882

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked issues and discussions under Release Challenges, PR reviews, communication channels, and consensus for large changes. Done would require resolving or organizing those related topics into an agreed approach, but this issue names no files, tests, or concrete completion criteria.

Written by the indexing model from the issue text.

Assessment

Domain
developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.