Scaling Maintenance

Over the past six months the amount of interest in contributing to Dask has grown considerably. We should think about how to best harness and organize this work to improve the project and community needs.

### Historically

Historically Dask maintenance was handled by a smallish group of around four people, most of whom were under the same employer (Continuum/Anaconda) and so had a shared management chain. This group was small enough that everyone knew what was going on, and product/project management was the work of an individual (mostly me at the time), which made sense since we were all in the same corporate structure.

### Today

Today we have something like five-to-ten organizations showing up. We've tried a loose anarchic product/project management structure for a while in the form of

1. A rotating facilitator every week (lovingly called czar)
2. A "what does everyone want to work on" assignment policy
3. With occasional "it would be great if someone could help over here" peer-management (which is great to see)

### Issues (from my perspective)

This approach is nice because it democratizes out management in a friendly way. No one is "in charge" which reflects the employment reality. It feels community centric, which feels good to me.

However there are, I think, a few issues:

1. Because individuals don't track the entire project they don't have a good sense on what is important to work on. As a result, people usually self-select issues that seem easy to them. I get the sense that people are happy working on harder problems if they're more important, they just don't have this information.
2. There is no onboarding process for new engineers who join the group
3. There is no speciation of labor. Some tasks require in-depth Dask expertise. Some require finicky programming and problem solving skills. Some require emotional sensitivity and communication skills.
4. Handling larger or harder projects is rare. This is appropriate given that our original goal is to handle the onslaught of issues, but maybe now we have enough dev-power to handle some larger issues.
5. Maintenance issues still get left behind. We're inconsistent about handling old issues and PRs, which often get left behind.

I get the sense that with more structure we could get a lot more done, which would be exciting, and might grow this team even further.

### Some options

Two extremes in project management might be the following:

1. The current model, which I'll call "anarchy"
2. The original model from years ago, which I'll call "dictatorship", where someone with full view over the project tells people what to do. Historically this was me, but today could also be someone like @TomAugspurger or @jrbourbeau who both seem to track all of the issue trackers and have a good holistic view of what's going on in the project.

Personally I don't think that either anarchy or dictatorship are appropriate today. I think that something in between probably makes sense. For example, I'll propose "federation" (I've been watching Star Trek recently) where we might split off into teams, each with its own czar. These teams might assemble for something like a month at a time and have clear direction by a single individual. They might meet weekly (or whatever makes sense for them) and then at the end of the month report out work done in a blogpost (responsibility of Czar) and at the monthly meeting.

One of these teams would always be issue/PR triage, and in some cases the triage team might ask another team to roll something into their work. I suspect that a team of 4-5 people is about right for this task, especially if there is a single Maintenance Czar for that month who is comfortable tracking everything and managing work.

The topics for other teams might come out of a monthly planning issue, followed on by the monthly meeting. Here are some example topics that might serve as a good month-long project for a small team.

1. High level query optimization
2. Array performance with task fusion, as applied to both Xarray problems and optimization problems
3. Plan, deploy, and analyze a user survey
4. Generalize the Dask-XGBoost pattern so that it can be more effectively used by others
5. Profile and reduce scheduler overhead
6. Improve Dask + Tensorflow or Dask + PyTorch integration. Both by surveying users, fixing bugs, and maybe making a small package
7. Dask + Timeseries. Survey users, fix bugs, maybe make some small package
8. Outreach. Give a tutorial and have office hours every week over some web conference to the general public
9. Stability. The team tries to break the Dask scheduler, and then tries to fix whatever it broke
10. Work with the Napari community to make sure that their system is as smooth as possible

### Teams

Regardless of what we choose, I think that there are two parts of the plan above that I like

1. Teams: we're at a point where having 10 people in a weekly call probably doesn't make sense. I think that breaking people into smaller teams is probably better for organization, and helps to develop comraderie. I think it's probably also good for onboarding if we are intentional about mixing newer and older maintainers, and about mixing people from different companies.

2. Monthly rather than weekly cycles. I think that switching czar on a weekly basis means that things get dropped, and that people tend not to develop a sense for what's going on broadly within the project.

### Corporate Engagement

Some of the topics above correspond to pain points by certain companies. As two examples:

- Stability. The team tries to break the Dask scheduler, and then tries to fix whatever it broke. This is important for many, but Blue Yonder in particular
- Profile and reduce scheduler overhead. This is important for many, but NVIDIA and Pangeo in particular

We might invite corporate involvement, where they dedicate more resources than typical, but we mix maintainers in to the effort in order to act as spirit guide and make sure that the community side is smooth.