[Initiative]: Ecosystem-Scale Open Source Development Pattern Analysis

### Name

Ecosystem Development Pattern Analysis

### Short description

Analyze large-scale open source activity data (such as the GitHub Archive) using advanced data processing techniques to identify and understand development workflows and collaboration patterns, building on existing academic research.

### Responsible group

TOC

### Does the initiative belong to a subproject?

Yes

### Subproject name

Contributor Strategy

### Primary contact

Matt Young

### Additional contacts

_No response_

### Initiative description

# Open Source Development Pattern Analysis

## Description

Analyze large-scale open source activity data (such as the GitHub Archive) using advanced data processing techniques to identify and understand development workflows and collaboration patterns, potentially building on existing academic research.

While vast amounts of open-source activity data are available (e.g., GitHub Archive), processing this data at scale and extracting meaningful insights into complex development workflows (beyond simple metrics) remains challenging. Understanding these patterns can inform best practices and tooling for large collaborative projects.

Each hour, [millions of events](https://docs.github.com/en/rest/using-the-rest-api/github-event-types?apiVersion=2022-11-28) happen on GitHub (gh event backgrounder). This event stream captures much of the activity in a project's Issues, PR's, discussions and codebase. If these are assembled into a graph, contributor interaction patterns and workflows can be observed. Building on and expanding [rudimentary statistical and metrics based models](https://github.com/cncf/toc/discussions/1219) for project topologies and community health towards something more meaningful.

At the University of British Columbia (https://www.cs.ubc.ca), [Dr. Dongwook Yoon](https://dwyoon.com/), [Emilie Ma](https://kewbi.sh) ([@kewbish](https://github.com/kewbish)) and others have published a paper entitled "*Revealing Software Development Work Patterns with PR-Issue Graph Topologies*" ([https://www.cs.ubc.ca/\~bestchai/papers/fse24-pr-issue-graphs.pdf](https://www.cs.ubc.ca/~bestchai/papers/fse24-pr-issue-graphs.pdf)).

This work was presented recently here:

[NODES 2024 - The GitHub Graph: Open-Source Collaboration Patterns and Neo4j](https://www.youtube.com/watch?v=xFtwCuRj-Wg)

## Scope

### In Scope

- Access and process large-scale open source activity datasets (e.g., GitHub Archive)
- Apply or develop models (e.g., graph-based representations of PR/Issue interactions) to capture development workflows and contributor interactions from the raw data
- Utilize scalable data processing techniques suitable for the volume and velocity of the data
- Perform analysis to identify, quantify, and characterize recurring development patterns within a significant scope (e.g., CNCF projects, wider open source ecosystem)
- Collaborate with academic or industry researchers to validate methodologies and interpret findings

### Out of Scope

- Building real-time monitoring systems
- Creating project-specific health dashboards
- Developing predictive models initially

## Goals

- Protect Contributor Privacy by design. See more here: https://github.com/cncf/toc/discussions/1218
- Demonstrate the ability to process large-scale development activity streams effectively
- Implement and populate a data store or analytical model representing development interactions and workflows
- Identify and document significant, potentially non-obvious, patterns of collaboration or workflow within the analyzed dataset, potentially replicating or extending existing research findings
- Foster collaboration between the CNCF community and researchers studying software development practices

## Non-Goals

- Providing prescriptive recommendations for all projects based on the analysis
- Developing project management or developer productivity tools directly
- Archiving the raw activity data long-term within this initiative's scope

## Implementation Plan

- Build on the research paper "Revealing Software Development Work Patterns with PR-Issue Graph Topologies" from UBC researchers. The paper includes the Cypher queries used to identify patterns. This forms a great starting point.
- Implement a scalable GitHub Archive processing pipeline that leverages existing resources (CNCF Infra, LFX Insights team and data stores. The data set used by the researchers was activity data from 50 large open source projects and was time-constrained. The paper calls out the difficulty obtaining github activity in a more arbitrary way as a primary limitation. This work can unblock that technical barrier.
- Create a graph database that captures the relationships between Issues, PRs, and contributors for CNCF projects.
- Analyze patterns of collaboration and workflow in CNCF projects initially
- Document the patterns identified and make the analysis tools available to the community

## Resource Requirements

- Expertise in large-scale data processing (stream/batch), software repository mining, graph analytics, statistical analysis
- Significant compute and storage resources appropriate for large datasets like the GitHub Archive
- Collaboration time with academic or industry research partners
- Integrations with existing data sources (LFX Insights, gharchive.org)

## Timeline

Estimate: 9-12+ months, given the scale of data and complexity of analysis.

## Additional References

1. [GitHub Archive](https://www.gharchive.org/) \- Source of GitHub event data
2. [Revealing Software Development Work Patterns with PR-Issue Graph Topologies](https://www.cs.ubc.ca/~bestchai/papers/fse24-pr-issue-graphs.pdf) \- UBC Research Paper
3. [NODES 2024 \- The GitHub Graph: Open-Source Collaboration Patterns and Neo4j](https://www.youtube.com/watch?v=xFtwCuRj-Wg) \- Video presentation on GitHub graph analysis
4. [Neo4j Graphs4Good](https://neo4j.com/graphs4good/) \- Potential collaboration opportunity
5. [TODO Group's OSPO Research](https://todogroup.org/resources/research/) \- Related research on open source program offices
6. [DevStats \- CNCF Project Analytics](https://devstats.cncf.io/) \- Existing CNCF metrics platform to integrate with

### Deliverable(s) or exit criteria

- Successful processing of CNCF project activity and history using methods described above.
- Data store populated with derived interaction/workflow data
- Analysis yields documented, quantifiable insights into development patterns, potentially validating/extending existing research
- Active collaboration with relevant research communities established and next steps defined.