[Initiative]: CNCF Community Knowledge Graph (projects, blogs, youtube, packages, threats, landscape data) subproject application

### Name

Form the proposal for a CNCF Knowledge Graph - Projects and Community Activity

### Short description

Produce the Subproject application for a CNCF Knowledge Graph - Projects and Community Activity

### Responsible group

TOC

### Does the initiative belong to a subproject?

Yes

### Subproject name

Contributor Strategy

### Primary contact

Matt Young

### Additional contacts

_No response_

### Initiative description

# CNCF Knowledge Graph: Projects and Community Activity

## Description
Develop infrastructure and processes to ingest, integrate, and analyze diverse data streams related to CNCF community activities, content, and project evolution to create a unified knowledge base.

## Relevant TAGs/WGs
Most TAG's would have an interest as consumers and Entities in the graph. TAG Security, TAG Operational Resilience, and Contributor Strategy are key stakeholders.

## Problem Statement
Valuable information about the CNCF community—including technical content (talks, blogs), project interactions (commits, issues, PRs), contributor activities, and communication (mailing lists, Slack)—is dispersed across numerous platforms. There is no unified way to connect and analyze this data to understand community trends, knowledge flows, or project relationships effectively.

Examples of questions that are presently difficult, or costly, to answer include:

## Scope

### In Scope
- Identify and prioritize key CNCF community data sources, including the LFX Insights team and CNCF supported infrastructure and data assets.
- Develop methods and pipelines for extracting, transforming, and integrating data from these diverse sources (structured and unstructured)
- Implement or leverage a suitable data store (e.g., a graph database, data warehouse) capable of representing the complex relationships between people, projects, content, and activities
- Develop example analyses and queries to demonstrate insights into community structure, content discovery, contributor journeys, or project interactions
- Document the data sources, integration methods, and potential analyses

### Out of Scope
- Real-time analysis of all communication channels
- Building a public-facing social network or full analytics platform
- Deep NLP analysis initially

## Goals
- Protect Contributor Privacy by design. See more here: https://github.com/cncf/toc/discussions/1218
- Establish processes for ingesting data from at least 3-4 prioritized community sources
- Deploy and populate a data store that integrates these diverse data types
- Create and document example queries or visualizations that reveal non-obvious insights about the CNCF ecosystem (e.g., identifying subject matter experts, mapping technology adoption across projects, visualizing collaboration patterns)
- Provide a foundation for better understanding and supporting the CNCF community

## Non-Goals
- Replacing existing tools like DevStats entirely
- Monitoring individual contributor performance
- Providing real-time operational dashboards for all community metrics

## Implementation Plan

Process and integrate CNCF community data sources:

- YouTube videos, blogs, books, conference talks
- Project evolution over time
- Dependencies between projects
- Integration with other Linux Foundation landscapes

The CNCF has a project (lower case 'p') that uses Neo4j [landscape-graph [short]](https://docs.google.com/presentation/d/1ZXlBr-XOboNDOU2CtjTlJ1wYMQcdZv_oxT0owxobTQg/edit?usp=sharing) and defines a "supergraph" composed of many "SubGraph Modules" (SGM).

Each of these submodules are based on a stream of activity (e.g. blogs, talks, meetings, social media posts, etc.) that needs to be processed continuously as it arrives.

For example, the core entities in the CNCF are modelled thus:

The organizing principles for structuring this [super]Graph (composed of sub-graphs) is based on what was Apollo's federation standard, which has since formed the basis of the GraphQL Foundation's (first) open standard for GraphQL Federation.

#### More information regarding GraphQL federation:

* [Announcing the Composite Schemas Working Group](https://graphql.org/blog/2024-05-16-composite-schemas-announcement/)
* [Open Federation Initiative](https://open-federation.org/)
Open Federation is a community-driven open source initiative to create and maintain a specification for federated GraphQL APIs. Be part of the future; let's craft together.
* https://spec.open-federation.org/draft

### Infrastructure

- Collaborate w/ Neo4j, building on existing contact
- https://github.com/cncf/landscape-graph/issues/40#issuecomment-1111662444
- https://neo4j.com/graphs4good/
- LF - negotiate w/ cloud providers, in concert w/ Neo4j to host multi-cloud Neo4j instance to help enable the open source software ecosystems
- CNCF Infra.

## Success Metrics
- Successful ingestion and integration of data from the selected sources into the central data store
- Demonstrated ability to query the integrated data to answer meaningful questions about community activity and knowledge connections
- Data sources, integration architecture, and example analyses are documented

## Resource Requirements
- Expertise in data integration, ETL, various data formats (APIs, web content, potentially video metadata), and analytical data stores (graph DBs, warehouses)
- Access to relevant CNCF data sources/APIs
- Compute/storage resources for data processing and storage

## Timeline

* To form the proposal for the subproject, 4-6 weeks to solicit feedback and finalize the Subproject application
* To execute on the proposed Subproject's launch - Estimate: 6-9 months for establishing initial data integration and analysis capabilities.

Current Date: May 11, 2025

## Additional References
1. [CNCF Landscape Graph Project](https://github.com/cncf/landscape-graph) - Existing CNCF effort to graph project relationships
2. [Neo4j Graph Data Science Library](https://neo4j.com/product/graph-data-science/) - Advanced analytics for knowledge graphs
3. [CNCF DevStats](https://devstats.cncf.io/) - Current metrics for CNCF projects
4. [Knowledge Graphs: Fundamentals, Techniques, and Applications](https://www.amazon.com/Knowledge-Graphs-Fundamentals-Techniques-Applications/dp/0262045095) - Academic reference on knowledge graph applications
5. [CNCF YouTube Channel](https://www.youtube.com/c/cloudnativefdn) - Source of video content to index
6. [Linux Foundation Landscape Tools](https://github.com/cncf/landscapeapp) - Infrastructure for visualization
7. [Graph Databases: New Opportunities for Connected Data](https://www.oreilly.com/library/view/graph-databases-2nd/9781491930891/) - O'Reilly book on graph database implementation
8. [YouTube playlist: Graph Databases](https://www.youtube.com/playlist?list=PLd13Ui6RDb8l9aFn_1jBjtmJssdG0rj36) 158 curated videos of background in Neo4j, Cypher, and Graph Data Science algorithms. These may prove useful for future contributors to this effort.
9. [YouTube Playlist: GraphQL Foundation Composite Schema WG](https://www.youtube.com/playlist?list=PLP1igyLx8foFjxyTg6wPn4pUkZwuAk2GR)
10. [YouTube Playlist: GraphQL Foundation OpenTelemetry WG](https://www.youtube.com/playlist?list=PLP1igyLx8foFO2xFpWp7IturLnInoyWv1) - 📣 _First video posted 3 weeks ago!_ 📣

### Deliverable(s) or exit criteria

* Produce and submit an application to the TOC to create a new sub-project as described above.
* Actively solicit and incorporate feedback from CNCF Stakeholders and Community Leaders into the Subproject application.