hackforla / hackforla/website

Analysis: Collecting Data on Issue Completion per Prework Author and Creating Looker Dashboards to Uncover Insights

Open
#4,152 17 comments 0 reactions 0 assignees View on GitHub
Complexity: Small Dependency Feature: Board/GitHub Maintenance feature: skills / productivity ladder dashboard role: data analyst size: 2pt status: Updated
Dominant language
JavaScript
Stars
363
Forks
872
Avg merge
2d 21h
Merged PRs (30d)
18

Description

### Dependency
- #4921. Resume when the dashboard is ready

### Overview

We need to collect data on the authors of all the prework issues in our repository to perform data analysis.

### Action Items

- [x] #7758
- [x] Read relevant sections in GitHub API documentation on retrieving data with REST APIs
- [x] Search for other resources on platforms or libraries and syntax to use to retrieve data with GitHub REST APIs
- [x] Download Postman to retrieve needed JSON data via GitHub REST API (based on online tutorials)
- [x] Read documentation on rate limiting
- [x] Retrieve data on all prework issues (date range from Nov 1, 2021 to now) using REST API in Jupyter Notebook
- [x] Put JSON data in a tabular format and clean data
- [x] Get distribution of issues completed by each complexity level for each prework author:
Put data in columns: GitHub Handle, Date Prework Closed, No. of Good First Issues Completed, No. of Good Second Issues Completed, No. of Small Complexity Issues Completed, No. of Medium Complexity Issues Completed, No. of Large Complexity Issues Completed
- [x] Export data as Excel file and add to Google Drive folder (GitHub Data Analysis)
- [x] Manually check accuracy of numbers in dataset/spreadsheet)
- [x] Write documentation on process and considerations (not complete yet)

- [x] Duplicate data in another spreadsheet and perform following analysis:
1. Of the 202 people, how many people left the team?
2. How many people started and got to "Complexity: Large" issues (completed at least 2 for combined first and second good issue, and one of every other complexity type)?
- [x] Perform above analysis again on only closed prework issues.

- [x] Clean data and get number and percentage of closed large issues that were unassigned in Google Sheets

- [x] Create Google spreadsheet with list of issues that have more than one complexity label and unassigned closed large issues.

- [x] Perform cohort analysis on closed prework authors
1. Clean data from API and create dataset
2. Import into Google Drive and visualize data in Google Sheets

- [x] Research how to connect data to Looker Studio in a way that new data can come in and Looker visualizations are automatically updated.

- [x] Create new repository with Sophia and Chelsey's help that has GitHub Actions that perform cron job so that Python script can be run automatically daily for fresh data.

- [ ] Add automation components to Python script and verify data cleaning accuracy.

- [x] Create Looker dashboard with data pulled in.
- [ ] Refine the Looker dashboard so that it is more intuitive

- [ ] Investigate correlation between number of issues available and cohort performance:
- [ ] Design analysis and investigate where/how data can be obtained

### Might be separated into another issue
- [x] Get project board column data from GitHub and clean the data
- [x] Set up data source and create Looker dashboard to show live number of issues available per role
- [ ] Create separate dashboard pages for developers (front end, back end, front and back end, and dev lead)
- [x] Create documentation of process for GitHub class using Hack for LA template
- [ ] Set up automation of running of Python script so that dashbboard updates automatically

### Resources/Instructions

1. [GitHub API Documentation](https://docs.github.com/en/rest?apiVersion=2022-11-28)
4. [GitHub Rate Limiting](https://docs.github.com/en/rest/rate-limit?apiVersion=2022-11-28)
5. [Link to GitHub Data Analysis Folder](https://drive.google.com/drive/folders/1rk8PGeJ_S0Gvn4mpzsoNVKPOPPTgkddW?usp=share_link)
6. [Spreadsheet with accurate numbers as of 03/26/2023](https://docs.google.com/spreadsheets/d/1guNth22yDEphNI7kyA8RcsIzHnUjG9DhJd1Ea3-_Yzw/edit?usp=sharing)
7. [Link to process documentation](https://docs.google.com/document/d/1SBs4C1KGLjOPMaVBV3JjP3pijXwLjrw6KSbR0XwwxdY/edit?usp=sharing)
8. Using Google Sheets API to add and refresh dataframe in Python to Google Sheets:
https://www.youtube.com/watch?v=sVURhxyc6jE
https://medium.com/@jb.ranchana/write-and-append-dataframes-to-google-sheets-in-python-f62479460cf0
https://www.youtube.com/watch?v=3wC-SCdJK2c
9. [Slides documentation process from Python to GitHub](https://docs.google.com/presentation/d/1hZCNMZewl4L_so1ceGMOuP0mTbaBkUuwzAbBPgcGYfU/edit)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.