redhat-developer / redhat-developer/rhdh-plugins

Extend the scorecard github module with a new AI adoption metric provider

Open
#4,579 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement feature triaged workspace/scorecard
Dominant language
TypeScript
Stars
48
Forks
120
Avg merge
2d 6h
Merged PRs (30d)
337

Description

Scorecard is a backstage plugin and supports multiple metric, grouped as modules in folder workspaces/scorecard/plugins.

Extend the existing GitHub plugin/module under plugins/github with a new GithubAiAdoptionMetricProvider.

  • getProviderDatasourceId: "github"

  • getProviderId: "github.aiAdoption"

  • getMetrics: 3 metrics, for the AI adoption rate for the last 7 days, 30 days and 90 days. ID should be calculated from this.getProviderId() + "Rate[7d]" (etc.). The threshold for all 3 metrics should be defined in the provider with:

    export const AI_ADOPTION_RATE_TIME_RANGES = ["7d", "30d", "90d"];
    
    export const AI_ADOPTION_RATE_THRESHOLD: ThresholdConfig = {
      rules: [
        { key: 'success', expression: '>=0.2' },
        { key: 'warning', expression: '>=0.1' },
        { key: 'error', expression: '>=0' },
      ],
    };
    
  • catalogFilter similar to the other GitHub providers:

      getCatalogFilter(): Record<string, string | symbol | (string | symbol)[]> {
        return {
          'metadata.annotations.github.com/project-slug': CATALOG_FILTER_EXISTS,
        };
      }
    
  • calculateMetrics should request the commit history via GraphQL from GitHub for the biggest time range that is defined (currently 90 days).

    Then it should take a look into all received commit messages and check if they was AI assisted or not. It could probably ignore merge commits.

    It should check commit message lines that starts Assisted-by: or Co-Authored-By: (or Co-authored-by: ). When the value starts with a known AI tool, like Claude, Cursor it should count this message as AI assisted. Add extra conditions that identifies clearly AI assistents.

    Log at the end how many commits was analyied, how many commits was ignored, marked as ai assisted commits and NOT marked as ai assisted commits.

    Finally return the Map with the metric Ids and the ratio (0-1) per time range.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing GitHub providers under workspaces/scorecard/plugins/github and trace how they fetch commit history through GraphQL and expose metric maps. Done means a GithubAiAdoptionMetricProvider defines the requested IDs, thresholds, catalog filter and 7d/30d/90d ratios, while handling qualifying commit messages and logging the requested counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, graphql, typescript
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.