Potential Pipeline Metrics for Risk Assessment

In preparation of the "regulatory readiness" project at the upcoming nf-core hackathon, we've started brainstorming potential metrics that would potentially be needed/useful in that context.

• Maintenance

  • Number of solved issues -> aggregate from issue_stats
  • Median time to bug closure
    • median time to issue closure can be aggregated from issue_stats
    • issue labels are not properly captured, so we can't distinguish between bugs and other issues #84
  • Number of releases #83
  • Date of last release -> nfcore_pipelines.last_release_date
  • Has issue tracker -> all nf-core repos have an issue tracker
  • Has maintainer #82
    • this is also a governance topic

• Popularity

  • Number of contributors -> aggregate from contributor_stats
  • Number of downloads clones -> aggregate from traffic_stats.clones_uniques
  • Number of Views -> aggregate from traffic_stats.views
  • GitHub Stars -> nfcore_pipelines.stargazers_count
  • GitHub Forks -> nfcore_pipelines.forks_count
  • Citations #79

• Quality

  • Dependencies containerized and containers versioned https://github.com/nf-core/stats/issues/121
  • Test coverage of modules #80
  • Pipeline end-to-end tests
    • needs to be manually checked for the given set of input parameters, hard to flag this globally in a use-case agnostic manner
  • PR reviews mandatory? / 4-eye principle (nf-core guidelines reference) #89
    https://nf-co.re/pipelines.json
    -> This is also a governance topic.
  • Size of codebase #81

We plan to scan existing issues and PRs to identify which of these metrics are already being tracked or discussed. Feel free to comment, suggest additions, or flag any metrics that might be difficult to track or interpret.

CC @apeltzer @grst