Kepler Model for Kubernetes Clusters

### Type of project

Building a plug-in for Impact Framework

### Overview

### Context ###
[Kepler](https://github.com/sustainable-computing-io/kepler) is a CNCF project for measuring energy consumption of bare metal Kubernetes clusters using RAPL. A Kepler model for Impact Framework would query Prometheus for Kepler metrics and allow calculating the SCI score for applications deployed to Kubernetes clusters when Kepler is deployed.

### Idea/Problem statement ###
We are thinking of building a pipeline which has following steps
1. Prometheus importer plugin - Pulls Kepler metrics from prometheus server
2. Kepler plugin - Analyze the Kepler prometheus metrics and normalize the data to consume by official IF SCI models
3. IF official SCI Models - Calculates SCI score
4. Prometheus Metrics Publisher plugin - Finailly, publishes the SCI score back to the prometheus server

### Scope ###
For hackathon, it would be limited to

1. Measuring SCI score for Kubernetes workload running on bare metal
2. We would require Kepler and Prometheus/Grafana to be installed on the Kubernetes cluster
3. IF pipeline to calculate SCI score will be invoked via a Kubernetes cron job

### Questions to be answered

1. What programming languages can we use?
2. Are there any generic models we can build on? e.g for Prometheus or Kubernetes

### Have you got a project team yet?

Yes and we are still open to extras members

### Project team

@greenscale-nandesh @rossf7 @SFulpius @tmcclell

### Terms of Participation

- [X] I agree to the hackathon [Rules & Terms](https://github.com/Green-Software-Foundation/hack/wiki/Rules-and-Terms) and [Code of Conduct](https://github.com/Green-Software-Foundation/hack/wiki/Code-of-Conduct)

___

## Project Submission
### Summary
The Kepler Importer plugin is an essential tool for Kubernetes users who are serious about understanding the environmental impact of their workloads. With this powerful plugin, users can achieve a deeper insight into how their workloads affect the environment, and make informed decisions to reduce their impact.

### Problem
[Kepler](https://github.com/sustainable-computing-io/kepler) (Kubernetes-based Efficient Power Level Exporter) is an impressive tool that leverages eBPF to probe performance counters and system stats. Using advanced ML models, it accurately estimates energy consumption based on these stats and exports them as Prometheus metrics. However, it currently lacks support for assessing the environmental impact of K8s workloads.

### Application
The Kepler Importer plugin is a powerful tool that enables easy and efficient importing of Kepler metrics using Impact Framework. By leveraging this plugin, users can effortlessly calculate the SCI score using official plugins- a crucial metric for evaluating the environment impact of applications deployed to Kubernetes clusters. With its advanced features and user-friendly interface, the Kepler Importer plugin is the perfect solution for businesses looking to optimize their Kubernetes deployments.

### Prize category
📦 Best Plugin

### Judging Criteria
**Overall Impact**
This plugin could increase the practice of measuring energy consumption in Kubernetes clusters to enable reducing their footprint. It could help increase the adoption of both Impact Framework and Kepler. This is important because in 2022 [a report](https://enterprisersproject.com/article/2022/10/kubernetes-statistics-2022) stated 70% of organizations had adopted Kubernetes.

**Opportunity**
It opens the door to using Impact Framework and Kepler across both Kubernetes and the wider cloud native ecosystem. Kepler supports using both RAPL on bare metal and can use machine learning when deployed to the cloud and RAPL is not available. Our plugin supports both these modes.

**Modular**
We have followed the Unix philosophy to keep a small scope and ensure the plugin just integrates with Prometheus. We also tested it with the other IF plugins to ensure they can use the energy value the plugin provides.

### Video
https://www.youtube.com/watch?v=sn0zrhghrjE
https://docs.google.com/presentation/d/1xXVTVBOXM1ZJXWYE67anPl9cWEwAis8-KZoJL4p-UM8/edit?usp=sharing

### Artifacts
Plugin - https://github.com/greenscale-nandesh/if-unofficial-models/tree/main/src/lib/kepler-importer
Manifest - https://github.com/greenscale-nandesh/if-unofficial-models/blob/main/kepler-pipeline-manifest.yaml

### Usage
https://github.com/greenscale-nandesh/if-unofficial-models/blob/main/src/lib/kepler-importer/README.md

### Process
We started by looking at the existing plugins and agreeing the design in a Miro board and then created GitHub issues. We then found a Prometheus SDK for TypeScript and used the WG Green Reviews cluster to test with Kepler deployed. We then created a manifest with a full SCI pipeline to integrate with the other plugins. During development we held twice a week team calls to sync with weekly calls prior to the start to plan.

### Inspiration
Some of our teammates are part of CNCF [Green Reviews](https://github.com/cncf/tag-env-sustainability/blob/main/working-groups/green-reviews/charter.md) working group. During the development of the monitoring pipeline they realized that having an Impact Framework plugin to generate SCI score for K8s workloads using Kepler would be useful for the working group and also the wider cloud native community.

### Challenges
One challenge was to debug input and output between plugins. Initially we didn’t realize we needed to pass all inputs to the next stage in the pipeline. Another challenge was Kepler provides metrics in Joules which we needed to convert to kWh.

### Accomplishments
- Great team work - Working across geo-locations to deliver this plugin (even thru the time change twice)
- Integrating Kepler Plugin with rest of the IF official plugins to generate the SCI score

### Learnings
- Getting a deeper understanding of how SCI scores are calculated
- How to use Impact Framework and write manifests
- Learning Kepler’s design and the metrics it exposes
- How to calculate embodied carbon using the Boavizta API

### What's next?
- Discuss with Green Reviews WG whether they can use the plugin for calculating SCI scores for CNCF projects
- Discuss with the Kepler team if there are any enhancements they would recommend and how to increase adoption of both the plugin and Kepler itself