Green-Software-Foundation / Green-Software-Foundation/hack
GCP Importer
- Dominant language
- No language data
- Stars
- 16
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
### Prize category
Best Plugin
### Overview
### Story
As a user of a Google Cloud Platform, I want to be able to retrieve usage metrics for my application. Right now, IF only enables me to do this for Azure VMs.
### Rationale
We have implemented an Azure importer in IF, but no such model exists for Google Cloud Platform. This will address the current limitation of IF having a Azure-only importer and enable GCP users to retrieve observations of GCP resources via Google Monitor (or similar APIs) and pipeline that utilisation data into other IF models. As suggested here (https://github.com/Green-Software-Foundation/hack/discussions/30)
### Scope
This should be a model plugin that conforms to the IF [model plugin interface](https://if.greensoftware.foundation/tutorials/how-to-build-models).
### Implementation guidelines
This model MUST conform to the IF model plugin interface
The model MUST return metadata about the application and usage data (e.g. CPU-util, RAM-used, etc) similar to the [azure importer](https://if.greensoftware.foundation/tutorials/how-to-use-azure-vm)
The model must return data in the expected units (kWh, seconds, GB...)
The model must accept duration in units of seconds an input.
The model SHOULD be written in Typescript
The model MUST have unit tests that demonstrate correct execution
The model MUST have documentation in the form of a README that explains how to use.
### Questions to be answered
No
### Have you got a project team yet?
I am working on this project alone
### Project team
_No response_
### Terms of Participation
- [X] I agree to the hackathon [Rules & Terms](https://github.com/Green-Software-Foundation/hack/wiki/Rules-and-Terms) and [Code of Conduct](https://github.com/Green-Software-Foundation/hack/wiki/Code-of-Conduct)
Project Submission
=====
### Summary
As a user of Google Cloud Platform, I want to be able to retrieve usage metrics for my application. GCP importer Plugin enables GCP users to retrieve observations of GCP resources via Google Monitor APIs and automatically populate your manifest with usage metrics that can then be passed along a plugin pipeline to calculate energy and carbon impacts.
### Problems
To enable GCP users to retrieve observations of GCP resources, below problems has been addressed
1. Authentication with GCP APIs
2. Retrieve the monitor metrics of Compute engine VMs from all regions of one GCP projects, eg: CPU and memory usage
3. Populate usage metrics that can be passed along with other IF plugins
### Application
- **Method**: Provide one method `GcpImporter` to retrieve CPU utilization and memory data from GCP, the method is a `execute` type.
- **Config**: accept one GCP project ID. GCP importer plugin will list usage data for all compute engine VMs of one GCP project.
- **Inputs**: accept `timestamp` and `duration` as the input. `timestamp` is the start time of usage data and `duration` is the seconds that the usage data covers.
- **Validation**: the GCP importer plugin will validate the config and inputs data. `ConfigValidationError` error will be raised if validation not passed
- **GCP APIs**: the first version of GCP plugin use Google Compute API and Google Monitoring API. Google Compute API is used to retrieve all compute engine VMs of one GCP projects and some instance details such as instance name and machine type. Google Monitoring API is used to CPU utilization and memory utilization of VMs.
- **Enrich Outputs**: the GCP plugin populate the manifest with usage metrics can be passed to plugins. eg ccf.
### Prize category
Best Plugin
### Judging criteria
#### Overall Impact
GCP is the world's Top3 cloud provide, the GCP importer plugin can enable GCP uses to use the Impact Framework
#### Opportunity
The GCP importer is easy to use for systems hosted in Google Compute engine VMs. And the GCP importer plugin is easy to be extended to support more GCP services such as App engine, Cloud Run
#### Modular
The plugin is built based on the arch of [Azure importer](https://github.com/Green-Software-Foundation/if-unofficial-plugins/blob/main/src/lib/azure-importer/README.md), and it works well with other plugins such as [CCF](https://github.com/Green-Software-Foundation/if-unofficial-plugins/blob/main/src/lib/ccf/README.md)
### Video
https://youtu.be/q4mPejLqj-A
### Artefacts
https://github.com/tianbiao/if-unofficial-plugins/tree/gcp-importer
### Usage
https://github.com/tianbiao/if-unofficial-plugins/blob/gcp-importer/src/lib/gcp-importer/README.md
### Process
- Ideation: As I am a GCP user, I want to build a plugin to let me to use Impact Framework.
- Research: Understand the current unofficial plugins and follow the design of Azure Importer plugin as we are similar plugin for different cloud provider.
- Code
### Inspiration
1. I am a GCP user for 4 years
2. I'd like to understand and contribute to Green Software open source tools.
### Challenges
1. Choosing the right Google monitor metrics when retrieving CPU and memory utilizations
2. As the GCP provides a lot of compute services, for example Compute engine, App engine and Kubernetes Engine, it's hard to implement the GCP importer plugin to support them all during such short hack time.Finally, I start to build the plugin from Compute engine usage metrics.
### Accomplishments
The GCP importer plugin can response enough details of the usage data for Carbon calculations.
### Learnings
- Understand IF and it's plugins
- How to build IF plugins
### What's next?
1. Introduce to my teams that is using GCP to use IF and this plugin to calculate Carbon
2. Extend the current plugin to support more GCP resources
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.