Monitoring of Spark emissions via Spark plugin

Hi,

I am working on a prototype of a Spark plugin to report the energy consumption of executors. The logic behind is similar to CodeCarbon's, although the computation method differs slightly: the executor process scheduling is sampled regularly, converted to Wh with the TDP (provided or inferred), and aggregated by the driver. The total energy is published as a Spark metric, accessible via the REST API.

I wanted to know if you'd be interested in integrating it with CodeCarbon, for example with Spark cluster as a new type of resource alongside CPU, GPU, or RAM. It would let CC factor in the energy mix and cloud provider data, which could be cumbersome to access from a private Spark cluster (it's better not to assume internet connectivity). And it would benefit from CC's ease of use, which is a strong factor of adoption.

In any case, it's a prototype, it needs more testing and validation, and only handles CPU for now (but many data engineering pipelines don't use GPUs anyway). Here it is: https://github.com/tvial/ccspark (Apache 2.0 license). Note that it embeds your CPU database for the TDPs, I'm open to remove it if you think it's a bad idea :)

Let me know if it can be of any help
Thanks

Start by reviewing the linked ccSpark prototype and the existing CodeCarbon support for CPU, GPU, or RAM resources. Validate how Spark executor sampling, TDP-based conversion, driver aggregation, and REST-exposed metrics could map to CodeCarbon's model. Done should include a defined integration scope and testing and validation for Spark workloads.