RFC: Extending AI Energy Score to Agentic Workloads with Verified Useful Work/Joule

I've been thinking about how **AI Energy Score** could evolve as AI workloads become increasingly agentic.

Today, AI Energy Score is well positioned around a very important question: **how much energy does an AI workload consume, under a comparable evaluation setup?**

As workloads move from single-model inference toward agents that reason, call tools, browse, execute code, retry, and verify their own work, I wonder whether it could be useful to complement the existing energy measurements with an **outcome-normalized energy metric**.

The core question would be:

> **How much verified useful work can a system produce per joule?**

## Motivation

Consider two coding agents given the same GitHub issue.

**Agent A**

* 27 model calls
* 182k tokens
* 42 kJ consumed
* Issue successfully fixed
* Tests pass

**Agent B**

* 8 model calls
* 61k tokens
* 11 kJ consumed
* Issue successfully fixed
* Tests pass

Both agents achieve the same verified outcome, but Agent B uses roughly one quarter of the energy.

If we only report task success, both systems score the same.

If we only report raw energy, we know which system consumed less energy, but not whether it actually accomplished the task.

Combining the two gives us a potentially useful additional view: **energy per verified outcome**.

## Proposed concept: Verified Useful Work per Joule

One possible metric is:

`VUW/J = Sum(w_i * V_i) / Sum(E_i)`

where:

* `V_i` is the verified completion score for task `i`
* `w_i` is an optional task weight or difficulty factor
* `E_i` is the total energy consumed while attempting task `i`

The inverse may be even easier to interpret operationally:

`J/VT = Total Energy / Number of Verified Tasks`

This gives two complementary views:

* **Verified Tasks / kJ** — useful work produced from a fixed energy budget
* **J / Verified Task** — energy required to complete one verified unit of work

I see this as a complement to AI Energy Score's existing energy measurements, not a replacement for them.

## Benchmark-defined verifiers

The key idea would be that each workload defines how useful work is verified.

For example:

* Coding agent → tests, build checks, regression checks
* Browser agent → target state successfully reached
* Data agent → expected artifact or query result produced
* Research agent → reference-backed evaluator or rubric
* Robotics agent → expected environment state reached
* Traditional QA → existing correctness grader

This allows very different workloads to share a common high-level abstraction:

`Task -> Agent/System -> Verifier -> Energy-normalized outcome`

## Why this may matter for agentic workloads

For a single model inference benchmark, the workload boundary is relatively clear.

For an agent, the workload may look more like:

`Goal -> Reasoning -> Tool Use -> Observation -> Retry -> Verification -> Result`

The total energy cost of producing the result may therefore span much more than a single model forward pass.

Two agents can reach the same outcome through radically different trajectories. One may require many retries, model calls, browser interactions, or code executions, while another may complete the same task much more directly.

A verified-work metric would make that difference visible.

## Latency should remain visible

Optimizing energy alone can create another problem: a system might reduce power consumption simply by operating very slowly.

So it may be useful to report outcome-normalized energy under a latency constraint, for example:

* `VUW/J @ 30s`
* `VUW/J @ 5min`
* or more generally `VUW/J subject to latency <= T`

I would prefer keeping latency visible rather than collapsing everything into one opaque scalar score.

## Defining the energy boundary

For model benchmarking, accelerator energy can be a reasonable and reproducible boundary.

For agentic systems, however, useful work may involve:

`Model inference + CPU + memory + browser/runtime + tools + code execution + networking`

It may therefore be useful to distinguish measurement scopes explicitly, for example:

* **Accelerator Energy**
* **Compute Node Energy**
* **Whole Agent System Energy**

This would preserve comparability without implying that every result measures exactly the same physical boundary.

## Why I think this could fit AI Energy Score

AI Energy Score already provides a foundation for making AI energy consumption measurable and comparable.

As workloads become more agentic, an outcome-normalized layer could make the results even more useful for system-level comparisons.

Instead of asking only:

> How much energy did this workload consume?

we could additionally ask:

> **What verified outcome did we get for that energy?**

This also helps move evaluation away from proxy metrics such as parameter count, token count, or raw FLOPs and toward the physical cost of accomplishing useful work.

A larger system is not necessarily less efficient if it reaches the correct outcome directly. A smaller system is not necessarily more efficient if it requires many retries and ultimately consumes more energy to solve the same task.

## Questions

I'd be very interested in hearing thoughts on a few points:

1. Would an outcome-normalized metric such as **Verified Useful Work/Joule** fit naturally alongside AI Energy Score's existing measurements?
2. Could workload-specific verifiers be integrated into the current evaluation framework?
3. Would metrics such as `Verified Tasks / kJ` and `J / Verified Task` be useful for agentic workloads?
4. Should latency constraints be reported together with energy efficiency?
5. For agentic workloads, should the measurement boundary eventually expand beyond accelerator energy to whole-system energy?
6. More broadly, could **verified useful work** provide a common abstraction for comparing energy efficiency across different classes of agents?

I'm curious whether this direction has already been discussed, and whether it might fit the project's roadmap as AI workloads become increasingly agentic.

This is an RFC proposing Verified Useful Work per Joule for agentic workloads, including workload-specific verifiers, latency constraints, and broader energy boundaries. No files, tests, or entry points are named; start by reviewing the existing AI Energy Score measurements and evaluation framework. Done would require a decided direction for whether and how the proposal fits the project roadmap.

Written by the indexing model from the issue text.