[FT] Problems with CorpusLevelMetricGrouping

## Issue encountered
I’ve been working on setting up a custom evaluation task using a corpus-level metric, and while I’ve managed to get it working after a lot of trial and error, I’m still unclear on whether I’m doing it the right way — or just hacking around it.

Specifically, I have an external evaluation function that processes the entire dataset and returns multiple metrics. It works independently, and I’d like to integrate it into a custom task in lighteval.

Since the evaluation is the fastest at the corpus level, I initially tried to use corpus_level_fn with CorpusLevelMetric. However, because the evaluator returns multiple metrics, I need some grouping.
Therefore, CorpusLevelMetricGrouping felt like the correct solution — but I haven’t been able to get it working.

Below is what I currently have. This works, but I believe it’s not the intended approach, and it would be way better if i corpus_level_fn would.

```py
from typing import List

import numpy as np
from datasets import get_dataset_config_names
from evaluate import load
from lighteval.metrics.utils.metric_utils import (
MetricUseCase,
MetricCategory,
SampleLevelMetricGrouping,
)
from lighteval.tasks.lighteval_task import LightevalTaskConfig
from lighteval.tasks.requests import Doc

symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")

def generate_pass_at_k_n_sample_level_metric(k: int, n: int):
"""Factory function to create pass@k metrics and detailed metrics"""

assert k >= 1
assert n >= k

def prolog_data_preparator(
predictions: List[str], golds: List[str], formatted_doc: Doc
):
references = [
{
"validation_program": gold,
"evaluation_config": {
"positive_predicate": "eastbound",
"negative_predicate": "westbound",
},
}
for gold in golds
]
results = symbolic_judge.compute(predictions=predictions, references=references)
return {
f"accuracy@{k}:{n}": results["accuracy"],
f"partial_score@{k}:{n}": results["partial_score"],
f"syntax_score@{k}:{n}": results["syntax_score"],
}

def top_k_mean_fn(values):
return np.mean(np.sort(values)[-k:])

metric_directions = {
"accuracy": True,
"partial_score": True,
"syntax_score": True,
}
metric_names = [
f"{metric}@{k}:{n}" for metric, direction in metric_directions.items()
]

return SampleLevelMetricGrouping(
metric_name=metric_names,
sample_level_fn=prolog_data_preparator,
category=MetricCategory.GENERATIVE,
use_case=MetricUseCase.REASONING,
corpus_level_fn=dict.fromkeys(metric_names, top_k_mean_fn),
higher_is_better=dict.fromkeys(metric_names, True),
)

def prompt_fn(line: dict, task_name: str):
return Doc(
task_name=task_name,
query=line["prompt"],
choices=[line["validation program"]],
gold_index=0,
)

available_subsets = get_dataset_config_names("ahmad21omar/MetaBench")

TASKS_TABLE = []

for subset in available_subsets:
_task_name = f"V-LOL-Benchmark:{subset}"
task_config = LightevalTaskConfig(
name=_task_name,
suite=["custom"],
prompt_function=prompt_fn,
hf_repo="ahmad21omar/SRL-Bench",
hf_subset=subset,
hf_avail_splits=["validation"],
evaluation_splits=["validation"],
few_shots_split=None,
few_shots_select=None,
metric=[
generate_pass_at_k_n_sample_level_metric(1, 1),
generate_pass_at_k_n_sample_level_metric(2, 8),
generate_pass_at_k_n_sample_level_metric(4, 32),
],
trust_dataset=True,
generation_size=32768,
version=1,
)
TASKS_TABLE.append(task_config)
```

## My questions
- Is CorpusLevelMetricGrouping the right tool here? If so, are there any working examples or documentation that I could look at?

## Motivation
In theory, this use case is simple: I have an evaluator that takes in the full corpus and returns a dict of scores. And I want lighteval to just calculates `"{metric}@{n}"`.
This cant be that complicated, right?