Question about Request Count and Generative Sampling

Hi everyone,

First off, thanks for this great library! I've been using it extensively recently and really appreciate the work that has gone into it. While analyzing the code, I came across a couple of questions and would love your insights.

1. Request Count for Metrics
I'm evaluating a custom community summarization task with the following metrics: BERTScore, METEOR, BLEU, ROUGE-1/2/L, Extractiveness, and a custom LLM-as-Judge metric.

Given a set of samples, I noticed that the number of requests being generated is twice the number of samples. I have two metric categories: `MetricCategory.GENERATIVE` and `MetricCategory.LLM_AS_JUDGE`. Does this mean the model gets called twice per sample, effectively doubling costs when using an endpoint? I see that LiteLLM has caching, but even when running the model locally, it seems like each sample is processed twice thus generating an overhead.

Would switching my judge metric to MetricCategory.GENERATIVE prevent this duplication, or is there a better way to ensure only one evaluation pass per sample? And would this change have any other side effect that I am not seeing currently

2. Generative Sampling Behavior
I want to use sampling (`do_sample=True`) when generating locally, but the setting gets overwritten unless the metric category is `MetricCategory.GENERATIVE_SAMPLING`. To work around this, I currently instantiate my metrics and then dynamically change the category, but this feels like a hack.

However, when I do this, I suddenly see three times the number of requests—one for `GENERATIVE`, one for `GENERATIVE_SAMPLING`, and one for `LLM_AS_JUDGE`. I'm guessing something is off in how I'm modifying the category dynamically.

Is there a cleaner way to enable sampling without unintended extra requests?

Thanks in advance for your help! I really appreciate your time and insights.