[BUG] alpaca_eval_v2:bpb::tulu yields incorrect, inflated BPB scores due to num_chars defaulting to 1

Title: alpaca_eval_v2:bpb::tulu yields incorrect, inflated BPB scores due to num_chars defaulting to 1
Description:
I encountered a critical bug when running the likelihood-based evaluation (alpaca_eval_v2:bpb::tulu) using OLMES. The computed Bits Per Byte (BPB) score is astronomically high (~250) because the evaluator fails to correctly calculate the length of the target text, treating num_chars as 1 regardless of the actual sequence length.
Reproduction Command:
olmes \
--model allenai/Llama-3.1-Tulu-3-8B \
--output-dir eval_results/debug_bpb \
--batch-size 256 \
--model-args "trust_remote_code=True,max_length=8192" \
--task alpaca_eval_v2:bpb::tulu \
--model-type vllm
Observed Behavior (Logs):
In the logs, despite the model processing a full continuation string (e.g., "1. Julie Andrews..."), the metrics dictionary reports num_chars: 1. This causes the negative log-likelihood sum to be divided by 1, resulting in an impossible BPB score.
Relevant log snippet:
code
JSON
{
"metrics": {
"sum_logits_corr": -173.1932856232157,
"bits_per_byte_corr": 249.86509428405864
},
"model_output": [
{
"continuation": "1. Julie Andrews - Before becoming a household name...",
"num_tokens": 452,
"sum_logits": -129.47763001374045,
"num_chars": 1 <-- THE BUG IS HERE
}
]
}
Analysis:
The metric calculates BPB roughly as (Total Negative Log Probs) / (Number of Bytes).
Actual Calculation: -173.19 / 1 = -173.19 (converted to bits ≈ 249.8).
Expected Calculation: If the text is ~450 tokens, the length in bytes should be ~1500+. The score should be -173.19 / 1500 ≈ 0.11.
This results in the instructed model appearing significantly "worse" (higher BPB) than base models, rendering the metric unusable for this task configuration.
Environment:
Model: allenai/Llama-3.1-Tulu-3-8B
Framework: OLMES / OE-Eval
Task: alpaca_eval_v2:bpb::tulu