Difference between latest lm-eval-harness and lit-gpt eval

Hi,

I was trying to evaluate the Pythia-160M model against some tasks in lm-eval-harness and noticed that the results produced by the code in lit-gpt/eval and the latest version of lm-eval-harness are different. Here're the outputs of the two commands.

Command

python eval/lm_eval_harness.py \
    --checkpoint_dir checkpoints/EleutherAI/pythia-160m \
    --eval_tasks "[hellaswag,openbookqa,winogrande,boolq,piqa]" \
    --save_filepath pythia_160m_results.json

Result

{
    "results": {
        "winogrande": {
            "acc": 0.5185477505919495,
            "acc_stderr": 0.014042813708888378
        },
        "boolq": {
            "acc": 0.43700305810397555,
            "acc_stderr": 0.008675365793227082
        },
        "openbookqa": {
            "acc": 0.152,
            "acc_stderr": 0.01607198236791175,
            "acc_norm": 0.248,
            "acc_norm_stderr": 0.019332342821239103
        },
        "hellaswag": {
            "acc": 0.28141804421429994,
            "acc_stderr": 0.0044877188433302805,
            "acc_norm": 0.3053176658036248,
            "acc_norm_stderr": 0.004596006250433537
        },
        "piqa": {
            "acc": 0.5979325353645266,
            "acc_stderr": 0.011439867127267531,
            "acc_norm": 0.5908596300326442,
            "acc_norm_stderr": 0.011471593460443312
        }
    },
    "versions": {
        "winogrande": 0,
        "boolq": 1,
        "openbookqa": 0,
        "hellaswag": 0,
        "piqa": 0
    },
    "config": {
        "model": "pythia-160m",
        "batch_size": 16,
        "device": "cuda:0",
        "num_fewshot": 0,
        "limit": null,
        "bootstrap_iters": 100000,
        "no_cache": true
    }
}

Command

lm_eval --model hf --model_args pretrained=EleutherAI/pythia-160m \
    --tasks hellaswag,openbookqa,winogrande,boolq,piqa \
    --device cuda:0 \
    --batch_size 16

Result

|  Tasks   |Version|Filter|n-shot| Metric |Value |   |Stderr|
|----------|-------|------|-----:|--------|-----:|---|-----:|
|boolq     |Yaml   |none  |     0|acc     |0.5688|±  |0.0087|
|hellaswag |Yaml   |none  |     0|acc     |0.2838|±  |0.0045|
|          |       |none  |     0|acc_norm|0.3027|±  |0.0046|
|openbookqa|Yaml   |none  |     0|acc     |0.1500|±  |0.0160|
|          |       |none  |     0|acc_norm|0.2680|±  |0.0198|
|piqa      |Yaml   |none  |     0|acc     |0.6230|±  |0.0113|
|          |       |none  |     0|acc_norm|0.6192|±  |0.0113|
|winogrande|Yaml   |none  |     0|acc     |0.5130|±  |0.0140|

As you can see for some tasks like BoolQ and PIQA the results are quite different. I wonder what could cause such a big difference.

Best,
Ajinkya

Reproduce the two commands in the issue, then compare eval/lm_eval_harness.py with the installed lm-eval-harness behavior for the listed tasks. Check why BoolQ and PIQA differ while the other results are closer; done means identifying the cause and making the evaluation results consistent or documenting the remaining difference.