Issue with Perplexity Score Using `max_length > 1024` in lm-evaluation-harness

I am using a custom model in the lm-evaluation-harness script and have configured the tokenizer to use `GPT-2` with a `max_length` of 2048. However, I’ve noticed that when I increase the max_length beyond 1024, the perplexity score unexpectedly decreases. This reduction in perplexity doesn’t align with my expectations, as the perplexity should not inherently decrease with a larger sequence length.

Could someone kindly investigate and help understand why this is happening?

Start by reproducing the custom-model evaluation with the GPT-2 tokenizer at max_length 1024 and 2048, then trace how perplexity is computed for each sequence length. Done means identifying why the score changes unexpectedly and documenting or correcting the behavior with a regression test.

Written by the indexing model from the issue text.