Poor regression test performance

Howdy,
I get models with 0.97 test r squared, but when I test "accuracy" with the following formula, it's something like 50%.

in_range = (y >= p - avg_mae) & (y <= p + avg_mae)
accuracy = sum(in_range) / length(in_range)

I didn't read this in the docs, but are we meant to shuffle before? Also, what specifically does subsample do? I know it randomly selects subset, but does it ignore the rest?