Slider pulls focus back to itself when I try to manually move it elsewhere onChangeEnd

### Discussed in https://github.com/adobe/react-spectrum/discussions/4550

https://codesandbox.io/s/spring-glitter-329tzn?file=/src/App.js

1. Using the mouse, move the slider
2. Press Esc

See in the console that the button briefly receives focus before it loses it back to the thumb, which displays the focus ring after hitting Esc.

The reproduction is in the CodeSandbox's src/App.js; first read the linked discussion and reproduce the mouse-move/Esc sequence while watching the console. Trace the slider's onChangeEnd focus behavior and verify completion when the button keeps focus instead of focus returning to the thumb.

Written by the indexing model from the issue text.