Sometimes the state is not updating on swipe

## Environment
```
System:
OS: Linux 5.4 Ubuntu 18.04.6 LTS (Bionic Beaver)
CPU: (8) x64 Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
Memory: 369.14 MB / 15.48 GB
Shell: 4.4.20 - /bin/bash
Binaries:
Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
Yarn: Not Found
npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
Android SDK:
API Levels: 22, 23, 24, 27, 28, 29, 30
Build Tools: 23.0.2, 25.0.0, 27.0.1, 27.0.3, 28.0.2, 28.0.3, 29.0.2, 30.0.2, 31.0.0
System Images: android-28 | Google Play Intel x86 Atom, android-29 | Intel x86 Atom, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play ARM 64 v8a, android-29 | Google Play Intel x86 Atom, android-30 | Android TV Intel x86 Atom, android-30 | Google TV Intel x86 Atom, android-30 | Google APIs Intel x86 Atom, android-30 | Google APIs Intel x86 Atom_64, android-30 | Google Play Intel x86 Atom, android-30 | Google Play Intel x86 Atom_64
Android NDK: Not Found
IDEs:
Android Studio: Not Found
Languages:
Java: 15.0.2 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.64 => 0.64.2
npmGlobalPackages:
*react-native*: Not Found
```

## Description

When drag, sometimes the state is not getting updated. It still remains `idle`. I'm showing a loader on page swipe. For that, I update the status of the loader as `true`, when the state is `dragging`. Again I update the loader as `false`, when the state changes to `idle`. Sometimes after a couple of page swipes, the state doesn't change to `dragging` from `idle` it still showing `idle` so the loader is not getting displayed.

## Reproducible Demo

https://user-images.githubusercontent.com/80961628/134028480-ac77704c-b29a-44da-a7d4-a9f7c95b8fc1.mp4

Start by reproducing the behavior from the linked demo with repeated page swipes, then trace how swipe events update the pager state from idle to dragging. Done means the state reliably enters dragging during swipes so the loader behavior described in the issue works consistently.

Written by the indexing model from the issue text.