Sometimes the state is not updating on swipe

## Environment
```
System:
OS: Linux 5.4 Ubuntu 18.04.6 LTS (Bionic Beaver)
CPU: (8) x64 Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
Memory: 369.14 MB / 15.48 GB
Shell: 4.4.20 - /bin/bash
Binaries:
Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
Yarn: Not Found
npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
Android SDK:
API Levels: 22, 23, 24, 27, 28, 29, 30
Build Tools: 23.0.2, 25.0.0, 27.0.1, 27.0.3, 28.0.2, 28.0.3, 29.0.2, 30.0.2, 31.0.0
System Images: android-28 | Google Play Intel x86 Atom, android-29 | Intel x86 Atom, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play ARM 64 v8a, android-29 | Google Play Intel x86 Atom, android-30 | Android TV Intel x86 Atom, android-30 | Google TV Intel x86 Atom, android-30 | Google APIs Intel x86 Atom, android-30 | Google APIs Intel x86 Atom_64, android-30 | Google Play Intel x86 Atom, android-30 | Google Play Intel x86 Atom_64
Android NDK: Not Found
IDEs:
Android Studio: Not Found
Languages:
Java: 15.0.2 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.64 => 0.64.2
npmGlobalPackages:
*react-native*: Not Found
```

## Description

When drag, sometimes the state is not getting updated. It still remains `idle`. I'm showing a loader on page swipe. For that, I update the status of the loader as `true`, when the state is `dragging`. Again I update the loader as `false`, when the state changes to `idle`. Sometimes after a couple of page swipes, the state doesn't change to `dragging` from `idle` it still showing `idle` so the loader is not getting displayed.

## Reproducible Demo

https://user-images.githubusercontent.com/80961628/134028480-ac77704c-b29a-44da-a7d4-a9f7c95b8fc1.mp4

まず、リンクされたデモでページを繰り返しスワイプして動作を再現し、その後、スワイプイベントが pager の状態を idle から dragging に更新する流れを追跡します。スワイプ中に状態が確実に dragging に遷移し、issue で説明されている loader の動作が一貫して機能すれば完了です。

索引モデルが issue の本文から書いたものです。