[go_router] pop method does not pop route when using refresh method at same frame

### What package does this bug report belong to?

go_router

### What target platforms are you seeing this bug on?

Android, iOS, Web, macOS, Linux, Windows

### Have you already upgraded your packages?

Yes

### Dependency versions

_No response_

### Steps to reproduce

1. Add go_router: ^12.1.3 in pubspec.yaml
2. Paste sample code to tests/xxx_test.dart in Flutter project
3. Execute test by `flutter test --plain-name 'GoRouter'`

### Expected results

The test will success, which indicates the route popped by `pop` method.

### Actual results

The test will be failed with `expect` failure, which indicates the `pop` method does not pop route.

### Code sample

go_router_test.dart

```dart
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:go_router/go_router.dart';

final GlobalKey rootNavigatorKey =
GlobalKey(debugLabel: 'rootNavigatorKey');

void main() {
// flutter test --plain-name 'GoRouter'
testWidgets("GoRouter", (tester) async {
await tester.pumpWidget(MaterialApp.router(
routerConfig: GoRouter(
navigatorKey: rootNavigatorKey,
debugLogDiagnostics: true,
initialLocation: '/',
routes: [
GoRoute(
path: '/',
builder: (_, __) {
return Scaffold(
appBar: AppBar(
title: const Text("home"),
),
body: const Center(
child: Text("home"),
),
);
},
),
GoRoute(
path: "/a",
builder: (_, __) {
return Scaffold(
appBar: AppBar(
title: const Text("a"),
),
body: const Center(
child: Text("a"),
),
);
}),
],
redirect: (BuildContext context, GoRouterState state) {
return null;
},
),
));

GoRouter.of(rootNavigatorKey.currentContext!).push('/a');
await tester.pumpAndSettle();

GoRouter.of(rootNavigatorKey.currentContext!).pop();
//await tester.pumpAndSettle(); // If pop and refresh executed at different frame, of course the results will be as expected.
GoRouter.of(rootNavigatorKey.currentContext!).refresh();
await tester.pumpAndSettle();

// My assumption is that the "a" text is not displayed,
// because the route "/a" is already popped.
// However, this expect will fail.
expect(find.widgetWithText(Center, "a"), findsNothing);
});
}

```

### Screenshots or Videos

Screenshots / Video demonstration

[Upload media here]

### Logs

Logs

```console
% flutter test --plain-name 'GoRouter'
00:03 +0: /Users/xxx/test/go_router_test.dart: GoRouter
[GoRouter] Full paths for routes:
=> /
=> /a

[GoRouter] setting initial location /
[GoRouter] Using MaterialApp configuration
[GoRouter] pushing /a
[GoRouter] popping /
[GoRouter] refreshing /
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: no matching candidates
Actual: _AncestorWidgetFinder:
Which: means one was found but none were expected
```

### Flutter Doctor output

I have not installed the Android environment, but it is not related to the test.

```console
flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.16.9, on macOS 14.1.1 23B81 darwin-arm64, locale ja-JP)
[✗] Android toolchain - develop for Android devices
✗ Unable to locate Android SDK.
Install Android Studio from: https://developer.android.com/studio/index.html
On first launch it will assist you in installing the Android SDK components.
(or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
If the Android SDK has been installed to a custom location, please use
`flutter config --android-sdk` to update to that location.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
[✓] Chrome - develop for the web
[!] Android Studio (not installed)
[✓] VS Code (version 1.85.2)
[✓] Connected device (3 available)
[✓] Network resources

```