Pan Functionality Fails to Activate When Initially Disabled

When you start the chart with the pan functionality disabled and later enable it, the pan won't work. All the samples start with pan enabled, which is why it works in those cases.

In the next part, I will reproduce the issue in the simplest way by disabling pan functionality from the start.

## Steps to Reproduce
(Apologies for the awkward setup—this was just the easiest way for me to test it)

1. Go to the samples page: [https://www.chartjs.org/chartjs-plugin-zoom/samples/basic.html](https://www.chartjs.org/chartjs-plugin-zoom/samples/basic.html).
2. Open the F12 developer tools and locate the chunk file `17.49536e53.js`, which runs the basic example.
3. Navigate to line 24, where the sample code is defined, and change the following piece starting at column 1435:
```javascript
pan: {\n enabled: true,
```
to
```javascript
pan: {\n enabled: false,
```
4. Override the file to ensure the change survives a refresh.
5. Refresh the page. Pan functionality is now disabled, but when you toggle it, it won't activate.

https://github.com/user-attachments/assets/dac35237-6453-406a-8087-87ac8ae69986

## Expected Behavior

The pan functionality should activate when toggled after being initially disabled.

## Additional Notes

My guess is that this issue might be related to how mouse events are registered.

Also, note that this behavior doesn't occur for other options, like `zoom.wheel`; it only happens with the pan functionality.