Holding Option/Alt does not allow selection of text inside Link

### Provide a general summary of the issue here

While holding Option/Alt, you can usually select text inside links without triggering them.

### 🤔 Expected Behavior?

Being able to select text while holding Option/Alt like on native links.
Also the link should neither be followed nor a download should be triggered.

### 😯 Current Behavior

No selection possible, also the link gets triggered as soon as you lift the mouse button.
When keeping Option/Alt pressed, it triggers a download of the link, otherwise it navigates.

### 💁 Possible Solution

_No response_

### 🔦 Context

_No response_

### 🖥️ Steps to Reproduce

Hold `Option` while trying to select the text of the RAC Link and then the native link.

https://codesandbox.io/p/sandbox/crazy-hooks-sfs3lr

https://github.com/adobe/react-spectrum/assets/9491603/178bdd6b-47bf-4518-a4c7-e4fa9ddd9228

### Version

React Aria Components 1.2.1

### What browsers are you seeing the problem on?

Microsoft Edge

### If other, please specify.

_No response_

### What operating system are you using?

macOS

### 🧢 Your Company/Team

_No response_

### 🕷 Tracking Issue

_No response_