Unit in NumberField cannot be deleted (for some units/ unit-displays)

# 🐛 Bug Report
I am using the useNumberField with the "formatOptions" option to add units to the number fields.

Using the following option works fine:
```typescript
formatOptions={{
style: "unit",
unit: "gram",
}}
```

But when I am using the "milliliter" unit, the unit's letters cannot be deleted (by pressing the backspace or the delete key).
```typescript
formatOptions={{
style: "unit",
unit: "milligram",
}}
```

When using `unitDisplay="long` only the last letter can be deleted.

## 🤔 Expected Behavior

Pressing the backspace or the delete key should delete unit's letters inside the input.

## 😯 Current Behavior
Pressing the backspace or the delete key does not always delete the unit's letters inside the input.

## 💁 Possible Solution

## 🔦 Context

## 💻 Code Sample

[Codesandbox reproduction](https://codesandbox.io/s/react-spectrum-number-field-unit-issues-e8i4y?file=/src/App.js)

Go to the codesandbox and focus one NumberField after another to see which letters of which units can be deleted and which cannot.

## 🌍 Your Environment

| Software | Version(s) |
| ---------------- | ---------- |
| react-spectrum |3.9.0
| Browser |
| Operating System |