[Bug] [Broker] The destinationBroker parameter not work in concurrent lookup when unloading bundle

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

- broker version: 3.3.6, still not solved on master.

### Issue Description

1. `pulsar-admin namespaces unload --bundle aaa --destinationBroker xxx public/default`
2. broker setNamespaceBundleAffinityAsync when internalUnloadNamespaceBundleAsync.
4. broker load bundle request or lookup request will get and remove this bundle. The first request will get the correct destinationBroker and remove it. But the second request wil lose destinationBroker and getLeastLoadedBroker.

find same issue in closed pr #19260

### Error messages

```text

```

### Reproducing the issue

`pulsar-admin namespaces unload --bundle aaa --destinationBroker xxx public/default`

Bundle aaa is not always loaded to broker xxx

### Additional information

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Start by tracing the `pulsar-admin namespaces unload --bundle aaa --destinationBroker xxx public/default` path through `setNamespaceBundleAffinityAsync` and `internalUnloadNamespaceBundleAsync`. Compare the concurrent load-bundle and lookup requests with closed PR #19260, focusing on when the destination broker is removed. Done means the bundle is consistently loaded to the requested broker rather than the least-loaded broker.

Written by the indexing model from the issue text.