Workqueue depth and duration metrics are not accurate

### What happened?

During scalability testing efforts leading https://github.com/crossplane-contrib/provider-kubernetes/pull/203, I noticed that the controller runtime metrics like `workqueue_depth` and `workqueue_queue_duration_seconds` are not accurate or not reflecting the state of the system as expected.

See the workqueue depth and duration graphs for "1m, 10" [here](https://github.com/crossplane-contrib/provider-kubernetes/pull/203#issuecomment-1953712346).

### How can we reproduce it?

Checkout https://github.com/turkenh/provider-kubernetes-scalability/tree/repro-xp-no-metric

```
just setup
just create_x_objects 1 1000
```

```
just help_launch_prometheus
```

```
just help_launch_grafana
# import dashboard json there
```

### What environment did it happen in?
Crossplane version: v1.14.5
Provider Kubernetes: v0.11.4