Failing builds with pip in build environment

### Solution to issue cannot be found in the documentation.

- [X] I checked the documentation.

### Issue

Due to an issue with [`conda-libmamba-solver`<->`conda-build` interaction](https://github.com/conda/conda-libmamba-solver/issues/393), we had `pip` installed in the `build` environment for the `numpy=1.26.2` update PR which caused the cross-compiled builds to fail with
```sh
Successfully built numpy-1.26.2-cp311-cp311-linux_aarch64.whl
+ pip install dist/numpy-1.26.2-cp311-cp311-linux_aarch64.whl
ERROR: numpy-1.26.2-cp311-cp311-linux_aarch64.whl is not a supported wheel on this platform.
```

@isuruf asked me to open an issue here too.

refs:
- https://github.com/conda-forge/numpy-feedstock/pull/305#issuecomment-1807452173 ff.
- https://github.com/conda-forge/conda-smithy/issues/1805
- https://github.com/conda/conda-libmamba-solver/issues/393

### Installed packages

```shell
-
```

### Environment info

```shell
-
```