Cross compilation failures since PYO3_CROSS env variables are set

### Comment:

I'm not sure if this is the correct place for is, if you think it isn't feel free to close this.

Since #55 was merged, I get compilation errors when cross compiling a package using pyo3 for osx-arm64, and the errors go away if I manually unset the `PYO3_CROSS*` environment variables to undo the changes in that PR.

I see the following error:
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=611557&view=logs&j=1b8be447-c2bd-5772-b66a-a1146441bf88&t=0da7bde9-21e8-5d19-f418-738876121ca7&l=273

This doesn't seem to happen to all projects using pyo3, polars for instance seems to build fine, so this might also just be a bug in my recipe...

Logs of a current (working) polars build:
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=612122&view=logs&j=c35410bf-a820-56fd-e231-d3cd4bead63c&t=b92b255f-9dfe-5c14-4730-460f7a671001&l=212

But since the change that broke it happend here, I though I'll ask...
(cc @isuruf)