Cranelift: JIT assertion failure when using `ArgumentPurpose::StructArgument` on macOS (A64)

### `.clif` Test Case

```
function %foo(i64 sarg(16)) -> i32 apple_aarch64 {
block0(v0: i64):
v1 = load.i32 notrap aligned v0
return v1
}

function %main() -> i32 apple_aarch64 {
ss0 = explicit_slot 4, align = 16
sig0 = (i64 sarg(16)) -> i32 apple_aarch64
fn0 = colocated u0:0 sig0

block0:
v0 = iconst.i32 42
stack_store v0, ss0 ; v0 = 42
v1 = stack_addr.i64 ss0
v2 = call fn0(v1)
return v2
}
```

### Steps to Reproduce

I've created a MRE [here](https://github.com/NotAFlyingGoose/cranelift-jit-aggr-args)

The basic issue is caused when declaring a function with `AbiParam::special(ptr_ty, ArgumentPurpose::StructArgument(aggr_pointee_size))` as a parameter.

This is not an issue with `cranelift_object`. This assertion failure specifically happens in `cranelift_jit`

### Expected Results

I expected the above MRE to compile and print the number 42 to the screen.
This happens when using x86, `cranelift_object`, or not using `ArgumentPurpose::StructArgument`.
When using A64, `cranelift_jit`, and `ArgumentPurpose::StructArgument`, cranelift panics

### Actual Results

When calling `module.finalize_definitions()`, the MRE reaches an assertion failure

```
thread 'main' panicked at /Users/goose/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cranelift-jit-0.109.0/src/compiled_blob.rs:90:21:
assertion failed: (diff >> 26 == -1) || (diff >> 26 == 0)
stack backtrace:
0: rust_begin_unwind
at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:652:5
1: core::panicking::panic_fmt
at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:72:14
2: core::panicking::panic
at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:146:5
3: cranelift_jit::compiled_blob::CompiledBlob::perform_relocations
at /Users/goose/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cranelift-jit-0.109.0/src/compiled_blob.rs:90:21
4: cranelift_jit::backend::JITModule::finalize_definitions
at /Users/goose/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cranelift-jit-0.109.0/src/backend.rs:475:13
5: cl_aggr_args_mac::main
at ./src/main.rs:41:5
6: core::ops::function::FnOnce::call_once
at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

### Versions and Environment

Cranelift version or commit: I just updated the MRE to 0.109 this morning and the error is still present

Operating system: macOS 14.3.1

Architecture: Apple M2 (Arm64)
It's important to note that this ONLY occurs on Arm64, which makes sense as that's where the assertion failure is.
The MRE should work as expected on x86.

### Extra Info

The exact line in the MRE to look at is [main.rs:152](https://github.com/NotAFlyingGoose/cranelift-jit-aggr-args/blob/822842b5ddb5ccc7f70a1dd1530e78b2a07719ed/src/main.rs#L152-L153)