Performance overhead

Hi, consider the following MWE:
```julia
using FFTW
using LinearAlgebra
using Random
using BenchmarkTools
using Base.Threads

N, M = 512, 16
Random.seed!(42)
a = rand(2N-1, M)
b = zeros(ComplexF64, N, M)
c = zeros(2N-1, M)

function foo(plan, a, b, c)
mul!(b, plan, a)
ldiv!(c, plan, b)
end

for t ∈ [1, 2, 4, 8, 12, 18, 24]
print("Julia threads: $(nthreads()), FFT threads: $(t)")
FFTW.set_num_threads(t)
p = plan_rfft(a, 1; flags=FFTW.PATIENT)
@btime foo($p, $a, $b, $c);
end
```

I tested on a workstation with 2 Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz (2 * 12 CPUs).

```
Julia 1.5.4
Julia threads: 12, FFT threads: 1 384.595 μs (2 allocations: 160 bytes)
Julia threads: 12, FFT threads: 2 226.376 μs (55 allocations: 4.31 KiB)
Julia threads: 12, FFT threads: 4 133.087 μs (83 allocations: 7.44 KiB)
Julia threads: 12, FFT threads: 8 117.697 μs (140 allocations: 13.81 KiB)
Julia threads: 12, FFT threads: 12 101.563 μs (556 allocations: 46.81 KiB)
Julia threads: 12, FFT threads: 18 89.751 μs (254 allocations: 26.59 KiB)
Julia threads: 12, FFT threads: 24 90.119 μs (254 allocations: 26.59 KiB)

Julia threads: 24, FFT threads: 1 385.778 μs (2 allocations: 160 bytes)
Julia threads: 24, FFT threads: 2 238.616 μs (55 allocations: 4.31 KiB)
Julia threads: 24, FFT threads: 4 152.831 μs (83 allocations: 7.44 KiB)
Julia threads: 24, FFT threads: 8 109.802 μs (141 allocations: 13.84 KiB)
Julia threads: 24, FFT threads: 12 109.869 μs (141 allocations: 13.84 KiB)
Julia threads: 24, FFT threads: 18 152.472 μs (256 allocations: 26.66 KiB)
Julia threads: 24, FFT threads: 24 149.632 μs (256 allocations: 26.66 KiB)

Julia 1.6.1
Julia threads: 12, FFT threads: 1 372.699 μs (2 allocations: 160 bytes)
Julia threads: 12, FFT threads: 2 372.061 μs (2 allocations: 160 bytes)
Julia threads: 12, FFT threads: 4 407.617 μs (43 allocations: 2.69 KiB)
Julia threads: 12, FFT threads: 8 131.019 μs (141 allocations: 9.34 KiB)
Julia threads: 12, FFT threads: 12 125.367 μs (141 allocations: 9.34 KiB)
Julia threads: 12, FFT threads: 18 123.347 μs (254 allocations: 17.59 KiB)
Julia threads: 12, FFT threads: 24 138.969 μs (254 allocations: 17.59 KiB)

Julia threads: 24, FFT threads: 1 374.955 μs (2 allocations: 160 bytes)
Julia threads: 24, FFT threads: 2 376.429 μs (2 allocations: 160 bytes)
Julia threads: 24, FFT threads: 4 375.154 μs (2 allocations: 160 bytes)
Julia threads: 24, FFT threads: 8 142.737 μs (140 allocations: 9.31 KiB)
Julia threads: 24, FFT threads: 12 219.083 μs (141 allocations: 9.34 KiB)
Julia threads: 24, FFT threads: 18 196.197 μs (254 allocations: 17.59 KiB)
Julia threads: 24, FFT threads: 24 228.277 μs (256 allocations: 17.66 KiB)
```

It looks that (1) allocating more Julia threads hurts performance; (2) Julia 1.6 performs worse than Julia 1.5. Is there any suggestion to improve this kind of computation?

Thank you!