considerable performance degradation vs. threads and idle CPU

Hello,
this might get long so I'll do a bit of an intro. We've done some work on tremor (github.com/wayfair-tremor/tremor-runtime) attempting to move for actix and threads to a more async-std implementation - we love <3 async-std! And a good part of the move went swimmingly and reduced code and complexity - huzza!

When doing performance testing on the result however there was a bit of a surprise, it was significantly slower. We test with more cores than hot threads/tasks so we did expect some degradation but it ended up being two-digit percentages, 40% for completely replacing threads with tasks and 20% for just replacing web socket handling.

Now, this in itself might not be worth an issue, the interesting part is to follow. When benchmarking we used task pin to limit the number of cores used and when pinning the number of cores to the number of hot tasks/threads a significant portion of the CPU remained unused, about 40% and 20% respectively.

So I hope that finding the reason for the under-utilization might add some performance.

When reading the code I noticed a few things which might be part of this but I obviously don't have the same understanding of the underpinnings then people here, I'll still throw out the observations for the same of discussion.

1) Worker tasks are not core pinned - this means the operating system might move them between cores while they are working invalidating cpu caches in the process. As the worker is task stealing it also means that there is the potential for two algorithms to schedule work conflicting with each other.

2) I think the worker tries to keep the active task running for a few rounds but will eventually put it back on the queue even if the queue is empty, I'm not 100% sure I understood this correctly, but I am thinking not returning it on an empty queue might save some cycles.

3) I think the work steeler will steal work from a worker even if the stolen from worker could handle the task, in my mind this leads to unnecessary task migrations between workers, which when avoided, could again save a few cycles.