Worker task deserialization and ThreadPoolExecutor interactions

I've had users report issues where a task arrives at a worker and is deserialized on the worker's main thread (here https://github.com/dask/distributed/blob/2bff61d9bee59e0bf655937922d9d4c37e49820a/distributed/worker.py#L1328) which causes code to be executed that interacts poorly with other tasks running concurrently on the worker `ThreadPoolExecutor`. Is there an existing mechanism that could help prevent this task deserialization and task execution from happening at the same time?

One potential development could be to have task deserialization happen on the `ThreadPoolExecutor` right before the task is executed. This won't fix the problem in general, but would allow for limiting the worker's `ThreadPoolExecutor` to a single thread so that task deserialization and task execution never happen concurrently. This approach also has the added benefit of not deserializing tasks that are stolen before being executed. That said, I'm sure this approach also has some associated down sides and may not be appropriate for the common case.

Does this approach seem like a reasonable way forward (I'm not sure if it is)? Is improving non-threadsafe code execution in scope for `distributed`?