Retry messages are lost

I have a workflow execution with dynamic fork and the first time the workflow executes, the tasks are polled by the worker and the workflow runs to execution.

If the first execution of the task is failed manually via UI, the retry tasks are created and they are visible in the queue. These retry tasks queue becomes empty the moment the worker is started and the worker polls no task and the workflow does not move forward.