workflow stuck forever in "Job is about to start running on the hosted runner: GitHub Actions 7 (hosted)"

we are experiencing many times that a workflow gets stuck in some step. or canceled during a step with no indication why.
viewing the raw logs i get this:

2023-08-18T09:36:00.0703962Z Requested labels: ubuntu-22.04
2023-08-18T09:36:00.0704258Z Job defined at: digma-ai/digma-intellij-plugin/.github/workflows/build-workflow.yml@refs/heads/main
2023-08-18T09:36:00.0704585Z Reusable workflow chain:
2023-08-18T09:36:00.0704734Z digma-ai/digma-intellij-plugin/.github/workflows/build-main.yml@refs/heads/main (375d967cdf74aecc90f9e9b0dc2987c507b8192b)
2023-08-18T09:36:00.0704899Z -> digma-ai/digma-intellij-plugin/.github/workflows/build-workflow.yml@refs/heads/main (375d967cdf74aecc90f9e9b0dc2987c507b8192b)
2023-08-18T09:36:00.0705064Z Waiting for a runner to pick up this job...
2023-08-18T09:36:00.3344275Z Job is waiting for a hosted runner to come online.
2023-08-18T09:36:04.8994951Z Job is about to start running on the hosted runner: GitHub Actions 7 (hosted)

but the workflow already started and it gets stuck in some step, not always the same step.

sometimes the workflow is just canceled during a step and there is no indication why.

100% of the times a rerun succeeds. but sometimes it takes ages before the workflow stops and we need to cancel it manually.
sometimes its a workflow that can't be rerun because it started publishing a plugin to jetbrains, in that case we need to delete the published plugin and restart the workflow. very annoying.

This is happening many times and became very annoying in the past weeks, the average is that every second workflow gets stuck and we need to rerun. usually there is nothing in the logs.

Start with the reusable workflow chain in .github/workflows/build-main.yml and .github/workflows/build-workflow.yml, then review the raw logs around runner pickup and the reported stuck or canceled steps. Reproduce the intermittent behavior and determine what runner or workflow evidence is available when a job hangs or is canceled. Done means the failure has an identifiable cause or an actionable diagnostic path.