Inconsistent cache between github webhook server pods causing scaling errors

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

v1.11.2

Deployment Method

Helm

cert-manager installation

probably not relevant, the bug is isolated to github-webhook-server not webhook delivery itself

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

described below

To Reproduce

described below

Describe the bug

I have a deployment of github-webhook-server with 3 replicas. It works absolutely fine in general and correctly scales thousands of jobs per day across hundreds of RunnerDeployments

My github-webhook-server deployment has 3 replicas

I had an issue with 2 particular runnerDeployment CRs only:

• github-watchdog-c24c5 with spec.repository: my-org-name/github-watchdog
• github-watchdog-proxy-3a834 with spec.repository: my-org-name/github-watchdog-proxy

Something which might be relevant later is that previously (a few days ago), github-watchdog-proxy-3a834 also used to have spec.repository: my-org-name/github-watchdog but this was a mistake and it was then changed to the current value of my-org-name/github-watchdog-proxy

I suspect that 2 out of the 3 github webhook server pods did not pick up this modification for the reasons I describe below:

Against each pod in turn (by forwarding a distinct local port), I replayed a github event JSON payload which works absolutely fine to scale up these RunnerDeployments from zero.

However, I get strange and inconsistent results between the 3 replicas

Pod 1 = actions-runner-controller-github-webhook-server-776f4f656-6wj62
age = 31d
result = ERROR: finds 2 matching HRAs and might scale up either

{"severity":"debug","ts":"2023-06-19T16:30:16Z","logger":"controllers.webhookbasedautoscaler","message":"Found 2 HRAs by key","key":"my-org-name/github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:30:16Z","logger":"controllers.webhookbasedautoscaler","message":"job scale up target is repository-wide runners","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499,"repository":"github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:30:16Z","logger":"controllers.webhookbasedautoscaler","message":"scaled github-watchdog-c24c5 by 1","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499}
{"severity":"debug","ts":"2023-06-19T16:30:18Z","logger":"controllers.webhookbasedautoscaler","message":"Patching hra github-watchdog-c24c5 for capacityReservations update","before":4,"expired":1,"added":1,"completed":0,"after":3}

{"severity":"debug","ts":"2023-06-19T16:33:11Z","logger":"controllers.webhookbasedautoscaler","message":"Found 2 HRAs by key","key":"my-org-name/github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:33:11Z","logger":"controllers.webhookbasedautoscaler","message":"job scale up target is repository-wide runners","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499,"repository":"github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:33:11Z","logger":"controllers.webhookbasedautoscaler","message":"scaled github-watchdog-proxy-3a834 by 1","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499}
{"severity":"debug","ts":"2023-06-19T16:33:13Z","logger":"controllers.webhookbasedautoscaler","message":"Patching hra github-watchdog-proxy-3a834 for capacityReservations update","before":1,"expired":0,"added":1,"completed":0,"after":1}

Pod 2 = actions-runner-controller-github-webhook-server-776f4f656-n7wsp
age = 4d3h

result = ERROR: finds 2 matching HRAs and might scale up either

{"severity":"debug","ts":"2023-06-19T16:38:52Z","logger":"controllers.webhookbasedautoscaler","message":"Found 2 HRAs by key","key":"my-org-name/github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:38:52Z","logger":"controllers.webhookbasedautoscaler","message":"job scale up target is repository-wide runners","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499,"repository":"github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:38:52Z","logger":"controllers.webhookbasedautoscaler","message":"scaled github-watchdog-proxy-3a834 by 1","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499}
{"severity":"debug","ts":"2023-06-19T16:38:53Z","logger":"controllers.webhookbasedautoscaler","message":"Patching hra github-watchdog-proxy-3a834 for capacityReservations update","before":1,"expired":-1,"added":1,"completed":0,"after":2}

{"severity":"debug","ts":"2023-06-19T16:38:57Z","logger":"controllers.webhookbasedautoscaler","message":"Found 2 HRAs by key","key":"my-org-name/github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:38:57Z","logger":"controllers.webhookbasedautoscaler","message":"job scale up target is repository-wide runners","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499,"repository":"github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:38:57Z","logger":"controllers.webhookbasedautoscaler","message":"scaled github-watchdog-c24c5 by 1","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499}
{"severity":"debug","ts":"2023-06-19T16:38:59Z","logger":"controllers.webhookbasedautoscaler","message":"Patching hra github-watchdog-c24c5 for capacityReservations update","before":14,"expired":-1,"added":1,"completed":0,"after":15}

Pod 3 = actions-runner-controller-github-webhook-server-776f4f656-hsrn4
age = 2d15h

result = OK: finds 1 matching HRA and scales it up

{"severity":"debug","ts":"2023-06-19T16:39:41Z","logger":"controllers.webhookbasedautoscaler","message":"Found 1 HRAs by key","key":"my-org-name/github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:39:41Z","logger":"controllers.webhookbasedautoscaler","message":"job scale up target is repository-wide runners","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499,"repository":"github-watchdog"}
{"severity":"info","ts":"2023-06-19T16:39:41Z","logger":"controllers.webhookbasedautoscaler","message":"scaled github-watchdog-c24c5 by 1","event":"workflow_job","hookID":"","delivery":"","workflowJob.status":"queued","workflowJob.labels":["self-hosted","build"],"repository.name":"github-watchdog","repository.owner.login":"my-org-name","repository.owner.type":"Organization","enterprise.slug":"my-parent-org","action":"queued","workflowJob.runID":5313379581,"workflowJob.ID":14377922499}
{"severity":"debug","ts":"2023-06-19T16:39:43Z","logger":"controllers.webhookbasedautoscaler","message":"Patching hra github-watchdog-c24c5 for capacityReservations update","before":15,"expired":-1,"added":1,"completed":0,"after":16}

I believe, from reading previous github issues, that github-webhook-server maintains an internal cache of the state of relevant resources in the cluster.

The different replica pods seem to contain different data in their cache: 2/3 incorrectly believe that there are 2 HRAs corresponding to my-org-name/github-watchdog and 1/3 correctly believes that there is 1 HRA matching that repo.

Describe the expected behavior

github-webhook-server pods contain the correct cached resource state and behaviour is consistent between them

Whole Controller Logs

some relevant logs are above, I can provide more if you think it's relevant

Whole Runner Pod Logs

some relevant logs are above, I can provide more if you think it's relevant

Additional Context

No response

No source file, test, or entry point is named. Start by tracing the github-webhook-server cache and its resource update handling, then reproduce the reported repository rename with three replicas; done means every pod observes the same HRA state and scales consistently.