Getting EOF on requests out of runner

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

v0.27.5

Helm Chart Version

0.23.5

CertManager Version

latest

Deployment Method

ArgoCD

cert-manager installation

yes

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    app.kubernetes.io/name: large-github-runner
  creationTimestamp: '2023-07-31T11:16:40Z'
  generation: 571
  labels:
    app.kubernetes.io/instance: large
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: large-github-runner
    app.kubernetes.io/type: large
    app.kubernetes.io/version: 1.0.0
    argocd.argoproj.io/instance: large
    helm.sh/chart: github-runner-0.1.2
  name: large-github-runner
  namespace: github-runners
  resourceVersion: '233679717'
  uid: a4817b40-a87a-4129-883f-8ad37a7269c9
spec:
  effectiveTime: null
  replicas: 1
  selector: null
  template:
    metadata: {}
    spec:
      dockerdContainerResources:
        limits:
          cpu: '6'
          memory: 12Gi
        requests:
          cpu: '6'
          memory: 8Gi
      dockerdWithinRunnerContainer: false
      env:
        - name: ARC_DOCKER_MTU_PROPAGATION
          value: 'true'
      ephemeral: true
      group: k8s
      image: summerwind/actions-runner
      imagePullPolicy: IfNotPresent
      labels:
        - large-github-runner
        - self-hosted
        - large
      nodeSelector:
        NodeType: XL
      organization: CloudinaryLtd
      resources:
        limits:
          cpu: '6'
          memory: 12Gi
        requests:
          cpu: '6'
          memory: 8Gi
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
          tolerationSeconds: 10
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              runner-deployment-name: large-github-runner
          maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
      workDir: /home/runner/work
status:
  availableReplicas: 1
  desiredReplicas: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

To Reproduce

Try to do long requests such as: argocd app sync

Describe the bug

When trying to do a sync action (or a curl to bring a big package) we get the following error: runner@large-github-runner-clsdh-98hzg:/$ argocd app sync -l service_name=sample-go-service --retry-limit 10 --retry-backoff-duration 20s FATA[0005] rpc error: code = Unknown desc = Post "https://argocd-service-name.company.com/application.ApplicationService/List": EOF

This issue does not occur on any other container or from laptop.
The issue is definitely from the runner as we eliminated any other option and check the ingresses logs.

To be more precise - this issue happens sporadically, every 3-5 times that we try.

Describe the expected behavior

We expect the runner to be able to perform requests and understand why this is happening.
we tried to change the MTU and implement the env variable 'ARC_DOCKER_MTU_PROPAGATION' with no luck.

Any assistance will be appreciated

Whole Controller Logs

│ 2023-10-05T15:04:44Z    DEBUG    horizontalrunnerautoscaler    Calculated desired replicas of 1  │

Whole Runner Pod Logs

ecr-proxy.company.com/go-tools-base:v1.1.1: failed to do request: Head "https://ecr-proxy.company.com/v2/go-tools-base/manifests/v1.1.1": EOF

Additional Context

'ecr-proxy.company.com/go-tools-base:v1.1.1: failed to do request: Head "https://ecr-proxy.company.com/v2/go-tools-base/manifests/v1.1.1": EOF'

runner@large-github-runner-clsdh-98hzg:/$ argocd app sync -l service_name=sample-go-service --retry-limit 10 --retry-backoff-duration 20s FATA[0005] rpc error: code = Unknown desc = Post "https://argocd-service-name.company.com/application.ApplicationService/List": EOF

Start by reviewing the RunnerDeployment configuration and reproducing a long outbound request such as argocd app sync from the runner pod. Compare the intermittent EOF responses with the runner pod logs and the reported MTU settings; done means identifying the runner-side cause and documenting a verified fix.