[Failing Test]: portable_runner_test in Python SDK

### What happened?

When packaging beam 2.62.0 in nix https://github.com/NixOS/nixpkgs/pull/379201, some tests fails with

`grpc_status:14, grpc_message:"Cancelling all calls"}`.

I'm not sure it's related to #22643 as a temporary fix is to skip `apache_beam/runners/portability/portable_runner_test.py`.
Any pointer would be great ,thanks !

### Issue Failure

Failure: Test is flaky

### Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

### Issue Components

- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner