[Bug]: Python Lots of fn runner test items cost exactly 5 seconds to run

### What happened?

Was investigating #22115 and find this interesting observation

pytest results with time consuming info:

```
----------------------------------
Original:
=========================================================================================== slowest 100 test durations ============================================================================================
10.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pack_combiners
9.82s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pack_combiners
5.41s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_side_and_main_outputs
5.20s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_side_outputs
5.20s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_to_element_pardo
5.17s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_unfusable_side_inputs_with_separation
5.16s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_callbacks_with_exception
5.16s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo_overlapping_windows
5.16s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_draining_sdf_with_sdf_initiated_checkpointing
5.15s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo_fusion_break
5.14s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_flatten_same_pcollections
5.14s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_checkpoint_sdf
5.13s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo
5.13s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_assert_that
5.13s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_checkpoint
5.13s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo_window_param
5.12s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_element_to_batch_pardo
5.12s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo_override_type_inference
5.11s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo_override_type_inference
5.11s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_batch_pardo_dofn_params
5.09s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_create
5.09s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_assert_that
5.09s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_crazy_draining_sdf
5.09s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_crazy_sdf
5.04s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_custom_merging_window
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_timers_clear
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_side_inputs
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_sdf_with_sdf_initiated_checkpointing
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_sdf_default_truncate_when_bounded
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_read
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_half_sdf
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_flatten
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_half_draining_sdf
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_error_traceback_includes_user_code
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_nosplit_sdf
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_sdf_with_truncate
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_state_timers_non_standard_coder
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_state_only
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_sdf_with_watermark_tracking
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_gbk_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_side_input_dependencies
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_unfusable_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pack_combiners
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_combine_per_key
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_sdf_with_dofn_as_restriction_provider
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_sdf_with_sdf_initiated_checkpointing
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_state_timers
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_sdf_with_check_done_failed
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_create_value_provider_pipeline_option
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo_fusion_break
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_error_traceback_includes_user_code
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo_window_param
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo_dofn_params
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_gbk_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_batch_pardo
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_batch_pardo_overlapping_windows
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_metrics
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_timers_clear
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_sdf_with_watermark_tracking
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_sdf_with_truncate
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_windowed_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_side_and_main_outputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_flatten
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_side_outputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_sdf_with_truncate
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_combine_per_key
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_callbacks_with_exception
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_no_subtransform_composite
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_state_only
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_group_by_key
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_flattened_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_batch_pardo_dofn_params
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_windowed_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_error_message_includes_stage
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_windowed_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_combine_per_key
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_state_timers_non_standard_coder
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_multimap_side_input_type_coercion
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_side_inputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_sdf
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_state_timers
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_side_and_main_outputs
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_timers
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_dynamic_timer
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_flattened_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_batch_pardo_overlapping_windows
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_sdf_default_truncate_when_unbounded
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_flatten_same_pcollections
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_state_timers
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_sdf_synthetic_source
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_flattened_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_metrics
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_batch_pardo_window_param
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_side_input_dependencies
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_multimap_side_input
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_pardo_state_only
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_error_message_includes_stage
============================================================================== 407 passed, 54 skipped, 24 warnings in 88.40 seconds ===============================================================================
```

There is a default 5 seconds to close the rpc server in https://github.com/apache/beam/blob/c14a4cf877908852c139f67c0c28218c1339269c/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py#L453
Change this to None (or mock this constant in tests), tests still pass and running time decrease dramatically:
```
============================================================================================ slowest 20 test durations ============================================================================================
5.25s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_checkpoint_sdf
5.15s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_half_draining_sdf
5.13s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_checkpoint
5.12s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_crazy_sdf
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_nosplit_sdf
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_crazy_draining_sdf
5.01s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_half_sdf
5.00s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_split_half
1.48s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pack_combiners
1.44s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpcAndMultiWorkers::test_pardo_side_and_main_outputs
1.38s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerSplitTest::test_checkpoint_draining_sdf
1.19s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_register_finalizations
1.18s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithGrpc::test_register_finalizations
1.17s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTest::test_register_finalizations
1.15s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithBundleRepeatAndMultiWorkers::test_pardo_side_and_main_outputs
1.14s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithBundleRepeatAndMultiWorkers::test_pack_combiners
1.06s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithMultiWorkers::test_pardo_side_and_main_outputs
0.93s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithMultiWorkers::test_pack_combiners
0.82s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pardo_side_and_main_outputs
0.74s call apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiRunnerTestWithDisabledCaching::test_pack_combiners
============================================================================== 407 passed, 54 skipped, 24 warnings in 22.88 seconds ===============================================================================
```

Suprisingly, if changing `_DEFAULT_SHUTDOWN_TIMEOUT_SECS` to a large number (100), the running time is IDENTICAL with setting it as 5 seconds---hundreds of tests costs exactly 5 seconds, likely there is a hard-coded waiting time of 5 seconds that prevented RpcServer from closing.

### Issue Priority

Priority: 2

### Issue Component

Component: testing