losing GPUs as workers after a while running

Hi,

I am using dask for doing some udf on string data.
_def func_on_string_column(df):
df[col_name_2]= df[col_name_1].map(map_func)
return df

def map_func(string_path):
reading content of files in "string_path"
return content

dataframe_2=dataframe.map_partitions(func_on_string_column, meta={.......})_

I have 8 GPUs and set the localcudacluster and client as below:

_os.environ["DASK_UCX__CUDA_COPY"] = "True"
os.environ["DASK_UCX__TCP"] = "True"
os.environ["DASK_UCX__NVLINK"] = "True"
os.environ["DASK_UCX__INFINIBAND"] = "True"
os.environ["DASK_UCX__NET_DEVICES"] = "enp1s0f0"

cluster = LocalCUDACluster(
protocol="ucx",
enable_tcp_over_ucx=False,
enable_infiniband=False,
enable_nvlink=True,
memory_limit='40GB'
)

client = Client(cluster)_

when I am checking dask dashboard , at the beginning of running the code, I have 8 workers. but after 10 hours when I checked again the dask dashboard I just see one worker.
I dont know why it happens and how can I fix it. if someone know please let me know too.

In addition, I have another issue with lots of garbage collection warning message during the map_partition process.
_distributed.utils_perf - WARNING - full garbage collections took 18% CPU time recently (threshold: 10%)_
how can increase threshold or fix this issue?

Thanks.