TypeError: __init__() got an unexpected keyword argument 'callback_timeout' ERROR 2022-07-04 13:39:43 +0200 master-replica-0 NoneType: None

Hi,

I am trying to train a PyTorch model on GCP and I am reading a csv file from a cloud storage bucket in the mean time.

I get the same error when reading the file in two different ways:

1 - simply using pd.read_csv:

code:

`GCS_BUCKET = "led-test-run"

GCS_BASE_ROOT = f"gs://{GCS_BUCKET}"

TRAIN_DIR = os.path.join(GCS_BASE_ROOT, "bigpatent_sample_corpus_BIG_train_df.csv")
VAL_DIR = os.path.join(GCS_BASE_ROOT, "bigpatent_sample_corpus_BIG_val_df.csv")
train_dataset = pd.read_csv(TRAIN_DIR)`

error:

Exception ignored in: <function ClientSession.__del__ at 0x7f1ede3bd710> ERROR 2022-07-04 13:39:43 +0200 master-replica-0 Traceback (most recent call last): ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/aiohttp/client.py", line 326, in __del__ ERROR 2022-07-04 13:39:43 +0200 master-replica-0 if not self.closed: ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/aiohttp/client.py", line 963, in closed ERROR 2022-07-04 13:39:43 +0200 master-replica-0 return self._connector is None or self._connector.closed ERROR 2022-07-04 13:39:43 +0200 master-replica-0 AttributeError: 'ClientSession' object has no attribute '_connector' ERROR 2022-07-04 13:39:43 +0200 master-replica-0 ERROR:root:Traceback (most recent call last): ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main ERROR 2022-07-04 13:39:43 +0200 master-replica-0 "__main__", mod_spec) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code ERROR 2022-07-04 13:39:43 +0200 master-replica-0 exec(code, run_globals) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 78, in <module> ERROR 2022-07-04 13:39:43 +0200 master-replica-0 main() ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 74, in main ERROR 2022-07-04 13:39:43 +0200 master-replica-0 experiment.run(args) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 66, in run ERROR 2022-07-04 13:39:43 +0200 master-replica-0 train_dataset, test_dataset = utils.load_data() ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/trainer/utils.py", line 65, in load_data ERROR 2022-07-04 13:39:43 +0200 master-replica-0 train_dataset = pd.read_csv(TRAIN_DIR) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 610, in read_csv ERROR 2022-07-04 13:39:43 +0200 master-replica-0 return _read(filepath_or_buffer, kwds) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 462, in _read ERROR 2022-07-04 13:39:43 +0200 master-replica-0 parser = TextFileReader(filepath_or_buffer, **kwds) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 819, in __init__ ERROR 2022-07-04 13:39:43 +0200 master-replica-0 self._engine = self._make_engine(self.engine) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 1050, in _make_engine ERROR 2022-07-04 13:39:43 +0200 master-replica-0 return mapping[engine](self.f, **self.options) # type: ignore[call-arg] ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 1867, in __init__ ERROR 2022-07-04 13:39:43 +0200 master-replica-0 self._open_handles(src, kwds) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 1368, in _open_handles ERROR 2022-07-04 13:39:43 +0200 master-replica-0 storage_options=kwds.get("storage_options", None), ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 563, in get_handle ERROR 2022-07-04 13:39:43 +0200 master-replica-0 storage_options=storage_options, ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 334, in _get_filepath_or_buffer ERROR 2022-07-04 13:39:43 +0200 master-replica-0 filepath_or_buffer, mode=fsspec_mode, **(storage_options or {}) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/core.py", line 476, in open ERROR 2022-07-04 13:39:43 +0200 master-replica-0 **kwargs, ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/core.py", line 306, in open_files ERROR 2022-07-04 13:39:43 +0200 master-replica-0 expand=expand, ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/core.py", line 657, in get_fs_token_paths ERROR 2022-07-04 13:39:43 +0200 master-replica-0 fs = cls(**options) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/spec.py", line 76, in __call__ ERROR 2022-07-04 13:39:43 +0200 master-replica-0 obj = super().__call__(*args, **kwargs) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 270, in __init__ ERROR 2022-07-04 13:39:43 +0200 master-replica-0 self.loop, get_client, callback_timeout=self.callback_timeout ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-04 13:39:43 +0200 master-replica-0 raise return_result ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-04 13:39:43 +0200 master-replica-0 result[0] = await coro ERROR 2022-07-04 13:39:43 +0200 master-replica-0 File "/root/.local/lib/python3.7/site-packages/fsspec/implementations/http.py", line 29, in get_client ERROR 2022-07-04 13:39:43 +0200 master-replica-0 return aiohttp.ClientSession(**kwargs) ERROR 2022-07-04 13:39:43 +0200 master-replica-0 TypeError: __init__() got an unexpected keyword argument 'callback_timeout' ERROR 2022-07-04 13:39:43 +0200 master-replica-0 NoneType: None

2 - using gcsfs.GCSFileSystem:

code:

fs = gcsfs.GCSFileSystem(project='master-thesis-351911') with fs.open('led-test-run/bigpatent_sample_corpus_BIG_train_df.csv') as f: train_dataset = pd.read_csv(f)

error:

The replica master 0 exited with a non-zero status of 1. ERROR 2022-07-04 14:27:34 +0200 service Traceback (most recent call last): ERROR 2022-07-04 14:27:34 +0200 service File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main ERROR 2022-07-04 14:27:34 +0200 service "__main__", mod_spec) ERROR 2022-07-04 14:27:34 +0200 service File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code ERROR 2022-07-04 14:27:34 +0200 service exec(code, run_globals) ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 3, in <module> ERROR 2022-07-04 14:27:34 +0200 service from trainer import experiment ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 13, in <module> ERROR 2022-07-04 14:27:34 +0200 service from trainer import model, metadata, utils ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/trainer/utils.py", line 24, in <module> ERROR 2022-07-04 14:27:34 +0200 service fs = gcsfs.GCSFileSystem(project='master-thesis-351911') ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/fsspec/spec.py", line 76, in __call__ ERROR 2022-07-04 14:27:34 +0200 service obj = super().__call__(*args, **kwargs) ERROR 2022-07-04 14:27:34 +0200 service File "/opt/conda/lib/python3.7/site-packages/gcsfs/core.py", line 270, in __init__ ERROR 2022-07-04 14:27:34 +0200 service self.loop, get_client, callback_timeout=self.callback_timeout ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync ERROR 2022-07-04 14:27:34 +0200 service raise return_result ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner ERROR 2022-07-04 14:27:34 +0200 service result[0] = await coro ERROR 2022-07-04 14:27:34 +0200 service File "/root/.local/lib/python3.7/site-packages/fsspec/implementations/http.py", line 29, in get_client ERROR 2022-07-04 14:27:34 +0200 service return aiohttp.ClientSession(**kwargs) ERROR 2022-07-04 14:27:34 +0200 service TypeError: __init__() got an unexpected keyword argument 'callback_timeout'

Start with the traceback paths in gcsfs/core.py and fsspec/implementations/http.py, then inspect the dependency versions used by the trainer in trainer/utils.py. Reproduce the failure with both the pandas read_csv and GCSFileSystem examples. Done means the supported dependency combination no longer passes callback_timeout incompatibly and both GCS access paths work.