fsspec / fsspec/filesystem_spec

Pickle writing fails with `simplecache::` using xrootd paths (maybe with other remote paths as well).

Open
#1,671 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Something like this to reproduce:

import pickle
import fsspec

with fsspec.open("simplecache::root://cmseos.fnal.gov//store/user/ikrommyd/dummy/dummy.pkl", "wb") as f:
    d = {"1": 1, "2": 2}
    pickle.dump(d, f)

fails with:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
----> 1 with fsspec.open("simplecache::root://cmseos.fnal.gov//store/user/ikrommyd/dummy/dummy.pkl", "wb") as f:
      2     d = {"1": 1, "2": 2}
      3     pickle.dump(d, f)

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/core.py:134, in OpenFile.__exit__(self, *args)
    133 def __exit__(self, *args):
--> 134     self.close()

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/core.py:154, in OpenFile.close(self)
    152     if "r" not in self.mode and not f.closed:
    153         f.flush()
--> 154     f.close()
    155 self.fobjects.clear()

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/implementations/cached.py:911, in LocalTempFile.close(self)
    909 self.closed = True
    910 if self.autocommit:
--> 911     self.commit()

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/implementations/cached.py:918, in LocalTempFile.commit(self)
    917 def commit(self):
--> 918     self.fs.put(self.fn, self.path, **self.kwargs)

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    115 @functools.wraps(func)
    116 def wrapper(*args, **kwargs):
    117     self = obj or args[0]
--> 118     return sync(self.loop, func, *args, **kwargs)

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:589, in AsyncFileSystem._put(self, lpath, rpath, recursive, callback, batch_size, maxdepth, **kwargs)
    586     put_file = callback.branch_coro(self._put_file)
    587     coros.append(put_file(lfile, rfile, **kwargs))
--> 589 return await _run_coros_in_chunks(
    590     coros, batch_size=batch_size, callback=callback
    591 )

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:268, in _run_coros_in_chunks(coros, batch_size, callback, timeout, return_exceptions, nofiles)
    266     done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
    267     while done:
--> 268         result, k = await done.pop()
    269         results[k] = result
    271 return results

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:245, in _run_coros_in_chunks.<locals>._run_coro(coro, i)
    243 async def _run_coro(coro, i):
    244     try:
--> 245         return await asyncio.wait_for(coro, timeout=timeout), i
    246     except Exception as e:
    247         if not return_exceptions:

File ~/miniforge3/envs/egamma_dev/lib/python3.10/asyncio/tasks.py:408, in wait_for(fut, timeout)
    405 loop = events.get_running_loop()
    407 if timeout is None:
--> 408     return await fut
    410 if timeout <= 0:
    411     fut = ensure_future(fut, loop=loop)

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/callbacks.py:81, in Callback.branch_coro.<locals>.func(path1, path2, **kwargs)
     78 @wraps(fn)
     79 async def func(path1, path2: str, **kwargs):
     80     with self.branched(path1, path2, **kwargs) as child:
---> 81         return await fn(path1, path2, callback=child, **kwargs)

File ~/miniforge3/envs/egamma_dev/lib/python3.10/site-packages/fsspec/asyn.py:517, in AsyncFileSystem._put_file(self, lpath, rpath, **kwargs)
    516 async def _put_file(self, lpath, rpath, **kwargs):
--> 517     raise NotImplementedError

NotImplementedError:

I haven't tried with other types of remote paths. Perhaps there's a more general problem that isn't only xrootd+simplecache.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the fsspec.open example with simplecache:: and the xrootd path. Trace LocalTempFile.commit through fs.put and AsyncFileSystem._put_file, as shown in the traceback, and compare the behavior with other remote paths if relevant. Done means writing and closing the pickle succeeds without NotImplementedError, with regression coverage for the fixed case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.