fsspec / fsspec/filesystem_spec

DirFileSystem does not propagate transaction context to underlying filesystem

Open
#1,823 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

When wrapping a transactional file:// backend in DirFileSystem (e.g. filesystem("dir", …, fs=filesystem("file"))), entering with fs.transaction: on the dir:// wrapper sets only the wrapper’s _intrans flag. Because DirFileSystem never delegates its transaction context down to the wrapped LocalFileSystem, all writes (even via fs.open(..., "wb")) commit immediately rather than being deferred to temp files and renamed on commit.

Steps to Reproduce

Create a transactional file:// filesystem and wrap it in dir://:

import os, tempfile, fsspec

tmp = tempfile.mkdtemp()
base = fsspec.filesystem("file")  
fs   = fsspec.filesystem("dir", path=tmp, fs=base)

Enter a transaction on the dir:// wrapper and write a file:

with fs.transaction:
    with fs.open("data.txt", "wb") as f:
        f.write(b"hello")
    exists_inside = os.path.exists(os.path.join(tmp, "data.txt"))
    print("Exists inside transaction?", exists_inside)

Observe that exists_inside is True, even though no commit has occurred yet.

Expected Behavior
  • Inside the with fs.transaction: block, no file should appear on disk.
  • After exiting the block without errors, the file should be atomically renamed into place.
  • On error, no partial files should remain.
Actual Behavior
  • The file is created on disk immediately, inside the transaction block.
  • There is no deferral to a temp file, and no atomic rename on commit
Proposed Fix

Override DirFileSystem.transaction to delegate to self.fs.transaction, so that the wrapper and wrapped FS share the same transaction context and _intrans flag.

Minimal Repro Code
import os, tempfile, fsspec

tmp  = tempfile.mkdtemp()
base = fsspec.filesystem("file")
fs   = fsspec.filesystem("dir", path=tmp, fs=base)

print("Before transaction:", base._intrans, fs._intrans)

with fs.transaction:
    print("Inside transaction:", base._intrans, fs._intrans)
    with fs.open("data.txt", "wb") as f:
        f.write(b"hello")
    # Should be False, but is True
    print("Exists inside?", os.path.exists(os.path.join(tmp, "data.txt")))

print("Clean up")
os.remove(os.path.join(tmp, "data.txt"))
os.rmdir(tmp)

Output

Before transaction: False False
Inside transaction: False True
Exists inside? True
Clean up

Environment

fsspec version: 2025.3.2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating DirFileSystem and its transaction handling, then inspect how fs.open writes through the wrapped filesystem. Reproduce the transaction with the provided example and add coverage for deferred visibility, commit renaming, and cleanup after an error; done means the wrapper delegates transaction context and the expected filesystem behavior holds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.