python / python/cpython

Fix portability of `sendfile(2)` support detection for Lustre filesystems

Open
#139,416 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

3.13 3.14 3.15 stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

On Lustre on Linux (specifically the AWS implementation) there is a rare and difficult to reproduce failure of shutil.copyfile in its sendfile(2) fast path because the syscall returns ENODATA. The situation is recoverable by falling back to the standard read/write file copy (patch forthcoming).

Reproducing the bug requires the filesystem to be in a specific state which I have not been able to reproduce at will but we've seen it on occasions by multiple users. Additionally similar bug has been seen on rust fs::copy which similarly does not trap the ENODATA error.

Following is sufficient to reproduce bug for the specific (rare) file that fails:

import shutil
shutil.copyfile('src', 'dst')

error:

sent = os.sendfile(outfd, infd, offset, blocksize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 61] No data available: '/shared/fsx1/filename' 
CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs
  • gh-139417

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at shutil.copyfile's sendfile fast path and review how os.sendfile errors are handled. Reproduce the failure with shutil.copyfile('src', 'dst') on the affected filesystem; done means an ENODATA failure falls back to the standard read/write copy path.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.