deepmodeling / deepmodeling/deepmd-kit

[BUG]_SSHFS_incompatibility_issue_with_DeepMD_3.1.2

Open
#6,014 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
649
Avg merge
6d 18h
Merged PRs (30d)
15

Description

### Bug summary

Hello There,

i am currently training a model using Deepmd 3.1.2 with dpgen and I am experiencing issue when training on a SSHFS mounted system. The reported error seems to have a problem regarding the PyTorch workers accessing the same file at the same time.
I have attached the complete message regarding the problem. I am wondering if this has something to do with the Deepmd 3.1.2 or the SSHFS 3.7.3?

### DeePMD-kit Version

3.1.2

### Backend and its version

PyTorch

### How did you download the software?

conda

### Input Files, Running Commands, Error Log, etc.

[Pasted text(20260908-131850).txt](https://github.com/user-attachments/files/32094818/Pasted.text.20260908-131850.txt)

### Steps to Reproduce

train normally on a SSHFS mounted system with the SSHFS version 3.7.3.

### Further Information, Files, and Links

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with Pasted.text.20260908-131850.txt and the reported training setup: DeePMD-kit 3.1.2, PyTorch, dpgen, and SSHFS 3.7.3. Reproduce training on an SSHFS-mounted system and determine whether concurrent worker access causes the reported error; done means identifying a reproducible incompatibility or a confirmed external cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.