huggingface / huggingface/datasets

Feature request: being able to disable chmod for flat permission filesystems

Open
#8,125 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

For flat permission file systems, `os.chmod` simply fails when trying to cache datasets - in some places such as: https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L4043-L4048 for arrow datasets.
This seems to be optional to obtained desired behavior as simply commenting it out makes our workload work as expected. Hence the feature request would be to see if it would be possible to expose an API to disable `os.chmod` operations when users asks to do so (e.g. through an env variable?)

### Motivation

Being able to save properly datasets on flat permission filesystems

### Your contribution

Can test the feature if implemented

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.