mars-project / mars-project/mars
[BUG] to_csv raises FileNotFoundError: [Errno 2] No such file or directory
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
The problem is that when execute to_csv in a distributed cluster, it may raises FileNotFoundError: [Errno 2] No such file or directory.

It is because the DataFrameToCSV operand will generate a DataFrameToCSVStat operand to prepare data file before the agg of DataFrameToCSV runs, each DataFrameToCSV writes the data to the offset of the file in OperandStage.agg stage.
But, the DataFrameToCSVStat and the DataFrameToCSV with stage=OperandStage.agg are scheduled to different nodes. So, when executing agg, it can't find the file created by DataFrameToCSVStat.
To Reproduce
To help us reproducing this bug, please provide information below:
- Your Python version 3.7.7
- The version of Mars you use Latest master
- Versions of crucial packages, such as numpy, scipy and pandas
- Full stack of the error.
- Minimized code to reproduce the error.
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the DataFrameToCSV and DataFrameToCSVStat operands, especially their scheduling around OperandStage.agg. Reproduce the distributed-cluster to_csv failure described in the issue and verify that the aggregation can read the data produced by the preparation stage when those stages run on different nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100