mars-project / mars-project/mars

[BUG] to_csv raises FileNotFoundError: [Errno 2] No such file or directory

Open
#3,038 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

The problem is that when execute to_csv in a distributed cluster, it may raises FileNotFoundError: [Errno 2] No such file or directory.

image

It is because the DataFrameToCSV operand will generate a DataFrameToCSVStat operand to prepare data file before the agg of DataFrameToCSV runs, each DataFrameToCSV writes the data to the offset of the file in OperandStage.agg stage.

But, the DataFrameToCSVStat and the DataFrameToCSV with stage=OperandStage.agg are scheduled to different nodes. So, when executing agg, it can't find the file created by DataFrameToCSVStat.

To Reproduce
To help us reproducing this bug, please provide information below:

  1. Your Python version 3.7.7
  2. The version of Mars you use Latest master
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  5. Minimized code to reproduce the error.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the DataFrameToCSV and DataFrameToCSVStat operands, especially their scheduling around OperandStage.agg. Reproduce the distributed-cluster to_csv failure described in the issue and verify that the aggregation can read the data produced by the preparation stage when those stages run on different nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.