dask / dask/distributed

AttributeError: 'str' object has no attribute 'copy'

Open
#6,315 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
When computing my distributed `DataFrame`: I encountered the error `AttributeError: 'str' object has no attribute 'copy' `

**What you expected to happen**:
The computation to succeed and save the result to parquet

**Minimal Complete Verifiable Example**:

Unfortunately I can't exactly reproduce this, it seems to happen intermittently, even under the exam circumstances. **I assume this is caused by a race condition**. I'm wondering if an experienced Dask user/dev can look at my stack trace and think about why this might be happening.

**Anything else we need to know?**:
The full stack trace is this:
```
Traceback (most recent call last):
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 930, in get_task_run_state
result = self.result.write(value, **formatting_kwargs)
File "/biosig/workspace/users/mmilton/DriverGenePipe/driverpipe/task/result.py", line 39, in write
value_.to_parquet(full_path, **save_kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/dataframe/core.py", line 4990, in to_parquet
return to_parquet(self, path, *args, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/dataframe/io/parquet/core.py", line 800, in to_parquet
i_offset, fmd, metadata_file_exists, extra_write_kwargs = engine.initialize_write(
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/dataframe/io/parquet/arrow.py", line 537, in initialize_write
_s = delayed_schema_from_pandas(
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/base.py", line 292, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/base.py", line 575, in compute
results = schedule(dsk, keys, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/client.py", line 3004, in get
results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/client.py", line 2178, in gather
return self.sync(
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/utils.py", line 318, in sync
return sync(
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/utils.py", line 385, in sync
raise exc.with_traceback(tb)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/utils.py", line 358, in f
result = yield future
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/distributed/client.py", line 2041, in _gather
raise exception.with_traceback(traceback)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/optimization.py", line 990, in __call__
return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/core.py", line 149, in get
result = _execute_task(task, cache)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
return func(*(_execute_task(a, cache) for a in args))
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/utils.py", line 40, in apply
return func(*args, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/dataframe/core.py", line 6431, in apply_and_enforce
df = func(*args, **kwargs)
File "/home/mmilton/.conda/envs/mmilton/envs/driverpipe/lib/python3.9/site-packages/dask/dataframe/utils.py", line 712, in drop_by_shallow_copy
df2 = df.copy(deep=False)
AttributeError: 'str' object has no attribute 'copy'
```

Now, I managed to get it to print the value of this string that is supposed to be a DataFrame, and what it returns is:
`('read-parquet-4df82c883c7ce5b52038ec47dbf08ba0\, 43)`. So it's a tuple indicating a task key. Is there some way that this might be accidentally passed instead of a real `DataFrame`?

The *approximate* sequence of functions I'm running is `some_dask_series.to_frame().drop(columns=[]).apply(some_func, row=1, meta=...).to_parquet("destination")`

Also I note that this is a similar bug to https://github.com/dask/distributed/issues/2323.

**Environment**:

- Dask version: dask-2022.5.0 + dask_jobqueue-0.7.3
- Python version: 3.9.12
- Operating System: Ubuntu 20.04.4 LTS
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.