dask / dask/distributed

Constantly having MemoryError exceptions when processing dataframe

Open
#4,144 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
I'm trying to process a large dataframe (1,5Go and millions of rows) with dask and dask distributed but very quickly i get MemoryError exceptions and i need to restart python or workers.

**What you expected to happen**:
My comprehension is that Dask is not supposed to persist data in memory unless when using "compute" so i don't expect to get memory erros when retreive few data from the dataframe.

**Minimal Complete Verifiable Example**:

```python
dfT = dd.read_csv(r'taxa.txt', sep='\t', dtype={'description':'str','modified':'str','subgenus':'str','verbatimTaxonRank':'str', 'taxonConceptID':'str', 'datasetID':'str'})
canceraspis = dfT.loc[dfT.taxonID==316502,:]
dfT.shape[0].compute()
```

**Anything else we need to know?**:
The memory error doesn't always occur at the same time so i guess it depends of what i did before.
Dataset can be found here http://www.catalogueoflife.org/DCA_Export/zip/archive-complete.zip

**Environment**:

- Dask version:2.28.0
- Python version:3.8.1
- Operating System:Windows 10
- Install method (conda, pip, source): pip

The code is run in jupterlab or in vs code notebooks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.