Constantly having MemoryError exceptions when processing dataframe
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
I'm trying to process a large dataframe (1,5Go and millions of rows) with dask and dask distributed but very quickly i get MemoryError exceptions and i need to restart python or workers.
**What you expected to happen**:
My comprehension is that Dask is not supposed to persist data in memory unless when using "compute" so i don't expect to get memory erros when retreive few data from the dataframe.
**Minimal Complete Verifiable Example**:
```python
dfT = dd.read_csv(r'taxa.txt', sep='\t', dtype={'description':'str','modified':'str','subgenus':'str','verbatimTaxonRank':'str', 'taxonConceptID':'str', 'datasetID':'str'})
canceraspis = dfT.loc[dfT.taxonID==316502,:]
dfT.shape[0].compute()
```
**Anything else we need to know?**:
The memory error doesn't always occur at the same time so i guess it depends of what i did before.
Dataset can be found here http://www.catalogueoflife.org/DCA_Export/zip/archive-complete.zip
**Environment**:
- Dask version:2.28.0
- Python version:3.8.1
- Operating System:Windows 10
- Install method (conda, pip, source): pip
The code is run in jupterlab or in vs code notebooks
Contributor guide
Assessment
This issue has not been assessed yet.