CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Cannot generate csv files larger than ~1 GB

Open
#1,687 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

### What went wrong?

When trying to download a larger dataset from the People & Parliament production server, I noticed that it's not possible to generate .csv files larger than ca. 1 GB. The downloads overview will show "error" after a while, and the celery logs show a SIGKILL (9). *Probably* caused by too much memory usage, though I couldn't see memory & CPU usage going up beyond 30% though.

### What did you expect to happen?

I thought I could give researchers download rights beyond 1million documents, and they would be able to harvest large amounts of data in one go.

### Screenshot

_No response_

### Where did you find the bug?

- [ ] https://ianalyzer.hum.uu.nl
- [X] https://peopleandparliament.hum.uu.nl
- [ ] https://peace.sites.uu.nl
- [ ] a server hosted elsewhere (i.e. not by the research software lab)
- [ ] a local server

### Version

_No response_

### Steps to reproduce

Attempt a download of >1million documents from parliament-netherlands with a user who has rights to do so.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.