CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Cannot generate csv files larger than ~1 GB
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
### What went wrong?
When trying to download a larger dataset from the People & Parliament production server, I noticed that it's not possible to generate .csv files larger than ca. 1 GB. The downloads overview will show "error" after a while, and the celery logs show a SIGKILL (9). *Probably* caused by too much memory usage, though I couldn't see memory & CPU usage going up beyond 30% though.
### What did you expect to happen?
I thought I could give researchers download rights beyond 1million documents, and they would be able to harvest large amounts of data in one go.
### Screenshot
_No response_
### Where did you find the bug?
- [ ] https://ianalyzer.hum.uu.nl
- [X] https://peopleandparliament.hum.uu.nl
- [ ] https://peace.sites.uu.nl
- [ ] a server hosted elsewhere (i.e. not by the research software lab)
- [ ] a local server
### Version
_No response_
### Steps to reproduce
Attempt a download of >1million documents from parliament-netherlands with a user who has rights to do so.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.