DumpExporter fills up RAM
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 412
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 3
Description
cmsDumpExporter will put everything in a tempdir (i.e. usually /tmp) before packing it up in an archive at the desired path:
https://github.com/cms-dev/cms/blob/7d962150d9969281148d3eaba1ab35823fdfb1b6/cmscontrib/DumpExporter.py#L189-L190
On some systems, including Arch, /tmp is in RAM by default, and limited to half the RAM's size. At least on personal computers, this will usually cause the system to freeze or crash. Of course, CMS writes to /tmp on other occasions, but in normal operation, it will cleanup shortly afterwards and never write something as large as the whole database at once.
It seems that by setting one of the TMPDIR, TEMP, or TMP environment variables, one can specify another location, so circumventing the issue is technically easy. However, not everyone will know this and think of it beforehand, and a system freeze is annoying, as is finding out the cause of the problem.
Thus, I would like to suggest improving this somehow.
- The files have to be written somewhere of course, and using
tempfile.mkdtempmakes sense. Easy and still helpful would be a warning, so at least one will know why the system froze (or be able to hitSTRG+Cif fast enough...). This should include suggesting to setTMPDIRto somewhere big enough. - Maybe it could even explicitly ask along the lines of
Path to temporary directory (default is system temp dir; sufficient free space is needed): - Or it could use the directory where it shall export the archive. Drawback: If the exporter crashes, it won't be cleaned up.
This sounds similar to #322, but I think that was more about importing and exporting specific contests and is unrelated.
Happy to hear your thoughts!
PS: This has been in the back of my head since forever and I haven't seen someone else mention it, so who knows if I'm the only one with this problem. 😄
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read cmscontrib/DumpExporter.py around lines 189-190 and the Python tempfile.gettempdir documentation. Compare the proposed temporary-directory approaches and define how large exports should avoid exhausting RAM; completion should include an explicit, decided behavior rather than leaving the implementation choice open.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100