cms-dev / cms-dev/cms

DumpExporter fills up RAM

Open
#1,226 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

import_export
Dominant language
Python
Stars
1k
Forks
412
Avg merge
6d 10h
Merged PRs (30d)
3

Description

cmsDumpExporter will put everything in a tempdir (i.e. usually /tmp) before packing it up in an archive at the desired path:
https://github.com/cms-dev/cms/blob/7d962150d9969281148d3eaba1ab35823fdfb1b6/cmscontrib/DumpExporter.py#L189-L190

On some systems, including Arch, /tmp is in RAM by default, and limited to half the RAM's size. At least on personal computers, this will usually cause the system to freeze or crash. Of course, CMS writes to /tmp on other occasions, but in normal operation, it will cleanup shortly afterwards and never write something as large as the whole database at once.

It seems that by setting one of the TMPDIR, TEMP, or TMP environment variables, one can specify another location, so circumventing the issue is technically easy. However, not everyone will know this and think of it beforehand, and a system freeze is annoying, as is finding out the cause of the problem.

Thus, I would like to suggest improving this somehow.

  • The files have to be written somewhere of course, and using tempfile.mkdtemp makes sense. Easy and still helpful would be a warning, so at least one will know why the system froze (or be able to hit STRG+C if fast enough...). This should include suggesting to set TMPDIR to somewhere big enough.
  • Maybe it could even explicitly ask along the lines of Path to temporary directory (default is system temp dir; sufficient free space is needed):
  • Or it could use the directory where it shall export the archive. Drawback: If the exporter crashes, it won't be cleaned up.

This sounds similar to #322, but I think that was more about importing and exporting specific contests and is unrelated.

Happy to hear your thoughts!

PS: This has been in the back of my head since forever and I haven't seen someone else mention it, so who knows if I'm the only one with this problem. 😄

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read cmscontrib/DumpExporter.py around lines 189-190 and the Python tempfile.gettempdir documentation. Compare the proposed temporary-directory approaches and define how large exports should avoid exhausting RAM; completion should include an explicit, decided behavior rather than leaving the implementation choice open.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.