UCL / UCL/TLOmodel

Files downloaded in BatchRunner appear to be unnecessarily large

Open
#615 1 comment 0 reactions 1 assignee View on GitHub

@tamuri is already working on this.

Since Jul 1, 2026.

enhancement framework
Dominant language
Python
Stars
35
Forks
17
PR merge metrics
No merged PRs in 30d

Description

I note that the content of the folder for each run is duplicative and I wonder if there is an opportunity to reduce the size of the downloads, without incurring a heavy additional load for processing locally (which we definitely wouldn't want).

The folder for each run looks something like this (this is specially in the case of scenario_effect_of_each_treatment-2022-05-31T160914Z).

It appears to contain the same information 3 times:

  • The complete simulation log (compressed): ~27MB
  • The individual log files for each module (compressed): total ~27MB
  • The pickles of the results of running parse_log on each of the above: > 300MB

The result is that what is downloaded is about 10x bigger than what is strictly needed and the biggest files (the pickles) are not compressed.

(NB. I only notice on a set of runs that were very large and had probably too many logs included, and the download has taken many hours. This is probably an edge case, so this may not be a high priority).

image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.