GCArullo / GCArullo/JenpyROQ

Optimisation and memory usage: pre-compute waveforms at each enrichment cycle

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Waveforms of [each enrichment cycle](https://github.com/GCArullo/JenpyROQ/blob/master/JenpyROQ/jenpyroq.py#L440) should be stored and not re-generated at each while call. Currently, they are [re-generated every time](https://github.com/GCArullo/JenpyROQ/blob/master/JenpyROQ/jenpyroq.py#L316) a new basis element is added, which is also a different algorithm than the standard PyROQ one.
Note that this will induce a lot of memory overhead (see discussion in Smith+ and related downsampling-upsampling solution), hence solving this issue is connected with solving #22.

However, keep in mind that since we only keep the outliers during the cycle, the number of re-computed waveforms at each step will not be as large as the size of the training set.

--------------

Memory usage

Each waveform has Xf frequency points.
Each batch is composed of Xb waveforms.
Each waveform entry is a complex double, so occupies 2*8=16 bytes

Hence, if you just store all of the waveforms, you are using:

mem = Xf * Xb * 16 bytes.

Examples:

GW170817 LVK
Xf = 2.5 * 10^5 (with seglen 128, f_min=23, f_max=2048)
Xb = 10^6

mem = 2.5 * 10^5 * 10^6 * 16 = 1.6 * 2.5 * 10^12 = 4 * 10^12 bytes = 4 TB

--------------

Alternative solutions:

Instead of storing on RAM (when you define a variable), save to pickle and read it at each step (should be very quick), so that you occupy physical memory, and there 4TB are easy.

Use down-sampling as in Smith+.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.