Optimisation and memory usage: pre-compute waveforms at each enrichment cycle
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Waveforms of [each enrichment cycle](https://github.com/GCArullo/JenpyROQ/blob/master/JenpyROQ/jenpyroq.py#L440) should be stored and not re-generated at each while call. Currently, they are [re-generated every time](https://github.com/GCArullo/JenpyROQ/blob/master/JenpyROQ/jenpyroq.py#L316) a new basis element is added, which is also a different algorithm than the standard PyROQ one.
Note that this will induce a lot of memory overhead (see discussion in Smith+ and related downsampling-upsampling solution), hence solving this issue is connected with solving #22.
However, keep in mind that since we only keep the outliers during the cycle, the number of re-computed waveforms at each step will not be as large as the size of the training set.
--------------
Memory usage
Each waveform has Xf frequency points.
Each batch is composed of Xb waveforms.
Each waveform entry is a complex double, so occupies 2*8=16 bytes
Hence, if you just store all of the waveforms, you are using:
mem = Xf * Xb * 16 bytes.
Examples:
GW170817 LVK
Xf = 2.5 * 10^5 (with seglen 128, f_min=23, f_max=2048)
Xb = 10^6
mem = 2.5 * 10^5 * 10^6 * 16 = 1.6 * 2.5 * 10^12 = 4 * 10^12 bytes = 4 TB
--------------
Alternative solutions:
Instead of storing on RAM (when you define a variable), save to pickle and read it at each step (should be very quick), so that you occupy physical memory, and there 4TB are easy.
Use down-sampling as in Smith+.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.