simonsobs / simonsobs/sotodlib
Preprocess axis manager restrict appears to result in memory duplication
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 23
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 14
Description
While working on some preprocessing profiling, I noticed that the restrict step in certain select functions results in the axis manager data being seemingly duplicated until the end of preprocessing even with in_place=True when restricting. I have added tracemalloc snapshots in #1269 in the preprocess pipeline run function in the order of:
start tracing
loop over processes
process()
snapshot()
calc_and_save()
snapshot()
select()
snapshot()
end tracing
Tested with nproc=1 to avoid issues with parallelization.
For select functions in such steps as fp_flags and det_bias_flags the line:
meta.restrict("dets", meta.dets.vals[keep])
results in the memory duplicating. The memory is not cleared until preprocess is done, unlike other steps that clearly show memory being freed. See the figure as an example where that is the final snapshot from the list above (ignore the jump after subtract_hwpss, that is a config file typo). Commenting out the restrict line results in the expected memory usage. However this only occurs once, such that if fp_flags is called before det_bias flags, you do not see a second duplication from det_bias flags. Tracemalloc points to it being this line:
for where the memory allocation is coming from.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the preprocessing pipeline with nproc=1 and the listed tracemalloc snapshot order. Inspect sotodlib/preprocess/processes.py at the fp_flags and det_bias_flags select calls, then follow sotodlib/core/axisman.py around line 860; done means restrict with in_place=True no longer retains the apparent duplicated axis-manager memory through preprocessing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100