simonsobs / simonsobs/sotodlib

Preprocess axis manager restrict appears to result in memory duplication

Open
#1,280 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

preprocess
Dominant language
Python
Stars
19
Forks
23
Avg merge
1d 5h
Merged PRs (30d)
14

Description

While working on some preprocessing profiling, I noticed that the restrict step in certain select functions results in the axis manager data being seemingly duplicated until the end of preprocessing even with in_place=True when restricting. I have added tracemalloc snapshots in #1269 in the preprocess pipeline run function in the order of:

start tracing
loop over processes
   process()
   snapshot()
   calc_and_save()
   snapshot()
   select()
   snapshot()
end tracing

Tested with nproc=1 to avoid issues with parallelization.

For select functions in such steps as fp_flags and det_bias_flags the line:

meta.restrict("dets", meta.dets.vals[keep])

results in the memory duplicating. The memory is not cleared until preprocess is done, unlike other steps that clearly show memory being freed. See the figure as an example where that is the final snapshot from the list above (ignore the jump after subtract_hwpss, that is a config file typo). Commenting out the restrict line results in the expected memory usage. However this only occurs once, such that if fp_flags is called before det_bias flags, you do not see a second duplication from det_bias flags. Tracemalloc points to it being this line:

https://github.com/simonsobs/sotodlib/blob/9d8e1051133c6cad789ee0bf9d0b208e949574bd/sotodlib/core/axisman.py#L860

for where the memory allocation is coming from.

Image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the preprocessing pipeline with nproc=1 and the listed tracemalloc snapshot order. Inspect sotodlib/preprocess/processes.py at the fp_flags and det_bias_flags select calls, then follow sotodlib/core/axisman.py around line 860; done means restrict with in_place=True no longer retains the apparent duplicated axis-manager memory through preprocessing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.