simonsobs / simonsobs/sotodlib
preproccess: Cut portions of TODs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 23
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 14
Description
Right now the preprocess select will cut entire detectors, but there are cases when the data is only bad for part of the TOD.
For example this LAT obs starts off a jumpy nightmare:
and then turns into this reasonable data
but there are enough jumps in that first bit that a lot of dets get cut even though they eventually look fine.
I think it is absolutely true that we want to trash that first portion of the data but trashing the whole thing seems like a waste, we should have some way of marking only part of as garbage and keeping the rest.
One way of doing this is splitting up obs ahead of entering the preproc pipeline into some predetermined "sub-obs" and handle stitching these back together on the metadata end. But my suggested work flow would be to:
- Change select to be on some sort of rolling flag density basis
- Have a thresh at which the whole det gets tossed (I think you want to check that its uniformly bad here)
- Linear gapfill the tossed regions in the kept (so that other flags dont freak out)
- Include a final validity flag so we can use that to not project these samples when mapmaking
- Have some sort of helper function to split TODs up if there are time ranges where most dets are flagged.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the preprocess select entry point and the existing TOD flag handling. Compare the proposed rolling flag-density threshold, whole-detector threshold, gap filling, validity flag, and TOD-splitting helper with the current pipeline and mapmaking expectations. Done means partial bad regions can be excluded while valid samples remain usable, with a final validity signal and a defined approach for mostly flagged time ranges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100