simonsobs / simonsobs/sotodlib

preproccess: Cut portions of TODs

Open
#801 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

preprocess
Dominant language
Python
Stars
19
Forks
23
Avg merge
1d 5h
Merged PRs (30d)
14

Description

Right now the preprocess select will cut entire detectors, but there are cases when the data is only bad for part of the TOD.

For example this LAT obs starts off a jumpy nightmare:
image
and then turns into this reasonable data
image

but there are enough jumps in that first bit that a lot of dets get cut even though they eventually look fine.
I think it is absolutely true that we want to trash that first portion of the data but trashing the whole thing seems like a waste, we should have some way of marking only part of as garbage and keeping the rest.

One way of doing this is splitting up obs ahead of entering the preproc pipeline into some predetermined "sub-obs" and handle stitching these back together on the metadata end. But my suggested work flow would be to:

  • Change select to be on some sort of rolling flag density basis
  • Have a thresh at which the whole det gets tossed (I think you want to check that its uniformly bad here)
  • Linear gapfill the tossed regions in the kept (so that other flags dont freak out)
  • Include a final validity flag so we can use that to not project these samples when mapmaking
  • Have some sort of helper function to split TODs up if there are time ranges where most dets are flagged.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the preprocess select entry point and the existing TOD flag handling. Compare the proposed rolling flag-density threshold, whole-detector threshold, gap filling, validity flag, and TOD-splitting helper with the current pipeline and mapmaking expectations. Done means partial bad regions can be excluded while valid samples remain usable, with a final validity signal and a defined approach for mostly flagged time ranges.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.