simonsobs / simonsobs/sotodlib

Toast / AxisManager Translation

Open
#1,693 0 comments 0 reactions 1 assignee View on GitHub

@tskisner is already working on this.

Since Jul 9, 2026.

Dominant language
Python
Stars
19
Forks
23
Avg merge
1d 5h
Merged PRs (30d)
14

Description

This issue tracks previous discussion about translation between toast.Observation containers and AxisManager hierarchies. The goal is to support the limited case where a toast process group is set to a size of "1". This will be inefficient for some workflows, but still useful.

Phase 1

  • Extract the general AxisManager to Observation translation code in sotodlib.toast.ops.LoadContext into stand-alone routines that can be called from multiple places in the code.
  • Extract the specific Observation to AxisManager translation code from sotodlib.toast.ops.MLMapmaker into stand-alone routines. Make these more general.
  • Write round-trip tests that go from simulated toast Observation to AxisManager and back.
  • Note: translation needs to work with both demodulated and un-demodulated data.

User-facing Features Enabled:

  • The LoadContext operator would now support running a site-pipeline-on-load that goes all the way up through demodulation.

Phase 2

  • Add a new toast operator in sotodlib.toast.ops which takes a site-pipeline config file and runs it. This will translate from Observation to AxisManager, call PreProcPipe.run() and translate back.
  • Add a new ToastPipeline class that derives from _Preprocess and which can be used in a site pipeline job. The toast config information can be in an external file or included directly as nested yaml in a larger config file.
  • Add helper script to sync a toast volume index (sqlite DB, with possible extra columns like PWV, etc) into an "extended" obsdb and obsfiledb.
  • Add a new context loader (toast_hdf5). This will access the obsdb / obsfiledb created in the previous step. After loading an Observation, it will be translated to an AxisManager and returned.

User-facing Features Enabled:

  • Any code loading from a context (including toast workflows) would be able to load toast HDF5 files from disk through the context system. In the case of simulated data, one would disable any other metadata products in the context. In the case of real data, the context would look the same as a standard context, just with a different loader specified.
  • Toast workflows with group size == 1 could call arbitrary site pipelines, in the same way we call the MLMapmaker currently.
  • Site pipeline configs could call single-process toast pipelines for simulation or analysis steps.

Phase 3

  • Synthetic simulation updates: for historical reasons, our synthetic simulation tools use naming schemes that do not always match the deployed conventions. Fixing these would relieve many headaches. Mainly this involves bookkeeping and re-organizing some metadata in the toast Observations to match their name / locations when loaded from real data.

User-facing Features Enabled:

  • Less pain when trying to simulate data from a specific telescope using a synthetic observing schedule.
  • Easier integration testing when checking roundtrip data translation.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.