simonsobs / simonsobs/sotodlib

Unsure how exactly to use `ArchivePolicy` in site_pipeline scripts

Open
#670 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19
Forks
23
Avg merge
1d 5h
Merged PRs (30d)
14

Description

For the smurf_caldb script, I'm looking into how to use ArchivePolicy to determine the idx path, and there are some things that I'm confused about.

Different ArchivePolicies currently return different data from the get_dest function, with the ArchivePolicy class returning (h5path, dataset_addr) and DirectoryArchivePolicy returning just h5path. This means that if you create an archive policy using ArchivePolicy.from_params, you need to know which kind of archive policy you have before interpreting the get_dest results.

I think that all implementations of ArchivePolicy such as here: https://github.com/simonsobs/sotodlib/blob/0749f9f8910c56b0ed216c1d88536cb9eccf9c57/sotodlib/site_pipeline/make_uncal_beam_map.py#L329-L332
are assuming a particular ArchivePolicy type, and would fail if the other is used, so they should probably not be using the general from_params constructor.

I think this should probably be fixed by making sure all ArchivePolicies return the same data structure from get_dest, so that code doesn't need to assume a particular archive policy style.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the smurf_caldb script and the ArchivePolicy.from_params and get_dest implementations, then inspect the usage in site_pipeline/make_uncal_beam_map.py around lines 329-332. Compare the return values for ArchivePolicy and DirectoryArchivePolicy and determine a consistent structure so callers do not depend on the concrete policy type; done means the affected implementations and usages follow that contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.