RadioAstronomySoftwareGroup / RadioAstronomySoftwareGroup/pyuvdata

Build script for downselection on frequencies, polarizations, lsts, etc.

Open
#896 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

HERA UVData
Dominant language
Python
Stars
102
Forks
35
Avg merge
1d 20h
Merged PRs (30d)
14

Description

For the upcoming HERA season, we're going to need the RTP to perform a data downselection for internet transfer off site. We need a python script for performing this (which we can then build a shell wrapper around for OPM) and I think pyuvdata is reasonable place for it to live... though I'm open to other suggestions.

Here are the features I think we'll need:

  • Ability to take in a raw data file using partial I/O to save memory (probably it's easiest to do partial I/O over groups of baselines)
  • Ability to take in a UVFlag waterfall file, select on times to match the file (if the flags are for the whole night), and then apply those flags to the data.
  • Perform downselection on the following axes:
    • LST: given a list of pairs of floats that describe LST ranges, only keep data in those ranges
    • Solar altitude: keep only times when the sun is below some altitude parameter
    • Band: allow for a reduced band (or maybe subbands via spectral windows, though I don't personally need need that)
    • Polarization: allow for a reduction of polarizations from 4 pol to any subset of those four
    • Frequency Resolution: allow for a reduction of frequency resolution by binning together an integer number of channels. (Question: does this necessitate rephasing?)
    • Temporal Resolution: allow for data reduction via a common temporal resolution, or a reduction of temporal resolution by an integer factor (Question: does this necessitate rephasing? Probably...)
    • Baseline length: allow for a cut on minimum/maximum baseline length. (We might also want a separate cut on minimum/maximum EW length, but I'm fine punting on that)
    • Only keep autos or crosses
    • Others?
  • Write out to uvh5 using partial i/o

For both temporal/frequency downsampling, I think we should also decide how flags get handled. I can see a two two ways to do this:

  1. Average/sum all data that goes into a single freq/time bin, but if any of the bins that get combined are flagged, flag the whole bin.
  2. Average/sum only unflagged data, but keep track of how many unflagged samples went into each bin via nsamples. If nsamples == 0 then also flag.
    I think the user should be able to choose which of those two is used.

Thoughts on this design @bhazelton @nithyanandan?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing pyuvdata's existing partial-I/O support and its handling of UVFlag and uvh5 data. Define the entry point and scope for frequency, time, LST, solar-altitude, polarization, band, baseline, and auto/cross selection, including flag and nsamples behavior; done means the selected data can be written to uvh5 without requiring the full input in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.