isce-framework / isce-framework/isce3

`rfiLikelihood` metadata exceeds documented [0,1] interval for tone-rank algorithm

Open
#294 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
239
Forks
90
Avg merge
13d 1h
Merged PRs (30d)
5

Description

Summary

The rfiLikelihood metadata values need to be harmonized across RFI detection algorithms.

The tone-rank algorithm (introduced in #257) outputs ISR (Interference to Signal Ratio) values that can greatly exceed the documented [0,1] interval, while the ST-EVD algorithm correctly produces values within the expected range. (FDNF algorithm should also be considered.)

Problem

The rfiLikelihood metadata field in RSLC, GSLC, GCOV, and QA products is documented as having values in the interval [0,1], where:

  • 0 = lowest RFI severity
  • 1 = highest RFI severity
  • NaN = RFI detection was skipped

However, actual product data contains values far outside this range.

This metadata value is computed by the RSLC workflow and populated into the RSLC HDF5. This metadata is then copied by the downstream workflows and populated into their respective HDF5 files. A fix in RSLC's workflow should resolve the issue in the downstream products as well.

Example

Product: NISAR_L1_PR_RSLC_012_097_D_106_2005_DHDH_A_20260208T225030_20260208T225100_X05016_N_F_J_001.h5

RFI Mitigation Algorithm Used: tone-rank (from /science/LSAR/RSLC/metadata/processingInformation/algorithms/rfiMitigation)

Description attribute:

Severity of radio frequency interference (RFI) contamination in the data. Value is in the interval [0,1], where 0: lowest severity, and 1: highest severity (or NaN if RFI detection was skipped)

Actual values in the product:

Dataset Path Value Expected Range
science/LSAR/RSLC/metadata/calibrationInformation/frequencyA/HH/rfiLikelihood 259756.91 [0, 1]
science/LSAR/RSLC/metadata/calibrationInformation/frequencyA/HV/rfiLikelihood 534179.69 [0, 1]
science/LSAR/RSLC/metadata/calibrationInformation/frequencyB/HH/rfiLikelihood 2.95 [0, 1]
science/LSAR/RSLC/metadata/calibrationInformation/frequencyB/HV/rfiLikelihood 5.28 [0, 1]

All values exceed the documented maximum of 1, with frequency A values exceeding it by 5-6 orders of magnitude.

Root Cause

Different RFI detection algorithms produce values on different scales:

  1. ST-EVD algorithm: Generates RFI likelihood as the ratio of CPIs detected with RFI eigenvalues to total CPIs, producing values in [0,1] interval (focus.py:1162-1179)

  2. FDNF (Frequency Domain Notch Filter) algorithm: Returns the fraction of pulses contaminated by RFI (rfi_pulse_count_sum / (num_pulses * num_rng_blks)), producing values in [0,1] interval (rfi_freq_null.py:206)

  3. Tone-rank algorithm (introduced in #257): Assigns rfi_likelihood = np.max(isr) where isr is the Interference-to-Signal Ratio array (focus.py:1199-1206). ISR values can be >> 1, which violates the documented [0,1] interval.

The tone-rank algorithm assigns rfi_likelihood = np.max(isr) at focus.py:1206, where isr is the Interference-to-Signal Ratio array. This ISR value is then written to the rfiLikelihood dataset with the [0,1] interval description at SLC.py:1072-1078.

The algorithms have not been synchronized to produce consistent output scales.

Impact

  • Users cannot reliably interpret rfiLikelihood values
  • The metadata description is misleading
  • Downstream processing that assumes [0,1] range may fail or produce incorrect results
  • Products generated with different RFI algorithms are not comparable

cc: @hfattahi @bhawkins @hb6688 @Tyler-g-hudson @rad-eng-59 @gshiroma @seongsujeong

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python/packages/nisar/workflows/focus.py at the ST-EVD, FDNF, and tone-rank rfi_likelihood calculations, then inspect python/packages/isce3/signal/rfi_freq_null.py and python/packages/nisar/products/writers/SLC.py. Determine how the algorithms should produce a consistent [0,1] value, including the documented metadata description, and verify that RSLC and downstream product metadata remain interpretable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.