pytroll / pytroll/satpy

Dependency tree not propagating resolution

Open
#151 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
335
Avg merge
1d 18h
Merged PRs (30d)
10

Description

Code Sample, a minimal, complete, and verifiable piece of code
from datetime import datetime

from satpy.scene import Scene
from satpy.utils import debug_on
import glob

debug_on()

if __name__ == '__main__':
    scn = Scene(
        sensor="viirs",
        start_time=datetime(2015, 3, 11, 11, 15),
        end_time=datetime(2015, 3, 11, 11, 30),
        filenames=glob.glob("/home/a001673/data/satellite/Suomi-NPP/viirs/lvl1b/2015/03/11/SDR/*"),

        reader="viirs_sdr")
    composite = 'true_color_lowres'
    scn.load([composite])

Add a breakpoint at the beginning of the Scene.compute method, and print out the dependency tree:

pdb> print(self.dep_tree)
Problem description

Currently, the dependency search for composites doesn't propagate the correct resolution, leading to unnecessary complication of the composite and modifier configs in the yaml files.

For example, such a yaml definition:

  rayleigh_corrected:
    compositor: !!python/name:satpy.composites.PSPRayleighReflectance
    atmosphere: us-standard
    aerosol_type: marine_clean_aerosol
    prerequisites:
    - name: M05
      modifiers: [sunz_corrected]
    optional_prerequisites:
    - satellite_azimuth_angle
    - satellite_zenith_angle
    - solar_azimuth_angle
    - solar_zenith_angle

  true_color_lowres:
    compositor: !!python/name:satpy.composites.RGBCompositor
    prerequisites:
    - name: M05
      modifiers: [sunz_corrected, rayleigh_corrected]
    - name: M04
      modifiers: [sunz_corrected, rayleigh_corrected]
    - name: M03
      modifiers: [sunz_corrected, rayleigh_corrected]
    standard_name: true_color

where the angles are defined in both M- and I-band resolution leads to the wrong resolution being chosen under certain circumstances (for example when the I-band angles are defined first in the yaml dataset list).

If no modification is made to the above yaml, the presented code crashes down the line because of not having access to the M-band resolution angles.

Expected Behaviour

I'm expecting the dependency resolution to choose the right resolution when multiple resolutions are available for some datasets but not others.

Of course, creating per-resolution modifiers fixes the problem, but I find this ugly and non-optimal.

Actual Result, Traceback if applicable
ipdb> print(self.dep_tree)
None (No Data)
 +DatasetID(name='true_color_lowres', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=None)
 + +DatasetID(name='M05', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
 + + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + +DatasetID(name='M04', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
 + + +DatasetID(name='M04', wavelength=(0.545, 0.555, 0.565), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + +DatasetID(name='M03', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
 + + +DatasetID(name='M03', wavelength=(0.478, 0.488, 0.498), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
 + + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
 + + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
Versions of Python, package at hand and relevant dependencies

Python 2.7, 3.6, satpy 7.8 or develop as of today.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Scene.compute and trace how the dependency tree resolves composite prerequisites and modifier dependencies. Reproduce the provided true_color_lowres configuration with both M- and I-band angle resolutions; done means resolution selection is propagated correctly and the example no longer fails because M-band angles are unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.