Dependency tree not propagating resolution
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 335
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 10
Description
Code Sample, a minimal, complete, and verifiable piece of code
from datetime import datetime
from satpy.scene import Scene
from satpy.utils import debug_on
import glob
debug_on()
if __name__ == '__main__':
scn = Scene(
sensor="viirs",
start_time=datetime(2015, 3, 11, 11, 15),
end_time=datetime(2015, 3, 11, 11, 30),
filenames=glob.glob("/home/a001673/data/satellite/Suomi-NPP/viirs/lvl1b/2015/03/11/SDR/*"),
reader="viirs_sdr")
composite = 'true_color_lowres'
scn.load([composite])
Add a breakpoint at the beginning of the Scene.compute method, and print out the dependency tree:
pdb> print(self.dep_tree)
Problem description
Currently, the dependency search for composites doesn't propagate the correct resolution, leading to unnecessary complication of the composite and modifier configs in the yaml files.
For example, such a yaml definition:
rayleigh_corrected:
compositor: !!python/name:satpy.composites.PSPRayleighReflectance
atmosphere: us-standard
aerosol_type: marine_clean_aerosol
prerequisites:
- name: M05
modifiers: [sunz_corrected]
optional_prerequisites:
- satellite_azimuth_angle
- satellite_zenith_angle
- solar_azimuth_angle
- solar_zenith_angle
true_color_lowres:
compositor: !!python/name:satpy.composites.RGBCompositor
prerequisites:
- name: M05
modifiers: [sunz_corrected, rayleigh_corrected]
- name: M04
modifiers: [sunz_corrected, rayleigh_corrected]
- name: M03
modifiers: [sunz_corrected, rayleigh_corrected]
standard_name: true_color
where the angles are defined in both M- and I-band resolution leads to the wrong resolution being chosen under certain circumstances (for example when the I-band angles are defined first in the yaml dataset list).
If no modification is made to the above yaml, the presented code crashes down the line because of not having access to the M-band resolution angles.
Expected Behaviour
I'm expecting the dependency resolution to choose the right resolution when multiple resolutions are available for some datasets but not others.
Of course, creating per-resolution modifiers fixes the problem, but I find this ugly and non-optimal.
Actual Result, Traceback if applicable
ipdb> print(self.dep_tree)
None (No Data)
+DatasetID(name='true_color_lowres', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=None)
+ +DatasetID(name='M05', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
+ + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ +DatasetID(name='M04', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
+ + +DatasetID(name='M04', wavelength=(0.545, 0.555, 0.565), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ +DatasetID(name='M03', wavelength=None, resolution=None, polarization=None, calibration=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
+ + +DatasetID(name='M03', wavelength=(0.478, 0.488, 0.498), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='M05', wavelength=(0.662, 0.672, 0.682), resolution=742, polarization=None, calibration='reflectance', modifiers=('sunz_corrected',))
+ + +DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
+ + +DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, modifiers=())
Versions of Python, package at hand and relevant dependencies
Python 2.7, 3.6, satpy 7.8 or develop as of today.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Scene.compute and trace how the dependency tree resolves composite prerequisites and modifier dependencies. Reproduce the provided true_color_lowres configuration with both M- and I-band angle resolutions; done means resolution selection is propagated correctly and the example no longer fails because M-band angles are unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100