pytroll / pytroll/pyresample

YAML area configuration does not allow to specify the dtype and dtype is ignored in equality comparisons

Open
#590 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
385
Forks
102
Avg merge
4d 2h
Merged PRs (30d)
9

Description

Today I learned that an areadefinition has a dtype attribute. This seems to be not very well documented and not very well supported, but it's relevant. It is used at least in get_lonlats(), which is used in resampling, so changing the dtype can have considerable memory implications and make the difference between having enough RAM or not.

Code Sample, a minimal, complete, and verifiable piece of code
import pyresample

ar = pyresample.AreaDefinition(
        "area_id", "descr", "proj_id", 4326, 100, 100,
       [-100, -100, 100, 100], dtype="float32")
s = ar.dump()
print(s)
ar2 = pyresample.area_config.load_area_from_string(s)
print("equals?", ar == ar2)
print("dtypes", ar.dtype, ar2.dtype)
s2 = """area_id:
  description: descr
  projection:
    EPSG: 4326
  shape:
    height: 100
    width: 100
  area_extent:
    lower_left_xy: [-100, -100]
    upper_right_xy: [100, 100]
  dtype: float32
"""
ar3 = pyresample.area_config.load_area_from_string(s2)
print(ar3.dtype)

Problem description

Executing the code reveals several problems:

  • ar.dump() does not output the dtype
  • ar and ar2 are considered equal despite having different dtypes
  • ar3, loaded from a string that does encode the dtype, gets float64 despite the YAML definition stating float32.

It does not appear to be possible to specify the dtype in the YAML configuration (nor in pyresample.create_area_def()).

Expected Output
area_id:
  description: descr
  projection:
    EPSG: 4326
  shape:
    height: 100
    width: 100
  area_extent:
    lower_left_xy: [-100, -100]
    upper_right_xy: [100, 100]
  dtype: float32

equals? True
dtypes float32 <class 'numpy.float32'>
<class 'numpy.float32'>
Actual Result, Traceback if applicable
area_id:
  description: descr
  projection:
    EPSG: 4326
  shape:
    height: 100
    width: 100
  area_extent:
    lower_left_xy: [-100, -100]
    upper_right_xy: [100, 100]

equals? True
dtypes float32 <class 'numpy.float64'>
/data/gholl/checkouts/pyresample/pyresample/area_config.py:91: UserWarning: Unused/unexpected area definition parameter(s) for area_id: params={'dtype': 'float32'}
  area_list = parse_area_file(area_file_name, *regions)
<class 'numpy.float64'>
Versions of Python, package at hand and relevant dependencies

pyresample main (v1.28.2-2-g711f354)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pyresample/area_config.py and the AreaDefinition dump/load and equality entry points; trace how dtype is passed by load_area_from_string and create_area_def. Done means YAML serialization preserves dtype, YAML loading and create_area_def accept it, and differing dtypes affect equality, with regression coverage for the sample cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, yaml
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.