Open-EO / Open-EO/openeo-python-client

testing: support large arrays

Open
#871 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
217
Forks
56
Avg merge
1d 22h
Merged PRs (30d)
2

Description

We see tests going OOM when comparing large datasets.
There exists at least one solution that claims to solve this problem:
https://cubed-dev.github.io/cubed/examples/xarray.html

tests/test_benchmarks.py:146: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv/lib/python3.11/site-packages/openeo/testing/results.py:517: in assert_job_results_allclose
    issues = _compare_job_results(
venv/lib/python3.11/site-packages/openeo/testing/results.py:449: in _compare_job_results
    issues = _compare_xarray_dataarray(
venv/lib/python3.11/site-packages/openeo/testing/results.py:238: in _compare_xarray_dataarray
    threshold = abs(expected * rtol) + atol
                    ^^^^^^^^^^^^^^^
venv/lib/python3.11/site-packages/xarray/core/_typed_ops.py:551: in __mul__
    return self._binary_op(other, operator.mul)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
venv/lib/python3.11/site-packages/xarray/core/dataarray.py:4817: in _binary_op
    f(self.variable, other_variable_or_arraylike)
venv/lib/python3.11/site-packages/xarray/core/_typed_ops.py:934: in __mul__
    return self._binary_op(other, operator.mul)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <xarray.Variable (band: 1, y: 68355, x: 33904)> Size: 5GB
array([[[-32768, -32768, ..., -32768, -32768],
        [-327...lValue:           -32768
    scale_factor:         0.01
    add_offset:           0.0
    long_name:            biomass
other = 0.001, f = <built-in function mul>, reflexive = False

    def _binary_op(self, other, f, reflexive=False):
        if isinstance(other, xr.DataTree | xr.DataArray | xr.Dataset):
            return NotImplemented
        if reflexive and issubclass(type(self), type(other)):
            other_data, self_data, dims = _broadcast_compat_data(other, self)
        else:
            self_data, other_data, dims = _broadcast_compat_data(self, other)
        keep_attrs = _get_keep_attrs(default=False)
        attrs = self._attrs if keep_attrs else None
        with np.errstate(all="ignore"):
            new_data = (
>               f(self_data, other_data) if not reflexive else f(other_data, self_data)
                ^^^^^^^^^^^^^^^^^^^^^^^^
            )
E           numpy._core._exceptions._ArrayMemoryError: Unable to allocate 17.3 GiB for an array with shape (1, 68355, 33904) and data type float64

venv/lib/python3.11/site-packages/xarray/core/variable.py:2389: MemoryError

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/test_benchmarks.py:146 and trace the comparison path through openeo/testing/results.py, especially _compare_job_results and _compare_xarray_dataarray. Review the linked Cubed xarray example, then define and verify a comparison approach that avoids out-of-memory allocations for the large array case shown; the issue does not specify an exact implementation or acceptance test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.