Open-EO / Open-EO/openeo-python-driver

follow up performance problems of large EUGW process grap

Open
#429 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

architecture enhancement technical debt
Dominant language
Python
Stars
12
Forks
8
Avg merge
6d 3h
Merged PRs (30d)
4

Description

follow up performance problems/opportunities of the large process graph use case from #426/#427

this is for example profiling dump of running test test_very_large_graph
https://github.com/Open-EO/openeo-python-driver/blob/35a0b9f2f06b98ec614dac34115fb7a5c790fd73/tests/test_dry_run.py#L2955-L2957

(stripped the entries of pytest and other generic runners)


         416589268 function calls (307657453 primitive calls) in 153.792 seconds

   Ordered by: cumulative time
   List reduced from 9588 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  155.456  155.456 __main__.py:1(<module>)
...
        1    0.000    0.000  153.603  153.603 test_dry_run.py:2955(test_very_large_graph)
        1    0.001    0.001  153.602  153.602 ProcessGraphDeserializer.py:437(evaluate)
7422036/2    6.592    0.000  126.161   63.080 ProcessGraphDeserializer.py:544(convert_node)
1248656/2    6.244    0.000  126.161   63.080 ProcessGraphDeserializer.py:1916(apply_process)
1179216/2    1.779    0.000  126.160   63.080 ProcessGraphDeserializer.py:1943(<dictcomp>)
 730618/2    0.816    0.000  126.160   63.080 ProcessGraphDeserializer.py:603(<listcomp>)
   133952    0.753    0.000   27.443    0.000 ProcessGraphDeserializer.py:839(load_collection)
        1    0.547    0.547   26.902   26.902 dry_run.py:395(get_source_constraints)
  1395978    1.508    0.000   21.798    0.000 dry_run.py:550(_process)
  1395978    0.671    0.000   18.940    0.000 dry_run.py:273(process_traces)
  1395978    5.031    0.000   18.269    0.000 dry_run.py:275(<listcomp>)
   133952    0.945    0.000   17.803    0.000 dry_run.py:277(load_collection)
  4994625    2.735    0.000   17.146    0.000 version.py:45(__init__)
  1248656    0.715    0.000   16.699    0.000 dummy_backend.py:702(get_process_registry)
  1248656    0.944    0.000   15.984    0.000 ProcessGraphDeserializer.py:331(get_process_registry)
  1248656    0.499    0.000   14.607    0.000 version.py:115(at_least)
  1248656    0.672    0.000   14.108    0.000 version.py:95(__ge__)
  2497313    9.385    0.000   13.771    0.000 version.py:55(_parse)
  1248656    3.019    0.000   13.436    0.000 version.py:80(_pad)
  8084052    5.671    0.000   11.379    0.000 dry_run.py:216(__init__)
  1248656    0.938    0.000    9.485    0.000 utils.py:106(openeo_api_version)
  1248656    0.391    0.000    9.426    0.000 utils.py:80(collect_parameters)
4994624/1248656    3.321    0.000    9.035    0.000 utils.py:64(collect)
44604776/1406496    8.675    0.000    8.811    0.000 dry_run.py:226(get_arguments_by_operation)
  6516114    1.921    0.000    8.342    0.000 utils.py:53(get)
        1    0.000    0.000    8.047    8.047 dry_run.py:369(get_trace_leaves)
        1    0.684    0.684    8.047    8.047 dry_run.py:386(<listcomp>)
12260030/4175978    4.601    0.000    7.363    0.000 dry_run.py:376(get_leaves)

note the extremely high call counts of convert_node and related.

1248656 calls for get_process_registry(and related version parsing) also seems excessive

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/test_dry_run.py::test_very_large_graph and its profiling output. Inspect ProcessGraphDeserializer.py, especially convert_node, apply_process, and get_process_registry, along with the related paths in dry_run.py and version.py. Compare call counts and runtime while investigating the large process graph case; done means the reported performance problems or opportunities have been addressed and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.