aboutcode-org / aboutcode-org/purldb

purl2vcs: out of memory error

Offen
#489 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @TG1999 Auf GitHub ansehen
Vorherrschende Sprache
HTML
Sterne
67
Forks
69
Ø Merge
8 T. 8 Std.
Gemergte PRs (30 T.)
1

Beschreibung

The purldb webserver experienced an error that killed the gunicorn worker handling this request:

```
web-1 | [2024-07-02 21:31:42 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:10)
web-1 | [2024-07-02 21:31:42 +0000] [10] [ERROR] Error handling request /api/collect/index_packages/
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 135, in handle
web-1 | self.handle_request(listener, req, client, addr)
web-1 | File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/sync.py", line 178, in handle_request
web-1 | respiter = self.wsgi(environ, resp.start_response)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
web-1 | response = self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 140, in get_response
web-1 | response = self._middleware_chain(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
web-1 | response = wrapped_callback(request, *callback_args, **callback_kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
web-1 | return func(*args, **kwds)
web-1 | ^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
web-1 | return view_func(request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", line 124, in view
web-1 | return self.dispatch(request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
web-1 | response = handler(request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/app/packagedb/api.py", line 973, in index_packages
web-1 | get_source_package_and_add_to_package_set(package)
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 141, in get_source_package_and_add_to_package_set
web-1 | source_purl = get_source_repo(package=package)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 198, in get_source_repo
web-1 | repo_urls = list(get_repo_urls(package))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 225, in get_repo_urls
web-1 | source_urls = get_source_urls_from_package_data_and_resources(
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 244, in get_source_urls_from_package_data_and_resources
web-1 | metadata_urls = list(get_urls_from_package_data(package))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 345, in get_urls_from_package_data
web-1 | found_urls.extend(get_urls_from_text(text=homepage_text))
web-1 | File "/usr/local/lib/python3.11/site-packages/purl2vcs/find_source_repo.py", line 36, in get_urls_from_text
web-1 | for url in get_urls_from_location(location=lines)["urls"]:
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/scancode/api.py", line 134, in get_urls
web-1 | for urls, line_num in found_urls:
web-1 | File "/usr/local/lib/python3.11/site-packages/scancode/api.py", line 130, in
web-1 | found_urls = ((u, ln) for (u, ln) in find_urls(location) if u)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 257, in find_urls
web-1 | for _key, url, _line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 78, in unique_filter
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 576, in junk_urls_filter
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 553, in junk_url_hosts_filter
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 425, in canonical_url_cleaner
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 108, in re_filt
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 360, in user_pass_cleaning_filter
web-1 | for key, match, line, line_number in matches:
web-1 | File "/usr/local/lib/python3.11/site-packages/cluecode/finder.py", line 336, in scheme_adder
web-1 | yield key, match, line, line_number
web-1 | File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
web-1 | sys.exit(1)
web-1 | SystemExit: 1
web-1 | [2024-07-02 21:31:42 +0000] [10] [INFO] Worker exiting (pid: 10)
web-1 | [2024-07-02 21:31:43 +0000] [9] [ERROR] Worker (pid:10) was sent SIGKILL! Perhaps out of memory?
```

My initial guess is that there may be a regex explosion happening when parsing urls

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.