open-telemetry / open-telemetry/opentelemetry-python

Weaver tests are flaky on PyPy

Open
#5,176 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

They are skipped for now but would be nice to have them work reliably if possible. I guess we also have slow installing times due to missing wheels since failing runs took more than 10 minutes.

-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=53135): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 63%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor

FAILED                                                                   [ 72%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_no_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=54733): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 81%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_with_violations 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=39173): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [ 90%]
tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_report_span_statistics 
-------------------------------- live log call ---------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=34619): Failed to establish a new connection: [Errno 111] Connection refused")': /health
PASSED                                                                   [100%]

=================================== FAILURES ===================================
_________ TestSDKInitLiveCheck.test_end_and_check_raises_on_violations _________

self = <HTTPConnection(host='localhost', port=38821) at 0x1c237280>

    def _new_conn(self) -> socket.socket:
        """Establish a socket connection and set nodelay settings on it.
    
        :return: New socket connection.
        """
        try:
>           sock = connection.create_connection(
                (self._dns_host, self.port),
                self.timeout,
                source_address=self.source_address,
                socket_options=self.socket_options,
            )

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/connection.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/urllib3/util/connection.py:85: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('localhost', 38821), timeout = 5, source_address = None
socket_options = [(6, 1, 1)]

.tox/pypy3-test-opentelemetry-test-utils/lib/pypy3.10/site-packages/requests/adapters.py:678: ConnectionError

The above exception was the direct cause of the following exception:

self = <tests.test_weaver_live_check.TestSDKInitLiveCheck testMethod=test_end_and_check_raises_on_violations>

    def test_end_and_check_raises_on_violations(self):
        """end_and_check() raises LiveCheckError with the report attached."""
>       with WeaverLiveCheck(
            registry=_REGISTRY_DIR, policies_dir=_TESTDATA_DIR
        ) as weaver:

tests/opentelemetry-test-utils/tests/test_weaver_live_check.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:299: in __enter__
    return self.start()
tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:314: in start
    self._wait_for_ready()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <opentelemetry.test.weaver_live_check.WeaverLiveCheck object at 0x000000001cb5dc90>

    def _wait_for_ready(self) -> None:
        retry = Retry(
            total=10,
            backoff_factor=1,
            backoff_max=1,
            # Any non-2xx response from /health means weaver isn't ready yet.
            status_forcelist=list(range(300, 600)),
            raise_on_status=True,
            allowed_methods=["GET"],
        )
        session = Session()
        session.mount("http://", HTTPAdapter(max_retries=retry))
        try:
            session.get(
                f"http://localhost:{self._admin_port}/health", timeout=5
            )
        except Exception as exc:  # pylint: disable=broad-except
            if self._process is not None and self._process.poll() is not None:
>               raise RuntimeError(
                    f"WeaverLiveCheck process exited unexpectedly (code {self._process.returncode})"
                ) from exc
E               RuntimeError: WeaverLiveCheck process exited unexpectedly (code -6)

tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py:342: RuntimeError
------------------------------ Captured log call -------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
WARNING  urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError("HTTPConnection(host='localhost', port=38821): Failed to establish a new connection: [Errno 111] Connection refused")': /health
ERROR    opentelemetry.test.weaver_live_check:weaver_live_check.py:318 WeaverLiveCheck did not start: WeaverLiveCheck process exited unexpectedly (code -6), logs: 
E0424 15:12:55.913446    7686 ev_epoll1_linux.cc:371] (event_engine) Epoll1Poller:0x1d0db120 encountered epoll_wait error: Bad file descriptor
=========================== short test summary info ============================
FAILED tests/opentelemetry-test-utils/tests/test_weaver_live_check.py::TestSDKInitLiveCheck::test_end_and_check_raises_on_violations
======================== 1 failed, 10 passed in 16.23s =========================
pypy3-test-opentelemetry-test-utils: exit 1 (17.22 seconds) /home/runner/work/opentelemetry-python/opentelemetry-python> pytest /home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/tests -ra pid=7639
  pypy3-test-opentelemetry-test-utils: FAIL code 1 (779.65=setup[762.43]+cmd[17.22] seconds)
  evaluation failed :( (781.37 seconds)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/opentelemetry-test-utils/tests/test_weaver_live_check.py and tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py, then reproduce the skipped Weaver tests under PyPy. Trace the process startup and /health readiness path around the reported exit code -6 and epoll error. Done means the Weaver tests run reliably on PyPy without the current skip.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.