labthings / labthings/labthings-fastapi

Connection actively refused roughly 1 in 15 requests

Open
#69 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Suggest closing
Dominant language
Python
Stars
9
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Using v3 of the server running on an OFM, roughly 1 in 15 requests gives the error

[WinError 10061] No connection could be made because the target machine actively refused it

Full log below

{
	"name": "ConnectError",
	"message": "[WinError 10061] No connection could be made because the target machine actively refused it",
	"stack": "---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_exceptions.py:10, in map_exceptions(map)
      9 try:
---> 10     yield
     11 except Exception as exc:  # noqa: PIE786

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_backends\\sync.py:206, in SyncBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
    205 with map_exceptions(exc_map):
--> 206     sock = socket.create_connection(
    207         address,
    208         timeout,
    209         source_address=source_address,
    210     )
    211     for option in socket_options:

File ~\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\socket.py:851, in create_connection(address, timeout, source_address, all_errors)
    850 if not all_errors:
--> 851     raise exceptions[0]
    852 raise ExceptionGroup(\"create_connection failed\", exceptions)

File ~\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\socket.py:836, in create_connection(address, timeout, source_address, all_errors)
    835     sock.bind(source_address)
--> 836 sock.connect(sa)
    837 # Break explicitly a reference cycle

ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_transports\\default.py:66, in map_httpcore_exceptions()
     65 try:
---> 66     yield
     67 except Exception as exc:

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_transports\\default.py:228, in HTTPTransport.handle_request(self, request)
    227 with map_httpcore_exceptions():
--> 228     resp = self._pool.handle_request(req)
    230 assert isinstance(resp.stream, typing.Iterable)

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_sync\\connection_pool.py:268, in ConnectionPool.handle_request(self, request)
    267         self.response_closed(status)
--> 268     raise exc
    269 else:

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_sync\\connection_pool.py:251, in ConnectionPool.handle_request(self, request)
    250 try:
--> 251     response = connection.handle_request(request)
    252 except ConnectionNotAvailable:
    253     # The ConnectionNotAvailable exception is a special case, that
    254     # indicates we need to retry the request on a new connection.
   (...)
    258     # might end up as an HTTP/2 connection, but which actually ends
    259     # up as HTTP/1.1.

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_sync\\connection.py:99, in HTTPConnection.handle_request(self, request)
     98         self._connect_failed = True
---> 99         raise exc
    100 elif not self._connection.is_available():

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_sync\\connection.py:76, in HTTPConnection.handle_request(self, request)
     75 try:
---> 76     stream = self._connect(request)
     78     ssl_object = stream.get_extra_info(\"ssl_object\")

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_sync\\connection.py:124, in HTTPConnection._connect(self, request)
    123 with Trace(\"connect_tcp\", logger, request, kwargs) as trace:
--> 124     stream = self._network_backend.connect_tcp(**kwargs)
    125     trace.return_value = stream

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_backends\\sync.py:205, in SyncBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
    200 exc_map: ExceptionMapping = {
    201     socket.timeout: ConnectTimeout,
    202     OSError: ConnectError,
    203 }
--> 205 with map_exceptions(exc_map):
    206     sock = socket.create_connection(
    207         address,
    208         timeout,
    209         source_address=source_address,
    210     )

File ~\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    157 try:
--> 158     self.gen.throw(typ, value, traceback)
    159 except StopIteration as exc:
    160     # Suppress StopIteration *unless* it's the same exception that
    161     # was passed to throw().  This prevents a StopIteration
    162     # raised inside the \"with\" statement from being suppressed.

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpcore\\_exceptions.py:14, in map_exceptions(map)
     13     if isinstance(exc, from_exc):
---> 14         raise to_exc(exc) from exc
     15 raise

ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
Cell In[43], line 60
     58     tot_mag_xpos = tot_mag_xpos + (np.sqrt((delta[1])**2+(delta[0])**2)*pixel_um) #converts to um
     59     totMag_eachStep_xpos.append(tot_mag_xpos)
---> 60     stage_coord_xpos.append(stage.position.values())
     62 stage.move_absolute(x = starting_pos[0], y = starting_pos[1], z = starting_pos[2])
     63 pos = starting_pos.copy()

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\labthings_fastapi\\client\\__init__.py:165, in property_descriptor.<locals>.__get__(self, obj, _objtype)
    163 if obj is None:
    164     return self
--> 165 return obj.get_property(self.name)

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\labthings_fastapi\\client\\__init__.py:66, in ThingClient.get_property(self, path)
     65 def get_property(self, path: str) -> Any:
---> 66     r = self.client.get(urljoin(self.path, path))
     67     r.raise_for_status()
     68     return r.json()

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:1041, in Client.get(self, url, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1024 def get(
   1025     self,
   1026     url: URLTypes,
   (...)
   1034     extensions: typing.Optional[RequestExtensions] = None,
   1035 ) -> Response:
   1036     \"\"\"
   1037     Send a `GET` request.
   1038 
   1039     **Parameters**: See `httpx.request`.
   1040     \"\"\"
-> 1041     return self.request(
   1042         \"GET\",
   1043         url,
   1044         params=params,
   1045         headers=headers,
   1046         cookies=cookies,
   1047         auth=auth,
   1048         follow_redirects=follow_redirects,
   1049         timeout=timeout,
   1050         extensions=extensions,
   1051     )

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:814, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
    799     warnings.warn(message, DeprecationWarning)
    801 request = self.build_request(
    802     method=method,
    803     url=url,
   (...)
    812     extensions=extensions,
    813 )
--> 814 return self.send(request, auth=auth, follow_redirects=follow_redirects)

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:901, in Client.send(self, request, stream, auth, follow_redirects)
    893 follow_redirects = (
    894     self.follow_redirects
    895     if isinstance(follow_redirects, UseClientDefault)
    896     else follow_redirects
    897 )
    899 auth = self._build_request_auth(request, auth)
--> 901 response = self._send_handling_auth(
    902     request,
    903     auth=auth,
    904     follow_redirects=follow_redirects,
    905     history=[],
    906 )
    907 try:
    908     if not stream:

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:929, in Client._send_handling_auth(self, request, auth, follow_redirects, history)
    926 request = next(auth_flow)
    928 while True:
--> 929     response = self._send_handling_redirects(
    930         request,
    931         follow_redirects=follow_redirects,
    932         history=history,
    933     )
    934     try:
    935         try:

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:966, in Client._send_handling_redirects(self, request, follow_redirects, history)
    963 for hook in self._event_hooks[\"request\"]:
    964     hook(request)
--> 966 response = self._send_single_request(request)
    967 try:
    968     for hook in self._event_hooks[\"response\"]:

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_client.py:1002, in Client._send_single_request(self, request)
    997     raise RuntimeError(
    998         \"Attempted to send an async request with a sync Client instance.\"
    999     )
   1001 with request_context(request=request):
-> 1002     response = transport.handle_request(request)
   1004 assert isinstance(response.stream, SyncByteStream)
   1006 response.request = request

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_transports\\default.py:227, in HTTPTransport.handle_request(self, request)
    213 assert isinstance(request.stream, SyncByteStream)
    215 req = httpcore.Request(
    216     method=request.method,
    217     url=httpcore.URL(
   (...)
    225     extensions=request.extensions,
    226 )
--> 227 with map_httpcore_exceptions():
    228     resp = self._pool.handle_request(req)
    230 assert isinstance(resp.stream, typing.Iterable)

File ~\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    156     value = typ()
    157 try:
--> 158     self.gen.throw(typ, value, traceback)
    159 except StopIteration as exc:
    160     # Suppress StopIteration *unless* it's the same exception that
    161     # was passed to throw().  This prevents a StopIteration
    162     # raised inside the \"with\" statement from being suppressed.
    163     return exc is not value

File c:\\Users\\Administrator\\Source\\height_map\\venv\\Lib\\site-packages\\httpx\\_transports\\default.py:83, in map_httpcore_exceptions()
     80     raise
     82 message = str(exc)
---> 83 raise mapped_exc(message) from exc

ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it"
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the intermittent failure against a v3 server running on an OFM, using the reported stage.position.values() call as the trigger. Trace the server and client request path associated with labthings_fastapi.client.ThingClient.get_property; done means identifying why connections are refused roughly 1 in 15 times and confirming a reliable fix with repeated requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.