Fire Behaviour Calculator: Response json ending prematurely
- Dominant language
- Python
- Stars
- 65
- Forks
- 11
- Avg merge
- 21h 25m
- Merged PRs (30d)
- 70
Description
**Describe the Bug**
The response json for endpoint /api/fba-calc/stations sometimes fails to decode. It seems as if the json is incomplete (see how json snippet below ends weirdly:
```“displayLabel” : “Weather Station - GOES”,
“displayOrder” : 5,
“effectiveDate” : “2000-01-01T08:00:00.000+0000”,
“expiryDate” : “9999-12-31T08:00:00.000+0000”,
“createdBy” : “DATA_LOAD”,
“createdDate” : “2020-02-25T08:00:00.000+0000”,
“lastModifiedBy” : “DATA_LOAD”,
“lastModifiedDate” : “2020-02-25T08:00:00.000+0000”
},
“weatherZone” : {
“id” : 12,
“displayL
```
Example stack trace:
```
Exception occurred POST /api/fba-calc/stations
wps-prod-458-qcv4f (45): ‘utf-8’ codec can’t decode byte 0xff in position 99695: invalid start byte
[‘Traceback (most recent call last):\n’, ‘ File “/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py”, line 81, in receive\n return self.receive_nowait()\n’, ‘ File “/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py”, line 76, in receive_nowait\n raise WouldBlock\n’, ‘anyio.WouldBlock\n’, ‘\nDuring handling of the above exception, another exception occurred:\n\n’, ‘Traceback (most recent call last):\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py”, line 41, in call_next\n message = await recv_stream.receive()\n’, ‘ File “/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py”, line 101, in receive\n raise EndOfStream\n’, ‘anyio.EndOfStream\n’, ‘\nDuring handling of the above exception, another exception occurred:\n\n’, ‘Traceback (most recent call last):\n’, ‘ File “/app/app/main.py”, line 89, in catch_exception_middleware\n return await call_next(request)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py”, line 44, in call_next\n raise app_exc\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py”, line 34, in coro\n await self.app(scope, request.receive, send_stream.send)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 92, in call\n await self.simple_response(scope, receive, send, request_headers=headers)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 147, in simple_response\n await self.app(scope, receive, send)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/exceptions.py”, line 82, in call\n raise exc\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/exceptions.py”, line 71, in call\n await self.app(scope, receive, sender)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py”, line 21, in call\n raise e\n’, ‘ File “/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py”, line 18, in call\n await self.app(scope, receive, send)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 656, in call\n await route.handle(scope, receive, send)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 259, in handle\n await self.app(scope, receive, send)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 61, in app\n response = await func(request)\n’, ‘ File “/usr/local/lib/python3.9/site-packages/fastapi/routing.py”, line 227, in app\n raw_response = await run_endpoint_function(\n’, ‘ File “/usr/local/lib/python3.9/site-packages/fastapi/routing.py”, line 160, in run_endpoint_function\n return await dependant.call(**values)\n’, ‘ File “/app/app/routers/fba_calc.py”, line 211, in get_stations_data\n dailies_by_station_id = {raw_daily.get('stationId'): raw_daily async for raw_daily in dailies}\n’, ‘ File “/app/app/routers/fba_calc.py”, line 211, in \n dailies_by_station_id = {raw_daily.get('stationId'): raw_daily async for raw_daily in dailies}\n’, ‘ File “/app/app/wildfire_one/wildfire_fetchers.py”, line 68, in fetch_paged_response_generator\n response_json = await _fetch_cached_response(session, headers, url, params, cache_expiry_seconds)\n’, ‘ File “/app/app/wildfire_one/wildfire_fetchers.py”, line 39, in _fetch_cached_response\n response_json = await response.json()\n’, ‘ File “/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1119, in json\n return loads(stripped.decode(encoding))\n’, “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xff in position 99695: invalid start byte\n”]
```
** Steps To Reproduce**
It's difficult to reproduce, it does not consistently happen. I suspect it may be a thread blocking issue. (long actions may be blocking the response from being read correctly)
**Additional context**
- If it's due to long running threads - I suspect it's due to known issue with R. (see #1926)
Contributor guide
Research direction
Start with app/routers/fba_calc.py at get_stations_data and trace the paged data through app/wildfire_one/wildfire_fetchers.py, especially lines 39 and 68 where the cached response is read and decoded. Exercise POST /api/fba-calc/stations and compare failures with the R-related context in issue #1926; done means the endpoint consistently returns complete, decodable JSON.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100