bug: Internal Server Error while uploading custom ArduPilot Firmware

### Bug description

We are trying to upload some custom ArduPilot firmware to a CUAV-7-Nano autopilot using the firmware upload page built into BlueOS, however every time we try to do the firmware upload it fails with an *Internal Server Error*.

### Steps to reproduce

To reproduce the issue, go through the normal firmware upload workflow.

1. Go to the *Autopilot Firmware* tab
2. Expand the *Firmware Update* widget
3. Click *Upload firmware from file*
4. Select your firmware file (see attached)
5. Click the *Install Firmware* button
6. Wait a couple of seconds and you'll get an *Internal Server Error* popup

https://github.com/user-attachments/assets/c9a5e66e-8d75-4901-9c44-6e26c00e0e1d

To help with reproducing, here is my exact firmware binary:

[ardusub-firmware.zip](https://github.com/user-attachments/files/19185078/ardusub-firmware.zip)

The only difference between that firmware and the main ardupilot repo is that we've added a custom frame file. The exact commit is https://github.com/sunfish-robotics/ardupilot/commit/19ae91d244951bc5bd83e067bb4170161c87ab18, and once you've got the dev environment set up, you can build it using `make sunfish`.

### Primary pain point(s)

We are encountering this on an autonomous underwater vehicle where the only way to upload new ardupilot firmware is via BlueOS. If we can't use the autopilot firmware manager for updating firmware, it means we'd need physical access to the device... which isn't possible when it's in a sealed container at the bottom of the ocean.

### Additional context

I managed to look through the logs, and it seems like the exception is triggered from here:

https://github.com/bluerobotics/BlueOS/blob/028ce715e7552a1cd22664b6f5863dbd5fe7a067/core/services/ardupilot_manager/firmware/FirmwareManagement.py#L107

(click to see the full stack trace)

```
2025-03-11 13:31:26.164 | INFO | logging:callHandlers:1706 - 127.0.0.1:46056 - "POST /v1.0/install_firmware_from_file?board_name=CUAV-7-Nano HTTP/1.0" 500
2025-03-11 13:31:26.165 | ERROR | logging:callHandlers:1706 - Exception in ASGI application

Traceback (most recent call last):

File "/home/pi/services/ardupilot_manager/main.py", line 46, in
loop.run_until_complete(server.serve())
│ │ │ └
│ │ └
│ └
└ <_UnixSelectorEventLoop running=True closed=False debug=False>

File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
self.run_forever()
│ └
└ <_UnixSelectorEventLoop running=True closed=False debug=False>
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
self._run_once()
│ └
└ <_UnixSelectorEventLoop running=True closed=False debug=False>
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
handle._run()
│ └
└ )>
File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
│ │ │ │ │ └
│ │ │ │ └ )>
│ │ │ └
│ │ └ )>
│ └
└ )>
> File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
result = await app(self.scope, self.receive, self.send)
│ │ │ │ │ │ └
│ │ │ │ │ └
│ │ │ │ └
│ │ │ └
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
return await self.app(scope, receive, send)
│ │ │ │ └ >
│ │ │ └ >
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 65, in __call__
raise exc from None
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 61, in __call__
await self.app(scope, inner_receive, inner_send)
│ │ │ │ └ .inner_send at 0xf32cc5c8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
await super().__call__(scope, receive, send)
│ │ └ .inner_send at 0xf32cc5c8>
│ └ .inner_receive at 0xf3349488>
└ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
│ │ │ │ └ .inner_send at 0xf32cc5c8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
│ │ │ │ └ ._send at 0xf32cc758>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
│ │ │ │ └ .sender at 0xf32cc848>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc848>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc848>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└ Mount(path='/v1.0', name='', app=)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 443, in handle
await self.app(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc848>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└ Mount(path='/v1.0', name='', app=)
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
await super().__call__(scope, receive, send)
│ │ └ .sender at 0xf32cc848>
│ └ .inner_receive at 0xf3349488>
└ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc848>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
│ │ │ │ └ ._send at 0xf32ccca8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
│ │ │ │ └ .sender at 0xf32cc7a8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc7a8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc7a8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └
└ VersionedAPIRoute(path='/install_firmware_from_file', name='install_firmware_from_file', methods=['POST'])
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
│ │ │ │ └ .sender at 0xf32cc7a8>
│ │ │ └ .inner_receive at 0xf3349488>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
│ └ .app at 0xf6235898>
└ VersionedAPIRoute(path='/install_firmware_from_file', name='install_firmware_from_file', methods=['POST'])
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
│ └
└ .app at 0xf6235848>
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
raw_response = await run_endpoint_function(
└
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
│ │ └ {'board_name': 'CUAV-7-Nano', 'binary': , 'parameters': None}
│ └
└

File "/home/pi/libs/commonwealth/commonwealth/utils/decorators.py", line 63, in wrapper
return await function(*args, **kwargs)
│ │ └ {'board_name': 'CUAV-7-Nano', 'binary': , 'parameters': None}
│ └ ()
└

File "/home/pi/services/ardupilot_manager/api/v1/routers/index.py", line 161, in install_firmware_from_file
autopilot.install_firmware_from_file(custom_firmware, await target_board(board_name), parameters)
│ │ │ │ │ └ None
│ │ │ │ └ 'CUAV-7-Nano'
│ │ │ └
│ │ └ PosixPath('/root/.config/ardupilot-manager/firmware/custom_firmware')
│ └
└

File "/home/pi/services/ardupilot_manager/autopilot_manager.py", line 652, in install_firmware_from_file
self.firmware_manager.install_firmware_from_file(firmware_path, board, default_parameters)
│ │ │ │ │ └ None
│ │ │ │ └ FlightController(name='CUAV-7-Nano', manufacturer='ArduPilot', platform=, path='/dev...
│ │ │ └ PosixPath('/root/.config/ardupilot-manager/firmware/custom_firmware')
│ │ └
│ └
└

File "/home/pi/services/ardupilot_manager/firmware/FirmwareManagement.py", line 112, in install_firmware_from_file
raise FirmwareInstallFail("Could not install firmware.") from error
└

exceptions.FirmwareInstallFail: Could not install firmware.
```

Unfortunately, the code does `raise FirmwareInstallFail("Could not install firmware.") from error` and the logs aren't printing the original exception, so I wasn't able to troubleshoot any further.

### Prerequisites

- [x] I have checked to make sure that a similar request has not already been filed or fixed.

Start in core/services/ardupilot_manager/firmware/FirmwareManagement.py at line 107 and trace the POST /v1.0/install_firmware_from_file endpoint. Reproduce the failure with the attached firmware, using the provided sunfish ArduPilot commit and make sunfish build instructions as needed. Done means the custom firmware upload no longer returns an Internal Server Error and can proceed through the firmware update workflow.

Written by the indexing model from the issue text.