bug: Internal Server Error while uploading custom ArduPilot Firmware

[Michael-F-Bryan]

Bug description

We are trying to upload some custom ArduPilot firmware to a CUAV-7-Nano autopilot using the firmware upload page built into BlueOS, however every time we try to do the firmware upload it fails with an Internal Server Error.

Steps to reproduce

To reproduce the issue, go through the normal firmware upload workflow.

1. Go to the Autopilot Firmware tab
2. Expand the Firmware Update widget
3. Click Upload firmware from file
4. Select your firmware file (see attached)
5. Click the Install Firmware button
6. Wait a couple of seconds and you'll get an Internal Server Error popup

2025-03-11_22-05-04.mp4

To help with reproducing, here is my exact firmware binary:

ardusub-firmware.zip

The only difference between that firmware and the main ardupilot repo is that we've added a custom frame file. The exact commit is sunfish-robotics/ardupilot@19ae91d, and once you've got the dev environment set up, you can build it using make sunfish.

Primary pain point(s)

We are encountering this on an autonomous underwater vehicle where the only way to upload new ardupilot firmware is via BlueOS. If we can't use the autopilot firmware manager for updating firmware, it means we'd need physical access to the device... which isn't possible when it's in a sealed container at the bottom of the ocean.

Additional context

I managed to look through the logs, and it seems like the exception is triggered from here:

BlueOS/core/services/ardupilot_manager/firmware/FirmwareManagement.py

Line 107 in 028ce71

self.firmware_installer.install_firmware(new_firmware_path, board)

(click to see the full stack trace)

2025-03-11 13:31:26.164 | INFO     | logging:callHandlers:1706 - 127.0.0.1:46056 - "POST /v1.0/install_firmware_from_file?board_name=CUAV-7-Nano HTTP/1.0" 500
2025-03-11 13:31:26.165 | ERROR    | logging:callHandlers:1706 - Exception in ASGI application

Traceback (most recent call last):

  File "/home/pi/services/ardupilot_manager/main.py", line 46, in <module>
    loop.run_until_complete(server.serve())
    │    │                  │      └ <function Server.serve at 0xf6ddb708>
    │    │                  └ <uvicorn.server.Server object at 0xf624c470>
    │    └ <function BaseEventLoop.run_until_complete at 0xf71c5078>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>

  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
    │    └ <function BaseEventLoop.run_forever at 0xf71c5028>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
    │    └ <function BaseEventLoop._run_once at 0xf71c5f28>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
    handle._run()
    │      └ <function Handle._run at 0xf7264848>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
    │    │            │    │           │    └ <member '_args' of 'Handle' objects>
    │    │            │    │           └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    │            │    └ <member '_callback' of 'Handle' objects>
    │    │            └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    └ <member '_context' of 'Handle' objects>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>
> File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
                   │   │    │      │    │        │    └ <function RequestResponseCycle.send at 0xf61d2bb8>
                   │   │    │      │    │        └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xf3399110>
                   │   │    │      │    └ <function RequestResponseCycle.receive at 0xf61d2c08>
                   │   │    │      └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xf3399110>
                   │   │    └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
                   │   └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xf3399110>
                   └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0xf626d370>
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
                 │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xf3399110>>
                 │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0xf3399110>>
                 │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
                 │    └ <uvicorn.middleware.message_logger.MessageLoggerMiddleware object at 0xf626d050>
                 └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0xf626d370>
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 65, in __call__
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 61, in __call__
    await self.app(scope, inner_receive, inner_send)
          │    │   │      │              └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xf32cc5c8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.applications.FastAPI object at 0xf623feb0>
          └ <uvicorn.middleware.message_logger.MessageLoggerMiddleware object at 0xf626d050>
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xf32cc5c8>
                           │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <function MessageLoggerMiddleware.__call__.<locals>.inner_send at 0xf32cc5c8>
          │    │                │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xf625fcd0>
          └ <fastapi.applications.FastAPI object at 0xf623feb0>
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0xf32cc758>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0xf625fd30>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xf625fcd0>
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0xf625fb30>
          └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0xf625fd30>
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0xf6243130>
          └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0xf625fb30>
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
          │     │      │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │     └ <function Mount.handle at 0xf632bb18>
          └ Mount(path='/v1.0', name='', app=<fastapi.applications.FastAPI object at 0xf6243390>)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.applications.FastAPI object at 0xf6243390>
          └ Mount(path='/v1.0', name='', app=<fastapi.applications.FastAPI object at 0xf6243390>)
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
                           │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc848>
          │    │                │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xf61ea9f0>
          └ <fastapi.applications.FastAPI object at 0xf6243390>
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0xf32ccca8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0xf61eaa90>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0xf61ea9f0>
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc7a8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0xf61eaab0>
          └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0xf61eaa90>
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc7a8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <fastapi.routing.APIRouter object at 0xf6243470>
          └ <fastapi.middleware.asyncexitstack.AsyncExitStackMiddleware object at 0xf61eaab0>
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc7a8>
          │     │      │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │     └ <function Route.handle at 0xf632b708>
          └ VersionedAPIRoute(path='/install_firmware_from_file', name='install_firmware_from_file', methods=['POST'])
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0xf32cc7a8>
          │    │   │      └ <function MessageLoggerMiddleware.__call__.<locals>.inner_receive at 0xf3349488>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.0', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <function request_response.<locals>.app at 0xf6235898>
          └ VersionedAPIRoute(path='/install_firmware_from_file', name='install_firmware_from_file', methods=['POST'])
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
                     │    └ <starlette.requests.Request object at 0xf33999d0>
                     └ <function get_request_handler.<locals>.app at 0xf6235848>
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                         └ <function run_endpoint_function at 0xf632bf28>
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
                 │         │      └ {'board_name': 'CUAV-7-Nano', 'binary': <starlette.datastructures.UploadFile object at 0xe9f74eb0>, 'parameters': None}
                 │         └ <function install_firmware_from_file at 0xf629c1b8>
                 └ <fastapi.dependencies.models.Dependant object at 0xf6231b70>

  File "/home/pi/libs/commonwealth/commonwealth/utils/decorators.py", line 63, in wrapper
    return await function(*args, **kwargs)
                 │         │       └ {'board_name': 'CUAV-7-Nano', 'binary': <starlette.datastructures.UploadFile object at 0xe9f74eb0>, 'parameters': None}
                 │         └ ()
                 └ <function install_firmware_from_file at 0xf629c168>

  File "/home/pi/services/ardupilot_manager/api/v1/routers/index.py", line 161, in install_firmware_from_file
    autopilot.install_firmware_from_file(custom_firmware, await target_board(board_name), parameters)
    │         │                          │                      │            │            └ None
    │         │                          │                      │            └ 'CUAV-7-Nano'
    │         │                          │                      └ <function target_board at 0xf62e0c08>
    │         │                          └ PosixPath('/root/.config/ardupilot-manager/firmware/custom_firmware')
    │         └ <function AutoPilotManager.install_firmware_from_file at 0xf64da8e8>
    └ <autopilot_manager.AutoPilotManager object at 0xf652daf0>

  File "/home/pi/services/ardupilot_manager/autopilot_manager.py", line 652, in install_firmware_from_file
    self.firmware_manager.install_firmware_from_file(firmware_path, board, default_parameters)
    │    │                │                          │              │      └ None
    │    │                │                          │              └ FlightController(name='CUAV-7-Nano', manufacturer='ArduPilot', platform=<Platform.GenericSerial: 'GenericSerial'>, path='/dev...
    │    │                │                          └ PosixPath('/root/.config/ardupilot-manager/firmware/custom_firmware')
    │    │                └ <function FirmwareManager.install_firmware_from_file at 0xf660b438>
    │    └ <firmware.FirmwareManagement.FirmwareManager object at 0xf36fa610>
    └ <autopilot_manager.AutoPilotManager object at 0xf652daf0>

  File "/home/pi/services/ardupilot_manager/firmware/FirmwareManagement.py", line 112, in install_firmware_from_file
    raise FirmwareInstallFail("Could not install firmware.") from error
          └ <class 'exceptions.FirmwareInstallFail'>

exceptions.FirmwareInstallFail: Could not install firmware.

Unfortunately, the code does raise FirmwareInstallFail("Could not install firmware.") from error and the logs aren't printing the original exception, so I wasn't able to troubleshoot any further.

Prerequisites

• I have checked to make sure that a similar request has not already been filed or fixed.