Supplementaty Material for Replication and Future Research

Data collected and processed to build the AutoSimTest Framework

S-Agent Knowledge Base

sUAS incident data collected from multiple sources and used to construct the knowledge base of the S-agent can be accessed here: Go to AutoSIMTestFramework-->knowledge_base.csv

A-Agent Knowledge Base

Data collected and used to build the knowledge-base of Analytics Agent can be found [here](Go to Px4-Flight-Controller-Params

Data prepared to analyze the agents of the AutoSimTest Framework

Px4 Flight Logs Prepared to analyze Anlaytics-Agent

These flight logs were generated by injecting failures in the PX4 flight controller. The logs were used to analyze the performance of the Analytics-Agent. Please access the flight logs here:

• Go to FlightLogs-FailureInjection →

The code developed to simulate run-time sensor errors during Simulation can be found Go to SuT → px4 → standalone

Model-Based Design of Mission Inputs for 5 Different sUAS Use-Cases

The mission inputs were generated using the model-based design approach for 5 different sUAS use-cases. The mission inputs can be accessed Go to Sample-of-Expected-Output → Mission-Samples-For-Each-Use-Case. This data was used to analyze whether the Mission-Agent can generate correct mission inputs for different sUAS use-cases.

Implementation of Px4 and Ardupilot based SuT.

We designed our own System-Under-Tests. The implementation of these SuTs can be found Go to SuT. We tested our SuT using the AutoSimTest Framework.

Go to Analysis → Feasibility → Px4_SuT → Autonomous_Navigation → missions.json (Sample of Expected Mission Input by Px4)

Go to SuT → px4 → airsim-setting-sample.json (Sample of Airsim Settings to execute a Px4 Mission)

Go to Analysis → Feasibility → Ardu_SuT → Autonomous_Navigation → missions.json (Sample of Expected Mission Input by Ardupilot)

Go to Analysis → Feasibility → Ardu → Waypoint_Navigation → environment.json (Sample of Ardupilot-Sitl Simulator Input)

Instructions to Run the SuT

• Go to SuT → px4 → README.md
• Go to SuT → ardu → README.md

---

Data Generated by the AutoSimTest Framework

Scenarios Generated by S-Agent Across 5 sUAS Use-Cases

Go to Analysis → Generalizability → S-Agent-Scenario.xlsx

Mission and Environment Config Files Generated by the Mission and Environment Agents Across the 5 Use-Cases

• Go to Agent_output → mission_json (JSON Generated by M-Agent for 25 scenarios)
• Go to Agent_output → Environment_json (JSON Generated by Env-Agent for 25 scenarios)

Analytics Report Generated by the Analytics-Agent

Go to Agent_output → analytics_agent_output (Report Generated by Analytics Agent)

AutoSimTestFramework - Impementation and Set-up Details

AUTOSIMTEST Framework Codebase

The implementation of the AutoSimTest Framework including all Agents and communication between then is available here Go to AutoSimTestFramework for replication and further research and analysis.

AutoSimTestFramework Setup Guide

Follow these steps to set up and run the AUTOSIMTEST Framework Codebase.

---

Prerequisites

• Install Conda.
• Ensure you have Python installed and available in your system PATH.

---

Installation Steps

1. Clone the Repository

   git clone https://github.com/UAVLab-SLU/AutoSimTestFramework.git
   cd AutoSimTestFramework/AutoSimTestFramework

2. Create and Activate a Virtual Environment

   conda create -n autosim_env python=3.8
   conda activate autosim_env

3. Install Dependencies

   pip install -r requirements.txt

---

Running the Framework

1. Run the Flask Backend

   python app.py

2. Run the Gradio Frontend

   • Launch the Gradio-based frontend for communicating with the S-Agent, Env-Agent, and M-Agent

   python frontend2.py

   Run the Gradio Frontend

   • Launch the Gradio-based frontend for communicating with Analytics-Agent

   python Analytics_Agent_front_end.py