Merge pull request #3 from lp6m/dev

v1.0

Author: lp6m

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "yolov5"]
+	path = yolov5
+	url = https://github.com/ultralytics/yolov5

README.md

@@ -1,40 +1,90 @@
-# yolov5s_android  
-The implementation of yolov5s on android for the [yolov5s export contest](https://github.com/ultralytics/yolov5/discussions/3213)  
+# yolov5s_android:rocket: 
+<div align="center">
+<img src="https://github.com/lp6m/yolov5s_android/raw/media/android_app.gif" width=30%>
+<img src="https://github.com/lp6m/yolov5s_android/raw/media/screenshot.jpg" width=30%>
+</div>
 
+The implementation of yolov5s on android for the [yolov5s export contest](https://github.com/ultralytics/yolov5/discussions/3213).    
+Download the latest android apk from [release](https://github.com/lp6m/yolov5s_android/releases) and install your device.
 
-## Overview
+## Environment
+- Host Ubuntu18.04
+- Docker 
+    * Tensorflow 2.4.0
+    * PyTorch 1.7.0
+    * OpenVino 2021.3
+- Android App
+    * Android Studio 4.2.1
+    * minSdkVersion 28
+    * targetSdkVersion 29
+    * TfLite 2.4.0
+- Android Device
+    * Xiaomi Mi11 (Storage 128GB/ RAM8GB)
+    * OS MUI 12.5.8  
+  
+We use docker container for host evaluation and model conversion.
+```sh
+git clone --recursive https://github.com/lp6m/yolov5s_android
+cd yolov5s_android
+docker build ./ -f ./docker/Dockerfile  -t yolov5s_android
+docker run -it --gpus all -v `pwd`:/workspace yolov5s_anrdoid bash
+```
+
+## Files
+- `./app`
+    * Android application.
+    * To build application by yourself, copy `./tflite_model/*.tflite` to `app/tflite_yolov5_test/app/src/main/assets/`, and build on Android Studio.
+    * The app can perform inference with various configurations of input size, inference accuracy, and model accuracy. 
+    * For 'Open Directory Mode', save the detected bounding boxes results as a json file in coco format.
+    * Realtime deteciton from camera image (precision and input size is fixed to int8/320). Achieved FPS is about **15FPS** on Mi11.
+- `./benchmark` 
+    * Benchmark script and results by [TFLite Model Benchmark Tool with C++ Binary](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#profiling-model-operators).
+- `./convert_model`
+    * Model conversion guide and model quantization script.
+- `./docker`
+    * Dockerfile for the evaluation and model conversion environment.
+- `./host`
+    * `detect.py`  : Run detection for image with TfLite model on host environment.
+    * `evaluate.py`: Run evaluation with coco validation dataset and inference results.
+- `./tflite_model`
+    * Converted TfLite Model.
 
 ## Performance
 ### Latency (inference)
-These results are measured by [TFLite Model Benchmark Tool with C++ Binary](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#profiling-model-operators).  
-The latency does not contain the image pre-processing time and data transfer time.  
+These results are measured on `Xiaomi Mi11`.  
+Please refer [`benchmark/README.md`](https://github.com/lp6m/yolov5s_android/tree/master/benchmark) about the detail of benchmark command.  
+The latency does not contain the pre/post processing time and data transfer time.  
 #### float32 model  
-When using NNAPI with the float32 model, the results are the same as when `qti-gpu` is specified, even if accelerator-name is not specified.
 
-| delegatem option 　　　　 | latency [ms] |
-| :-----------------------  | :----------- |
-| None (CPU)                | xxx |
-| NNAPI (qti-default, fp32) | xxx |
-| NNAPI (qti-default, fp16) | xxx |
+|       delegate        | 640x640 [ms] | 320x320 [ms] |
+| :-------------------- | -----------: | -----------: |
+| None (CPU)            |          249 |           61 |
+| NNAPI (qti-gpu, fp32) |          156 |          112 |
+| NNAPI (qti-gpu, fp16) |           92 |           79 |
 
 #### int8 model
+We tried to accelerate the inference process by using `NNAPI (qti-dsp)` and offload calculation to Hexagon DSP, but it doesn't work for now. Please see [here](https://github.com/lp6m/yolov5s_android/tree/dev/benchmark#nnapi-qti-dsp-not-working) in detail.
 
-| delegate             | latency [ms] |
-| :---------           | :----------- |
-| None (CPU)           | xxx |
-| NNAPI  (qti-default) | xxx |
-| NNAPI  (qti-dsp)     | Not worked |
-
-### Latency (inference + postprocess)
+|       delegate       | 640x640 [ms] | 320x320 [ms] |
+| :------------------- | -----------: | -----------: |
+| None (CPU)           |           95 |           23 |
+| NNAPI  (qti-default) |  Not working |  Not working |
+| NNAPI  (qti-dsp)     |  Not working |  Not working |
 
-### Accuracy
-#### Evaluation on host
-
-#### Evaluation on android
+## Accuracy
+Please refer [host/README.md](https://github.com/lp6m/yolov5s_android/tree/master/host#example2) about the evaluation method.    
+We set `conf_thresh=0.25` and `iou_thresh=0.45` for nms parameter.
+|      device, model, delegate      | 640x640 mAP | 320x320 mAP |
+| :-------------------------------- | ----------: | ----------: |
+| host GPU (Tflite + PyTorch, fp32) |        27.8 |        26.6 |
+| host CPU (Tflite + PyTorch, int8) |        26.6 |        25.5 |
+| NNAPI  (qti-gpu, fp16)            |        28.5 |        26.8 |
+| CPU    (int8)                     |        27.2 |        25.8 |
 
 
 ## Model conversion
-This project focuses on obtaining a tflite model by model conversion from PyTorch original implementation, rather than doing its own implementation in tflite.  
-
+This project focuses on obtaining a tflite model by **model conversion from PyTorch original implementation, rather than doing own implementation in tflite**.  
+We convert models in this way: `PyTorch -> ONNX -> OpenVino -> TfLite`.  
+To convert the model from OpenVino to TfLite, we use [openvino2tensorflow](https://github.com/PINTO0309/openvino2tensorflow).
+Please refer [convert_model/README.md](https://github.com/lp6m/yolov5s_android/tree/master/convert_model) about the model conversion.
 
-## TODO

app/tflite_yolov5_test/.gitignore

@@ -0,0 +1,18 @@
+*.iml
+.gradle
+/local.properties
+/.idea/caches
+/.idea/libraries
+/.idea/modules.xml
+/.idea/workspace.xml
+/.idea/navEditor.xml
+/.idea/assetWizardSettings.xml
+.DS_Store
+/build
+/captures
+.externalNativeBuild
+.cxx
+local.properties
+
+app/src/main/assets/*.tflite
+app/release

app/tflite_yolov5_test/.idea/.gitignore

@@ -0,0 +1 @@
+/build

app/tflite_yolov5_test/.idea/compiler.xml

@@ -0,0 +1,34 @@
+cmake_minimum_required(VERSION 3.4.1)
+
+add_library( # Sets the name of the library.
+             native-lib
+
+             # Sets the library as a shared library.
+             SHARED
+
+             # Provides a relative path to your source file(s).
+             src/main/cpp/postprocess.cpp )
+
+# Searches for a specified prebuilt library and stores the path as a
+# variable. Because CMake includes system libraries in the search path by
+# default, you only need to specify the name of the public NDK library
+# you want to add. CMake verifies that the library exists before
+# completing its build.
+
+find_library( # Sets the name of the path variable.
+              log-lib
+
+              # Specifies the name of the NDK library that
+              # you want CMake to locate.
+              log )
+
+# Specifies libraries CMake should link to your target library. You
+# can link multiple libraries, such as libraries you define in this
+# build script, prebuilt third-party libraries, or system libraries.
+
+target_link_libraries( # Specifies the target library.
+                       native-lib
+
+                       # Links the target library to the log library
+                       # included in the NDK.
+                       ${log-lib} )

app/tflite_yolov5_test/.idea/gradle.xml

@@ -0,0 +1,67 @@
+plugins {
+    id 'com.android.application'
+}
+
+android {
+    compileSdkVersion 29
+    buildToolsVersion "30.0.3"
+
+    defaultConfig {
+        ndk {
+            abiFilters 'armeabi-v7a', 'arm64-v8a'
+        }
+        applicationId "com.example.tflite_yolov5_test"
+        minSdkVersion 28
+        targetSdkVersion 29
+        versionCode 1
+        versionName "1.0"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+        externalNativeBuild {
+            cmake {
+                cppFlags "-O3"
+            }
+        }
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+        }
+    }
+    compileOptions {
+        sourceCompatibility JavaVersion.VERSION_1_8
+        targetCompatibility JavaVersion.VERSION_1_8
+    }
+    externalNativeBuild {
+        cmake {
+            path "CMakeLists.txt"
+        }
+    }
+    buildFeatures {
+        viewBinding true
+    }
+}
+
+dependencies {
+    implementation 'org.nanohttpd:nanohttpd:2.3.1'
+    implementation 'androidx.appcompat:appcompat:1.2.0'
+    implementation 'com.google.android.material:material:1.2.1'
+    implementation 'androidx.constraintlayout:constraintlayout:2.0.1'
+    implementation 'androidx.lifecycle:lifecycle-livedata-ktx:2.2.0'
+    implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.2.0'
+    implementation 'androidx.navigation:navigation-fragment:2.3.0'
+    implementation 'androidx.navigation:navigation-ui:2.3.0'
+
+    implementation 'org.tensorflow:tensorflow-lite:2.4.0'
+    //implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
+    //implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'
+    //implementation 'org.tensorflow:tensorflow-lite-select-tf-ops:0.0.0-nightly'
+    //implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly-SNAPSHOT'
+    //implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly-SNAPSHOT'
+
+    testImplementation 'junit:junit:4.+'
+    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
+    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'
+}

app/tflite_yolov5_test/.idea/jarRepositories.xml

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

app/tflite_yolov5_test/.idea/misc.xml

@@ -0,0 +1,26 @@
+package com.example.tflite_yolov5_test;
+
+import android.content.Context;
+
+import androidx.test.platform.app.InstrumentationRegistry;
+import androidx.test.ext.junit.runners.AndroidJUnit4;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+import static org.junit.Assert.*;
+
+/**
+ * Instrumented test, which will execute on an Android device.
+ *
+ * @see <a href="http://d.android.com/tools/testing">Testing documentation</a>
+ */
+@RunWith(AndroidJUnit4.class)
+public class ExampleInstrumentedTest {
+    @Test
+    public void useAppContext() {
+        // Context of the app under test.
+        Context appContext = InstrumentationRegistry.getInstrumentation().getTargetContext();
+        assertEquals("com.example.tflite_yolov5_test", appContext.getPackageName());
+    }
+}