Yolov4_Object_Detector.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Yolov4 Object Detector",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMkHz4h84HGlA3XRt8AcRJe",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Mhmdaris15/Yolov4-Object-Detection/blob/main/Yolov4_Object_Detector.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xv1tSdZdJzH"
      },
      "source": [
        "# Yolo v4 - The Fastest and most Accurate Object Detector\n",
        "Hello everyone, my name is Aris. This is my code to make Object Detector using Yolo v4, which is the fastest and most accurate object detector ever currently now. This simulation also involve Webcam that can detect with your own camera. wanna know more about this Yolo, check this link <br> \n",
        "<a href='https://github.com/AlexeyAB/darknet'>Darknet Yolo</a><br>\n",
        "![picture](https://cdn-images-1.medium.com/max/1098/1*i-QSgEXjpi2VQngAIkbn1w.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6PPDumO5gJ_8"
      },
      "source": [
        "# import needed libraries\n",
        "from IPython.display import display, Javascript, Image\n",
        "from google.colab.output import eval_js\n",
        "from google.colab.patches import cv2_imshow\n",
        "from base64 import b64decode, b64encode\n",
        "import cv2\n",
        "import numpy as np\n",
        "import PIL\n",
        "import io\n",
        "import html\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_nq-SKu-gPO9"
      },
      "source": [
        "## Cloning and Setting Up Darknet for YOLOv4\n",
        "I will be using the famous AlexeyAB's darknet repository in this tutorial to perform YOLOv4 detections."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e_h-5uwlgdd-"
      },
      "source": [
        "# clone darknet repository\n",
        "!git clone https://github.com/AlexeyAB/darknet"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2FPpN2PSgkzu"
      },
      "source": [
        "# change makefile to have GPU, OPENCV and LIBSO enabled\n",
        "%cd darknet\n",
        "!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n",
        "!sed -i 's/GPU=0/GPU=1/' Makefile\n",
        "!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n",
        "!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n",
        "!sed -i 's/LIBSO=0/LIBSO=1/' Makefile"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qcj3NIdmgm-r"
      },
      "source": [
        "# make darknet (builds darknet in order to you can then use the darknet.py file and have its dependencies)\n",
        "!make"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ztGkI1oUgyF4"
      },
      "source": [
        "# get the scaled yolov4 weights file that is pre-trained to detect 80 classes (objects) from shared google drive\n",
        "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1V3vsIaxAlGWvK4Aar9bAiK5U0QFttKwq' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1V3vsIaxAlGWvK4Aar9bAiK5U0QFttKwq\" -O yolov4-csp.weights && rm -rf /tmp/cookies.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGqgvwzShIlE"
      },
      "source": [
        "## Darknet for Python\n",
        "In order to utilize YOLOv4 with Python code we will use some of pre-built functions in darknet.py by importing the functions into our workstation. Feel free to checkout the darknet.py file to see the function codes in detail! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pnhK6IrQhYQA"
      },
      "source": [
        "# import darknet functions to perform object detections\n",
        "from darknet import *\n",
        "# load in our YOLOv4 architecture network\n",
        "network, class_names, class_colors = load_network(\"cfg/yolov4-csp.cfg\", \"cfg/coco.data\", \"yolov4-csp.weights\")\n",
        "width = network_width(network)\n",
        "height = network_height(network)\n",
        "\n",
        "# darknet helper function to run detection on image\n",
        "def darknet_helper(img, width, height):\n",
        "  darknet_image = make_image(width, height, 3)\n",
        "  img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
        "  img_resized = cv2.resize(img_rgb, (width, height),\n",
        "                              interpolation=cv2.INTER_LINEAR)\n",
        "\n",
        "  # get image ratios to convert bounding boxes to proper size\n",
        "  img_height, img_width, _ = img.shape\n",
        "  width_ratio = img_width/width\n",
        "  height_ratio = img_height/height\n",
        "\n",
        "  # run model on darknet style image to get detections\n",
        "  copy_image_from_bytes(darknet_image, img_resized.tobytes())\n",
        "  detections = detect_image(network, class_names, darknet_image)\n",
        "  free_image(darknet_image)\n",
        "  return detections, width_ratio, height_ratio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YgFsRSDChdmS"
      },
      "source": [
        "## YOLOv4 Example on Test Image\n",
        "Let's make sure our model has successfully been loaded and that we can make detections properly on a test image. You can try with your own, with upload your image to this google colab. and then make a little change with its path. Let's enjoy it !"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWaCFPb5ht5Q"
      },
      "source": [
        "# run test on giraffe.jpg image that comes with repository\n",
        "image = cv2.imread(\"data/giraffe.jpg\")\n",
        "detections, width_ratio, height_ratio = darknet_helper(image, width, height)\n",
        "\n",
        "for label, confidence, bbox in detections:\n",
        "  left, top, right, bottom = bbox2points(bbox)\n",
        "  left, top, right, bottom = int(left * width_ratio), int(top * height_ratio), int(right * width_ratio), int(bottom * height_ratio)\n",
        "  cv2.rectangle(image, (left, top), (right, bottom), class_colors[label], 2)\n",
        "  cv2.putText(image, \"{} [{:.2f}]\".format(label, float(confidence)),\n",
        "                    (left, top - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5,\n",
        "                    class_colors[label], 2)\n",
        "cv2_imshow(image)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iavfIk3Jh0Ul"
      },
      "source": [
        "## Helper Functions\n",
        "Here are a few helper functions defined that will be used to convinient convert between different image types within our later steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WeJfbs8giBXC"
      },
      "source": [
        "# function to convert the JavaScript object into an OpenCV image\n",
        "def js_to_image(js_reply):\n",
        "  \"\"\"\n",
        "  Params:\n",
        "          js_reply: JavaScript object containing image from webcam\n",
        "  Returns:\n",
        "          img: OpenCV BGR image\n",
        "  \"\"\"\n",
        "  # decode base64 image\n",
        "  image_bytes = b64decode(js_reply.split(',')[1])\n",
        "  # convert bytes to numpy array\n",
        "  jpg_as_np = np.frombuffer(image_bytes, dtype=np.uint8)\n",
        "  # decode numpy array into OpenCV BGR image\n",
        "  img = cv2.imdecode(jpg_as_np, flags=1)\n",
        "\n",
        "  return img\n",
        "\n",
        "# function to convert OpenCV Rectangle bounding box image into base64 byte string to be overlayed on video stream\n",
        "def bbox_to_bytes(bbox_array):\n",
        "  \"\"\"\n",
        "  Params:\n",
        "          bbox_array: Numpy array (pixels) containing rectangle to overlay on video stream.\n",
        "  Returns:\n",
        "        bytes: Base64 image byte string\n",
        "  \"\"\"\n",
        "  # convert array into PIL image\n",
        "  bbox_PIL = PIL.Image.fromarray(bbox_array, 'RGBA')\n",
        "  iobuf = io.BytesIO()\n",
        "  # format bbox into png for return\n",
        "  bbox_PIL.save(iobuf, format='png')\n",
        "  # format return string\n",
        "  bbox_bytes = 'data:image/png;base64,{}'.format((str(b64encode(iobuf.getvalue()), 'utf-8')))\n",
        "\n",
        "  return bbox_bytes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ctfFNs3qiJj0"
      },
      "source": [
        "## YOLOv4 on Wecam Images\n",
        "Running YOLOv4 on images taken from webcam is fairly straight-forward. We will utilize code within Google Colab's **Code Snippets** that has a variety of useful code functions to perform various tasks.\n",
        "\n",
        "We will be using the code snippet for **Camera Capture** which runs JavaScript code to utilize your computer's webcam. The code snippet will take a webcam photo, which we will then pass into our YOLOv4 model for object detection.\n",
        "\n",
        "Below is a function to take the webcam picture using JavaScript and then run YOLOv4 on it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FY7aIMuuiVFs"
      },
      "source": [
        "def take_photo(filename='photo.jpg', quality=0.8):\n",
        "  js = Javascript('''\n",
        "    async function takePhoto(quality) {\n",
        "      const div = document.createElement('div');\n",
        "      const capture = document.createElement('button');\n",
        "      capture.textContent = 'Capture';\n",
        "      div.appendChild(capture);\n",
        "\n",
        "      const video = document.createElement('video');\n",
        "      video.style.display = 'block';\n",
        "      const stream = await navigator.mediaDevices.getUserMedia({video: true});\n",
        "\n",
        "      document.body.appendChild(div);\n",
        "      div.appendChild(video);\n",
        "      video.srcObject = stream;\n",
        "      await video.play();\n",
        "\n",
        "      // Resize the output to fit the video element.\n",
        "      google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);\n",
        "\n",
        "      // Wait for Capture to be clicked.\n",
        "      await new Promise((resolve) => capture.onclick = resolve);\n",
        "\n",
        "      const canvas = document.createElement('canvas');\n",
        "      canvas.width = video.videoWidth;\n",
        "      canvas.height = video.videoHeight;\n",
        "      canvas.getContext('2d').drawImage(video, 0, 0);\n",
        "      stream.getVideoTracks()[0].stop();\n",
        "      div.remove();\n",
        "      return canvas.toDataURL('image/jpeg', quality);\n",
        "    }\n",
        "    ''')\n",
        "  display(js)\n",
        "\n",
        "  # get photo data\n",
        "  data = eval_js('takePhoto({})'.format(quality))\n",
        "  # get OpenCV format image\n",
        "  img = js_to_image(data) \n",
        "  \n",
        "  # call our darknet helper on webcam image\n",
        "  detections, width_ratio, height_ratio = darknet_helper(img, width, height)\n",
        "\n",
        "  # loop through detections and draw them on webcam image\n",
        "  for label, confidence, bbox in detections:\n",
        "    left, top, right, bottom = bbox2points(bbox)\n",
        "    left, top, right, bottom = int(left * width_ratio), int(top * height_ratio), int(right * width_ratio), int(bottom * height_ratio)\n",
        "    cv2.rectangle(img, (left, top), (right, bottom), class_colors[label], 2)\n",
        "    cv2.putText(img, \"{} [{:.2f}]\".format(label, float(confidence)),\n",
        "                      (left, top - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5,\n",
        "                      class_colors[label], 2)\n",
        "  # save image\n",
        "  cv2.imwrite(filename, img)\n",
        "\n",
        "  return filename"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aIKufWpFiXJI"
      },
      "source": [
        "try:\n",
        "  filename = take_photo('photo.jpg')\n",
        "  print('Saved to {}'.format(filename))\n",
        "  \n",
        "  # Show the image which was just taken.\n",
        "  display(Image(filename))\n",
        "except Exception as err:\n",
        "  # Errors will be thrown if the user does not have a webcam or if they do not\n",
        "  # grant the page permission to access it.\n",
        "  print(str(err))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ysOFnOnUiZ9c"
      },
      "source": [
        "## YOLOv4 on Webcam Videos\n",
        "Running YOLOv4 on webcam video is a little more complex than images. We need to start a video stream using our webcam as input. Then we run each frame through our YOLOv4 model and create an overlay image that contains bounding box of detection(s). We then overlay the bounding box image back onto the next frame of our video stream. \n",
        "\n",
        "YOLOv4 is so fast that it can run the detections in real-time!\n",
        "\n",
        "<a href='https://drive.google.com/file/d/1fUHKYsAUeDOUC3AECZJRF55OxjuZQ5Dx/view?usp=sharing'>Yolo Webcam Workflow</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TXHH1vmflq56"
      },
      "source": [
        "# JavaScript to properly create our live video stream using our webcam as input\n",
        "def video_stream():\n",
        "  js = Javascript('''\n",
        "    var video;\n",
        "    var div = null;\n",
        "    var stream;\n",
        "    var captureCanvas;\n",
        "    var imgElement;\n",
        "    var labelElement;\n",
        "    \n",
        "    var pendingResolve = null;\n",
        "    var shutdown = false;\n",
        "    \n",
        "    function removeDom() {\n",
        "       stream.getVideoTracks()[0].stop();\n",
        "       video.remove();\n",
        "       div.remove();\n",
        "       video = null;\n",
        "       div = null;\n",
        "       stream = null;\n",
        "       imgElement = null;\n",
        "       captureCanvas = null;\n",
        "       labelElement = null;\n",
        "    }\n",
        "    \n",
        "    function onAnimationFrame() {\n",
        "      if (!shutdown) {\n",
        "        window.requestAnimationFrame(onAnimationFrame);\n",
        "      }\n",
        "      if (pendingResolve) {\n",
        "        var result = \"\";\n",
        "        if (!shutdown) {\n",
        "          captureCanvas.getContext('2d').drawImage(video, 0, 0, 640, 480);\n",
        "          result = captureCanvas.toDataURL('image/jpeg', 0.8)\n",
        "        }\n",
        "        var lp = pendingResolve;\n",
        "        pendingResolve = null;\n",
        "        lp(result);\n",
        "      }\n",
        "    }\n",
        "    \n",
        "    async function createDom() {\n",
        "      if (div !== null) {\n",
        "        return stream;\n",
        "      }\n",
        "\n",
        "      div = document.createElement('div');\n",
        "      div.style.border = '2px solid black';\n",
        "      div.style.padding = '3px';\n",
        "      div.style.width = '100%';\n",
        "      div.style.maxWidth = '600px';\n",
        "      document.body.appendChild(div);\n",
        "      \n",
        "      const modelOut = document.createElement('div');\n",
        "      modelOut.innerHTML = \"<span>Status:</span>\";\n",
        "      labelElement = document.createElement('span');\n",
        "      labelElement.innerText = 'No data';\n",
        "      labelElement.style.fontWeight = 'bold';\n",
        "      modelOut.appendChild(labelElement);\n",
        "      div.appendChild(modelOut);\n",
        "           \n",
        "      video = document.createElement('video');\n",
        "      video.style.display = 'block';\n",
        "      video.width = div.clientWidth - 6;\n",
        "      video.setAttribute('playsinline', '');\n",
        "      video.onclick = () => { shutdown = true; };\n",
        "      stream = await navigator.mediaDevices.getUserMedia(\n",
        "          {video: { facingMode: \"environment\"}});\n",
        "      div.appendChild(video);\n",
        "\n",
        "      imgElement = document.createElement('img');\n",
        "      imgElement.style.position = 'absolute';\n",
        "      imgElement.style.zIndex = 1;\n",
        "      imgElement.onclick = () => { shutdown = true; };\n",
        "      div.appendChild(imgElement);\n",
        "      \n",
        "      const instruction = document.createElement('div');\n",
        "      instruction.innerHTML = \n",
        "          '<span style=\"color: red; font-weight: bold;\">' +\n",
        "          'When finished, click here or on the video to stop this demo</span>';\n",
        "      div.appendChild(instruction);\n",
        "      instruction.onclick = () => { shutdown = true; };\n",
        "      \n",
        "      video.srcObject = stream;\n",
        "      await video.play();\n",
        "\n",
        "      captureCanvas = document.createElement('canvas');\n",
        "      captureCanvas.width = 640; //video.videoWidth;\n",
        "      captureCanvas.height = 480; //video.videoHeight;\n",
        "      window.requestAnimationFrame(onAnimationFrame);\n",
        "      \n",
        "      return stream;\n",
        "    }\n",
        "    async function stream_frame(label, imgData) {\n",
        "      if (shutdown) {\n",
        "        removeDom();\n",
        "        shutdown = false;\n",
        "        return '';\n",
        "      }\n",
        "\n",
        "      var preCreate = Date.now();\n",
        "      stream = await createDom();\n",
        "      \n",
        "      var preShow = Date.now();\n",
        "      if (label != \"\") {\n",
        "        labelElement.innerHTML = label;\n",
        "      }\n",
        "            \n",
        "      if (imgData != \"\") {\n",
        "        var videoRect = video.getClientRects()[0];\n",
        "        imgElement.style.top = videoRect.top + \"px\";\n",
        "        imgElement.style.left = videoRect.left + \"px\";\n",
        "        imgElement.style.width = videoRect.width + \"px\";\n",
        "        imgElement.style.height = videoRect.height + \"px\";\n",
        "        imgElement.src = imgData;\n",
        "      }\n",
        "      \n",
        "      var preCapture = Date.now();\n",
        "      var result = await new Promise(function(resolve, reject) {\n",
        "        pendingResolve = resolve;\n",
        "      });\n",
        "      shutdown = false;\n",
        "      \n",
        "      return {'create': preShow - preCreate, \n",
        "              'show': preCapture - preShow, \n",
        "              'capture': Date.now() - preCapture,\n",
        "              'img': result};\n",
        "    }\n",
        "    ''')\n",
        "\n",
        "  display(js)\n",
        "  \n",
        "def video_frame(label, bbox):\n",
        "  data = eval_js('stream_frame(\"{}\", \"{}\")'.format(label, bbox))\n",
        "  return data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inmso2f5lvYY"
      },
      "source": [
        "### Running on Webcam Video"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_9PrL9x5lzOY"
      },
      "source": [
        "# start streaming video from webcam\n",
        "video_stream()\n",
        "# label for video\n",
        "label_html = 'Capturing...'\n",
        "# initialze bounding box to empty\n",
        "bbox = ''\n",
        "count = 0 \n",
        "while True:\n",
        "    js_reply = video_frame(label_html, bbox)\n",
        "    if not js_reply:\n",
        "        break\n",
        "\n",
        "    # convert JS response to OpenCV Image\n",
        "    frame = js_to_image(js_reply[\"img\"])\n",
        "\n",
        "    # create transparent overlay for bounding box\n",
        "    bbox_array = np.zeros([480,640,4], dtype=np.uint8)\n",
        "\n",
        "    # call our darknet helper on video frame\n",
        "    detections, width_ratio, height_ratio = darknet_helper(frame, width, height)\n",
        "\n",
        "    # loop through detections and draw them on transparent overlay image\n",
        "    for label, confidence, bbox in detections:\n",
        "      left, top, right, bottom = bbox2points(bbox)\n",
        "      left, top, right, bottom = int(left * width_ratio), int(top * height_ratio), int(right * width_ratio), int(bottom * height_ratio)\n",
        "      bbox_array = cv2.rectangle(bbox_array, (left, top), (right, bottom), class_colors[label], 2)\n",
        "      bbox_array = cv2.putText(bbox_array, \"{} [{:.2f}]\".format(label, float(confidence)),\n",
        "                        (left, top - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5,\n",
        "                        class_colors[label], 2)\n",
        "\n",
        "    bbox_array[:,:,3] = (bbox_array.max(axis = 2) > 0 ).astype(int) * 255\n",
        "    # convert overlay of bbox into bytes\n",
        "    bbox_bytes = bbox_to_bytes(bbox_array)\n",
        "    # update bbox so next frame gets new overlay\n",
        "    bbox = bbox_bytes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WmSBdmHml1y3"
      },
      "source": [
        "## Hope You Enjoyed!\n",
        "\n",
        "Have a great day!"
      ]
    }
  ]
}