notebooks/210-slowfast-video-recognition/210-slowfast-video-recognition.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "95b8b645",
   "metadata": {},
   "source": [
    "# Video Recognition using SlowFast and OpenVINO™ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "168eea46",
   "metadata": {},
   "source": [
    "Teaching machines to detect, understand and analyze the contents of images has been one of the more well-known and well-studied problems in computer vision. However, analyzing videos to understand what is happening in them and detecting objects of interest are equally important and challenging tasks that have widespread applications in several areas including autonomous driving, healthcare, security, and many more.\n",
    "\n",
    "The SlowFast model puts forth an interesting approach to analyzing videos based on the intuition that videos typically contain static as well as dynamic elements- use a slow pathway operating at a low frame rate to analyze the static content and a fast pathway operating at a high frame rate to capture dynamic content. Its strength lies in its ability to effectively capture both fast and slow-motion information in video sequences, making it particularly well-suited to tasks that require a temporal and spatial understanding of the data.\n",
    "\n",
    "![](https://user-images.githubusercontent.com/34324155/143044111-94676f64-7ba8-4081-9011-f8054bed7030.png)\n",
    "\n",
    "\n",
    "More details about the network can be found in the original [paper](https://openaccess.thecvf.com/content_ICCV_2019/html/Feichtenhofer_SlowFast_Networks_for_Video_Recognition_ICCV_2019_paper.html) and [repository](https://github.com/facebookresearch/SlowFast).\n",
    "\n",
    "\n",
    "In this notebook, we will see how to convert and run a ResNet-50 based SlowFast model using OpenVINO. \n",
    "\n",
    "This tutorial consists of the following steps\n",
    "\n",
    "- Preparing the PyTorch model\n",
    "- Download and prepare data\n",
    "- Check inference with the PyTorch model\n",
    "- Convert to ONNX format\n",
    "- Convert ONNX Model to OpenVINO Intermediate Representation\n",
    "- Verify inference with the converted model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "04c22429",
   "metadata": {},
   "source": [
    "### Table of content:\n",
    "- [Prepare PyTorch Model](#Prepare-PyTorch-Model-Uparrow)\n",
    "    - [Install necessary packages](#Install-necessary-packages-Uparrow)\n",
    "    - [Imports and Settings](#Imports-and-Settings-Uparrow)\n",
    "- [Export to ONNX](#Export-to-ONNX-Uparrow)\n",
    "- [Convert ONNX to OpenVINO™ Intermediate Representation](#Convert-ONNX-to-OpenVINO-Intermediate-Representation-Uparrow)\n",
    "- [Select inference device](#Select-inference-device-Uparrow)\n",
    "- [Verify Model Inference](#Verify-Model-Inference-Uparrow)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07ffb626",
   "metadata": {},
   "source": [
    "## Prepare PyTorch Model [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aad46f1-068b-4cd5-a6a6-900c08cbf632",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Install necessary packages [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a70381fd-c1a8-4b5b-9657-de26feb587fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -q \"openvino>=2023.1.0\"\n",
    "!pip install -q fvcore"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9983ce7",
   "metadata": {},
   "source": [
    "### Imports and Settings [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "183a7c0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import math\n",
    "import sys\n",
    "import cv2\n",
    "import torch\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "from typing import Any, List, Dict\n",
    "from IPython.display import Video\n",
    "import openvino as ov\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "98f9d857",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = Path(\"data/\")\n",
    "MODEL_DIR = Path(\"model/\")\n",
    "MODEL_DIR.mkdir(exist_ok=True)\n",
    "DATA_DIR.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1dfa528",
   "metadata": {},
   "source": [
    "To begin, we download the PyTorch model from the PyTorchVideo repository. In this notebook, we will be using a SlowFast Network based on the ResNet-50 architecture trained on the Kinetics 400 dataset. Kinetics 400 is a large-scale dataset for action recognition in videos, containing 400 human action classes, with at least 400 video clips for each action. Read more about the dataset and the paper [here](https://www.deepmind.com/open-source/kinetics)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6a54c628",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "MODEL_NAME = \"slowfast_r50\"\n",
    "MODEL_REPOSITORY = \"facebookresearch/pytorchvideo\"\n",
    "DEVICE = \"cpu\"\n",
    "\n",
    "# load the pretrained model from the repository\n",
    "model = torch.hub.load(\n",
    "    repo_or_dir=MODEL_REPOSITORY, model=MODEL_NAME, pretrained=True, skip_validation=True\n",
    ")\n",
    "\n",
    "# set the device to allocate tensors to. for example, \"cpu\" or \"cuda\"\n",
    "model.to(DEVICE)\n",
    "\n",
    "# set the model to eval mode\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c1d0087",
   "metadata": {},
   "source": [
    "Now that we have loaded our pre-trained model, we will check inference with it. Since the model returns the detected class IDs, we download the ID to class label mapping for the Kinetics 400 dataset and load the mapping to a dict for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8b4e66e7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a44914bb1ab498da1e8435d72b2c5fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "data/kinetics_classnames.json: 0.00B [00:00, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "CLASSNAMES_SOURCE = (\n",
    "    \"https://dl.fbaipublicfiles.com/pyslowfast/dataset/class_names/kinetics_classnames.json\"\n",
    ")\n",
    "CLASSNAMES_FILE = \"kinetics_classnames.json\"\n",
    "\n",
    "download_file(url=CLASSNAMES_SOURCE, directory=DATA_DIR, show_progress=True)\n",
    "\n",
    "# load from json\n",
    "with open(DATA_DIR / CLASSNAMES_FILE, \"r\") as f:\n",
    "    kinetics_classnames = json.load(f)\n",
    "\n",
    "# load dict of id to class label mapping\n",
    "kinetics_id_to_classname = {}\n",
    "for k, v in kinetics_classnames.items():\n",
    "    kinetics_id_to_classname[v] = str(k).replace('\"', \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91ec867c",
   "metadata": {},
   "source": [
    "Let us download a sample video to run inference on, and take a look at the downloaded video. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "40f3c1a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "VIDEO_SOURCE = \"https://dl.fbaipublicfiles.com/pytorchvideo/projects/archery.mp4\"\n",
    "VIDEO_NAME = \"archery.mp4\"\n",
    "VIDEO_PATH = DATA_DIR / VIDEO_NAME\n",
    "\n",
    "download_file(url=VIDEO_SOURCE, directory=DATA_DIR, show_progress=True)\n",
    "Video(VIDEO_PATH, embed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6441640c",
   "metadata": {},
   "source": [
    "The sample video requires some preprocessing before we can run inference on it. During preprocessing, the video is normalized and scaled to size. Additionally, this preprocessing pipeline also involves sampling frames from the video to pass through the two pathways. The slow pathway can be any convolutional network that uses a large temporal stride on the input frames. The fast pathway is another convolutional network that uses a temporal stride smaller by a factor alpha($\\alpha$). In our model, both pathways use a 3D ResNet model. We define the following helper functions to implement the preprocessing steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e67c6a2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scale_short_side(size: int, frame: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Scale the short side of the frame to size and return a float\n",
    "    array.\n",
    "    \"\"\"\n",
    "    height = frame.shape[0]\n",
    "    width = frame.shape[1]\n",
    "    # return unchanged if short side already scaled\n",
    "    if (width <= height and width == size) or (height <= width and height == size):\n",
    "        return frame\n",
    "    new_width = size\n",
    "    new_height = size\n",
    "    if width < height:\n",
    "        new_height = int(math.floor((float(height) / width) * size))\n",
    "    else:\n",
    "        new_width = int(math.floor((float(width) / height) * size))\n",
    "    scaled = cv2.resize(frame, (new_width, new_height), interpolation=cv2.INTER_LINEAR)\n",
    "    return scaled.astype(np.float32)\n",
    "\n",
    "\n",
    "def center_crop(size: int, frame: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Center crop the input frame to size.\n",
    "    \"\"\"\n",
    "    height = frame.shape[0]\n",
    "    width = frame.shape[1]\n",
    "    y_offset = int(math.ceil((height - size) / 2))\n",
    "    x_offset = int(math.ceil((width - size) / 2))\n",
    "    cropped = frame[y_offset:y_offset + size, x_offset:x_offset + size, :]\n",
    "    assert cropped.shape[0] == size, \"Image height not cropped properly\"\n",
    "    assert cropped.shape[1] == size, \"Image width not cropped properly\"\n",
    "    return cropped\n",
    "\n",
    "\n",
    "def normalize(array: np.ndarray, mean: List[float], std: List[float]) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Normalize a given array by subtracting the mean and dividing the std.\n",
    "    \"\"\"\n",
    "    if array.dtype == np.uint8:\n",
    "        array = array.astype(np.float32)\n",
    "        array = array / 255.0\n",
    "    mean = np.array(mean, dtype=np.float32)\n",
    "    std = np.array(std, dtype=np.float32)\n",
    "    array = array - mean\n",
    "    array = array / std\n",
    "    return array\n",
    "\n",
    "\n",
    "def pack_pathway_output(frames: np.ndarray, alpha: int = 4) -> List[np.ndarray]:\n",
    "    \"\"\"\n",
    "    Prepare output as a list of arrays, each corresponding\n",
    "    to a unique pathway.\n",
    "    \"\"\"\n",
    "    fast_pathway = frames\n",
    "    # Perform temporal sampling from the fast pathway.\n",
    "    slow_pathway = np.take(\n",
    "        frames,\n",
    "        indices=np.linspace(0, frames.shape[1] - 1, frames.shape[1] // alpha).astype(np.int_),\n",
    "        axis=1\n",
    "    )\n",
    "    frame_list = [slow_pathway, fast_pathway]\n",
    "    return frame_list\n",
    "\n",
    "\n",
    "def process_inputs(\n",
    "    frames: List[np.ndarray],\n",
    "    num_frames: int,\n",
    "    crop_size: int,\n",
    "    mean: List[float],\n",
    "    std: List[float],\n",
    ") -> List[np.ndarray]:\n",
    "    \"\"\"\n",
    "    Performs normalization and applies required transforms\n",
    "    to prepare the input frames and returns a list of arrays.\n",
    "    Specifically the following actions are performed\n",
    "    1. scale the short side of the frames\n",
    "    2. center crop the frames to crop_size\n",
    "    3. perform normalization by subtracting mean and dividing std\n",
    "    4. sample frames for specified num_frames\n",
    "    5. sample frames for slow and fast pathways\n",
    "    \"\"\"\n",
    "    inputs = [scale_short_side(size=crop_size, frame=frame) for frame in frames]\n",
    "    inputs = [center_crop(size=crop_size, frame=frame) for frame in inputs]\n",
    "    inputs = np.array(inputs).astype(np.float32) / 255\n",
    "    inputs = normalize(array=inputs, mean=mean, std=std)\n",
    "    # T H W C -> C T H W\n",
    "    inputs = inputs.transpose([3, 0, 1, 2])\n",
    "    # Sample frames for num_frames specified\n",
    "    indices = np.linspace(0, inputs.shape[1] - 1, num_frames).astype(np.int_)\n",
    "    inputs = np.take(inputs, indices=indices, axis=1)\n",
    "    # prepare pathways for the model\n",
    "    inputs = pack_pathway_output(inputs)\n",
    "    inputs = [np.expand_dims(inp, 0) for inp in inputs]\n",
    "    return inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103ea967",
   "metadata": {},
   "source": [
    "Another helper method to run inference on a custom video using the given model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c10a0751",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_inference(\n",
    "    model: Any,\n",
    "    video_path: str,\n",
    "    top_k: int,\n",
    "    id_to_label_mapping: Dict[str, str],\n",
    "    num_frames: int,\n",
    "    sampling_rate: int,\n",
    "    crop_size: int,\n",
    "    mean: List[float],\n",
    "    std: List[float],\n",
    ") -> List[str]:\n",
    "    \"\"\"\n",
    "    Run inference on the video given by video_path using the given model.\n",
    "    First, the video is loaded from source. Frames are collected, processed\n",
    "    and fed to the model. The top top_k predicted class IDs are converted to class\n",
    "    labels and returned as a list of strings.\n",
    "    \"\"\"\n",
    "    video_cap = cv2.VideoCapture(video_path)\n",
    "    frames = []\n",
    "    seq_length = num_frames * sampling_rate\n",
    "    # get the list of frames from the video\n",
    "    ret = True\n",
    "    while ret and len(frames) < seq_length:\n",
    "        ret, frame = video_cap.read()\n",
    "        frames.append(frame)\n",
    "    # prepare the inputs\n",
    "    inputs = process_inputs(\n",
    "        frames=frames, num_frames=num_frames, crop_size=crop_size, mean=mean, std=std\n",
    "    )\n",
    "\n",
    "    if isinstance(model, ov.CompiledModel):\n",
    "        # openvino compiled model\n",
    "        output_blob = model.output(0)\n",
    "        predictions = model(inputs)[output_blob]\n",
    "    else:\n",
    "        # pytorch model\n",
    "        predictions = model([torch.from_numpy(inp) for inp in inputs])\n",
    "        predictions = predictions.detach().cpu().numpy()\n",
    "\n",
    "    def softmax(x):\n",
    "        return (np.exp(x) / np.exp(x).sum(axis=None))\n",
    "\n",
    "    # apply activation\n",
    "    predictions = softmax(predictions)\n",
    "\n",
    "    # top k predicted class IDs\n",
    "    topk = 5\n",
    "    pred_classes = np.argsort(-1 * predictions, axis=1)[:, :topk]\n",
    "\n",
    "    # Map the predicted classes to the label names\n",
    "    pred_class_names = [id_to_label_mapping[int(i)] for i in pred_classes[0]]\n",
    "    return pred_class_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92fc4a34",
   "metadata": {},
   "source": [
    "We define model-specific parameters for processing the input and run inference using the same. The top 5 predictions can be seen below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e1c04d11",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "NUM_FRAMES = 32\n",
    "SAMPLING_RATE = 2\n",
    "CROP_SIZE = 256\n",
    "MEAN = [0.45, 0.45, 0.45]\n",
    "STD = [0.225, 0.225, 0.225]\n",
    "TOP_K = 5\n",
    "\n",
    "predictions = run_inference(\n",
    "    model=model,\n",
    "    video_path=str(VIDEO_PATH),\n",
    "    top_k=TOP_K,\n",
    "    id_to_label_mapping=kinetics_id_to_classname,\n",
    "    num_frames=NUM_FRAMES,\n",
    "    sampling_rate=SAMPLING_RATE,\n",
    "    crop_size=CROP_SIZE,\n",
    "    mean=MEAN,\n",
    "    std=STD,\n",
    ")\n",
    "\n",
    "print(f\"Predicted labels: {', '.join(predictions)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e10aa8",
   "metadata": {},
   "source": [
    "## Export to ONNX [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92b1e21c",
   "metadata": {},
   "source": [
    "Now that we have obtained our trained model and checked inference with it, we export the PyTorch model to Open Neural Network Exchange(ONNX) format, an open format for representing machine learning models, so that we can use model conversion API to convert it to OpenVINO Intermediate Representation format(IR). This can be later used to run inference using the OpenVINO Runtime. Note that although the OpenVINO Runtime supports running ONNX models directly, converting to IR format enables us to take advantage of OpenVINO's optimization features including model quantization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f308f8ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "ONNX_MODEL_PATH = MODEL_DIR / \"slowfast-r50.onnx\"\n",
    "dummy_input = [torch.randn((1, 3, 8, 256, 256)), torch.randn([1, 3, 32, 256, 256])]\n",
    "torch.onnx.export(\n",
    "    model=model,\n",
    "    args=dummy_input,\n",
    "    f=ONNX_MODEL_PATH,\n",
    "    export_params=True,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8e750dee",
   "metadata": {},
   "source": [
    "## Convert ONNX to OpenVINO Intermediate Representation [$\\Uparrow$](#Table-of-content:)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3fe42b9",
   "metadata": {},
   "source": [
    "Now that our ONNX model is ready, we can convert it to IR format. In this format, the network is represented using two files: an `xml` file describing the network architecture and an accompanying binary file that stores constant values such as convolution weights in a binary format. We can use model conversion API for converting into IR format as follows. The `ov.convert_model` method returns an `ov.Model` object that can either be compiled and inferred or serialized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "de7ce812",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ov.convert_model(ONNX_MODEL_PATH)\n",
    "IR_PATH = MODEL_DIR / \"slowfast-r50.xml\"\n",
    "\n",
    "# serialize model for saving IR\n",
    "ov.save_model(model=model, output_model=str(IR_PATH), compress_to_fp16=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc8390f3",
   "metadata": {},
   "source": [
    "Next, we read and compile the serialized model using OpenVINO runtime. The `read_model` function expects the `.bin` weights file to have the same filename and be located in the same directory as the `.xml` file. If the weights file has a different filename, it can be specified using the `weights` parameter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "59bc7623",
   "metadata": {},
   "outputs": [],
   "source": [
    "core = ov.Core()\n",
    "\n",
    "# read converted model\n",
    "conv_model = core.read_model(str(IR_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a5909db-c74d-473a-8e38-4b5604b505e5",
   "metadata": {},
   "source": [
    "## Select inference device [$\\Uparrow$](#Table-of-content:)\n",
    "\n",
    "select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3ab5ce7b-b2ba-4a9a-8a01-5e4434e9612e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c23771ad1bab41deb4cb5052c82b0078",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value='AUTO',\n",
    "    description='Device:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8cf021e9-e231-4c05-bc01-d7a351086edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load model on device\n",
    "compiled_model = core.compile_model(model=conv_model, device_name=device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9970f436",
   "metadata": {},
   "source": [
    "## Verify Model Inference [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9f211f7",
   "metadata": {},
   "source": [
    "Using the compiled model, we run inference on the same sample video and print the top 5 predictions again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "92bdc44e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_class_names = run_inference(\n",
    "    model=compiled_model,\n",
    "    video_path=str(VIDEO_PATH),\n",
    "    top_k=TOP_K,\n",
    "    id_to_label_mapping=kinetics_id_to_classname,\n",
    "    num_frames=NUM_FRAMES,\n",
    "    sampling_rate=SAMPLING_RATE,\n",
    "    crop_size=CROP_SIZE,\n",
    "    mean=MEAN,\n",
    "    std=STD,\n",
    ")\n",
    "print(f\"Predicted labels: {', '.join(pred_class_names)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}