codes/MMSelfSup_tutorials/【4】自监督预训练模型的评估：“分类” 下游任务 之 SVM 评估.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/mmselfsup_logo.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自监督预训练模型的评估：“分类” 下游任务 之 SVM 评估\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/open-mmlab/OpenMMLabCourse/blob/main/codes/MMSelfSup_tutorials/【4】自监督预训练模型的评估：“分类”%20下游任务%20之%20SVM%20评估.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MMSelfSup Repo**：[https://github.com/open-mmlab/mmselfsup](https://github.com/open-mmlab/mmselfsup)\n",
    "\n",
    "**MMSelfSup 官方文档链接**：[https://mmselfsup.readthedocs.io/en/latest](https://mmselfsup.readthedocs.io/en/latest)\n",
    "\n",
    "**MMSelfSup 视频教学**：[https://space.bilibili.com/1293512903/channel/collectiondetail?sid=657287](https://space.bilibili.com/1293512903/channel/collectiondetail?sid=657287)\n",
    "\n",
    "**MMSelfSup 代码库介绍 PPT 获取方式**：关注 OpenMMLab 公众号，后台回复：mmselfsup，即可获取课程 PPT\n",
    "\n",
    "**加入微信社群方式**：关注公众号，选择 “加入我们” -> “微信社区”，即可获取入群二维码。非常期待你的到来呀~\n",
    "\n",
    "**作者**：OpenMMLab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 任务介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在使用 “分类” 下游任务对自监督预训练模型进行评估的方法中，MMSelfSup 目前实现了以下几种方法：(详细请参考官方[文档](https://mmselfsup.readthedocs.io/zh_CN/latest/tutorials/6_benchmarks.html#id2))\n",
    "+ **VOC SVM / Low-shot SVM**\n",
    "+ 线性评估\n",
    "+ ImageNet 半监督分类\n",
    "+ ImageNet 最邻近分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVM 评估也是评估自监督预训练模型的常用基准之一。它的 **`基本流程`** 如下：\n",
    "1. 每张图像送入自监督预训练好的模型提取特征\n",
    "2. 用所有训练集图片的特征向量来训练支持向量机 SVM \n",
    "3. 将测试集图片的特征向送入训练好的 SVM，将其分类性能作为评估从自监督学习方法中学习图像特征质量的手段，与其他自监督模型进行比较"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 环境配置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 查看 Python、PyTorch 和 Torchvision 的版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check nvcc version\n",
    "!nvcc -V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check GCC version\n",
    "!gcc --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check PyTorch installation\n",
    "import torch, torchvision\n",
    "print(torch.__version__)\n",
    "print(torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 安装 MMSelfSup 的依赖库：MMCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install openmim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mim install mmcv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3  安装 MMSelfSup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/open-mmlab/mmselfsup.git\n",
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install MMSelfSup from source\n",
    "!pip install -e . "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 检查安装是否正确"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mmselfsup\n",
    "print(mmselfsup.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 准备数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**：目前，MMSelfSup 的 SVM 评估只支持了 `PASCAL VOC 2007` 数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.0 数据集介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本教程将在 `PASCAL VOC 2007` 数据集上对自监督预训练好的模型进行 SVM 评估。\n",
    "\n",
    "该数据集包含有 20 个类别。训练集有 5011 张图片，测试集有 4952 张图片。\n",
    "\n",
    "数据集官方下载地址：http://host.robots.ox.ac.uk/pascal/VOC/voc2007/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 下载数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用如下命令，自动将 PASCAL VOC 2007 下载到 `$YOUR_DATA_ROOT` 文件夹中。\n",
    "\n",
    "```shell\n",
    "bash tools/data_converters/prepare_voc07_cls.sh $YOUR_DATA_ROOT\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% cd /content/mmselfsup\n",
    "!bash tools/data_converters/prepare_voc07_cls.sh data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf data/VOCtest_06-Nov-2007.tar\n",
    "!rm -rf data/VOCtrainval_06-Nov-2007.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 准备自监督预训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目前，MMSelfSup 里 SVM 评估的模型对象既可以是 `自监督预训练模型提取 backbone 部分的权重文件`，也可以是 `自监督预训练过程中直接保存下来的名为 epoch_*.pth 的 checkpoint 文件`。所以，我们这里需要准备好两种模型文件，方便后面对这两种文件进行 SVM 评估的代码演示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 准备自监督预训练模型提取 backbone 部分的权重文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：MMSelfSup 的 [模型库](https://github.com/open-mmlab/mmselfsup/blob/master/docs/en/model_zoo.md) 中的模型文件都已经提取过 backbone 权值，不需要再次提取！**我们直接使用即可。\n",
    "\n",
    "在模型库中找到在 SimCLR 的预训练模型文件 `simclr_resnet50_8xb32-coslr-200e_in1k`，下载放在 `checkpoints` 文件夹里 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir checkpoints\n",
    "!wget https://download.openmmlab.com/mmselfsup/simclr/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth -P checkpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 准备自监督预训练过程中直接保存下来的 checkpoint 文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用第一个教程 `模型自监督预训练 之 SimCLR` 中训练保存下来的 `epoch_1.pth` 文件进行演示，该文件可以从 [这里](https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth) 下载，存放在文件夹 `mmselfsup/work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200` 里。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir -p work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200\n",
    "!wget https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth  -P  work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同时，准备好第一个教程 `模型自监督预训练 之 SimCLR` 中的自监督预训练配置文件 `simclr_resnet50_1xb32-coslr-1e_tinyin200.py`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile /content/mmselfsup/configs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200.py\n",
    "\n",
    "_base_ = 'simclr_resnet50_8xb32-coslr-200e_in1k.py'\n",
    "\n",
    "# dataset\n",
    "data = dict(\n",
    "    samples_per_gpu=32, \n",
    "    workers_per_gpu=2,\n",
    "    train=dict(\n",
    "        data_source=dict(\n",
    "            data_prefix='data/tiny-imagenet-200/train',\n",
    "            ann_file='data/tiny-imagenet-200/train.txt',\n",
    "        )\n",
    "    )\n",
    ")\n",
    "\n",
    "# optimizer\n",
    "optimizer = dict(\n",
    "    lr=0.3 * ((32 * 1) / (32 * 8)),\n",
    ")\n",
    "\n",
    "runner = dict(max_epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 自监督预训练模型的 SVM 评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面，我们分别演示对 `自监督预训练模型提取 backbone 部分的权重文件` 和 `自监督预训练过程中直接保存下来的名为 epoch_*.pth 的 checkpoint 文件` 两种文件进行 SVM 评估。\n",
    "\n",
    "相关代码文件见：[tools/benchmarks/classification/svm_voc07](https://github.com/open-mmlab/mmselfsup/tree/master/tools/benchmarks/classification/svm_voc07)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 对自监督预训练模型的 backbone 权重文件进行 SVM 评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在运行自监督预训练模型的 SVM 评估时，我们使用 [dist_test_svm_pretrain.sh](https://github.com/open-mmlab/mmselfsup/blob/master/tools/benchmarks/classification/svm_voc07/dist_test_svm_pretrain.sh) 脚本来启动自监督预训练模型 backbone 权重文件的评估。\n",
    "\n",
    "```shell\n",
    "bash tools/benchmarks/classification/svm_voc07/dist_test_svm_pretrain.sh ${SELFSUP_CONFIG} ${GPUS} ${PRETRAIN} ${FEATURE_LIST}\n",
    "\n",
    "```\n",
    "\n",
    "参数：\n",
    "- SELFSUP_CONFIG：自监督预训练的配置文件\n",
    "- GPUS：使用 GPU 的数量\n",
    "- PRETRAIN：自监督预训练模型的 backbone 权重文件所在的路径。\n",
    "- FEATURE_LIST：该参数的值是一个字符串，用于指定评估从 layer1 到 layer5 的特征，默认值为 `\"feat5\"`，表示评估 layer5 的特征。如果想评估 layer3，那么该参数的值是 `\"feat3\"`；如果想评估所有特征，那么该参数的值是 `\"feat1 feat2 feat3 feat4 feat5\"`（注意用空格分隔）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!bash tools/benchmarks/classification/svm_voc07/dist_test_svm_pretrain.sh \\\n",
    "configs/selfsup/simclr/simclr_resnet50_8xb32-coslr-200e_in1k.py \\\n",
    "1 \\\n",
    "checkpoints/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth \\\n",
    "\"feat5\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 对自监督预训练过程中直接保存下来的 checkpoint 文件进行 SVM 评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在运行自监督预训练模型的 SVM 评估时，我们使用 [dist_test_svm_epoch.sh](https://github.com/open-mmlab/mmselfsup/blob/master/tools/benchmarks/classification/svm_voc07/dist_test_svm_epoch.sh) 脚本来启动自监督预训练过程中直接保存下来 checkpoint 文件的评估。\n",
    "\n",
    "```shell\n",
    "GPUS=${GPUS} bash tools/benchmarks/classification/svm_voc07/dist_test_svm_epoch.sh ${SELFSUP_CONFIG} ${EPOCH} ${FEATURE_LIST}\n",
    "```\n",
    "\n",
    "参数：\n",
    "- GPUS：默认使用的 GPU 数量是 8，如果需要调整，需要加上该环境变量。\n",
    "- SELFSUP_CONFIG：自监督预训练的配置文件\n",
    "- EPOCH：想要测试 checkpoint 文件的 epoch 数。例如：该参数的值为 5，意味着测名为 `epoch_5.pth` 的 checkpoint 文件。\n",
    "- FEATURE_LIST：该参数的值是一个字符串，用于指定评估从 layer1 到 layer5 的特征，默认值为 `\"feat5\"`，表示评估 layer5 的特征。如果想评估 layer3，那么该参数的值是 `\"feat3\"`；如果想评估所有特征，那么该参数的值是 `\"feat1 feat2 feat3 feat4 feat5\"`（注意用空格分隔）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GPUS=1 bash tools/benchmarks/classification/svm_voc07/dist_test_svm_epoch.sh \\\n",
    "configs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200.py \\\n",
    "1 \\\n",
    "\"feat5\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}