3.klue-bert-hyperparameter-tuning-with-trainer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7eeab33",
   "metadata": {},
   "source": [
    "# Trainer API hyperparameter search\n",
    "모델 내부에서 결정되는 변수를 parameter라고 합니다. 예를 들어, 모델의 weight가 있습니다.   \n",
    "parameter는 데이터를 모델이 학습하는 과정에서 모델이 스스로 결정합니다.   \n",
    "\n",
    "모델을 정의할 때, 설계자가 직접 설정해주는 값을 hyper parameter라고 합니다. 예를들어 learning rate가 있습니다.  \n",
    "hyper parameter를 설정하는 방법에는 규칙이 없습니다. 따라서 모델에 따라서, 데이터에 따라서, 많은 변수에 의해 최고의 성능을 내는 hyper parameter의 값은 달라질 수 있습니다.  \n",
    "이러한 환경에서 모델의 성능을 최적화하는 hyper parameter를 찾는 방법은 Grid search, random search, Bayesian Optimization 등 여러가지 방법이 있습니다.  \n",
    "하지만 이러한 방식을 직접 구현하는데는 시간과 노력이 들어갑니다.  \n",
    "\n",
    "transformers 라이브러리의 Trainer를 사용하면 쉽게 hyper parameter를 찾을 수 있습니다.  \n",
    "`Trainer`의 `hyperparameter_search()`함수를 호출하는 것만으로 모델의 성능을 최적화 하는 hyper parameter를 찾을 수 있습니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "045d567d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datasets\n",
    "from datasets import load_metric\n",
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "warnings.filterwarnings(action='ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2c99a1",
   "metadata": {},
   "source": [
    "Reproduction을 위한 Seed 고정  \n",
    "출처 : https://dacon.io/codeshare/2363?dtype=vote&s_id=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b03c43a",
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_SEED = 42\n",
    "\n",
    "def seed_everything(seed: int = 42):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed) \n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.backends.cudnn.benchmark = True \n",
    "    \n",
    "seed_everything(RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c50a248b",
   "metadata": {},
   "source": [
    "이번 노트북에서는 klue/bert-base 모델을 사용합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a73d2b87",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint = \"klue/bert-base\"\n",
    "batch_size = 32\n",
    "save_checkpoint_path = \"./checkpoints\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f60b4519",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec520800",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_csv(\"data/train_data.csv\")\n",
    "test = pd.read_csv(\"data/test_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fdb89171",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_train, dataset_val = train_test_split(dataset,test_size = 0.2,random_state = RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "33b13e1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TrainDataset(Dataset):\n",
    "    def __init__(self, dataset, sent_key, label_key, bert_tokenizer):\n",
    "        self.sentences = [ bert_tokenizer(i,truncation=True,return_token_type_ids=False) for i in dataset[sent_key] ]\n",
    "        self.labels = [np.int64(i) for i in dataset[label_key]]\n",
    "\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        self.sentences[i][\"label\"] = self.labels[i]\n",
    "        return self.sentences[i]\n",
    "\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.sentences)\n",
    "    \n",
    "class TestDataset(Dataset):\n",
    "    def __init__(self, dataset, sent_key, bert_tokenizer):\n",
    "        self.sentences = [ bert_tokenizer(i,truncation=True,return_token_type_ids=False) for i in dataset[sent_key] ]\n",
    "        \n",
    "    def __getitem__(self, i):\n",
    "        return self.sentences[i]\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.sentences)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "57a107bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = TrainDataset(dataset_train, \"title\", \"topic_idx\", tokenizer)\n",
    "data_validation = TrainDataset(dataset_val, \"title\", \"topic_idx\", tokenizer)\n",
    "data_test = TestDataset(test, \"title\", tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e2c5986",
   "metadata": {},
   "source": [
    "모델의 성능을 측정하기 위한 metric(지표)를 불러옵니다.  \n",
    "수행할 과제는 Text Classification이기 때문에 비슷한 과제인 glue-qnli의 metric을 가져옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9456a5ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = load_metric(\"glue\", \"qnli\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4a5758a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "    predictions, labels = eval_pred\n",
    "    predictions = np.argmax(predictions, axis=1)\n",
    "    return metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d4b4fd1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_name = \"accuracy\"\n",
    "\n",
    "args = TrainingArguments(\n",
    "    save_checkpoint_path,\n",
    "    evaluation_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,\n",
    "    per_device_train_batch_size=batch_size,\n",
    "    per_device_eval_batch_size=batch_size,\n",
    "    num_train_epochs=3,\n",
    "    weight_decay=0.01,\n",
    "    load_best_model_at_end=True,\n",
    "    metric_for_best_model=metric_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ca9e650",
   "metadata": {},
   "source": [
    "Trainer의 `hyperparameter_search()`는 최적의 hyper parameter를 찾기 위해 여러번의 학습을 반복합니다.  \n",
    "학습을 반복할 때 모델의 parameter들을 초기화시켜주어야 하기 때문에 새로운 model을 정의하는 함수를 Trainer에 전달해주어야 합니다.  \n",
    "`model_init()`함수를 만들어 Trainer에 전달해주도록 하겠습니다. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "797d847a",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_labels = 7\n",
    "\n",
    "def model_init():\n",
    "    return AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c80d551b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file https://huggingface.co/klue/bert-base/resolve/main/config.json from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\fbd0b2ef898c4653902683fea8cc0dd99bf43f0e082645b913cda3b92429d1bb.7cee10e8ea7ffa278f8be4b141000263f2b18795e5ef5e025352b2af6851f8fb\n",
      "Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForPretraining\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\",\n",
      "    \"1\": \"LABEL_1\",\n",
      "    \"2\": \"LABEL_2\",\n",
      "    \"3\": \"LABEL_3\",\n",
      "    \"4\": \"LABEL_4\",\n",
      "    \"5\": \"LABEL_5\",\n",
      "    \"6\": \"LABEL_6\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0,\n",
      "    \"LABEL_1\": 1,\n",
      "    \"LABEL_2\": 2,\n",
      "    \"LABEL_3\": 3,\n",
      "    \"LABEL_4\": 4,\n",
      "    \"LABEL_5\": 5,\n",
      "    \"LABEL_6\": 6\n",
      "  },\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.8.2\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 32000\n",
      "}\n",
      "\n",
      "loading weights file https://huggingface.co/klue/bert-base/resolve/main/pytorch_model.bin from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\05b36ee62545d769939a7746eca739b844a40a7a7553700f110b58b28ed6a949.7cb231256a5dbe886e12b902d05cb1241f330d8c19428508f91b2b28c1cfe0b6\n",
      "Some weights of the model checkpoint at klue/bert-base were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at klue/bert-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model_init=model_init,\n",
    "    args=args,\n",
    "    train_dataset=data_train,\n",
    "    eval_dataset=data_validation,\n",
    "    tokenizer=tokenizer,\n",
    "    compute_metrics=compute_metrics\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "777ba7ba",
   "metadata": {},
   "source": [
    "transformers에서는 hyperparameter를 찾기 위해 `optuna`와 `ray` 라이브러리를 필요로 합니다.  \n",
    "pip를 통해서 설치해줍시다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ebdc981a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install optuna\n",
    "# !pip install ray[tune]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5866bbe",
   "metadata": {},
   "source": [
    "`hyperparameter_search()`의 인자로 `n_trials`를 조절하여 grid search를 하는 횟수를 조절할 수 있습니다.   \n",
    "`hyperparameter_search()` 함수는 hyperparameter를 조절해가며 `n_trials` 만큼 학습을 진행한 뒤 가장 성능이 좋았던 hyper parameter를 반환해줍니다.  \n",
    "학습에 시간이 오래걸려 2번만 search를 진행했습니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d1c0217c",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-08-15 22:21:41,466]\u001b[0m A new study created in memory with name: no-name-23d8a293-af53-4323-b728-069f3431ca44\u001b[0m\n",
      "Trial:\n",
      "loading configuration file https://huggingface.co/klue/bert-base/resolve/main/config.json from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\fbd0b2ef898c4653902683fea8cc0dd99bf43f0e082645b913cda3b92429d1bb.7cee10e8ea7ffa278f8be4b141000263f2b18795e5ef5e025352b2af6851f8fb\n",
      "Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForPretraining\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\",\n",
      "    \"1\": \"LABEL_1\",\n",
      "    \"2\": \"LABEL_2\",\n",
      "    \"3\": \"LABEL_3\",\n",
      "    \"4\": \"LABEL_4\",\n",
      "    \"5\": \"LABEL_5\",\n",
      "    \"6\": \"LABEL_6\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0,\n",
      "    \"LABEL_1\": 1,\n",
      "    \"LABEL_2\": 2,\n",
      "    \"LABEL_3\": 3,\n",
      "    \"LABEL_4\": 4,\n",
      "    \"LABEL_5\": 5,\n",
      "    \"LABEL_6\": 6\n",
      "  },\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.8.2\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 32000\n",
      "}\n",
      "\n",
      "loading weights file https://huggingface.co/klue/bert-base/resolve/main/pytorch_model.bin from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\05b36ee62545d769939a7746eca739b844a40a7a7553700f110b58b28ed6a949.7cb231256a5dbe886e12b902d05cb1241f330d8c19428508f91b2b28c1cfe0b6\n",
      "Some weights of the model checkpoint at klue/bert-base were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at klue/bert-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "***** Running training *****\n",
      "  Num examples = 36523\n",
      "  Num Epochs = 4\n",
      "  Instantaneous batch size per device = 32\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 32\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 4568\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='4568' max='4568' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [4568/4568 08:03, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.357700</td>\n",
       "      <td>0.336874</td>\n",
       "      <td>0.885664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.237800</td>\n",
       "      <td>0.355800</td>\n",
       "      <td>0.884679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.134700</td>\n",
       "      <td>0.439973</td>\n",
       "      <td>0.879750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.067300</td>\n",
       "      <td>0.533740</td>\n",
       "      <td>0.879860</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-0\\checkpoint-1142\n",
      "Configuration saved in test-nli\\run-0\\checkpoint-1142\\config.json\n",
      "Model weights saved in test-nli\\run-0\\checkpoint-1142\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-0\\checkpoint-1142\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-0\\checkpoint-1142\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-0\\checkpoint-2284\n",
      "Configuration saved in test-nli\\run-0\\checkpoint-2284\\config.json\n",
      "Model weights saved in test-nli\\run-0\\checkpoint-2284\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-0\\checkpoint-2284\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-0\\checkpoint-2284\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-0\\checkpoint-3426\n",
      "Configuration saved in test-nli\\run-0\\checkpoint-3426\\config.json\n",
      "Model weights saved in test-nli\\run-0\\checkpoint-3426\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-0\\checkpoint-3426\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-0\\checkpoint-3426\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-0\\checkpoint-4568\n",
      "Configuration saved in test-nli\\run-0\\checkpoint-4568\\config.json\n",
      "Model weights saved in test-nli\\run-0\\checkpoint-4568\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-0\\checkpoint-4568\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-0\\checkpoint-4568\\special_tokens_map.json\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "Loading best model from test-nli\\run-0\\checkpoint-1142 (score: 0.8856642207863322).\n",
      "\u001b[32m[I 2021-08-15 22:29:47,686]\u001b[0m Trial 0 finished with value: 0.8798598182017303 and parameters: {'learning_rate': 4.5352366623370486e-05, 'num_train_epochs': 4, 'seed': 35, 'per_device_train_batch_size': 32}. Best is trial 0 with value: 0.8798598182017303.\u001b[0m\n",
      "Trial:\n",
      "loading configuration file https://huggingface.co/klue/bert-base/resolve/main/config.json from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\fbd0b2ef898c4653902683fea8cc0dd99bf43f0e082645b913cda3b92429d1bb.7cee10e8ea7ffa278f8be4b141000263f2b18795e5ef5e025352b2af6851f8fb\n",
      "Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForPretraining\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\",\n",
      "    \"1\": \"LABEL_1\",\n",
      "    \"2\": \"LABEL_2\",\n",
      "    \"3\": \"LABEL_3\",\n",
      "    \"4\": \"LABEL_4\",\n",
      "    \"5\": \"LABEL_5\",\n",
      "    \"6\": \"LABEL_6\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0,\n",
      "    \"LABEL_1\": 1,\n",
      "    \"LABEL_2\": 2,\n",
      "    \"LABEL_3\": 3,\n",
      "    \"LABEL_4\": 4,\n",
      "    \"LABEL_5\": 5,\n",
      "    \"LABEL_6\": 6\n",
      "  },\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.8.2\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 32000\n",
      "}\n",
      "\n",
      "loading weights file https://huggingface.co/klue/bert-base/resolve/main/pytorch_model.bin from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\05b36ee62545d769939a7746eca739b844a40a7a7553700f110b58b28ed6a949.7cb231256a5dbe886e12b902d05cb1241f330d8c19428508f91b2b28c1cfe0b6\n",
      "Some weights of the model checkpoint at klue/bert-base were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at klue/bert-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "***** Running training *****\n",
      "  Num examples = 36523\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 4\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 4\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 27393\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='27393' max='27393' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [27393/27393 27:34, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.772300</td>\n",
       "      <td>0.671673</td>\n",
       "      <td>0.847552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.623400</td>\n",
       "      <td>0.750209</td>\n",
       "      <td>0.861680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.487400</td>\n",
       "      <td>0.698994</td>\n",
       "      <td>0.871098</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-1\\checkpoint-9131\n",
      "Configuration saved in test-nli\\run-1\\checkpoint-9131\\config.json\n",
      "Model weights saved in test-nli\\run-1\\checkpoint-9131\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-1\\checkpoint-9131\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-1\\checkpoint-9131\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-1\\checkpoint-18262\n",
      "Configuration saved in test-nli\\run-1\\checkpoint-18262\\config.json\n",
      "Model weights saved in test-nli\\run-1\\checkpoint-18262\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-1\\checkpoint-18262\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-1\\checkpoint-18262\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\run-1\\checkpoint-27393\n",
      "Configuration saved in test-nli\\run-1\\checkpoint-27393\\config.json\n",
      "Model weights saved in test-nli\\run-1\\checkpoint-27393\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\run-1\\checkpoint-27393\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\run-1\\checkpoint-27393\\special_tokens_map.json\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "Loading best model from test-nli\\run-1\\checkpoint-27393 (score: 0.8710984558098784).\n",
      "\u001b[32m[I 2021-08-15 22:57:24,837]\u001b[0m Trial 1 finished with value: 0.8710984558098784 and parameters: {'learning_rate': 7.378529464222357e-05, 'num_train_epochs': 3, 'seed': 24, 'per_device_train_batch_size': 4}. Best is trial 0 with value: 0.8798598182017303.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "best_run = trainer.hyperparameter_search(n_trials=2, direction=\"maximize\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e9fbf61",
   "metadata": {},
   "source": [
    "성능이 가장 좋았던 hyper parameter를 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a4e9bbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BestRun(run_id='0', objective=0.8798598182017303, hyperparameters={'learning_rate': 4.5352366623370486e-05, 'num_train_epochs': 4, 'seed': 35, 'per_device_train_batch_size': 32})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "624c47b2",
   "metadata": {},
   "source": [
    "trainer에 성능이 가장 좋았던 hyper parameter로 설정해준 뒤 학습을 진행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1892c285",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file https://huggingface.co/klue/bert-base/resolve/main/config.json from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\fbd0b2ef898c4653902683fea8cc0dd99bf43f0e082645b913cda3b92429d1bb.7cee10e8ea7ffa278f8be4b141000263f2b18795e5ef5e025352b2af6851f8fb\n",
      "Model config BertConfig {\n",
      "  \"architectures\": [\n",
      "    \"BertForPretraining\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\",\n",
      "    \"1\": \"LABEL_1\",\n",
      "    \"2\": \"LABEL_2\",\n",
      "    \"3\": \"LABEL_3\",\n",
      "    \"4\": \"LABEL_4\",\n",
      "    \"5\": \"LABEL_5\",\n",
      "    \"6\": \"LABEL_6\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0,\n",
      "    \"LABEL_1\": 1,\n",
      "    \"LABEL_2\": 2,\n",
      "    \"LABEL_3\": 3,\n",
      "    \"LABEL_4\": 4,\n",
      "    \"LABEL_5\": 5,\n",
      "    \"LABEL_6\": 6\n",
      "  },\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.8.2\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 32000\n",
      "}\n",
      "\n",
      "loading weights file https://huggingface.co/klue/bert-base/resolve/main/pytorch_model.bin from cache at C:\\Users\\or7l0/.cache\\huggingface\\transformers\\05b36ee62545d769939a7746eca739b844a40a7a7553700f110b58b28ed6a949.7cb231256a5dbe886e12b902d05cb1241f330d8c19428508f91b2b28c1cfe0b6\n",
      "Some weights of the model checkpoint at klue/bert-base were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at klue/bert-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "***** Running training *****\n",
      "  Num examples = 36523\n",
      "  Num Epochs = 4\n",
      "  Instantaneous batch size per device = 32\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 32\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 4568\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='4568' max='4568' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [4568/4568 08:06, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.357700</td>\n",
       "      <td>0.336874</td>\n",
       "      <td>0.885664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.237800</td>\n",
       "      <td>0.355800</td>\n",
       "      <td>0.884679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.134700</td>\n",
       "      <td>0.439973</td>\n",
       "      <td>0.879750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.067300</td>\n",
       "      <td>0.533740</td>\n",
       "      <td>0.879860</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\checkpoint-1142\n",
      "Configuration saved in test-nli\\checkpoint-1142\\config.json\n",
      "Model weights saved in test-nli\\checkpoint-1142\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\checkpoint-1142\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\checkpoint-1142\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\checkpoint-2284\n",
      "Configuration saved in test-nli\\checkpoint-2284\\config.json\n",
      "Model weights saved in test-nli\\checkpoint-2284\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\checkpoint-2284\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\checkpoint-2284\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\checkpoint-3426\n",
      "Configuration saved in test-nli\\checkpoint-3426\\config.json\n",
      "Model weights saved in test-nli\\checkpoint-3426\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\checkpoint-3426\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\checkpoint-3426\\special_tokens_map.json\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n",
      "Saving model checkpoint to test-nli\\checkpoint-4568\n",
      "Configuration saved in test-nli\\checkpoint-4568\\config.json\n",
      "Model weights saved in test-nli\\checkpoint-4568\\pytorch_model.bin\n",
      "tokenizer config file saved in test-nli\\checkpoint-4568\\tokenizer_config.json\n",
      "Special tokens file saved in test-nli\\checkpoint-4568\\special_tokens_map.json\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "Loading best model from test-nli\\checkpoint-1142 (score: 0.8856642207863322).\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=4568, training_loss=0.2147540268463763, metrics={'train_runtime': 486.3823, 'train_samples_per_second': 300.365, 'train_steps_per_second': 9.392, 'total_flos': 2123068942884006.0, 'train_loss': 0.2147540268463763, 'epoch': 4.0})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for n, v in best_run.hyperparameters.items():\n",
    "    setattr(trainer.args, n, v)\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b089631",
   "metadata": {},
   "source": [
    "학습이 잘 되었는지 확인하고 예측을 진행한 뒤 결과를 제출해줍니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4e704194",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='572' max='286' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [286/286 00:15]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.3368736207485199,\n",
       " 'eval_accuracy': 0.8856642207863322,\n",
       " 'eval_runtime': 7.5415,\n",
       " 'eval_samples_per_second': 1210.761,\n",
       " 'eval_steps_per_second': 37.923,\n",
       " 'epoch': 4.0}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bc8f3feb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Prediction *****\n",
      "  Num examples = 9131\n",
      "  Batch size = 32\n"
     ]
    }
   ],
   "source": [
    "pred = trainer.predict(data_test)\n",
    "pred = pred[0]\n",
    "pred = np.argmax(pred,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6d6d4111",
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.read_csv('data/sample_submission.csv')\n",
    "submission['topic_idx'] = pred\n",
    "submission.to_csv(\"results/klue-bert-base-0810.csv\",index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7466b1",
   "metadata": {},
   "source": [
    "참고문헌  \n",
    "transformers 공식문서 How to fine-tune a model on text classification\n",
    "https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb  \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}