notebooks/tdc-tcr-epitope-binding-affinity-model.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Don't forget to first create and activate the conda environment.\n",
    "\n",
    "##### Or the [Docker/Apptainer](../README.md#running-things-in-a-container) if you prefer that instead\n",
    "\n",
    "The [conda environment](../README.md#creating-the-conda-environment) will install everything you need to run this notebook.\n",
    "\n",
    "```\n",
    "conda env create -f environment.yml\n",
    "conda activate tdc-tcr-epitope-binding-affinity-env\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#NOTE:\n",
    "This is not a [Resilience](https://resilience.com) project. The code/opinions/recommendations are mine and mine alone.\n",
    "\n",
    "# Bioinformatics model for protein therapeutics\n",
    "\n",
    "We'll use the [Therapeutics Data Commons](https://tdcommons.ai/) Python package to download open-source ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) datasets that are meaningful in pharmaceutical research. In this notebook, we'll use a dataset called [TCR-Epitope Binding Affinity](https://tdcommons.ai/multi_pred_tasks/tcrepitope/).\n",
    "\n",
    "![TCR-epitope binding](tcr-epitope-binding.png)\n",
    "\n",
    "We show how to create a deep learning model for predicting if a T-cell receptor (TCR) and protein epitope will bind to each other. A model that can predict how well a TCR binds to an epitope can lead to more effective treatments that use immunotherapy. For example, in anti-cancer therapies it is important for the T-cell receptor to bind to the protein marker in the cancer cell so that the T-cell (actually the T-cell's friends in the immune system) can kill the cancer cell.\n",
    "\n",
    "We'll see how to use HuggingFace's [transformers](https://huggingface.co/facebook/esm2_t36_3B_UR50D) package to run state-of-the-art AI models for embedding the protein sequences. In this case, we use Facebook's open-source [Evolutionary Scale Model (ESM-2)](https://github.com/facebookresearch/esm). These embeddings turn the protein sequences into a vector of numbers that the computer can use in a mathematical model. The vector of numbers uniquely encodes (aka embeds) a protein sequence in the same way that the [Dewey Decimal System](https://en.wikipedia.org/wiki/Dewey_Decimal_Classification) and [ISBN](https://en.wikipedia.org/wiki/ISBN) uniquely encode a book into a set of numbers (and letters). This representation is also referred to as a [latent space](https://en.wikipedia.org/wiki/Latent_space#:~:text=A%20latent%20space%2C%20also%20known,another%20in%20the%20latent%20space).\n",
    "\n",
    "Then, we'll show how to combine this embedding with a simple neural network to create a [binary classifier](https://en.wikipedia.org/wiki/Binary_classification) for the TCR-epitope binding affinity prediction (True=\"They Bind\", False=\"They don't bind\").\n",
    "\n",
    "![encoder-decoder Dewey Decimal](encoder-decoder.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the dataset\n",
    "\n",
    "We are using the TDC dataset for [TCR-Epitope Binding Affinity Prediction Task](https://tdcommons.ai/multi_pred_tasks/tcrepitope/).\n",
    "\n",
    "From the TDC website:\n",
    "\n",
    ">T-cells are an integral part of the adaptive immune system, whose survival, proliferation, activation and function are all governed by the interaction of their T-cell receptor (TCR) with immunogenic peptides (epitopes). A large repertoire of T-cell receptors with different specificity is needed to provide protection against a wide range of pathogens. This new task aims to predict the binding affinity given a pair of TCR sequence and epitope sequence.\n",
    ">\n",
    ">Weber et al.\n",
    "Dataset Description: The dataset is from Weber et al. who assemble a large and diverse data from the VDJ database and ImmuneCODE project. It uses human TCR-beta chain sequences. Since this dataset is highly imbalanced, the authors exclude epitopes with less than 15 associated TCR sequences and downsample to a limit of 400 TCRs per epitope. The dataset contains amino acid sequences either for the entire TCR or only for the hypervariable CDR3 loop. Epitopes are available as amino acid sequences. Since Weber et al. proposed to represent the peptides as SMILES strings (which reformulates the problem to protein-ligand binding prediction) the SMILES strings of the epitopes are also included. 50% negative samples were generated by shuffling the pairs, i.e. associating TCR sequences with epitopes they have not been shown to bind.\n",
    ">\n",
    ">Task Description: Binary classification. Given the epitope (a peptide, either represented as amino acid sequence or as SMILES) and a T-cell receptor (amino acid sequence, either of the full protein complex or only of the hypervariable CDR3 loop), predict whether the epitope binds to the TCR.\n",
    ">\n",
    ">Dataset Statistics: 47,182 TCR-Epitope pairs between 192 epitopes and 23,139 TCRs.\n",
    ">\n",
    ">References:\n",
    ">\n",
    "> 1. Weber, Anna, Jannis Born, and María Rodriguez Martínez. “TITAN: T-cell receptor specificity prediction with bimodal attention networks.” Bioinformatics 37.Supplement_1 (2021): i237-i244.\n",
    ">\n",
    "> 2. Bagaev, Dmitry V., et al. “VDJdb in 2019: database extension, new analysis infrastructure and a T-cell receptor motif compendium.” Nucleic Acids Research 48.D1 (2020): D1057-D1062.\n",
    ">\n",
    "> 3. Dines, Jennifer N., et al. “The immunerace study: A prospective multicohort study of immune response action to covid-19 events with the immunecode™ open access database.” medRxiv (2020).\n",
    ">\n",
    ">Dataset License: CC BY 4.0.\n",
    ">\n",
    ">Contributed by: Anna Weber and Jannis Born.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the TCR-epitope dataset\n",
    "\n",
    "Download and split randomly into 70% training data, 10% validation data, and 20% testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tdc.multi_pred import TCREpitopeBinding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found local copy...\n",
      "Loading...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "data = TCREpitopeBinding(name=\"weber\", path=\"./data\")  # Download the dataset\n",
    "split = data.get_split(\n",
    "    method=\"random\", seed=816, frac=[0.7, 0.1, 0.2]\n",
    ")  # Split the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train dataset size: \t\t33,028 proteins\n",
      "Validation dataset size: \t 4,718 proteins\n",
      "Test dataset size: \t\t 9,436 proteins\n"
     ]
    }
   ],
   "source": [
    "print(f\"Train dataset size: \\t\\t{len(split['train']):6,d} proteins\")\n",
    "print(f\"Validation dataset size: \\t{len(split['valid']):6,d} proteins\")\n",
    "print(f\"Test dataset size: \\t\\t{len(split['test']):6,d} proteins\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epitope_aa</th>\n",
       "      <th>epitope_smi</th>\n",
       "      <th>tcr</th>\n",
       "      <th>tcr_full</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FLKEKGGL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...</td>\n",
       "      <td>CSVWGTGKTYEQYF</td>\n",
       "      <td>SAVISQKPSRDICQRGTSLTIQCQVDSQVTMMFWYRQQPGQSLTLI...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FLKEKGGL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...</td>\n",
       "      <td>CSVWGEGRSYEQYF</td>\n",
       "      <td>SAVISQKPSRDICQRGTSLTIQCQVDSQVTMMFWYRQQPGQSLTLI...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FLKEKGGL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...</td>\n",
       "      <td>CSATILAGVPYGEQYF</td>\n",
       "      <td>GAVVSQHPSWVICKSGTSVKIECRSLDFQATTMFWYRQFPKQSLML...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>FLKEKGGL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...</td>\n",
       "      <td>CASSFDREVTGELFF</td>\n",
       "      <td>GAGVSQTPSNKVTEKGKYVELRCDPISGHTALYWYRQSLGQGPEFL...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FLKEKGGL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...</td>\n",
       "      <td>CASSVGAGTEAFF</td>\n",
       "      <td>DGGITQSPKYLFRKEGQNVTLSCEQNLNHDAMYWYRQDPGQGLRLI...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33023</th>\n",
       "      <td>KLMNIQQKL</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)[C@H](CC(N)=O)NC(=O)[C@H...</td>\n",
       "      <td>CASSKPGLTDTQYF</td>\n",
       "      <td>NAGVTQTPKFQVLKTGQSMTLQCAQDMNHEYMSWYRQDPGMGLRLI...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33024</th>\n",
       "      <td>TLIGDCATV</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)[C@H](CC(C)C)NC(=O)[C@@H...</td>\n",
       "      <td>CASSPGQGRTHYGYTF</td>\n",
       "      <td>NAGVTQTPKFRILKIGQSMTLQCAQDMNHNYMYWYRQDPGMGLKLI...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33025</th>\n",
       "      <td>LLFGYPVYV</td>\n",
       "      <td>CC(C)C[C@H](N)C(=O)N[C@@H](CC(C)C)C(=O)N[C@@H]...</td>\n",
       "      <td>CASSGGSLNTEAFF</td>\n",
       "      <td>NAGVTQTPKFQVLKTGQSMTLQCAQDMNHEYMSWYRQDPGMGLRLI...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33026</th>\n",
       "      <td>ISPRTLNAW</td>\n",
       "      <td>CC[C@H](C)[C@H](N)C(=O)N[C@@H](CO)C(=O)N1CCC[C...</td>\n",
       "      <td>CASSPSAAMNTEAFF</td>\n",
       "      <td>TVSWYQQALGQGPQFIFQYYREEENGRGNSPPRFSGLQFPNYSSEL...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33027</th>\n",
       "      <td>ILHCANFNV</td>\n",
       "      <td>CC[C@H](C)[C@H](N)C(=O)N[C@@H](CC(C)C)C(=O)N[C...</td>\n",
       "      <td>CASWGSPSSNTQYF</td>\n",
       "      <td>TVSWYQQALGQGPQFIFQYYREEENGRGNSPPRFSGLQFPNYSSEL...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33028 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      epitope_aa                                        epitope_smi  \\\n",
       "0       FLKEKGGL  CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...   \n",
       "1       FLKEKGGL  CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...   \n",
       "2       FLKEKGGL  CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...   \n",
       "3       FLKEKGGL  CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...   \n",
       "4       FLKEKGGL  CC(C)C[C@H](NC(=O)CNC(=O)CNC(=O)[C@H](CCCCN)NC...   \n",
       "...          ...                                                ...   \n",
       "33023  KLMNIQQKL  CC[C@H](C)[C@H](NC(=O)[C@H](CC(N)=O)NC(=O)[C@H...   \n",
       "33024  TLIGDCATV  CC[C@H](C)[C@H](NC(=O)[C@H](CC(C)C)NC(=O)[C@@H...   \n",
       "33025  LLFGYPVYV  CC(C)C[C@H](N)C(=O)N[C@@H](CC(C)C)C(=O)N[C@@H]...   \n",
       "33026  ISPRTLNAW  CC[C@H](C)[C@H](N)C(=O)N[C@@H](CO)C(=O)N1CCC[C...   \n",
       "33027  ILHCANFNV  CC[C@H](C)[C@H](N)C(=O)N[C@@H](CC(C)C)C(=O)N[C...   \n",
       "\n",
       "                    tcr                                           tcr_full  \\\n",
       "0        CSVWGTGKTYEQYF  SAVISQKPSRDICQRGTSLTIQCQVDSQVTMMFWYRQQPGQSLTLI...   \n",
       "1        CSVWGEGRSYEQYF  SAVISQKPSRDICQRGTSLTIQCQVDSQVTMMFWYRQQPGQSLTLI...   \n",
       "2      CSATILAGVPYGEQYF  GAVVSQHPSWVICKSGTSVKIECRSLDFQATTMFWYRQFPKQSLML...   \n",
       "3       CASSFDREVTGELFF  GAGVSQTPSNKVTEKGKYVELRCDPISGHTALYWYRQSLGQGPEFL...   \n",
       "4         CASSVGAGTEAFF  DGGITQSPKYLFRKEGQNVTLSCEQNLNHDAMYWYRQDPGQGLRLI...   \n",
       "...                 ...                                                ...   \n",
       "33023    CASSKPGLTDTQYF  NAGVTQTPKFQVLKTGQSMTLQCAQDMNHEYMSWYRQDPGMGLRLI...   \n",
       "33024  CASSPGQGRTHYGYTF  NAGVTQTPKFRILKIGQSMTLQCAQDMNHNYMYWYRQDPGMGLKLI...   \n",
       "33025    CASSGGSLNTEAFF  NAGVTQTPKFQVLKTGQSMTLQCAQDMNHEYMSWYRQDPGMGLRLI...   \n",
       "33026   CASSPSAAMNTEAFF  TVSWYQQALGQGPQFIFQYYREEENGRGNSPPRFSGLQFPNYSSEL...   \n",
       "33027    CASWGSPSSNTQYF  TVSWYQQALGQGPQFIFQYYREEENGRGNSPPRFSGLQFPNYSSEL...   \n",
       "\n",
       "       label  \n",
       "0          1  \n",
       "1          1  \n",
       "2          1  \n",
       "3          1  \n",
       "4          1  \n",
       "...      ...  \n",
       "33023      0  \n",
       "33024      0  \n",
       "33025      0  \n",
       "33026      0  \n",
       "33027      0  \n",
       "\n",
       "[33028 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = split[\"train\"]\n",
    "train_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What do these columns mean?\n",
    "\n",
    "The **epitope_aa** and the **tcr_full** columns are the protein (peptide) sequences for the epitope and the T-cell receptor, respectively. The letters correspond to the [standard amino acid codes](https://en.wikipedia.org/wiki/DNA_and_RNA_codon_tables).\n",
    "\n",
    "The **epitope_smi** column is the [SMILES](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system) notation for the chemical structure of the epitope. We won't use this information. Instead, the ESM-2 embedder should be sufficient for the input to our binary classification model.\n",
    "\n",
    "The **tcr** column is the CDR3 hyperloop. That's the part of the TCR that actually attaches to the epitope (assuming it binds).\n",
    "\n",
    "The **label** column is whether the two proteins bind. 0 = No. 1 = Yes.\n",
    "\n",
    "Our binary classification model should:\n",
    "* Use the **epitope_aa** and **tcr_full** embeddings as the input\n",
    "* Make a prediction if the **epitope_aa** and **tcr_full** will bind. This is the classification model's output. (0 = No, 1 = Yes)\n",
    "* Use the **label** as the ground truth of the binding (i.e. what the scientific experiment says)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The data is not shuffled\n",
    "\n",
    "In the original datasets, the rows are sorted by label. We should randomize the row order when training our classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomize row order\n",
    "train_data = split[\"train\"].sample(frac=1, random_state=816)\n",
    "validation_data = split[\"valid\"].sample(frac=1, random_state=816)\n",
    "test_data = split[\"test\"].sample(frac=1, random_state=816)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create embedding function\n",
    "\n",
    "Here we create an embedding function that we can use in the `pandas.apply` method."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting embedding vectors for the protein sequences\n",
    "\n",
    "HuggingFace provides a \"one-stop shop\" to train and deploy AI models. In this case, we use Facebook's open-source [Evolutionary Scale Model (ESM-2)](https://github.com/facebookresearch/esm). These embeddings turn the protein sequences into a vector of numbers that the computer can use in a mathematical model.\n",
    "\n",
    "https://huggingface.co/facebook/esm2_t36_3B_UR50D\n",
    "\n",
    "**Checkpoint name**|**Number of layers**|**Number of parameters**\n",
    ":-----:|:-----:|:-----:\n",
    "esm2\\_t48\\_15B\\_UR50D|\t48|\t15B\n",
    "esm2\\_t36\\_3B\\_UR50D|\t36|\t3B\n",
    "esm2\\_t33\\_650M\\_UR50D|\t33|\t650M\n",
    "esm2\\_t30\\_150M\\_UR50D|\t30|\t150M\n",
    "esm2\\_t12\\_35M\\_UR50D|\t12|\t35M\n",
    "esm2\\_t6\\_8M\\_UR50D|\t6|\t8M\n",
    "\n",
    "The Facebook ESM-2 model has the MIT license and was published in:\n",
    "\n",
    "> Zeming Lin et al, Evolutionary-scale prediction of atomic-level protein structure with a language model, Science (2023). DOI: 10.1126/science.ade2574  https://www.science.org/doi/10.1126/science.ade2574\n",
    "\n",
    "We're using the smaller ESM2 model simply because it won't take as long to complete the embedding. However, if you have the time, then try to see if the larger ESM2 models will give you and boost in performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU = True\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using GPU = {torch.cuda.is_available()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of EsmModel were not initialized from the model checkpoint at facebook/esm2_t6_8M_UR50D and are newly initialized: ['esm.pooler.dense.bias', 'esm.pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "EsmModel(\n",
       "  (embeddings): EsmEmbeddings(\n",
       "    (word_embeddings): Embedding(33, 320, padding_idx=1)\n",
       "    (dropout): Dropout(p=0.0, inplace=False)\n",
       "    (position_embeddings): Embedding(1026, 320, padding_idx=1)\n",
       "  )\n",
       "  (encoder): EsmEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0-5): 6 x EsmLayer(\n",
       "        (attention): EsmAttention(\n",
       "          (self): EsmSelfAttention(\n",
       "            (query): Linear(in_features=320, out_features=320, bias=True)\n",
       "            (key): Linear(in_features=320, out_features=320, bias=True)\n",
       "            (value): Linear(in_features=320, out_features=320, bias=True)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "            (rotary_embeddings): RotaryEmbedding()\n",
       "          )\n",
       "          (output): EsmSelfOutput(\n",
       "            (dense): Linear(in_features=320, out_features=320, bias=True)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "          )\n",
       "          (LayerNorm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)\n",
       "        )\n",
       "        (intermediate): EsmIntermediate(\n",
       "          (dense): Linear(in_features=320, out_features=1280, bias=True)\n",
       "        )\n",
       "        (output): EsmOutput(\n",
       "          (dense): Linear(in_features=1280, out_features=320, bias=True)\n",
       "          (dropout): Dropout(p=0.0, inplace=False)\n",
       "        )\n",
       "        (LayerNorm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)\n",
       "      )\n",
       "    )\n",
       "    (emb_layer_norm_after): LayerNorm((320,), eps=1e-05, elementwise_affine=True)\n",
       "  )\n",
       "  (pooler): EsmPooler(\n",
       "    (dense): Linear(in_features=320, out_features=320, bias=True)\n",
       "    (activation): Tanh()\n",
       "  )\n",
       "  (contact_head): EsmContactPredictionHead(\n",
       "    (regression): Linear(in_features=120, out_features=1, bias=True)\n",
       "    (activation): Sigmoid()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, AutoModel\n",
    "\n",
    "# model_name = \"facebook/esm2_t36_3B_UR50D\"  # 3 billion parameter model, embedding size 2560\n",
    "model_name = (\n",
    "    \"facebook/esm2_t6_8M_UR50D\"  # 8 million parameter model, embedding size 320\n",
    ")\n",
    "#model_name = \"facebook/esm2_t33_650M_UR50D\"\n",
    "esm_tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "esm_model = AutoModel.from_pretrained(model_name)\n",
    "esm_model = esm_model.to(device=device)\n",
    "esm_model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embeddings(sequence, model, tokenizer):\n",
    "    \"\"\"\n",
    "    Compute embeddings for each token in the sequence\n",
    "\n",
    "    Args:\n",
    "       sequence (str): The input protein sequence\n",
    "       model: The HuggingFace model\n",
    "       tokenizer: The HuggingFace tokenizer\n",
    "\n",
    "    Returns:\n",
    "      numpy.ndarray: A matrix where each row is the embedding of a token in the sequence\n",
    "    \"\"\"\n",
    "\n",
    "    inputs = tokenizer(\n",
    "        sequence, return_tensors=\"pt\", truncation=True, padding=True, max_length=1024\n",
    "    ).to(device)\n",
    "    with torch.no_grad():\n",
    "        outputs = model(**inputs)\n",
    "\n",
    "    # Return embeddings after removing CLS and EOS tokens.\n",
    "    # Only take the mean of the sequence so that all embeddings are the same length\n",
    "    # i.e. Otherwise they would be a 2D array where the first dimension is sequence length\n",
    "    return outputs.last_hidden_state[:, 1:-1, :].squeeze(0).mean(0).cpu().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sequence length is 30\n",
      "The shape of each embedding vector is: (320,)\n"
     ]
    }
   ],
   "source": [
    "# Sequence from: https://www.uniprot.org/uniprot/P58426\n",
    "sequence = \"DDCGKLFSGCDTNADCCEGYVCRLWCKLDW\"\n",
    "embedding = get_embeddings(sequence, esm_model, esm_tokenizer)\n",
    "\n",
    "print(f\"The sequence length is {len(sequence)}\")\n",
    "print(f\"The shape of each embedding vector is: {np.shape(embedding)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_embeddings(epitope_sequence: str, tcr_sequence: str, cdr3_sequence: str):\n",
    "    \"\"\"Calculate embeddings for different parts of the protein sequences\"\"\"\n",
    "\n",
    "    epitope_embedding = get_embeddings(epitope_sequence, esm_model, esm_tokenizer)\n",
    "    tcr_embedding = get_embeddings(tcr_sequence, esm_model, esm_tokenizer)\n",
    "    cdr3_embedding = get_embeddings(cdr3_sequence, esm_model, esm_tokenizer)\n",
    "\n",
    "    return pd.Series([epitope_embedding, tcr_embedding, cdr3_embedding])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get embedding for the protein sequences\n",
    "\n",
    "Returns the embedding vectors for the **epitope** and **tcr**. We'll use these embeddings as input to our model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding speed\n",
    "\n",
    "It takes over 2 hours to embed the training dataset using an NVidia RTX2000 Ada generation with model `esm2_t33_650M_UR50D` (650M parameters). The smaller `esm2_t6_8M_UR50D` (8M parameter) model takes about 30 minutes.\n",
    "\n",
    "#### Pre-trained embeddings\n",
    "\n",
    "The embeddings can take a long time to process. You can skip this step by downloading my pre-calculated ones from [Zenodo](https://zenodo.org/records/11894560)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(model_name, exist_ok=True)   # Create a directory to save the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tqdm.pandas(\n",
    "    desc=\"Embedding protein sequences\"\n",
    ")  # Create fancy progress bar for Pandas apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bf1978500244352bda3e17e2442dbf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Embedding protein sequences:   0%|          | 0/33028 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "training_data_filename = os.path.join(model_name, \"train_data.pkl\")\n",
    "if not os.path.exists(training_data_filename):\n",
    "    train_data[[\"epitope_vector\", \"tcr_vector\", \"cd3r_vector\"]] = (\n",
    "        train_data.progress_apply(\n",
    "            lambda x: get_all_embeddings(x[\"epitope_aa\"], x[\"tcr_full\"], x[\"tcr\"]),\n",
    "            axis=\"columns\",\n",
    "            result_type=\"expand\",\n",
    "        )\n",
    "    )\n",
    "\n",
    "    train_data.to_pickle(training_data_filename)\n",
    "else:\n",
    "    print(\"Loading previously saved train dataset embeddings.\")\n",
    "    train_data = pd.read_pickle(training_data_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de7762803b7a4abdb724105d8ce52a3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Embedding protein sequences:   0%|          | 0/4718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "validation_data_filename = os.path.join(model_name, \"validation_data.pkl\")\n",
    "if not os.path.exists(validation_data_filename):\n",
    "    validation_data[[\"epitope_vector\", \"tcr_vector\", \"cd3r_vector\"]] = (\n",
    "        validation_data.progress_apply(\n",
    "            lambda x: get_all_embeddings(x[\"epitope_aa\"], x[\"tcr_full\"], x[\"tcr\"]),\n",
    "            axis=\"columns\",\n",
    "            result_type=\"expand\",\n",
    "        )\n",
    "    )\n",
    "    validation_data.to_pickle(validation_data_filename)\n",
    "else:\n",
    "    print(\"Loading previously saved validation dataset embeddings.\")\n",
    "    validation_data = pd.read_pickle(validation_data_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "edf5f795eb3f42629c2cb35cac88dded",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Embedding protein sequences:   0%|          | 0/9436 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_data_filename = os.path.join(model_name, \"test_data.pkl\")\n",
    "if not os.path.exists(test_data_filename):\n",
    "    test_data[[\"epitope_vector\", \"tcr_vector\", \"cd3r_vector\"]] = (\n",
    "        test_data.progress_apply(\n",
    "            lambda x: get_all_embeddings(x[\"epitope_aa\"], x[\"tcr_full\"], x[\"tcr\"]),\n",
    "            axis=\"columns\",\n",
    "            result_type=\"expand\",\n",
    "        )\n",
    "    )\n",
    "    test_data.to_pickle(test_data_filename)\n",
    "else:\n",
    "    print(\"Loading previously saved test dataset embeddings.\")\n",
    "    test_data = pd.read_pickle(test_data_filename)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's look at our new dataframe with the protein embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epitope_aa</th>\n",
       "      <th>epitope_smi</th>\n",
       "      <th>tcr</th>\n",
       "      <th>tcr_full</th>\n",
       "      <th>label</th>\n",
       "      <th>epitope_vector</th>\n",
       "      <th>tcr_vector</th>\n",
       "      <th>cd3r_vector</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11113</th>\n",
       "      <td>LVLSVNPYV</td>\n",
       "      <td>CC(C)C[C@H](N)C(=O)N[C@H](C(=O)N[C@@H](CC(C)C)...</td>\n",
       "      <td>CASGAPSYEQYF</td>\n",
       "      <td>DAGITQSPRYKITETGRQVTLMCHQTWSHSYMFWYRQDLGHGLRLI...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.038534015, -0.32389918, 0.3308991, 0.134337...</td>\n",
       "      <td>[-0.28208765, -0.14748743, -0.099499434, 0.044...</td>\n",
       "      <td>[0.10006699, -0.24881284, 0.16725007, 0.158485...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27246</th>\n",
       "      <td>RLRPGGKKR</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)[C@@H](N)CCCNC(=N)N)C(=O)N[C...</td>\n",
       "      <td>CASSLLAGDFSYNEQFF</td>\n",
       "      <td>GAGVSQSPRYKVTKRGQDVALRCDPISGHVSLYWYRQALGQGPEFL...</td>\n",
       "      <td>0</td>\n",
       "      <td>[-0.0021749537, -0.5705189, 0.34297302, 0.6072...</td>\n",
       "      <td>[-0.2839003, -0.14871292, 0.082863465, 0.05347...</td>\n",
       "      <td>[0.07825382, -0.25411108, 0.16184156, 0.093944...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32541</th>\n",
       "      <td>TLDSKTQSL</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)[C@H](CO)NC(=O)[C@H](CCC(N)=...</td>\n",
       "      <td>CSVGWRQNTEAFF</td>\n",
       "      <td>TIQCQVDSQVTMIFWYRQQPGQSLTLIATANQGSEATYESGFVIDK...</td>\n",
       "      <td>0</td>\n",
       "      <td>[0.12294577, -0.2256919, 0.22606874, 0.0216736...</td>\n",
       "      <td>[-0.17999662, -0.13155857, 0.03158948, -0.1171...</td>\n",
       "      <td>[0.077383354, -0.26157582, 0.23503162, 0.17523...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2925</th>\n",
       "      <td>GILGFVFTL</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)CN)C(=O)N[C@@H](CC(C)C)C...</td>\n",
       "      <td>CASKARGSPLHF</td>\n",
       "      <td>EAQVTQNPRYLITVTGKKLTVTCSQNMNHEYMSWYRQDPGLGLRQI...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.1166945, -0.12083704, 0.19408055, 0.1212016...</td>\n",
       "      <td>[-0.34745646, -0.15962036, -0.021790126, 0.139...</td>\n",
       "      <td>[0.03470412, -0.3783253, 0.13022633, 0.2472901...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8283</th>\n",
       "      <td>ITEEVGHTDLMAAY</td>\n",
       "      <td>CC[C@H](C)[C@H](N)C(=O)N[C@H](C(=O)N[C@@H](CCC...</td>\n",
       "      <td>CASSQGPGGDGELFF</td>\n",
       "      <td>HLVMGMTNKKSLKCEQHMGHRAMYWYKQKAKKPPELMFVYSYEKLS...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.17735769, -0.085576214, 0.23910408, 0.02161...</td>\n",
       "      <td>[-0.29679978, -0.19048814, 0.009169238, 0.1410...</td>\n",
       "      <td>[0.083958775, -0.39204308, 0.22613832, 0.13092...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11255</th>\n",
       "      <td>KLSYGIATV</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)CNC(=O)[C@H](Cc1ccc(O)cc...</td>\n",
       "      <td>CASSRDYNTGELFF</td>\n",
       "      <td>TAGITQAPTSQILAAGRSMTLRCTQDMRHNAMYWYRQDLGLGLRLI...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.107314385, -0.15555699, 0.24414475, 0.14851...</td>\n",
       "      <td>[-0.23373993, -0.13337035, 0.007332661, -0.028...</td>\n",
       "      <td>[0.07891652, -0.24792679, 0.14765465, 0.163737...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19674</th>\n",
       "      <td>SFHSLHLLF</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)[C@H](CC(C)C)NC(=O)[C@H](Cc1...</td>\n",
       "      <td>CASSPRGSGESTDTQYF</td>\n",
       "      <td>HLVMGMTNKKSLKCEQHMGHRAMYWYKQKAKKPPELMFVYSYEKLS...</td>\n",
       "      <td>0</td>\n",
       "      <td>[-0.041375518, -0.39945617, 0.21541002, 0.1111...</td>\n",
       "      <td>[-0.30409825, -0.21730398, -0.065790065, 0.147...</td>\n",
       "      <td>[0.060847964, -0.36856344, 0.088047996, 0.1679...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15677</th>\n",
       "      <td>WICLLQFAY</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)[C@@H](N)Cc1c[nH]c2ccccc...</td>\n",
       "      <td>CASSAGTGGITDTQYF</td>\n",
       "      <td>DAEITQSPRHKITETGRQVTLACHQTWNHNNMFWYRQDLGHGLRLI...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.03950087, -0.3644118, 0.22049274, 0.2395057...</td>\n",
       "      <td>[-0.28494298, -0.17834686, 0.03679241, -0.0048...</td>\n",
       "      <td>[0.14439249, -0.15367289, 0.07951622, 0.108295...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>LLQTGIHVRVSQPSL</td>\n",
       "      <td>CC[C@H](C)[C@H](NC(=O)CNC(=O)[C@@H](NC(=O)[C@H...</td>\n",
       "      <td>CSASLDSGTQETQYF</td>\n",
       "      <td>GAVVSQHPSWVICKSGTSVKIECRSLDFQATTMFWYRQFPKQSLML...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.09716663, -0.35961112, 0.27731243, 0.097459...</td>\n",
       "      <td>[-0.3185343, -0.1504636, 0.018437518, -0.01153...</td>\n",
       "      <td>[0.06731486, -0.2654356, 0.16570343, 0.0855510...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5373</th>\n",
       "      <td>EPLPQGQLTAY</td>\n",
       "      <td>CC(C)C[C@H](NC(=O)[C@H](CCC(N)=O)NC(=O)CNC(=O)...</td>\n",
       "      <td>CASSLPGGAQGYTF</td>\n",
       "      <td>DVKVTQSSRYLVKRTGEKVFLECVQDMDHENMFWYRQDPGLGLRLI...</td>\n",
       "      <td>1</td>\n",
       "      <td>[0.13108458, -0.36076045, 0.34040436, 0.177657...</td>\n",
       "      <td>[-0.2682216, -0.15916963, 0.0871031, 0.1132328...</td>\n",
       "      <td>[0.089325994, -0.24473035, 0.15014848, 0.14057...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33028 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            epitope_aa                                        epitope_smi  \\\n",
       "11113        LVLSVNPYV  CC(C)C[C@H](N)C(=O)N[C@H](C(=O)N[C@@H](CC(C)C)...   \n",
       "27246        RLRPGGKKR  CC(C)C[C@H](NC(=O)[C@@H](N)CCCNC(=N)N)C(=O)N[C...   \n",
       "32541        TLDSKTQSL  CC(C)C[C@H](NC(=O)[C@H](CO)NC(=O)[C@H](CCC(N)=...   \n",
       "2925         GILGFVFTL  CC[C@H](C)[C@H](NC(=O)CN)C(=O)N[C@@H](CC(C)C)C...   \n",
       "8283    ITEEVGHTDLMAAY  CC[C@H](C)[C@H](N)C(=O)N[C@H](C(=O)N[C@@H](CCC...   \n",
       "...                ...                                                ...   \n",
       "11255        KLSYGIATV  CC[C@H](C)[C@H](NC(=O)CNC(=O)[C@H](Cc1ccc(O)cc...   \n",
       "19674        SFHSLHLLF  CC(C)C[C@H](NC(=O)[C@H](CC(C)C)NC(=O)[C@H](Cc1...   \n",
       "15677        WICLLQFAY  CC[C@H](C)[C@H](NC(=O)[C@@H](N)Cc1c[nH]c2ccccc...   \n",
       "164    LLQTGIHVRVSQPSL  CC[C@H](C)[C@H](NC(=O)CNC(=O)[C@@H](NC(=O)[C@H...   \n",
       "5373       EPLPQGQLTAY  CC(C)C[C@H](NC(=O)[C@H](CCC(N)=O)NC(=O)CNC(=O)...   \n",
       "\n",
       "                     tcr                                           tcr_full  \\\n",
       "11113       CASGAPSYEQYF  DAGITQSPRYKITETGRQVTLMCHQTWSHSYMFWYRQDLGHGLRLI...   \n",
       "27246  CASSLLAGDFSYNEQFF  GAGVSQSPRYKVTKRGQDVALRCDPISGHVSLYWYRQALGQGPEFL...   \n",
       "32541      CSVGWRQNTEAFF  TIQCQVDSQVTMIFWYRQQPGQSLTLIATANQGSEATYESGFVIDK...   \n",
       "2925        CASKARGSPLHF  EAQVTQNPRYLITVTGKKLTVTCSQNMNHEYMSWYRQDPGLGLRQI...   \n",
       "8283     CASSQGPGGDGELFF  HLVMGMTNKKSLKCEQHMGHRAMYWYKQKAKKPPELMFVYSYEKLS...   \n",
       "...                  ...                                                ...   \n",
       "11255     CASSRDYNTGELFF  TAGITQAPTSQILAAGRSMTLRCTQDMRHNAMYWYRQDLGLGLRLI...   \n",
       "19674  CASSPRGSGESTDTQYF  HLVMGMTNKKSLKCEQHMGHRAMYWYKQKAKKPPELMFVYSYEKLS...   \n",
       "15677   CASSAGTGGITDTQYF  DAEITQSPRHKITETGRQVTLACHQTWNHNNMFWYRQDLGHGLRLI...   \n",
       "164      CSASLDSGTQETQYF  GAVVSQHPSWVICKSGTSVKIECRSLDFQATTMFWYRQFPKQSLML...   \n",
       "5373      CASSLPGGAQGYTF  DVKVTQSSRYLVKRTGEKVFLECVQDMDHENMFWYRQDPGLGLRLI...   \n",
       "\n",
       "       label                                     epitope_vector  \\\n",
       "11113      1  [0.038534015, -0.32389918, 0.3308991, 0.134337...   \n",
       "27246      0  [-0.0021749537, -0.5705189, 0.34297302, 0.6072...   \n",
       "32541      0  [0.12294577, -0.2256919, 0.22606874, 0.0216736...   \n",
       "2925       1  [0.1166945, -0.12083704, 0.19408055, 0.1212016...   \n",
       "8283       1  [0.17735769, -0.085576214, 0.23910408, 0.02161...   \n",
       "...      ...                                                ...   \n",
       "11255      1  [0.107314385, -0.15555699, 0.24414475, 0.14851...   \n",
       "19674      0  [-0.041375518, -0.39945617, 0.21541002, 0.1111...   \n",
       "15677      1  [0.03950087, -0.3644118, 0.22049274, 0.2395057...   \n",
       "164        1  [0.09716663, -0.35961112, 0.27731243, 0.097459...   \n",
       "5373       1  [0.13108458, -0.36076045, 0.34040436, 0.177657...   \n",
       "\n",
       "                                              tcr_vector  \\\n",
       "11113  [-0.28208765, -0.14748743, -0.099499434, 0.044...   \n",
       "27246  [-0.2839003, -0.14871292, 0.082863465, 0.05347...   \n",
       "32541  [-0.17999662, -0.13155857, 0.03158948, -0.1171...   \n",
       "2925   [-0.34745646, -0.15962036, -0.021790126, 0.139...   \n",
       "8283   [-0.29679978, -0.19048814, 0.009169238, 0.1410...   \n",
       "...                                                  ...   \n",
       "11255  [-0.23373993, -0.13337035, 0.007332661, -0.028...   \n",
       "19674  [-0.30409825, -0.21730398, -0.065790065, 0.147...   \n",
       "15677  [-0.28494298, -0.17834686, 0.03679241, -0.0048...   \n",
       "164    [-0.3185343, -0.1504636, 0.018437518, -0.01153...   \n",
       "5373   [-0.2682216, -0.15916963, 0.0871031, 0.1132328...   \n",
       "\n",
       "                                             cd3r_vector  \n",
       "11113  [0.10006699, -0.24881284, 0.16725007, 0.158485...  \n",
       "27246  [0.07825382, -0.25411108, 0.16184156, 0.093944...  \n",
       "32541  [0.077383354, -0.26157582, 0.23503162, 0.17523...  \n",
       "2925   [0.03470412, -0.3783253, 0.13022633, 0.2472901...  \n",
       "8283   [0.083958775, -0.39204308, 0.22613832, 0.13092...  \n",
       "...                                                  ...  \n",
       "11255  [0.07891652, -0.24792679, 0.14765465, 0.163737...  \n",
       "19674  [0.060847964, -0.36856344, 0.088047996, 0.1679...  \n",
       "15677  [0.14439249, -0.15367289, 0.07951622, 0.108295...  \n",
       "164    [0.06731486, -0.2654356, 0.16570343, 0.0855510...  \n",
       "5373   [0.089325994, -0.24473035, 0.15014848, 0.14057...  \n",
       "\n",
       "[33028 rows x 8 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epitope_vector</th>\n",
       "      <th>tcr_vector</th>\n",
       "      <th>cd3r_vector</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1966</th>\n",
       "      <td>[0.11756068, -0.17726119, 0.21559499, 0.226084...</td>\n",
       "      <td>[-0.2496988, -0.14253776, 0.040706888, 0.08691...</td>\n",
       "      <td>[0.07543397, -0.30791205, 0.15727155, 0.087190...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9408</th>\n",
       "      <td>[0.21328215, -0.06046487, 0.269702, 0.00173665...</td>\n",
       "      <td>[-0.32340783, -0.2267902, -0.012520636, 0.0924...</td>\n",
       "      <td>[0.04712369, -0.27774197, 0.14300615, 0.160680...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3168</th>\n",
       "      <td>[0.06054182, -0.31700128, 0.38526642, 0.282358...</td>\n",
       "      <td>[-0.26972562, -0.15555282, -0.044835523, 0.055...</td>\n",
       "      <td>[0.10473794, -0.22698924, 0.1669027, 0.1547409...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>956</th>\n",
       "      <td>[0.031733107, -0.38563797, 0.22444288, 0.13873...</td>\n",
       "      <td>[-0.2774681, -0.18532652, 0.059007324, 0.01290...</td>\n",
       "      <td>[0.10494517, -0.25884837, 0.15787368, 0.101360...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5034</th>\n",
       "      <td>[0.16600582, -0.32457992, 0.3533009, 0.1887973...</td>\n",
       "      <td>[-0.2218102, -0.102710515, 0.027087376, 0.0079...</td>\n",
       "      <td>[-0.0027738106, -0.1857487, 0.064010814, 0.176...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3290</th>\n",
       "      <td>[0.109696895, -0.31173432, 0.3281438, 0.103866...</td>\n",
       "      <td>[-0.33872184, -0.18544455, 0.03181725, 0.09333...</td>\n",
       "      <td>[0.06117232, -0.3019955, 0.2630141, 0.169023, ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5774</th>\n",
       "      <td>[0.046975505, -0.2649276, 0.23652273, 0.294588...</td>\n",
       "      <td>[-0.2833794, -0.21230242, 0.044284813, -0.0083...</td>\n",
       "      <td>[0.060584165, -0.23498924, 0.14196908, 0.11568...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>[0.015819773, 0.0018905882, 0.29173946, 0.1501...</td>\n",
       "      <td>[-0.2566507, -0.14838167, 0.09135117, 0.015357...</td>\n",
       "      <td>[0.042209413, -0.29709765, 0.21791954, 0.23540...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5373</th>\n",
       "      <td>[0.107314385, -0.15555699, 0.24414475, 0.14851...</td>\n",
       "      <td>[-0.27858907, -0.11695308, 0.06962588, 0.06203...</td>\n",
       "      <td>[0.1383814, -0.20685391, 0.13446483, 0.1321696...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8804</th>\n",
       "      <td>[0.14689134, -0.26006117, 0.24947928, 0.295612...</td>\n",
       "      <td>[-0.3175784, -0.15996759, -0.043165814, 0.1267...</td>\n",
       "      <td>[0.033301324, -0.3357919, 0.15983474, 0.163346...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9436 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         epitope_vector  \\\n",
       "1966  [0.11756068, -0.17726119, 0.21559499, 0.226084...   \n",
       "9408  [0.21328215, -0.06046487, 0.269702, 0.00173665...   \n",
       "3168  [0.06054182, -0.31700128, 0.38526642, 0.282358...   \n",
       "956   [0.031733107, -0.38563797, 0.22444288, 0.13873...   \n",
       "5034  [0.16600582, -0.32457992, 0.3533009, 0.1887973...   \n",
       "...                                                 ...   \n",
       "3290  [0.109696895, -0.31173432, 0.3281438, 0.103866...   \n",
       "5774  [0.046975505, -0.2649276, 0.23652273, 0.294588...   \n",
       "164   [0.015819773, 0.0018905882, 0.29173946, 0.1501...   \n",
       "5373  [0.107314385, -0.15555699, 0.24414475, 0.14851...   \n",
       "8804  [0.14689134, -0.26006117, 0.24947928, 0.295612...   \n",
       "\n",
       "                                             tcr_vector  \\\n",
       "1966  [-0.2496988, -0.14253776, 0.040706888, 0.08691...   \n",
       "9408  [-0.32340783, -0.2267902, -0.012520636, 0.0924...   \n",
       "3168  [-0.26972562, -0.15555282, -0.044835523, 0.055...   \n",
       "956   [-0.2774681, -0.18532652, 0.059007324, 0.01290...   \n",
       "5034  [-0.2218102, -0.102710515, 0.027087376, 0.0079...   \n",
       "...                                                 ...   \n",
       "3290  [-0.33872184, -0.18544455, 0.03181725, 0.09333...   \n",
       "5774  [-0.2833794, -0.21230242, 0.044284813, -0.0083...   \n",
       "164   [-0.2566507, -0.14838167, 0.09135117, 0.015357...   \n",
       "5373  [-0.27858907, -0.11695308, 0.06962588, 0.06203...   \n",
       "8804  [-0.3175784, -0.15996759, -0.043165814, 0.1267...   \n",
       "\n",
       "                                            cd3r_vector  label  \n",
       "1966  [0.07543397, -0.30791205, 0.15727155, 0.087190...      0  \n",
       "9408  [0.04712369, -0.27774197, 0.14300615, 0.160680...      0  \n",
       "3168  [0.10473794, -0.22698924, 0.1669027, 0.1547409...      0  \n",
       "956   [0.10494517, -0.25884837, 0.15787368, 0.101360...      1  \n",
       "5034  [-0.0027738106, -0.1857487, 0.064010814, 0.176...      1  \n",
       "...                                                 ...    ...  \n",
       "3290  [0.06117232, -0.3019955, 0.2630141, 0.169023, ...      1  \n",
       "5774  [0.060584165, -0.23498924, 0.14196908, 0.11568...      1  \n",
       "164   [0.042209413, -0.29709765, 0.21791954, 0.23540...      0  \n",
       "5373  [0.1383814, -0.20685391, 0.13446483, 0.1321696...      1  \n",
       "8804  [0.033301324, -0.3357919, 0.15983474, 0.163346...      1  \n",
       "\n",
       "[9436 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data[[\"epitope_vector\", \"tcr_vector\", \"cd3r_vector\", \"label\"]]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We want to merge the input vectors\n",
    "\n",
    "We have 2 inputs for our model: **epitope_vector** and **tcr_vector**. We have many choices of how to use these in the model. \n",
    "\n",
    "There's no real \"right\" way to put them together so let's just merge the epitope and TCR vectors into a single vector. \n",
    "\n",
    "This will be the input to our binary classification model.\n",
    "\n",
    "You can try any modifications you want if you don't like to concatenate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data[\"input\"] = train_data.apply(\n",
    "    lambda x: np.concatenate((x[\"epitope_vector\"], x[\"tcr_vector\"])), axis=\"columns\"\n",
    ")\n",
    "validation_data[\"input\"] = validation_data.apply(\n",
    "    lambda x: np.concatenate((x[\"epitope_vector\"], x[\"tcr_vector\"])), axis=\"columns\"\n",
    ")\n",
    "test_data[\"input\"] = test_data.apply(\n",
    "    lambda x: np.concatenate((x[\"epitope_vector\"], x[\"tcr_vector\"])), axis=\"columns\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pull out the X and y datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train_data[\"input\"]\n",
    "y_train = train_data[\"label\"]\n",
    "\n",
    "X_validation = validation_data[\"input\"]\n",
    "y_validation = validation_data[\"label\"]\n",
    "\n",
    "X_test = test_data[\"input\"]\n",
    "y_test = test_data[\"label\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We should scale the input data\n",
    "\n",
    "[StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) just removes the mean and normalizes by the standard deviation. It's usually a good idea for pre-processing the input dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "ss_train = StandardScaler()\n",
    "X_train = ss_train.fit_transform(train_data[\"input\"].to_list())\n",
    "\n",
    "ss_validation = StandardScaler()\n",
    "X_validation = ss_validation.fit_transform(validation_data[\"input\"].to_list())\n",
    "\n",
    "ss_test = StandardScaler()\n",
    "X_test = ss_test.fit_transform(test_data[\"input\"].to_list())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use XGBoost to create binary classifier\n",
    "\n",
    "[XGBoost](https://xgboost.readthedocs.io/en/stable/get_started.html) is a good Python library to train models using Gradient Boosted Trees. We'll use this to create our binary classifier. If you have a GPU, then you can add the `tree_method=\"gpu_hist\"` parameter. Training on the GPU takes only seconds. Training on the CPU takes ~6 minutes.\n",
    "\n",
    "NOTE: Scikit-learn is another good choice for training models. However, I found that it took ~ 45 minutes to train a SVM classifier and the results weren't any better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU acceleration\n"
     ]
    }
   ],
   "source": [
    "from xgboost import XGBClassifier\n",
    "\n",
    "from torch import cuda\n",
    "\n",
    "if cuda.is_available():  # GPU is available to use\n",
    "    print(\"Using GPU acceleration\")\n",
    "    clf = XGBClassifier(\n",
    "        tree_method=\"gpu_hist\", random_state=816, verbosity=2\n",
    "    )  # If GPU is available\n",
    "else:  # No GPU\n",
    "    clf = XGBClassifier(random_state=816, verbosity=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>XGBClassifier(base_score=None, booster=None, callbacks=None,\n",
       "              colsample_bylevel=None, colsample_bynode=None,\n",
       "              colsample_bytree=None, device=None, early_stopping_rounds=None,\n",
       "              enable_categorical=False, eval_metric=None, feature_types=None,\n",
       "              gamma=None, grow_policy=None, importance_type=None,\n",
       "              interaction_constraints=None, learning_rate=None, max_bin=None,\n",
       "              max_cat_threshold=None, max_cat_to_onehot=None,\n",
       "              max_delta_step=None, max_depth=None, max_leaves=None,\n",
       "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
       "              multi_strategy=None, n_estimators=None, n_jobs=None,\n",
       "              num_parallel_tree=None, random_state=816, ...)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;XGBClassifier<span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>XGBClassifier(base_score=None, booster=None, callbacks=None,\n",
       "              colsample_bylevel=None, colsample_bynode=None,\n",
       "              colsample_bytree=None, device=None, early_stopping_rounds=None,\n",
       "              enable_categorical=False, eval_metric=None, feature_types=None,\n",
       "              gamma=None, grow_policy=None, importance_type=None,\n",
       "              interaction_constraints=None, learning_rate=None, max_bin=None,\n",
       "              max_cat_threshold=None, max_cat_to_onehot=None,\n",
       "              max_delta_step=None, max_depth=None, max_leaves=None,\n",
       "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
       "              multi_strategy=None, n_estimators=None, n_jobs=None,\n",
       "              num_parallel_tree=None, random_state=816, ...)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "XGBClassifier(base_score=None, booster=None, callbacks=None,\n",
       "              colsample_bylevel=None, colsample_bynode=None,\n",
       "              colsample_bytree=None, device=None, early_stopping_rounds=None,\n",
       "              enable_categorical=False, eval_metric=None, feature_types=None,\n",
       "              gamma=None, grow_policy=None, importance_type=None,\n",
       "              interaction_constraints=None, learning_rate=None, max_bin=None,\n",
       "              max_cat_threshold=None, max_cat_to_onehot=None,\n",
       "              max_delta_step=None, max_depth=None, max_leaves=None,\n",
       "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
       "              multi_strategy=None, n_estimators=None, n_jobs=None,\n",
       "              num_parallel_tree=None, random_state=816, ...)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate how the model performs on unseen data\n",
    "\n",
    "We'll print out the [classification report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) and [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix). These are calculated on the test_data. The model was never trained on the test_data so it is a good approximation of how the model will perform on future unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASS_NAMES = [\"No bind\", \"Bind\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import (\n",
    "    confusion_matrix,\n",
    "    ConfusionMatrixDisplay,\n",
    "    classification_report,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     No bind       0.73      0.75      0.74      4708\n",
      "        Bind       0.75      0.72      0.74      4728\n",
      "\n",
      "    accuracy                           0.74      9436\n",
      "   macro avg       0.74      0.74      0.74      9436\n",
      "weighted avg       0.74      0.74      0.74      9436\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions_gradient = clf.predict(X_test)\n",
    "\n",
    "print(classification_report(y_test, predictions_gradient, target_names=CLASS_NAMES))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAHFCAYAAAAg3/mzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABMkUlEQVR4nO3deVyU1f4H8M8zLDOIgIEKqIgoooILCi5oue9LLqUoppJYuRSZqbiDK1pqLom7oqYlN5dyLXPpaqKJYi6QmYqggiAuKAoInN8f/pjrOIAzMDCD83n3el73zpnznOc7MMKX7znnGUkIIUBERESkJzJ9B0BERETGjckIERER6RWTESIiItIrJiNERESkV0xGiIiISK+YjBAREZFeMRkhIiIivWIyQkRERHrFZISIiIj0iskI6UV4eDgkSYJCocDNmzfVnm/bti3q16+vh8h0w9/fHzVq1FBpq1GjBvz9/Us1jri4OEiShPDwcI36X79+HZ9++inc3NxgYWGBcuXKwcPDA9OmTcPt27dLPNYePXrA1tYWkiRh7NixOr+GPr4HAHDs2DFIklTo96J9+/aQJEntfaOpbdu2YcmSJVqdo+37g6ikmOo7ADJumZmZmDZtGrZs2aLvUErcrl27YG1tre8wCrR3714MHDgQFStWxKefforGjRtDkiRcvHgRGzZswL59+xAdHV1i1//iiy9w+vRpbNiwAQ4ODnB0dNT5NfT9PbCyssL69evVEqIbN27g2LFjxYpt27ZtuHTpklZJnKOjIyIjI1GrVq0iX5dIF5iMkF517doV27Ztw/jx49GoUaMSu86zZ89gYWFRYuNronHjxnq9fmFu3LiBgQMHws3NDUePHoWNjY3yufbt2yMwMBC7du0q0RguXbqEZs2aoU+fPiV2DX1/D3x9fbFu3TpcvXoVtWvXVrZv2LABVatWRYMGDRATE1PiceTk5CA7OxtyuRwtWrQo8esRvQ6naUivJk6cCDs7OwQFBb22b0ZGBiZPngwXFxeYm5ujatWqGDNmDB4+fKjSr0aNGujZsyd27tyJxo0bQ6FQYObMmcpS+bZt2xAUFARHR0eUL18evXr1wt27d/H48WN8/PHHqFixIipWrIgPP/wQT548URl7xYoVaN26NSpXrgxLS0s0aNAAX331FZ4/f/7a+F+dImjbtq2ydP/q8XLZPCkpCZ988gmqVasGc3NzuLi4YObMmcjOzlYZ/86dOxgwYACsrKxgY2MDX19fJCUlvTYuAFi8eDHS09MRFhamkojkkSQJ/fr1U2nbsGEDGjVqBIVCAVtbW/Tt2xexsbEqffz9/VG+fHn8+++/6N69O8qXLw8nJyd8+eWXyMzMBPC/KYx///0XBw4cUH4N4uLilNN5cXFxKuPmnXPs2DFlW3R0NHr27InKlStDLpejSpUq6NGjB27duqXsk980TXx8PD744APlefXq1cOiRYuQm5ur7JM3nbFw4UIsXrwYLi4uKF++PHx8fHDq1CmNvsYA0KlTJzg5OWHDhg3KttzcXGzatAnDhg2DTKb+I1mT91zbtm2xb98+3Lx5U+V99HLsX331FebMmQMXFxfI5XIcPXpUbZomIyMDjRs3hqurKx49eqQcPykpCQ4ODmjbti1ycnI0fr1EmmJlhPTKysoK06ZNw+eff44jR46gffv2+fYTQqBPnz44fPgwJk+ejHfeeQcXLlxAcHAwIiMjERkZCblcrux/7tw5xMbGYtq0aXBxcYGlpSXS09MBAFOmTEG7du0QHh6OuLg4jB8/HoMGDYKpqSkaNWqE77//HtHR0ZgyZQqsrKywbNky5bjXrl2Dn5+fMiH666+/MHfuXPz9998qv2A0ERYWhrS0NJW26dOn4+jRo6hTpw6AF78EmjVrBplMhhkzZqBWrVqIjIzEnDlzEBcXh40bNwJ4Ufnp2LEj7ty5g9DQULi5uWHfvn3w9fXVKJZff/0V9vb2Gv+VHBoaiilTpmDQoEEIDQ1FamoqQkJC4OPjgzNnzqj81f/8+XO8++67CAgIwJdffon//ve/mD17NmxsbDBjxgw0adIEkZGR6Nu3L2rVqoWFCxcCgFbTNOnp6ejUqRNcXFywYsUK2NvbIykpCUePHsXjx48LPC8lJQUtW7ZEVlYWZs+ejRo1amDv3r0YP348rl27hrCwMJX+K1asQN26dZVrM6ZPn47u3bvjxo0b+SZxr5LJZPD398f69esxZ84cmJiY4Ndff8WtW7fw4Ycf4vPPP1c7R5P3XFhYGD7++GNcu3atwArWsmXL4ObmhoULF8La2lrle5RHoVAgIiICXl5eGD58OHbs2IHc3FwMHjwYQgh8//33MDExee3rJNKaINKDjRs3CgDizJkzIjMzU9SsWVN4e3uL3NxcIYQQbdq0ER4eHsr+Bw8eFADEV199pTLO9u3bBQCxZs0aZZuzs7MwMTERV65cUel79OhRAUD06tVLpX3s2LECgAgMDFRp79Onj7C1tS3wNeTk5Ijnz5+LzZs3CxMTE3H//n3lc8OGDRPOzs4q/Z2dncWwYcMKHO/rr79Wey2ffPKJKF++vLh586ZK34ULFwoA4vLly0IIIVauXCkAiJ9++kml30cffSQAiI0bNxZ4XSGEUCgUokWLFoX2yfPgwQNhYWEhunfvrtIeHx8v5HK58PPzU7YNGzZMABAREREqfbt37y7q1Kmj0ubs7Cx69Oih0pb3Prlx44ZKe9738ujRo0IIIaKiogQAsXv37kJjf/V7MGnSJAFAnD59WqXfqFGjhCRJyvfQjRs3BADRoEEDkZ2drez3559/CgDi+++/L/S6efH+5z//EdevXxeSJIm9e/cKIYTo37+/aNu2rRBCiB49eqi9b15W2HuuoHPzYq9Vq5bIysrK97lX3x95/66WLFkiZsyYIWQymfj1118LfY1ExcFpGtI7c3NzzJkzB1FRUYiIiMi3z5EjRwBArcTev39/WFpa4vDhwyrtDRs2hJubW75j9ezZU+VxvXr1AAA9evRQa79//77KVE10dDTeffdd2NnZwcTEBGZmZhg6dChycnLwzz//vP7FFuD777/HxIkTMW3aNHz00UfK9r1796Jdu3aoUqUKsrOzlUe3bt0AAL///jsA4OjRo7CyssK7776rMq6fn1+RYypIZGQknj17pva9cHJyQvv27dW+F5IkoVevXiptDRs2zHcXVVG5urrirbfeQlBQEFatWqXxuosjR47A3d0dzZo1U2n39/eHEEL5vsvTo0cPlcpAw4YNAUCr1+Li4oK2bdtiw4YNSE1NxU8//YThw4cX2F9X77l3330XZmZmGvUdMGAARo0ahQkTJmDOnDmYMmUKOnXqpPG1iLTFZIQMwsCBA9GkSRNMnTo13/UXqampMDU1RaVKlVTaJUmCg4MDUlNTVdoLK/Hb2tqqPDY3Ny+0PSMjA8CLtQXvvPMObt++jaVLl+L48eM4c+YMVqxYAeDFVElRHD16FP7+/hg6dChmz56t8tzdu3exZ88emJmZqRweHh4AgHv37gF48fWxt7dXG9vBwUGjGKpXr44bN25o1Dfva53f17hKlSpq34ty5cpBoVCotMnlcuXXVRdsbGzw+++/w9PTE1OmTIGHhweqVKmC4ODgQtfzpKamFvg68p5/mZ2dncrjvKlBbb/3AQEB2LNnDxYvXgwLCwu8//77+fbT5XtO291Jw4cPx/Pnz2FqaorAwECtziXSFteMkEGQJAkLFixAp06dsGbNGrXn7ezskJ2djZSUFJWERAiBpKQkNG3aVG08Xdu9ezfS09Oxc+dOODs7K9vPnz9f5DEvXLiAPn36oE2bNli7dq3a8xUrVkTDhg0xd+7cfM/P+6VpZ2eHP//8U+15TRewdunSBcuXL8epU6deu24k7xdyYmKi2nN37txBxYoVNbqmJvKSmLzFrnnykrCXNWjQAD/88AOEELhw4QLCw8Mxa9YsWFhYYNKkSfmOb2dnV+DrAKDT1/Kyfv36YcyYMZg/fz4++uijAnd66fI9p82/ifT0dAwZMgRubm64e/cuRowYgZ9++knraxJpipURMhgdO3ZEp06dMGvWLLVdLB06dAAAfPfddyrtO3bsQHp6uvL5kpT3w/zlhbJCiHyTCE3Ex8ejW7duqFmzJnbs2JFvCb1nz564dOkSatWqBW9vb7UjLxlp164dHj9+jJ9//lnl/G3btmkUyxdffAFLS0uMHj1aZRdFHiGEcmGkj48PLCws1L4Xt27dwpEjR3T6vci7AdiFCxdU2l99nS+TJAmNGjXCN998gwoVKuDcuXMF9u3QoQNiYmLU+mzevBmSJKFdu3ZFD74QFhYWmDFjBnr16oVRo0YV2E+b95xcLi9yde5VI0eORHx8PHbu3In169fj559/xjfffKOTsYnyw8oIGZQFCxbAy8sLycnJyqkI4MWWyC5duiAoKAhpaWlo1aqVcjdN48aNMWTIkBKPrVOnTjA3N8egQYMwceJEZGRkYOXKlXjw4EGRxuvWrRsePnyIb7/9FpcvX1Z5rlatWqhUqRJmzZqFQ4cOoWXLlggMDESdOnWQkZGBuLg47N+/H6tWrUK1atUwdOhQfPPNNxg6dCjmzp2L2rVrY//+/fjll180isXFxQU//PADfH194enpqbzpGQDExMRgw4YNEEKgb9++qFChAqZPn44pU6Zg6NChGDRoEFJTUzFz5kwoFAoEBwcX6euRn6ZNm6JOnToYP348srOz8dZbb2HXrl04ceKESr+9e/ciLCwMffr0Qc2aNSGEwM6dO/Hw4cNC1zp88cUX2Lx5M3r06IFZs2bB2dkZ+/btQ1hYGEaNGlXguiNdGDduHMaNG1doH23ecw0aNMDOnTuxcuVKeHl5QSaTwdvbW+u41q1bh++++w4bN26Eh4cHPDw88OmnnyIoKAitWrVSW19DpBP6WztLxuzl3TSv8vPzEwBUdtMIIcSzZ89EUFCQcHZ2FmZmZsLR0VGMGjVKPHjwQKVffrsyhFDd0aBJLMHBwQKASElJUbbt2bNHNGrUSCgUClG1alUxYcIEceDAAZWdHUJotpsGQIHHy7sbUlJSRGBgoHBxcRFmZmbC1tZWeHl5ialTp4onT54o+926dUu89957onz58sLKykq899574uTJkxrtpslz7do1MXr0aOHq6irkcrmwsLAQ7u7uYty4cWo7WtatWycaNmwozM3NhY2Njejdu7dyd8/LXwdLS0u16+R9bV/9+uT3ffvnn39E586dhbW1tahUqZL47LPPxL59+1S+5n///bcYNGiQqFWrlrCwsBA2NjaiWbNmIjw8XO0ar+5ounnzpvDz8xN2dnbCzMxM1KlTR3z99dciJydH2Sdv18nXX3+tFh8AERwcrNb+soLee6/Kb0eMpu+5+/fvi/fff19UqFBBSJKk/PoWFvuru2kuXLggLCws1L5GGRkZwsvLS9SoUUPt3xuRLkhCCFGKuQ8RERGRCq4ZISIiIr1iMkJERER6xWSEiIiI9IrJCBEREekVkxEiIiLSKyYjREREpFe86Vkpy83NxZ07d2BlZVUitywnIqKSJYTA48ePUaVKFchkJfM3fUZGBrKysnQylrm5udrnQxkaJiOl7M6dO3ByctJ3GEREVEwJCQmoVq2azsfNyMiAhZUdkP1UJ+M5ODjgxo0bBp2QMBkpZVZWVgAAc/dhkEzM9RwNUcmIP7ZQ3yEQlZjHaWlwdXFS/jzXtaysLCD7KeTuw4Di/p7IyUJSzCZkZWUxGaH/yZuakUzMmYzQG8va2lrfIRCVuBKfajdVFPv3hJDKxtJQJiNERESGSAJQ3ISnjCxNZDJCRERkiCTZi6O4Y5QBZSNKIiIiemOxMkJERGSIJEkH0zRlY56GyQgREZEh4jQNERERUelgZYSIiMgQcZqGiIiI9EsH0zRlZAKkbERJREREbyxWRoiIiAwRp2mIiIhIr7ibhoiIiKh0sDJCRERkiDhNQ0RERHplRNM0TEaIiIgMkRFVRspGykRERERvLFZGiIiIDBGnaYiIiEivJEkHyQinaYiIiIhei5URIiIiQySTXhzFHaMMYDJCRERkiIxozUjZiJKIiIjeWKyMEBERGSIjus8IkxEiIiJDxGkaIiIiotLByggREZEh4jQNERER6ZURTdMwGSEiIjJERlQZKRspExEREb2xWBkhIiIyRJymISIiIr3iNA0RERFR6WAyQkREZJBk/5uqKepRhF/zYWFhcHFxgUKhgJeXF44fP15gX39/f0iSpHZ4eHho+0qJiIjI4ORN0xT30ML27dsxduxYTJ06FdHR0XjnnXfQrVs3xMfH59t/6dKlSExMVB4JCQmwtbVF//79tboukxEiIiICACxevBgBAQEYMWIE6tWrhyVLlsDJyQkrV67Mt7+NjQ0cHByUR1RUFB48eIAPP/xQq+syGSEiIjJEklT8aRotKiNZWVk4e/YsOnfurNLeuXNnnDx5UqMx1q9fj44dO8LZ2Vmrl8rdNERERIZIh1t709LSVJrlcjnkcrlK271795CTkwN7e3uVdnt7eyQlJb32UomJiThw4AC2bdumdZisjBAREb3hnJycYGNjozxCQ0ML7Cu9Uk0RQqi15Sc8PBwVKlRAnz59tI6PlREiIiJDpMP7jCQkJMDa2lrZ/GpVBAAqVqwIExMTtSpIcnKyWrXkVUIIbNiwAUOGDIG5ubnWYbIyQkREZIiKvV7kf9M81tbWKkd+yYi5uTm8vLxw6NAhlfZDhw6hZcuWhYb6+++/499//0VAQECRXiorI0RERIZID3dgHTduHIYMGQJvb2/4+PhgzZo1iI+Px8iRIwEAkydPxu3bt7F582aV89avX4/mzZujfv36RQqTyQgREREBAHx9fZGamopZs2YhMTER9evXx/79+5W7YxITE9XuOfLo0SPs2LEDS5cuLfJ1mYwQEREZIj19UN7o0aMxevTofJ8LDw9Xa7OxscHTp0+1vs7LmIwQEREZIn5QHhEREVHpYGWEiIjIAOV96FwxB9FNMCWMyQgREZEBMqZkhNM0REREpFesjBARERki6f+P4o5RBjAZISIiMkCcpiEiIiIqJayMEBERGSBjqowwGSEiIjJATEaIiIhIr4wpGeGaESIiItIrVkaIiIgMEbf2EhERkT5xmoaIiIiolLAyQkREZIAkCTqojOgmlpLGZISIiMgASdDBNE0ZyUY4TUNERER6xcoIERGRATKmBaxMRoiIiAyREW3t5TQNERER6RUrI0RERIZIB9M0gtM0REREVFS6WDNS/N04pYPJCBERkQEypmSEa0aIiIhIr1gZISIiMkRGtJuGyQgREZEB4jQNERERUSlhZYSIiMgAGVNlhMkIERGRATKmZITTNERERKRXrIwQEREZIGOqjDAZISIiMkRGtLWX0zRERESkV6yMEBERGSBO0xAREZFeMRkhIiIivTKmZIRrRoiIiEivWBkhIiIyREa0m4bJCBERkQHiNA0RERFRKXljKyMhISHYvXs3zp8/X2Aff39/PHz4ELt37y7WtSRJwq5du9CnT59ijUOaCXj/HXz2QQfYV7TB39cTMWXxDkSev5Zv3xXBH8CvZwu19tjriWjpOxcAMKhnc4QFD1Hr49BqLDKzsnUbPNFrrPvPf7H8u8O4e+8R6tZ0xLxx76FlY9d8++45ch4bdhzHxX9uI+t5NurWdEDQR93Rwcdd2WfbnlMYM+s7tXMTT3wDhdysxF4HFZ8xVUb0moz4+/tj06ZNCA0NxaRJk5Ttu3fvRt++fSGEKNHrL126tMSvQbrVt1MTzBv3HsYv2I7Tf12Hf7+3EbF0NHwGzMGtuw/U+k9e+CNmfvuT8rGpiQmOb52Mn36LVumX9uQZmr4/S6WNiQiVtp2/nsWUxTuwMMgXzRvVRPjOExjweRgiI6bBycFWrf/J6H/RtnldTB/9LmysLLB1zykMGrcav4WPR8M6Tsp+VpYKnPlxhsq5TEQMnwQdJCNlZNGI3qdpFAoFFixYgAcP1H+RlDQbGxtUqFCh1K9LRTfarz2++ykSW36KxD9xdzFl8Q7cvvsAw99/J9/+aekZSE59rDw861VHBWsLbNsTqdJPCKHSLzn1cWm8HCIVYduO4IPePhjapyXquDgg9Mv3UdX+LWz48Xi+/UO/fB+fD+2EJh7OqFW9MmaMeRe1nCrh4H8vqfSTJAn2Fa1VDiJDovdkpGPHjnBwcEBoaGih/Xbs2AEPDw/I5XLUqFEDixYt0mj81atXw8nJCeXKlUP//v3x8OFD5XP+/v4qUytt27ZFYGAgJk6cCFtbWzg4OCAkJERlvKtXr6J169ZQKBRwd3fHoUOHNH2pVExmpibwrOuEI6djVdqPno5Fs4YuGo0xpLcPjv15BQlJqsmvpYUcF36ehUt7Z+OHxSPRwK2azuIm0kTW82yc/zsB7ZvXU2lv17we/rxwQ6MxcnNz8fhpJirYlFNpT3+WiQa9psOjxzT4frESF64k6CxuKjl50zTFPcoCvScjJiYmmDdvHpYvX45bt27l2+fs2bMYMGAABg4ciIsXLyIkJATTp09HeHh4oWP/+++/iIiIwJ49e3Dw4EGcP38eY8aMKfScTZs2wdLSEqdPn8ZXX32FWbNmKROO3Nxc9OvXDyYmJjh16hRWrVqFoKCgIr1u0p5dhfIwNTVByn3VqkVK6mNUtnv9X3r2dtbo6OOOLT+dVGm/GncXY2Z9B78vV+OjaeHIyHqOg+vHoaZTJZ3GT1SY1IdPkJOTi0q2VirtleyskJyaptEY3249gqcZmejbsYmyrXYNe6yY8QG2LfoEa+f4Q2Fuhq4Bi3EtPlmn8VMJkHR0lAEGsYC1b9++8PT0RHBwMNavX6/2/OLFi9GhQwdMnz4dAODm5oaYmBh8/fXX8Pf3L3DcjIwMbNq0CdWqvfgrd/ny5ejRowcWLVoEBweHfM9p2LAhgoODAQC1a9fGt99+i8OHD6NTp0747bffEBsbi7i4OOWY8+bNQ7du3QqMITMzE5mZmcrHaWma/VChgr26zEeSJI3W/vj1aoFHT55h37ELKu1Rl+IQdSlO+fjUX9fx+3dB+HhAG0xa9KMuQibS2Kt/yAohNPrr9sdforBgzX5sXfixSkLTtIELmjb4X+WwRaOaaPPBAqyJ+B0LxvfXWdxExaH3ykieBQsWYNOmTYiJiVF7LjY2Fq1atVJpa9WqFa5evYqcnJwCx6xevboyaQAAHx8f5Obm4sqVKwWe07BhQ5XHjo6OSE5OVsaR35iFCQ0NhY2NjfJwcnIqtD8VLPXhE2Rn56CynepfjhVty6tVS/IzuFcLbN//J55nF/yeAV788D8XcxO1qrMyQqXHrkJ5mJjI1NYr3bv/RK1a8qqdv55F4Oyt2BA6HG2b1y20r0wmQxN3Z1yLTyl2zFSyOE2jB61bt0aXLl0wZcoUtefy+8ugKLtg8sYo7JtjZqa6wlySJOTm5hZ4zdd9oydPnoxHjx4pj4QEztUW1fPsHJz/OwHtXvlh27ZZ3dfOqbdqUhu1qlfGdz9HFtovTwO3aki6xyoWlR5zM1N41nXC0dN/q7Qf+/PvQtdE/fhLFMbM+g5r5/ijy9v1X3sdIQQu/nMLDlzEavCMKRkxiGmaPPPnz4enpyfc3NxU2t3d3XHixAmVtpMnT8LNzQ0mJiYFjhcfH487d+6gSpUqAIDIyEjIZDK18TXl7u6e75iFkcvlkMvlRboeqQvbdgSrZg5FdEw8zly8gWF9W6Gagy027nix22DGmHfhWMkGo0K2qJw3pLcPzly8gdhriWpjThzRDVGX4nAtIRlWlgp84tsWDdyqYcKCiFJ5TUR5Rvu1x8jgzWjsXh1NG7hg064/cCvpPj5878VusZnf/oTElEdYNXMogBeJyKjgzQj98n1413fB3f9PoBUKM9iUtwAALFi7H971a6CWU2U8Ts/A6u3HcPGfW/g6aIB+XiRpTJLUp+2KMkZZYFDJSIMGDTB48GAsX75cpf3LL79E06ZNMXv2bPj6+iIyMhLffvstwsLCCh1PoVBg2LBhWLhwIdLS0hAYGIgBAwYUuF7kdTp27Ig6depg6NChWLRoEdLS0jB16tQijUVFs+vQOdjaWGLiiG6wr2iN2GuJ8B0bptwdY1/RGtVeuR+DtaUCvdp7YnIB6z9srCywZMogVLazQtqTDFy4cgs9Pl6CczE3S/z1EL2sX2cv3H+Ujq/WHcDde2moV8sR25eMRnXHF+/pu/fScCvpvrJ/+M4TyM7JxYSvIjDhq/8lz4N6NEdYyIsb+T16/Axj532P5NTHsC6vQMM61bBvzVh4edQo1ddGVBiDSkYAYPbs2YiIUP2LtEmTJoiIiMCMGTMwe/ZsODo6YtasWYUuXgUAV1dX9OvXD927d8f9+/fRvXv31yYwhZHJZNi1axcCAgLQrFkz1KhRA8uWLUPXrl2LPCZpb/2Px7G+gPsujJmpfqfJtPQMVH1nXIHjTf1mJ6Z+s1Nn8REVx4j+rTGif+t8n8tLMPLsXT32tePNG/ce5o17TxehUSl7URkp7h1YdRRMCZMEb0FaqtLS0mBjYwN5g48gmZjrOxyiEvHgzLf6DoGoxKSlpcHezgaPHj2CtbXu197k/Z6oGfgjTOSWxRorJzMd15e9X2Kx6orBLGAlIiIi42Rw0zRERETED8ojIiIiPTOm3TScpiEiIiK9YmWEiIjIAMlkEmSy4pU2RDHPLy1MRoiIiAwQp2mIiIiISgkrI0RERAaIu2mIiIhIrzhNQ0RERHqlr0/tDQsLg4uLCxQKBby8vHD8eP4fv5EnMzMTU6dOhbOzM+RyOWrVqoUNGzZodU1WRoiIiAgAsH37dowdOxZhYWFo1aoVVq9ejW7duiEmJgbVq1fP95wBAwbg7t27WL9+PVxdXZGcnIzs7GytrstkhIiIyADpY83I4sWLERAQgBEjRgAAlixZgl9++QUrV65EaGioWv+DBw/i999/x/Xr12Fr++LTpWvUqKF1nJymISIiMkB5a0aKewAvPnzv5SMzM1PtellZWTh79iw6d+6s0t65c2ecPHky3xh//vlneHt746uvvkLVqlXh5uaG8ePH49mzZ1q9VlZGiIiI3nBOTk4qj4ODgxESEqLSdu/ePeTk5MDe3l6l3d7eHklJSfmOe/36dZw4cQIKhQK7du3CvXv3MHr0aNy/f1+rdSNMRoiIiAyQBB1M0+DF+QkJCbC2tla2y+Xygs955ZpCiALjyM3NhSRJ2Lp1K2xsbAC8mOp5//33sWLFClhYWGgUJ5MRIiIiA6TLrb3W1tYqyUh+KlasCBMTE7UqSHJyslq1JI+joyOqVq2qTEQAoF69ehBC4NatW6hdu7ZGcXLNCBEREcHc3BxeXl44dOiQSvuhQ4fQsmXLfM9p1aoV7ty5gydPnijb/vnnH8hkMlSrVk3jazMZISIiMkD6uM/IuHHjsG7dOmzYsAGxsbH44osvEB8fj5EjRwIAJk+ejKFDhyr7+/n5wc7ODh9++CFiYmLw3//+FxMmTMDw4cM1nqIBOE1DRERkkPRxB1ZfX1+kpqZi1qxZSExMRP369bF//344OzsDABITExEfH6/sX758eRw6dAifffYZvL29YWdnhwEDBmDOnDlaXZfJCBERESmNHj0ao0ePzve58PBwtba6deuqTe1oi8kIERGRAeIH5REREZFeGdMH5TEZISIiMkDGVBnhbhoiIiLSK1ZGiIiIDJEOpmlQNgojTEaIiIgMEadpiIiIiEoJKyNEREQGiLtpiIiISK84TUNERERUSlgZISIiMkCcpiEiIiK94jQNERERUSlhZYSIiMgAGVNlhMkIERGRAeKaESIiItIrY6qMcM0IERER6RUrI0RERAaI0zRERESkV5ymISIiIiolrIwQEREZIAk6mKbRSSQlj8kIERGRAZJJEmTFzEaKe35p4TQNERER6RUrI0RERAaIu2mIiIhIr4xpNw2TESIiIgMkk14cxR2jLOCaESIiItIrVkaIiIgMkaSDaZYyUhlhMkJERGSAjGkBK6dpiIiISK9YGSEiIjJA0v//V9wxygImI0RERAaIu2mIiIiISgkrI0RERAaINz17xbJlyzQeMDAwsMjBEBER0QvGtJtGo2Tkm2++0WgwSZKYjBAREZFWNEpGbty4UdJxEBER0UtkkgRZMUsbxT2/tBR5AWtWVhauXLmC7OxsXcZDRERE+N80TXGPskDrZOTp06cICAhAuXLl4OHhgfj4eAAv1orMnz9f5wESEREZo7wFrMU9ygKtk5HJkyfjr7/+wrFjx6BQKJTtHTt2xPbt23UaHBEREb35tN7au3v3bmzfvh0tWrRQybjc3d1x7do1nQZHRERkrLibphApKSmoXLmyWnt6enqZKQcREREZOi5gLUTTpk2xb98+5eO8BGTt2rXw8fHRXWRERERkFLSujISGhqJr166IiYlBdnY2li5disuXLyMyMhK///57ScRIRERkdKT/P4o7RlmgdWWkZcuW+OOPP/D06VPUqlULv/76K+zt7REZGQkvL6+SiJGIiMjoGNNumiJ9Nk2DBg2wadMmXcdCRERERqhIyUhOTg527dqF2NhYSJKEevXqoXfv3jA15efuERER6YJMenEUd4yyQOvs4dKlS+jduzeSkpJQp04dAMA///yDSpUq4eeff0aDBg10HiQREZGxMaZP7dV6zciIESPg4eGBW7du4dy5czh37hwSEhLQsGFDfPzxxyURIxEREb3BtK6M/PXXX4iKisJbb72lbHvrrbcwd+5cNG3aVKfBERERGbMyUtgoNq0rI3Xq1MHdu3fV2pOTk+Hq6qqToIiIiIwdd9O8Ii0tTfn/582bh8DAQISEhKBFixYAgFOnTmHWrFlYsGBByURJRERkZLiA9RUVKlRQya6EEBgwYICyTQgBAOjVqxdycnJKIEwiIiJ6U2mUjBw9erSk4yAiIqKXGNNuGo2SkTZt2pR0HERERPQSY7odfJHvUvb06VPEx8cjKytLpb1hw4bFDoqIiIiMh9bJSEpKCj788EMcOHAg3+e5ZoSIiKj4ZJIEWTGnWYp7fmnRemvv2LFj8eDBA5w6dQoWFhY4ePAgNm3ahNq1a+Pnn38uiRiJiIiMjiTp5igLtK6MHDlyBD/99BOaNm0KmUwGZ2dndOrUCdbW1ggNDUWPHj1KIk4iIiJ6Q2ldGUlPT0flypUBALa2tkhJSQHw4pN8z507p9voiIiIjJQx3fSsSHdgvXLlCgDA09MTq1evxu3bt7Fq1So4OjrqPEAiIiJjxGmaQowdOxaJiYkAgODgYHTp0gVbt26Fubk5wsPDdR0fERERveG0rowMHjwY/v7+AIDGjRsjLi4OZ86cQUJCAnx9fXUdHxERkVHK201T3ENbYWFhcHFxgUKhgJeXF44fP15g32PHjuU7NfT3339rdc0i32ckT7ly5dCkSZPiDkNEREQv0cU0i7bnb9++HWPHjkVYWBhatWqF1atXo1u3boiJiUH16tULPO/KlSuwtrZWPq5UqZJW19UoGRk3bpzGAy5evFirAIiIiEidPm4Hv3jxYgQEBGDEiBEAgCVLluCXX37BypUrERoaWuB5lStXRoUKFYocp0bJSHR0tEaDlZVVu0RERMYkLS1N5bFcLodcLldpy8rKwtmzZzFp0iSV9s6dO+PkyZOFjt+4cWNkZGTA3d0d06ZNQ7t27bSKjx+UpyexB+fD6qWSFtGb5K2Os/UdAlGJEdkZpXIdGYqwsDOfMQDAyclJpT04OBghISEqbffu3UNOTg7s7e1V2u3t7ZGUlJTv+I6OjlizZg28vLyQmZmJLVu2oEOHDjh27Bhat26tcZzFXjNCREREuqfLaZqEhASVNR2vVkXyOyePEKLAOOrUqYM6deooH/v4+CAhIQELFy7UKhkpbtJFREREBs7a2lrlyC8ZqVixIkxMTNSqIMnJyWrVksK0aNECV69e1So+JiNEREQGSJIAWTEPbQor5ubm8PLywqFDh1TaDx06hJYtW2o8TnR0tNY3QeU0DRERkQHKSyiKO4Y2xo0bhyFDhsDb2xs+Pj5Ys2YN4uPjMXLkSADA5MmTcfv2bWzevBnAi902NWrUgIeHB7KysvDdd99hx44d2LFjh1bXZTJCREREAABfX1+kpqZi1qxZSExMRP369bF//344OzsDABITExEfH6/sn5WVhfHjx+P27duwsLCAh4cH9u3bh+7du2t1XUkIIbQNdsuWLVi1ahVu3LiByMhIODs7Y8mSJXBxcUHv3r21Hc6opKWlwcbGBtdvp3I3Db2xnHoWfD8CorJOZGcg8495ePTokcqiUF3J+z0x5ocoyMuVL9ZYmU+fYMVA7xKLVVe0XjOycuVKjBs3Dt27d8fDhw+Rk5MDAKhQoQKWLFmi6/iIiIiMUnHXi+himqe0aJ2MLF++HGvXrsXUqVNhYmKibPf29sbFixd1GhwRERG9+bReM3Ljxg00btxYrV0ulyM9PV0nQRERERk7fXw2jb5oXRlxcXHB+fPn1doPHDgAd3d3XcRERERk9PT1qb36oHVlZMKECRgzZgwyMjIghMCff/6J77//HqGhoVi3bl1JxEhERGR0dHk7eEOndTLy4YcfIjs7GxMnTsTTp0/h5+eHqlWrYunSpRg4cGBJxEhERERvsCLdZ+Sjjz7CRx99hHv37iE3NxeVK1fWdVxERERGzZjWjBTrpmcVK1bUVRxERET0EhmKv+ZDhrKRjWidjLi4uBT6KYLXr18vVkBERERkXLRORsaOHavy+Pnz54iOjsbBgwcxYcIEXcVFRERk1DhNU4jPP/883/YVK1YgKiqq2AERERGRfj4oT190tuunW7duWn9KHxEREZHOPrX3xx9/hK2tra6GIyIiMmqShGIvYH1jp2kaN26ssoBVCIGkpCSkpKQgLCxMp8EREREZK64ZKUSfPn1UHstkMlSqVAlt27ZF3bp1dRUXERERGQmtkpHs7GzUqFEDXbp0gYODQ0nFREREZPS4gLUApqamGDVqFDIzM0sqHiIiIgIg6ei/skDr3TTNmzdHdHR0ScRCRERE/y+vMlLcoyzQes3I6NGj8eWXX+LWrVvw8vKCpaWlyvMNGzbUWXBERET05tM4GRk+fDiWLFkCX19fAEBgYKDyOUmSIISAJEnIycnRfZRERERGxpjWjGicjGzatAnz58/HjRs3SjIeIiIiwos/9Av7LDhNxygLNE5GhBAAAGdn5xILhoiIiIyPVmtGykqGRUREVNZxmqYAbm5ur01I7t+/X6yAiIiIiHdgLdDMmTNhY2NTUrEQERGREdIqGRk4cCAqV65cUrEQERHR/5NJUrE/KK+455cWjZMRrhchIiIqPca0ZkTjO7Dm7aYhIiIi0iWNKyO5ubklGQcRERG9TAcLWMvIR9Nofzt4IiIiKnkySJAVM5so7vmlhckIERGRATKmrb1af2ovERERkS6xMkJERGSAjGk3DZMRIiIiA2RM9xnhNA0RERHpFSsjREREBsiYFrAyGSEiIjJAMuhgmqaMbO3lNA0RERHpFSsjREREBojTNERERKRXMhR/+qKsTH+UlTiJiIjoDcXKCBERkQGSJAlSMedZint+aWEyQkREZIAkFP9Dd8tGKsJkhIiIyCDxDqxEREREpYSVESIiIgNVNuoaxcdkhIiIyAAZ031GOE1DREREesXKCBERkQHi1l4iIiLSK96BlYiIiKiUsDJCRERkgDhNQ0RERHplTHdg5TQNERER6RUrI0RERAaI0zRERESkV8a0m4bJCBERkQEypspIWUmaiIiI6A3FyggREZEBMqbdNExGiIiIDBA/KI+IiIiolDAZISIiMkAySDo5tBUWFgYXFxcoFAp4eXnh+PHjGp33xx9/wNTUFJ6enlpfk8kIERGRAcqbpinuoY3t27dj7NixmDp1KqKjo/HOO++gW7duiI+PL/S8R48eYejQoejQoUORXiuTESIiIgIALF68GAEBARgxYgTq1auHJUuWwMnJCStXriz0vE8++QR+fn7w8fEp0nWZjBARERkgSUf/aSorKwtnz55F586dVdo7d+6MkydPFnjexo0bce3aNQQHBxf5tXI3DRERkQHS5W6atLQ0lXa5XA65XK7Sdu/ePeTk5MDe3l6l3d7eHklJSfmOf/XqVUyaNAnHjx+HqWnRUwpWRoiIiN5wTk5OsLGxUR6hoaEF9n31rq1CiHzv5JqTkwM/Pz/MnDkTbm5uxYqPlREiIiIDJBVxN8yrYwBAQkICrK2tle2vVkUAoGLFijAxMVGrgiQnJ6tVSwDg8ePHiIqKQnR0ND799FMAQG5uLoQQMDU1xa+//or27dtrFCeTESIiIgOky2kaa2trlWQkP+bm5vDy8sKhQ4fQt29fZfuhQ4fQu3dvtf7W1ta4ePGiSltYWBiOHDmCH3/8ES4uLhrHyWSEiIjIAOnjDqzjxo3DkCFD4O3tDR8fH6xZswbx8fEYOXIkAGDy5Mm4ffs2Nm/eDJlMhvr166ucX7lyZSgUCrX212EyQkRERAAAX19fpKamYtasWUhMTET9+vWxf/9+ODs7AwASExNfe8+RopCEEELno1KB0tLSYGNjg+u3U2H1mpIZUVnl1LPgxXFEZZ3IzkDmH/Pw6NGj1059FEXe74ldf16HZXmrYo2V/uQx+jarWWKx6gorI0RERAZIJr04ijtGWcCtvURERKRXrIwQEREZIG3voFrQGGUBkxEiIiIDpI/dNPrCaRoiIiLSK1ZGiIiIDJCE4k+zlJHCCJMRIiIiQ8TdNERERESlxCgrI3FxcXBxcUF0dDQ8PT2LPE5ISAh2796N8+fP6yw2er1Nu05g9fdHkJyaBrcaDggO7IvmjWrl2/fA739hy+4/cPnqbWQ9z4abiwO++LAr2javp9JvXcQxbNn9B27ffQjbCpbo3qYRJn3SEwq5WWm8JCKlgHe98Fl/H9jbWeHvuBRMCfsFkZcS8u27YsK78OvSSK09Ni4FLUesAgAM7d4YAzs1RL0alQAA568mYvb6ozh35U7JvQjSCWPaTfNGVkb8/f0hSZLysLOzQ9euXXHhwgUALz5KOe82t1S2/Hz4HGYu24XPhnTCgfXj0axRTQydsBq37z7It//pv67hHe862PT1J9i/bjx8GtfG8EnrcOmfW8o+u36NwvzVezH2w644+t0kfB00EHuORGP+6r2l9bKIAAB927pj3qguWLTtBNqMXIvIi/GICPVDtcr53zlzctgvqNN/sfLwGLgE99Oe4qf/xij7vN3IGTuOXkKv8VvQOXAjbiWnYeeCwXC0K96dPank5e2mKe5RFryRyQgAdO3aFYmJiUhMTMThw4dhamqKnj17AgBMTEzg4OAAU1OjLAyVaWu3H4Nvj+YY1MsHtWs4ICSwH6pUroAtu07k2z8ksB9GDe4Az3rV4eJUCZM+6QmXapXw2x+XlH3OXo6Dd30X9O3kBSdHO7RpVhe9OzbBhSv5/zVKVFJGv9cC3x2MxpYD5/FP/D1MWfkrbienYXgv73z7p6VnIvlBuvLwdKuCCuUtsO3gX8o+H4fuxvqfz+LStbu4mpCKzxfvhSRJaN1E809UJf2QdHSUBW9sMiKXy+Hg4AAHBwd4enoiKCgICQkJSElJQVxcHCRJUk6vHDt2DJIk4fDhw/D29ka5cuXQsmVLXLlyRWXM+fPnw97eHlZWVggICEBGRoYeXpnxynqejYv/3ELrZnVV2ls3rYuoS3EajZGbm4snTzNQwdpS2da0QU1c/CcB0TE3AQA379zD0VMx6ODjrrPYiV7HzFQGTzdHHIm6rtJ+9Ow1NHOvptEYQ7p54ti560hIflRgn3JyM5iZyvAw7Vmx4iXSpTc2GXnZkydPsHXrVri6usLOzq7AflOnTsWiRYsQFRUFU1NTDB8+XPlcREQEgoODMXfuXERFRcHR0RFhYWGvvXZmZibS0tJUDiqa+4/SkZOTi0pvqZaXK75lhZT7mn1d1/xwDE8zstCzvaeyrXfHJhgf0B3vjVkGl7bj8LbvHPg0ro0xH3TUZfhEhbKzKQdTExlSHqSrtKc8SEdl2/KvPd/etjw6NnPFlgPnC+0XPKI9Eu89xrFz1wvtR/ongwSZVMyjjNRG3th5ir1796J8+Rf/gNPT0+Ho6Ii9e/dCJis4/5o7dy7atGkDAJg0aRJ69OiBjIwMKBQKLFmyBMOHD8eIESMAAHPmzMFvv/322upIaGgoZs6cqaNXRYD6HKiAgKTBxOju385i8caDWB8agIovJTSR0VexfMshzB33PjzdnRF3+x5Clu7EEjtrjPXvouvwiQr16gepS5Kk1pYfvy6N8OhJBvb98XeBfQIH+OC9dvXR68vNyHyeU+xYqWTpYpqlbKQib3BlpF27djh//jzOnz+P06dPo3PnzujWrRtu3rxZ4DkNGzZU/n9HR0cAQHJyMgAgNjYWPj4+Kv1ffZyfyZMn49GjR8ojIYHrEIrK1sYSJiYyJN9/rNKe+uCJSnKRn58Pn8OE+T9g5cxheMe7jspzX687gH6dvTGolw/q1aqCbq0bIujjnljx3W/Izc3V+esgyk/qo6fIzslVq4JUrFBOrVqSn8FdGmH7bxfwPDv/9+yn/VtgnN/b6DdpKy7fSNZJzES68sYmI5aWlnB1dYWrqyuaNWuG9evXIz09HWvXri3wHDOz/23jzPtLu7i/jORyOaytrVUOKhpzM1M0cKuG42dU1/IcP3MF3vVrFHje7t/OYty877F8xhB0aOmh9nxGRhZkr9wZSGYiQQhAgz9IiXTieXYuzv+TiHZeNVXa23rVxJ8xtwo464VWjZxRq5odvitgiuazAT6Y8ME7eH/yNpz/J1FXIVNJM6IVrG9sMvIqSZIgk8nw7FnRFm3Vq1cPp06dUml79TGVvI982+KHvafww75TuBqXhJBlu3A7+QE+6NMKADB/1R6MnfOdsv/u387iizlbMf3T3mjiUQPJqWlITk1D2pP/vQ86tvLAlt1/4KffziH+Tir+e+YKFq47gE5ve8DExGj+iZABCNtxCkO6Ncbgro3gVr0i5o7qhGqVbbBxz1kAwIyA9lgZ1FvtvCFdPXEm9hZi41LUngsc4IOp/m3x6cI9iE96iMpvWaLyW5awVPAeOoZO0tF/ZcEbu2YkMzMTSUlJAIAHDx7g22+/xZMnT9CrV68ijff5559j2LBh8Pb2xttvv42tW7fi8uXLqFmz5utPJp15t0MTPEh7iqXhvyA5NQ11XByx6atPUM3BFgBwNzVN5Z4jW386ieycXExb/COmLf5R2f5+16b4ZupgAEDg0M6QJAlfr9uPpJRHsKtgiY6t6mPiR91L98WR0dt1LAa21haY+EFr2NuWR2xcCnynfK/cHWNvW17tniPWlnL0eqceJof9ku+YAe96Q25uis3B/VXa52/+HQs2/7dkXgiRlt7YZOTgwYPKdR9WVlaoW7cu/vOf/6Bt27aIi4vTejxfX19cu3YNQUFByMjIwHvvvYdRo0bhl1/y/wFAJWdY37cxrO/b+T6Xl2Dk+c/yz147nqmpCb74sCu++LCrTuIjKo71P5/F+p/P5vvcmK9/VmtLS89E1Z7zCxyv0QfLdRYblTJd3LSsbBRGIAlNlmmTzqSlpcHGxgbXb6fCiutH6A3l1DNU3yEQlRiRnYHMP+bh0aNHJbIOMO/3xJHz8ShvVbzxnzxOQ3vP6iUWq65wQpyIiIj06o2dpiEiIirTjOhGI0xGiIiIDJAxfWovkxEiIiIDpItP3eWn9hIRERFpgJURIiIiA2RES0aYjBARERkkI8pGOE1DREREesXKCBERkQHibhoiIiLSK+6mISIiIiolrIwQEREZICNav8pkhIiIyCAZUTbCaRoiIiLSK1ZGiIiIDBB30xAREZFeGdNuGiYjREREBsiIloxwzQgRERHpFysjREREhsiISiNMRoiIiAyQMS1g5TQNERER6RUrI0RERAaIu2mIiIhIr4xoyQinaYiIiEi/WBkhIiIyREZUGmEyQkREZIC4m4aIiIiolLAyQkREZIC4m4aIiIj0yoiWjDAZISIiMkhGlI1wzQgRERHpFSsjREREBsiYdtMwGSEiIjJEOljAWkZyEU7TEBERkX6xMkJERGSAjGj9KpMRIiIig2RE2QinaYiIiEivWBkhIiIyQNxNQ0RERHplTLeD5zQNERER6RUrI0RERAbIiNavMhkhIiIySEaUjXCahoiIyABJOvpPW2FhYXBxcYFCoYCXlxeOHz9eYN8TJ06gVatWsLOzg4WFBerWrYtvvvlG62uyMkJEREQAgO3bt2Ps2LEICwtDq1atsHr1anTr1g0xMTGoXr26Wn9LS0t8+umnaNiwISwtLXHixAl88sknsLS0xMcff6zxdSUhhNDlC6HCpaWlwcbGBtdvp8LK2lrf4RCVCKeeofoOgajEiOwMZP4xD48ePYJ1Cfwcz/s9celGcrF/TzxOS0N9l8oax9q8eXM0adIEK1euVLbVq1cPffr0QWioZv+u+/XrB0tLS2zZskXjODlNQ0REZIAkHR3AiwTn5SMzM1PtellZWTh79iw6d+6s0t65c2ecPHlSo5ijo6Nx8uRJtGnTRqvXymSEiIjoDefk5AQbGxvlkV+V4969e8jJyYG9vb1Ku729PZKSkgodv1q1apDL5fD29saYMWMwYsQIreLjmhEiIiIDpMubniUkJKhM08jl8kLOUb2oEEKt7VXHjx/HkydPcOrUKUyaNAmurq4YNGiQxnEyGSEiIjJIutvba21t/do1IxUrVoSJiYlaFSQ5OVmtWvIqFxcXAECDBg1w9+5dhISEaJWMcJqGiIiIYG5uDi8vLxw6dEil/dChQ2jZsqXG4wgh8l2TUhhWRoiIiAyQPj6bZty4cRgyZAi8vb3h4+ODNWvWID4+HiNHjgQATJ48Gbdv38bmzZsBACtWrED16tVRt25dAC/uO7Jw4UJ89tlnWl2XyQgREZEB0scNWH19fZGamopZs2YhMTER9evXx/79++Hs7AwASExMRHx8vLJ/bm4uJk+ejBs3bsDU1BS1atXC/Pnz8cknn2gXJ+8zUrp4nxEyBrzPCL3JSus+I3/fTNHJfUbqOlcqsVh1hZURIiIiA6SPaRp9YTJCRERkgIr62TKvjlEWMBkhIiIyRPzUXiIiIqLSwcoIERGRATKiwgiTESIiIkNkTAtYOU1DREREesXKCBERkQHibhoiIiLSLyNaNMJpGiIiItIrVkaIiIgMkBEVRpiMEBERGSLupiEiIiIqJayMEBERGaTi76YpKxM1TEaIiIgMEKdpiIiIiEoJkxEiIiLSK07TEBERGSBjmqZhMkJERGSAjOl28JymISIiIr1iZYSIiMgAcZqGiIiI9MqYbgfPaRoiIiLSK1ZGiIiIDJERlUaYjBARERkg7qYhIiIiKiWsjBARERkg7qYhIiIivTKiJSNMRoiIiAySEWUjXDNCREREesXKCBERkQEypt00TEaIiIgMEBewUokRQgAAHj9O03MkRCVHZGfoOwSiEiOyM1/87///PC8paWnF/z2hizFKA5ORUvb48WMAQKO6LnqOhIiIiuPx48ewsbHR+bjm5uZwcHBAbRcnnYzn4OAAc3NznYxVUiRR0qkdqcjNzcWdO3dgZWUFqazUz8qwtLQ0ODk5ISEhAdbW1voOh0jn+B4vfUIIPH78GFWqVIFMVjL7QDIyMpCVlaWTsczNzaFQKHQyVklhZaSUyWQyVKtWTd9hGB1ra2v+oKY3Gt/jpaskKiIvUygUBp9A6BK39hIREZFeMRkhIiIivWIyQm80uVyO4OBgyOVyfYdCVCL4Hqc3ARewEhERkV6xMkJERER6xWSEiIiI9IrJCBEREekVkxF6I4WEhMDT07PQPv7+/ujTp0+xryVJEnbv3l3scYheJy4uDpIk4fz588UaR5N/H0SlickI6Y2/vz8kScL8+fNV2nfv3l0qd6ddunQpwsPDS/w6RJrK+zeRd9jZ2aFr1664cOECAMDJyQmJiYmoX7++niMl0i0mI6RXCoUCCxYswIMHD0r92jY2NqhQoUKpX5eoMF27dkViYiISExNx+PBhmJqaomfPngAAExMTODg4wNSUN8+mNwuTEdKrjh07wsHBAaGhoYX227FjBzw8PCCXy1GjRg0sWrRIo/FXr14NJycnlCtXDv3798fDhw+Vz706TdO2bVsEBgZi4sSJsLW1hYODA0JCQlTGu3r1Klq3bg2FQgF3d3ccOnRI05dKpBG5XA4HBwc4ODjA09MTQUFBSEhIQEpKito0zbFjxyBJEg4fPgxvb2+UK1cOLVu2xJUrV1TGnD9/Puzt7WFlZYWAgABkZPBTlcmwMBkhvTIxMcG8efOwfPly3Lp1K98+Z8+exYABAzBw4EBcvHgRISEhmD59+munWP79919ERERgz549OHjwIM6fP48xY8YUes6mTZtgaWmJ06dP46uvvsKsWbOUCUdubi769esHExMTnDp1CqtWrUJQUFCRXjeRJp48eYKtW7fC1dUVdnZ2BfabOnUqFi1ahKioKJiammL48OHK5yIiIhAcHIy5c+ciKioKjo6OCAsLK43wiTQniPRk2LBhonfv3kIIIVq0aCGGDx8uhBBi165d4uW3pp+fn+jUqZPKuRMmTBDu7u4Fjh0cHCxMTExEQkKCsu3AgQNCJpOJxMREtesLIUSbNm3E22+/rTJO06ZNRVBQkBBCiF9++SXfMQGIXbt2af7CiQowbNgwYWJiIiwtLYWlpaUAIBwdHcXZs2eFEELcuHFDABDR0dFCCCGOHj0qAIjffvtNOca+ffsEAPHs2TMhhBA+Pj5i5MiRKtdp3ry5aNSoUam8JiJNsDJCBmHBggXYtGkTYmJi1J6LjY1Fq1atVNpatWqFq1evIicnp8Axq1evrvIJyT4+PsjNzVUrYb+sYcOGKo8dHR2RnJysjCO/MYl0qV27djh//jzOnz+P06dPo3PnzujWrRtu3rxZ4Dkvv28dHR0BQOV9++r7lO9bMjRMRsggtG7dGl26dMGUKVPUnhNCqO2uEUX4FIO8MQrbqWNmZqZ2Tm5uboHXLI1dP2RcLC0t4erqCldXVzRr1gzr169Heno61q5dW+A5L79v896Tee9borKAyQgZjPnz52PPnj04efKkSru7uztOnDih0nby5Em4ubnBxMSkwPHi4+Nx584d5ePIyEjIZDK4ubkVKT53d/d8xyQqSZIkQSaT4dmzZ0U6v169ejh16pRK26uPifSN+8PIYDRo0ACDBw/G8uXLVdq//PJLNG3aFLNnz4avry8iIyPx7bffvnYRnkKhwLBhw7Bw4UKkpaUhMDAQAwYMgIODQ5Hi69ixI+rUqYOhQ4di0aJFSEtLw9SpU4s0FlFBMjMzkZSUBAB48OABvv32Wzx58gS9evUq0niff/45hg0bBm9vb7z99tvYunUrLl++jJo1a+oybKJiYWWEDMrs2bPVpkOaNGmCiIgI/PDDD6hfvz5mzJiBWbNmwd/fv9CxXF1d0a9fP3Tv3h2dO3dG/fr1i7WLQCaTYdeuXcjMzESzZs0wYsQIzJ07t8jjEeXn4MGDcHR0hKOjI5o3b44zZ87gP//5D9q2bVuk8Xx9fTFjxgwEBQXBy8sLN2/exKhRo3QbNFExSaIok+9EREREOsLKCBEREekVkxEiIiLSKyYjREREpFdMRoiIiEivmIwQERGRXjEZISIiIr1iMkJERER6xWSEyAiFhITA09NT+djf3x99+vQp9Tji4uIgSRLOnz9fYJ8aNWpgyZIlGo8ZHh6OChUqFDs2SZKwe/fuYo9DRK/HZITIQPj7+0OSJEiSBDMzM9SsWRPjx49Henp6iV976dKlCA8P16ivJgkEEZE2+Nk0RAaka9eu2LhxI54/f47jx49jxIgRSE9Px8qVK9X6Pn/+XO1ThovKxsZGJ+MQERUFKyNEBkQul8PBwQFOTk7w8/PD4MGDlVMFeVMrGzZsQM2aNSGXyyGEwKNHj/Dxxx+jcuXKsLa2Rvv27fHXX3+pjDt//nzY29vDysoKAQEByMjIUHn+1Wma3NxcLFiwAK6urpDL5ahevbryc3hcXFwAAI0bN4YkSSqfmbJx40bUq1cPCoUCdevWVfssoD///BONGzeGQqGAt7c3oqOjtf4aLV68GA0aNIClpSWcnJwwevRoPHnyRK3f7t274ebmBoVCgU6dOiEhIUHl+T179sDLywsKhQI1a9bEzJkzkZ2drXU8RFR8TEaIDJiFhQWeP3+ufPzvv/8iIiICO3bsUE6T9OjRA0lJSdi/fz/Onj2LJk2aoEOHDrh//z4AICIiAsHBwZg7dy6ioqLg6Oj42g8MnDx5MhYsWIDp06cjJiYG27Ztg729PYAXCQUA/Pbbb0hMTMTOnTsBAGvXrsXUqVMxd+5cxMbGYt68eZg+fTo2bdoEAEhPT0fPnj1Rp04dnD17FiEhIRg/frzWXxOZTIZly5bh0qVL2LRpE44cOYKJEyeq9Hn69Cnmzp2LTZs24Y8//kBaWhoGDhyofP6XX37BBx98gMDAQMTExGD16tUIDw/nBx8S6YsgIoMwbNgw0bt3b+Xj06dPCzs7OzFgwAAhhBDBwcHCzMxMJCcnK/scPnxYWFtbi4yMDJWxatWqJVavXi2EEMLHx0eMHDlS5fnmzZuLRo0a5XvttLQ0IZfLxdq1a/ON88aNGwKAiI6OVml3cnIS27ZtU2mbPXu28PHxEUIIsXr1amFrayvS09OVz69cuTLfsV7m7OwsvvnmmwKfj4iIEHZ2dsrHGzduFADEqVOnlG2xsbECgDh9+rQQQoh33nlHzJs3T2WcLVu2CEdHR+VjAGLXrl0FXpeIdIdrRogMyN69e1G+fHlkZ2fj+fPn6N27N5YvX6583tnZGZUqVVI+Pnv2LJ48eQI7OzuVcZ49e4Zr164BAGJjYzFy5EiV5318fHD06NF8Y4iNjUVmZiY6dOigcdwpKSlISEhAQEAAPvroI2V7dna2cj1KbGwsGjVqhHLlyqnEoa2jR49i3rx5iImJQVpaGrKzs5GRkYH09HRYWloCAExNTeHt7a08p27duqhQoQJiY2PRrFkznD17FmfOnFGphOTk5CAjIwNPnz5ViZGISh6TESID0q5dO6xcuRJmZmaoUqWK2gLVvF+2eXJzc+Ho6Ihjx46pjVXU7a0WFhZan5ObmwvgxVRN8+bNVZ4zMTEBAAghihTPy27evInu3btj5MiRmD17NmxtbXHixAkEBASoTGcBL7bmviqvLTc3FzNnzkS/fv3U+igUimLHSUTaYTJCZEAsLS3h6uqqcf8mTZogKSkJpqamqFGjRr596tWrh1OnTmHo0KHKtlOnThU4Zu3atWFhYYHDhw9jxIgRas+bm5sDeFFJyGNvb4+qVavi+vXrGDx4cL7juru7Y8uWLXj27Jky4SksjvxERUUhOzsbixYtgkz2YslbRESEWr/s7GxERUWhWbNmAIArV67g4cOHqFu3LoAXX7crV65o9bUmopLDZISoDOvYsSN8fHzQp08fLFiwAHXq1MGdO3ewf/9+9OnTB97e3vj8888xbNgweHt74+2338bWrVtx+fJl1KxZM98xFQoFgoKCMHHiRJibm6NVq1ZISUnB5cuXERAQgMqVK8PCwgIHDx5EtWrVoFAoYGNjg5CQEAQGBsLa2hrdunVDZmYmoqKi8ODBA4wbNw5+fn6YOnUqAgICMG3aNMTFxWHhwoVavd5atWohOzsby5cvR69evfDHH39g1apVav3MzMzw2WefYdmyZTAzM8Onn36KFi1aKJOTGTNmoGfPnnByckL//v0hk8lw4cIFXLx4EXPmzNH+G0FExcLdNERlmCRJ2L9/P1q3bo3hw4fDzc0NAwcORFxcnHL3i6+vL2bMmIGgoCB4eXnh5s2bGDVqVKHjTp8+HV9++SVmzJiBevXqwdfXF8nJyQBerMdYtmwZVq9ejSpVqqB3794AgBEjRmDdunUIDw9HgwYN0KZNG4SHhyu3ApcvXx579uxBTEwMGjdujKlTp2LBggVavV5PT08sXrwYCxYsQP369bF161aEhoaq9StXrhyCgoLg5+cHHx8fWFhY4IcfflA+36VLF+zduxeHDh1C06ZN0aJFCyxevBjOzs5axUNEuiEJXUzkEhERERURKyNERESkV0xGiIiISK+YjBAREZFeMRkhIiIivWIyQkRERHrFZISIiIj0iskIERER6RWTESIiItIrJiNERESkV0xGiIiISK+YjBAREZFeMRkhIiIivfo/HB7tATSIVXAAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# creating a confusion matrix\n",
    "cm_norm = confusion_matrix(y_test, predictions_gradient, normalize=\"true\")\n",
    "\n",
    "disp = ConfusionMatrixDisplay(confusion_matrix=cm_norm, display_labels=CLASS_NAMES)\n",
    "disp.plot(cmap=\"Blues\")\n",
    "\n",
    "disp.ax_.set_title(\"Normalized Confusion Matrix\")\n",
    "disp.ax_.grid(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "6efcf45f9bec60f919453d54d1718a4d1526f09dbabed9268bf472b0ad8085e9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}