cassandra-21-creatorz.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cassandra'21 Competition NOTEBOOK\n",
    "\n",
    "This notebook contains our approach on dealing with the dataset of Cassandra which have very less correlation in training and testing,We will later explain that the scores in private and public leadreboard is not a result of overfit but a result of randomness and less correlation in train and testing distribution which makes the cross validation of train data less trustworthy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "\n",
    "# lOading libraries\n",
    "import os\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import gc\n",
    "import optuna\n",
    "from tqdm.notebook import tqdm as tn\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold, learning_curve\n",
    "from lightgbm import LGBMClassifier\n",
    "from sklearn.metrics import f1_score, auc, roc_curve\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold, learning_curve\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta = pd.read_csv(\"../input/cassandra-udyam21/payment_history_data.csv\")\n",
    "train = pd.read_csv(\"../input/cassandra-udyam21/user_data_training.csv\")\n",
    "test = pd.read_csv(\"../input/cassandra-udyam21/user_data_testing.csv\")\n",
    "sample_Submission = pd.read_csv(\"../input/cassandra-udyam21/Sample Submission.csv\")\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train,head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Preprocessing on train and test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving on to the actual preprocessing **we done a most basic step is to remove extra noise and outliers based on the inter quartile range**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "def detect_outliers(df,n,features):\n",
    "    \"\"\"\n",
    "    Takes a dataframe df of features and returns a list of the indices\n",
    "    corresponding to the observations containing more than n outliers according\n",
    "    to the Tukey method.\n",
    "    \"\"\"\n",
    "    outlier_indices = []\n",
    "    \n",
    "    # iterate over features(columns)\n",
    "    for col in features:\n",
    "        # 1st quartile (25%)\n",
    "        Q1 = np.percentile(df[col], 25)\n",
    "        # 3rd quartile (75%)\n",
    "        \n",
    "        Q3 = np.percentile(df[col],75)\n",
    "        # Interquartile range (IQR)\n",
    "        IQR = Q3 - Q1\n",
    "        \n",
    "        # outlier step\n",
    "        outlier_step = 3.5 * IQR\n",
    "        \n",
    "        # Determine a list of indices of outliers for feature col\n",
    "        outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step )].index\n",
    "        \n",
    "        # append the found outlier indices for col to the list of outlier indices \n",
    "        outlier_indices.extend(outlier_list_col)\n",
    "        # select observations containing more than 2 outliers\n",
    "    outlier_indices = Counter(outlier_indices)        \n",
    "    multiple_outliers = list( k for k, v in outlier_indices.items() if v > n )\n",
    "    \n",
    "    return multiple_outliers   \n",
    "\n",
    "# detect outliers from Age, SibSp , Parch and Fare\n",
    "Outliers_to_drop = detect_outliers(train,2,train.columns[2:])  # we picked those rows which \n",
    "                                                            # contained outliers in more than 2 columns of it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found that there are around 100 such rows in train data out of which 90 have label 0 and 10 have 1 label so based on the data imbalance we decided to remove those 90 rows of 0, it made the imbalance as well as noise in data less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1484006679\n",
      "1486529479\n",
      "1490253206\n",
      "1489267459\n",
      "1487335291\n",
      "1491150284\n",
      "772423\n",
      "1486924223\n",
      "2153656\n",
      "1490862759\n",
      "1487547132\n",
      "1484012615\n",
      "1114\n",
      "1487069284\n",
      "1490044704\n",
      "1487894017\n",
      "1371217\n",
      "1489935259\n",
      "3241428\n",
      "1489211809\n",
      "1490571524\n",
      "1490902456\n",
      "1484008534\n",
      "1484907096\n",
      "1258433\n",
      "1486924594\n",
      "1488777739\n",
      "2512413\n",
      "1489154675\n",
      "1492283318\n",
      "1488123295\n",
      "1489086782\n",
      "1414995\n",
      "1490673178\n",
      "1491887461\n",
      "1491028225\n",
      "1487308579\n",
      "2172577\n",
      "825105\n",
      "1485296275\n",
      "765374\n",
      "1484025971\n",
      "1487368310\n",
      "1488158911\n",
      "1488028690\n",
      "1486255310\n",
      "1489550532\n",
      "1454692\n",
      "1490391960\n",
      "1193137\n",
      "1486569176\n",
      "2124718\n",
      "1488426402\n",
      "1487601298\n",
      "1485005411\n",
      "1459886\n",
      "1286258\n",
      "1484257846\n",
      "2328768\n",
      "1484541661\n",
      "1486066842\n",
      "1487550100\n",
      "1485517391\n",
      "1486566208\n",
      "949761\n",
      "1485407204\n",
      "1486057938\n",
      "1486571402\n",
      "1485826434\n",
      "1055125\n",
      "1485584171\n",
      "1486111362\n",
      "1485810110\n",
      "1488449404\n",
      "1488431967\n",
      "1484938260\n",
      "1206864\n",
      "586552\n",
      "3514855\n",
      "1488940237\n",
      "1485803061\n",
      "1486418179\n",
      "1485737023\n",
      "1489659235\n",
      "533870\n",
      "1488755850\n",
      "1487602411\n",
      "1326697\n",
      "1484017809\n",
      "1486201515\n",
      "1486911238\n",
      "2174432\n",
      "1872809\n",
      "1220591\n",
      "1487070026\n",
      "1488595578\n",
      "1485239141\n",
      "1488342185\n",
      "1485432803\n",
      "1484770568\n",
      "1484379534\n",
      "1492623896\n",
      "1490117420\n",
      "1487359035\n",
      "3440284\n",
      "1703262\n",
      "1491406645\n",
      "1485185346\n",
      "2421147\n",
      "1489590600\n",
      "1486878590\n",
      "2253455\n",
      "1491825875\n"
     ]
    }
   ],
   "source": [
    "## Now removing those outlier rows of 0 from train data as well as data of those ids from payment_history.csv\n",
    "\n",
    "# removed from payment history.\n",
    "for i in train.loc[Outliers_to_drop][train.loc[Outliers_to_drop]['label']==0].id.values:\n",
    "    print(i)\n",
    "    train_meta = train_meta[train_meta[\"id\"]!=i].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removed from train\n",
    "train.loc[Outliers_to_drop]['label'].values # Show the outliers rows\n",
    "# Drop outliers\n",
    "train = train.drop(train.loc[Outliers_to_drop][train.loc[Outliers_to_drop]['label']==0].index.values, axis = 0).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id\n",
       "1             18\n",
       "12615         11\n",
       "40811          2\n",
       "65668          4\n",
       "72717         10\n",
       "              ..\n",
       "1492771183     3\n",
       "1492824607     4\n",
       "1492844641     8\n",
       "1492854287     3\n",
       "1492861707     3\n",
       "Length: 1012, dtype: int64"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta.groupby([\"id\"]).size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>Kappa</th>\n",
       "      <th>Lambda</th>\n",
       "      <th>omikron</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1484353935</td>\n",
       "      <td>2615</td>\n",
       "      <td>1214.0</td>\n",
       "      <td>3</td>\n",
       "      <td>56000</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>101</td>\n",
       "      <td>5</td>\n",
       "      <td>60064</td>\n",
       "      <td>192.163993</td>\n",
       "      <td>0.029703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1490969236</td>\n",
       "      <td>2092</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>100000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>99</td>\n",
       "      <td>5</td>\n",
       "      <td>71003</td>\n",
       "      <td>199.974998</td>\n",
       "      <td>0.020202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1487077817</td>\n",
       "      <td>3661</td>\n",
       "      <td>1223.0</td>\n",
       "      <td>3</td>\n",
       "      <td>78000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>75</td>\n",
       "      <td>5</td>\n",
       "      <td>450007</td>\n",
       "      <td>173.205081</td>\n",
       "      <td>0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1485622755</td>\n",
       "      <td>2615</td>\n",
       "      <td>1214.0</td>\n",
       "      <td>3</td>\n",
       "      <td>68000</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>92</td>\n",
       "      <td>3</td>\n",
       "      <td>550044</td>\n",
       "      <td>173.205081</td>\n",
       "      <td>0.032609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1486796599</td>\n",
       "      <td>3661</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>63000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>84</td>\n",
       "      <td>5</td>\n",
       "      <td>60019</td>\n",
       "      <td>158.113883</td>\n",
       "      <td>0.023810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>405</th>\n",
       "      <td>1489702642</td>\n",
       "      <td>2092</td>\n",
       "      <td>1313.0</td>\n",
       "      <td>3</td>\n",
       "      <td>134000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>104</td>\n",
       "      <td>3</td>\n",
       "      <td>60054</td>\n",
       "      <td>244.948974</td>\n",
       "      <td>0.028846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>406</th>\n",
       "      <td>1835338</td>\n",
       "      <td>3661</td>\n",
       "      <td>1125.5</td>\n",
       "      <td>3</td>\n",
       "      <td>79000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "      <td>450007</td>\n",
       "      <td>187.082869</td>\n",
       "      <td>0.028037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>407</th>\n",
       "      <td>1236915</td>\n",
       "      <td>3661</td>\n",
       "      <td>1188.5</td>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>98</td>\n",
       "      <td>5</td>\n",
       "      <td>60017</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.030612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>408</th>\n",
       "      <td>1489208099</td>\n",
       "      <td>3661</td>\n",
       "      <td>1386.5</td>\n",
       "      <td>1</td>\n",
       "      <td>165000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>100</td>\n",
       "      <td>3</td>\n",
       "      <td>350098</td>\n",
       "      <td>173.205081</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>409</th>\n",
       "      <td>3788282</td>\n",
       "      <td>3661</td>\n",
       "      <td>1350.5</td>\n",
       "      <td>3</td>\n",
       "      <td>130000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>97</td>\n",
       "      <td>5</td>\n",
       "      <td>60016</td>\n",
       "      <td>173.205081</td>\n",
       "      <td>0.030928</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>410 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             id  Alpha    Beta  Gamma   Delta  Epsilon  Zeta  Eta  Theta  \\\n",
       "0    1484353935   2615  1214.0      3   56000        2    15    5    101   \n",
       "1    1490969236   2092     NaN      2  100000        2     8    9     99   \n",
       "2    1487077817   3661  1223.0      3   78000        2    11   -1     75   \n",
       "3    1485622755   2615  1214.0      3   68000        2    15    5     92   \n",
       "4    1486796599   3661     NaN      2   63000        2    11    5     84   \n",
       "..          ...    ...     ...    ...     ...      ...   ...  ...    ...   \n",
       "405  1489702642   2092  1313.0      3  134000        2     8    5    104   \n",
       "406     1835338   3661  1125.5      3   79000        2    11   -1    107   \n",
       "407     1236915   3661  1188.5      3   90000        2    11    5     98   \n",
       "408  1489208099   3661  1386.5      1  165000        2    11   -1    100   \n",
       "409     3788282   3661  1350.5      3  130000        1    11    5     97   \n",
       "\n",
       "     Iota   Kappa      Lambda   omikron  \n",
       "0       5   60064  192.163993  0.029703  \n",
       "1       5   71003  199.974998  0.020202  \n",
       "2       5  450007  173.205081  0.040000  \n",
       "3       3  550044  173.205081  0.032609  \n",
       "4       5   60019  158.113883  0.023810  \n",
       "..    ...     ...         ...       ...  \n",
       "405     3   60054  244.948974  0.028846  \n",
       "406     5  450007  187.082869  0.028037  \n",
       "407     5   60017    1.000000  0.030612  \n",
       "408     3  350098  173.205081  0.010000  \n",
       "409     5   60016  173.205081  0.030928  \n",
       "\n",
       "[410 rows x 13 columns]"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>Kappa</th>\n",
       "      <th>Lambda</th>\n",
       "      <th>omikron</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>602.000000</td>\n",
       "      <td>6.020000e+02</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>524.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "      <td>602.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.191030</td>\n",
       "      <td>1.051541e+09</td>\n",
       "      <td>2819.161130</td>\n",
       "      <td>1283.105916</td>\n",
       "      <td>2.395349</td>\n",
       "      <td>112624.584718</td>\n",
       "      <td>1.936877</td>\n",
       "      <td>10.686047</td>\n",
       "      <td>5.194352</td>\n",
       "      <td>100.682724</td>\n",
       "      <td>4.219269</td>\n",
       "      <td>127825.440199</td>\n",
       "      <td>125.127942</td>\n",
       "      <td>0.024092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.393439</td>\n",
       "      <td>6.779381e+08</td>\n",
       "      <td>702.800684</td>\n",
       "      <td>49.929709</td>\n",
       "      <td>0.849691</td>\n",
       "      <td>64477.564571</td>\n",
       "      <td>0.243386</td>\n",
       "      <td>2.631457</td>\n",
       "      <td>2.431169</td>\n",
       "      <td>11.621341</td>\n",
       "      <td>0.842957</td>\n",
       "      <td>120823.933689</td>\n",
       "      <td>110.557187</td>\n",
       "      <td>0.009095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.384700e+04</td>\n",
       "      <td>2092.000000</td>\n",
       "      <td>1116.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>15000.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>60000.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.008696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.126974e+06</td>\n",
       "      <td>2092.000000</td>\n",
       "      <td>1245.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>72250.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>90.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>60036.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.017544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.486809e+09</td>\n",
       "      <td>2615.000000</td>\n",
       "      <td>1281.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>104.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>72000.000000</td>\n",
       "      <td>173.205081</td>\n",
       "      <td>0.027027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.490172e+09</td>\n",
       "      <td>3661.000000</td>\n",
       "      <td>1314.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>135000.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>110.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>151300.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.492862e+09</td>\n",
       "      <td>3661.000000</td>\n",
       "      <td>1469.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>500000.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>115.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>591001.000000</td>\n",
       "      <td>707.106781</td>\n",
       "      <td>0.046875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            label            id        Alpha         Beta       Gamma  \\\n",
       "count  602.000000  6.020000e+02   602.000000   524.000000  602.000000   \n",
       "mean     0.191030  1.051541e+09  2819.161130  1283.105916    2.395349   \n",
       "std      0.393439  6.779381e+08   702.800684    49.929709    0.849691   \n",
       "min      0.000000  8.384700e+04  2092.000000  1116.500000    1.000000   \n",
       "25%      0.000000  3.126974e+06  2092.000000  1245.500000    2.000000   \n",
       "50%      0.000000  1.486809e+09  2615.000000  1281.500000    3.000000   \n",
       "75%      0.000000  1.490172e+09  3661.000000  1314.500000    3.000000   \n",
       "max      1.000000  1.492862e+09  3661.000000  1469.000000    3.000000   \n",
       "\n",
       "               Delta     Epsilon        Zeta         Eta       Theta  \\\n",
       "count     602.000000  602.000000  602.000000  602.000000  602.000000   \n",
       "mean   112624.584718    1.936877   10.686047    5.194352  100.682724   \n",
       "std     64477.564571    0.243386    2.631457    2.431169   11.621341   \n",
       "min     15000.000000    1.000000    6.000000   -1.000000   64.000000   \n",
       "25%     72250.000000    2.000000    8.000000    5.000000   90.000000   \n",
       "50%    100000.000000    2.000000   11.000000    5.000000  104.000000   \n",
       "75%    135000.000000    2.000000   11.000000    5.000000  110.000000   \n",
       "max    500000.000000    2.000000   15.000000   10.000000  115.000000   \n",
       "\n",
       "             Iota          Kappa      Lambda     omikron  \n",
       "count  602.000000     602.000000  602.000000  602.000000  \n",
       "mean     4.219269  127825.440199  125.127942    0.024092  \n",
       "std      0.842957  120823.933689  110.557187    0.009095  \n",
       "min      1.000000   60000.000000    1.000000    0.008696  \n",
       "25%      4.000000   60036.000000    1.000000    0.017544  \n",
       "50%      4.000000   72000.000000  173.205081    0.027027  \n",
       "75%      5.000000  151300.000000  200.000000    0.030000  \n",
       "max      5.000000  591001.000000  707.106781    0.046875  "
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# taking description of data now to find the distribution.\n",
    "train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our observations after outlier removal (Observation No. -1)\n",
    "\n",
    "1. we saw that some variables like Kappa and Lambda which shows less correlation with the data of around 0.01 nows \n",
    "showing decent correlation of around 0.12 and 0.09 which made us believe that actual outlier is a noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Beta', ylabel='Density'>"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAA88UlEQVR4nO3dd3zV9b348dc7OyE7hJUEEiDsPUURByo4qa1WrNaBrW3VXu+1t62jt+v+7K237e1Uq3VUrRUp1YrWva0iEDZhhrASRkISMghZJ+/fH98vGjGQdU6+Jyfv5+NxHvme78r7fEl457NFVTHGGGPaK8zrAIwxxvQsljiMMcZ0iCUOY4wxHWKJwxhjTIdY4jDGGNMhEV4H0B369u2r2dnZXodhjDE9xurVqw+ranprx3pF4sjOziYvL8/rMIwxpscQkT0nO2ZVVcYYYzrEEocxxpgOscRhjDGmQ3pFG4cxxnihsbGRoqIi6urqvA7lpGJiYsjMzCQyMrLd11jiMMaYACkqKiIhIYHs7GxExOtwPkdVKSsro6ioiJycnHZfZ1VVxhgTIHV1daSlpQVl0gAQEdLS0jpcIrLEYYwxARSsSeO4zsRnicOYINHka8aWOTA9gSUOYzy0eX8VdyxZx9T/foPh97zC2b98l/99dSu1DU1eh2a6UXx8/CmP7969m3HjxnXonjfccANLly7tSlgnZY3jxnig0dfMb97czgPv7iQ2MpwLxw1kUHIMG4srefC9nby55RAPfXUaOX37eB2qMZ9jJQ5jullNfRM3PL6S+9/ZyZenZrH8rrn86ssT+c4FI/nzjTN4ctEMDtc0cN1jKzhS2+B1uKYb1dTUMHfuXKZMmcL48eN54YUXPjnW1NTENddcw+jRo7niiiuora0FYPXq1Zx11llMnTqVefPmceDAgYDHaYnDmG5UU9/ENY+s4OPCcn5xxQTuu2ICSbGf7T9/Zm46j1w/jYOVdfz7s+tobrZ2j94iJiaG559/njVr1vDOO+/wne9855N2r23btnHLLbewZcsWEhMTeeCBB2hsbOTb3/42S5cuZfXq1SxatIh77rkn4HFaVZUx3cTXrPzbM2vZVFzJg9dM4YKxA0567pTBKfzXJWP44Qv5LFu/ny9MzujGSI1XVJW7776b999/n7CwMIqLizl06BAAWVlZnHHGGQBce+21/O53v2P+/Pls2rSJ888/HwCfz8fAgQMDHqclDmO6yb3/3MLbW0v47y+MO2XSOO7amUNYkrePX7y2jfnjBhATGd4NURovPf3005SWlrJ69WoiIyPJzs7+ZIzFid1mRQRVZezYsSxfvrxb47SqKmO6wV8+3sNjH+7ixjOy+eppQ9p1TViYcPeFoyk+cownPtod2ABNUKisrKRfv35ERkbyzjvvsGfPpzOb792795ME8de//pXZs2czcuRISktLP9nf2NhIfn5+wOMMaOIQkfkisk1ECkTkzlaOR4vIs+7xFSKS3eLYXe7+bSIyr8X+ZBFZKiJbRWSLiMwK5Gcwpqs+2FHKj5blc+6ofvzg4jEduvb04X2ZPbwvj324i0Zfc4AiNMHimmuuIS8vj/Hjx/Pkk08yatSoT46NHDmS+++/n9GjR1NRUcG3vvUtoqKiWLp0Kd///veZOHEikyZN4qOPPgp4nBKoAUciEg5sB84HioBVwNWqurnFObcAE1T1myKyELhcVa8SkTHAM8AMYBDwJjBCVX0i8gTwgao+IiJRQJyqHjlVLNOmTVNbyMl4oaCkmssf+IiM5FiWfut04qM7Xjv81pZD3PREHg9cM4WLxge+/tr4z5YtWxg9erTXYbSptThFZLWqTmvt/ECWOGYABapaqKoNwGJgwQnnLACecLeXAnPFqchbACxW1XpV3QUUADNEJAmYAzwKoKoNbSUNY7xSVlPPjX9eRXREOI9cP61TSQPg7JH9yEyJteoqEzQCmTgygH0t3he5+1o9R1WbgEog7RTX5gClwOMislZEHhERGyFlgk59k49v/mU1JVX1/Om6qWSmxHX6XuFhwldPG8KKXeUUlFT7MUpjOqenNY5HAFOAB1V1MnAU+FzbCYCI3CwieSKSV1pa2p0xml5OVbnz7xtZtbuCX315IpMHp3T5npdPySBMYNn6wA/uMqYtgUwcxUBWi/eZ7r5WzxGRCCAJKDvFtUVAkaqucPcvxUkkn6OqD6vqNFWdlp6e3sWPYkz7/f7tAp5fW8x3zh/BJRMG+eWe/RJimJmTxksb9ttEiMZzgUwcq4BcEclxG7EXAstOOGcZcL27fQXwtjq/FcuAhW6vqxwgF1ipqgeBfSIy0r1mLrAZY4LE4pV7+b83tvPFyRncdu5wv9774gkDKSw9ytaDVl1lvBWwxOG2WdwGvAZsAZaoar6I/FRELnNPexRIE5EC4A7caidVzQeW4CSFV4FbVdXnXvNt4GkR2QBMAn4WqM9gTEe8ln+Qu5/fyFkj0rnvigl+X4fhwnEDCBN4acN+v97XmI4KaBuHqr6sqiNUdZiq3uvu+6GqLnO361T1SlUdrqozVLWwxbX3uteNVNVXWuxf51ZBTVDVL6hqRSA/gzHtsaKwjG8/s5YJmck8eO0UIsP9/6uVFh/NaUPTeD3/kN/vbULbq6++ysiRIxk+fDg///nPu3y/ntY4bkzQ+deOw9z451VkpcTy+A3TiYsK3Ew+547qx46SGooqagP2PUxo8fl83Hrrrbzyyits3ryZZ555hs2bu1bDb4nDmC54ZeMBFv15FYNT43jm66eR0icqoN/v7JFOR493t1lPQdM+K1euZPjw4QwdOpSoqCgWLlz4menaO8MmOTSmk55ZuZd7nt/IpKxkHr9hBklxkW1f1EXD0uPJTInl3W2lXNvOOa9McPjJi/ls3l/l13uOGZTIjy4de8pziouLycr6tJNqZmYmK1asOMUVbbPEYUwH1TX6+MmLm3lm5V7OGpHOg9dOCWj1VEsiwtkj03luTTH1TT6iI2zGXNP9LHEY0wH7ymv51tOr2VRcxS1nD+OO80cQEYCG8FM5e0Q//vLxXlbvruD04X279XubzmurZBAoGRkZ7Nv36UQcRUVFZGR0bX0Xa+Mwpp3e3nqIS37/L/aU1fKn66bxvfmjuj1pAMwcmkqYwMeFZd3+vU3PM336dHbs2MGuXbtoaGhg8eLFXHbZZW1feApW4jCmDc3Nym/f2sFv39rBmIGJPHjtFIakeTdFWkJMJOMzkvi4sNyzGEzPERERwR/+8AfmzZuHz+dj0aJFjB3btdKPJQ5jTqG6rpE7lqznjc2H+OKUDH52+figWInvtKFpPP7hbuoafUERjwluF110ERdddJHf7mdVVcacxJ6yo1z+wEe8vbWEH14yhl9dOTFo/pOeOTSVBl8za/ba+FfT/SxxGNOKgpJqrvzjcg7X1PPUohksmp3j9ylEumJa9vF2DquuMt3PqqqMOcHuw0dZ+PDHiAhLvjGLEf0TvA7pcxJjIhmXkWQN5MYTVuIwpoXDNfVc//hKfM3K4ptPC8qkcdy0Iams33fE1iI33c4ShzGuJl8ztz69hkNVdTx6w3SGpcd7HdIpTR2SQn1Ts99HIxvTFkscxrh+8+YOVuwq594vjGeKH1btC7QpQ5IBrIHcdDtLHMYAq/eUc/+7BVw5NZMvTc30Opx2GZgUy6CkGFbvscRhTm7RokX069ePcePG+e2eljhMr1ff5OP7f9/IoKRYfnSZN9NCdNbkISms3XvE6zBMELvhhht49dVX/XpPSxym13v4vUIKSmr4f18YR3x0z+poOHVwCsVHjnGwss7rUEyQmjNnDqmpqX69Z8/6LTHGz0qq63jwvZ3MHzuAc0b18zqcDpsyxGmLWbu3ggvHD/Q4GnNKr9wJBzf6954DxsOFXV/Rr6OsxGF6tV+/sYOGpma+f+Eor0PplNEDE4gMFzYUV3odiulFrMRheq3C0hqeXbWX62Zlk9PXu0kLuyI6IpyRAxLYWGSJI+h5UDIIFCtxmF7rwXd3Ehkexq3nDPc6lC4Zn5HMxuJKVNXrUEwvYYnD9EpFFbU8v7aYq2cMJj0h2utwumRCZhKVxxrZV37M61BMELr66quZNWsW27ZtIzMzk0cffbTL97SqKtMrPfLBLkTg5jlDvQ6ly8ZnJAGwofgIg9PiPI7GBJtnnnnG7/e0EofpdarqGvlb3j4um5jBoORYr8PpshH9E4gKD7N2DtNtApo4RGS+iGwTkQIRubOV49Ei8qx7fIWIZLc4dpe7f5uIzGuxf7eIbBSRdSKSF8j4TWhamlfE0QYfN5ye7XUofhEVEcbogQlssMRhuknAEoeIhAP3AxcCY4CrRWTMCafdBFSo6nDg18B97rVjgIXAWGA+8IB7v+POUdVJqjotUPGb0NTcrDyxfDdTh6QwPjPJ63D8ZnxmEpv2V9LcbA3kwSbYOy10Jr5AljhmAAWqWqiqDcBiYMEJ5ywAnnC3lwJzxVktZwGwWFXrVXUXUODez5gu+aDgMHvKarlu1hCvQ/Gr8RlJVNc1sae81utQTAsxMTGUlZUFbfJQVcrKyoiJienQdYFsHM8A9rV4XwTMPNk5qtokIpVAmrv/4xOuzXC3FXhdRBR4SFUfbu2bi8jNwM0AgwcP7tonMSFjyap9pMRFMn/cAK9D8avxGckAbCg60mPHpISizMxMioqKKC0t9TqUk4qJiSEzs2MTe/bEXlWzVbVYRPoBb4jIVlV9/8ST3ITyMMC0adOCM92bblV+tIHXNx/k2tOGEB0RHGuH+0tu/3iiI5wG8gWTMtq+wHSLyMhIcnJyvA7D7wJZVVUMZLV4n+nua/UcEYkAkoCyU12rqse/lgDPY1VYpp2eX1tMo0+5anpW2yf3MJHhYYwemMhGm3rEdINAJo5VQK6I5IhIFE5j97ITzlkGXO9uXwG8rU5l4DJgodvrKgfIBVaKSB8RSQAQkT7ABcCmAH4GE0KeW1PEhMwkRg1I9DqUgJiQmcSmYmsgN4EXsMShqk3AbcBrwBZgiarmi8hPReQy97RHgTQRKQDuAO50r80HlgCbgVeBW1XVB/QH/iUi64GVwD9V1b8TzZuQtLO0hvz9VSFdjTM+I4mjDT4KDx/1OhQT4gLaxqGqLwMvn7Dvhy2264ArT3LtvcC9J+wrBCb6P1IT6l5afwARuDiEpx4/3r14Y/ERhvcL7vXSTc9mI8dNyFNVXtywn+nZqQxI6li3w55keLrTQJ5fXOV1KCbEWeIwIW/boWoKSmq4dOIgr0MJqIjwMEYNTGTTfmsgN4FlicOEvBfX7yc8TLgwxMZutGbsoETy91cF7YAzExoscZiQpqq8uP4Apw9Lo298z54+vT3GDXJGkNsU6yaQLHGYkLaxuJK95bVcOiG0q6mOGzvI6Wqcb9VVJoAscZiQ9uL6/USGC/PGhn41FcDIAQmEh4m1c5iAssRhQpaq8sqmg5yZm05SXKTX4XSLmMhwcvvFk7/felaZwLHEYULWtkPVFFUc44Ix/b0OpVuNHZRkicMElCUOE7Le3HwIgHNH9fM4ku41dlAipdX1lFTVeR2KCVGWOEzIenNLCROzkumXGLqD/lozzl2D3EodJlAscZiQVFJdx7p9Rzh/dO8qbQCMHpgAwCabKdcEiCUOE5Le3lICwNzRvat9AyAhJpLstDgrcZiAscRhQtKbW0rISI5l1IAEr0PxxNiMJOuSawLGEocJOccafPyroJTzRvfDWcK+9xk7KJGiimNU1jZ6HYoJQZY4TMj5sOAwdY3NnNfLuuG2NG7Q8QZyK3UY/7PEYULOW1sPER8dwcycNK9D8cynU49YO4fxP0scJqSoKm9vLWHOiL5ERfTeH++0+GgGJsVYicMERO/9zTIhaduhag5V1XP2iN7XDfdEYwclsslKHCYALHGYkPL+9lIAzhzR1+NIvDd2UBKFpTXUNjR5HYoJMZY4TEj5YMdhcvvFMzAp1utQPDd2UCLNClsOVHsdigkxljhMyDjW4GPFrnLmjEj3OpSgMNademSztXMYP7PEYULGil1lNDQ1c2auVVMBDEqKISUukk3F1s5h/MsShwkZ728/TFREWK/uhtuSiDhTrB+wEofxL0scJmR8sKOUmTmpxEaFex1K0Bibkcj2gzU0NDV7HYoJIQFNHCIyX0S2iUiBiNzZyvFoEXnWPb5CRLJbHLvL3b9NROadcF24iKwVkZcCGb/pOfYfOcaOkhrm5Fr7RktjByXR4GtmR4k1kBv/CVjiEJFw4H7gQmAMcLWIjDnhtJuAClUdDvwauM+9dgywEBgLzAcecO933O3AlkDFbnqeD3ZYN9zWjLMR5CYAAlnimAEUqGqhqjYAi4EFJ5yzAHjC3V4KzBVnVroFwGJVrVfVXUCBez9EJBO4GHgkgLGbHub97YfpnxjNyP69czbck8lO60OfqHDybW0O40eBTBwZwL4W74vcfa2eo6pNQCWQ1sa1vwG+B5yy0lZEbhaRPBHJKy0t7eRHMD1Bc7Py4c7DzB6e3mtnwz2ZsDBhzKBEK3EYv+pRjeMicglQoqqr2zpXVR9W1WmqOi093eq9Q9mWg1UcqW3k9GHWm6o1YwclsflAFb5m9ToUEyICmTiKgawW7zPdfa2eIyIRQBJQdoprzwAuE5HdOFVf54rIXwIRvOk5lu8sA2CWJY5WjRmUSG2Dj91lR70OxYSIQCaOVUCuiOSISBROY/eyE85ZBlzvbl8BvK2q6u5f6Pa6ygFygZWqepeqZqpqtnu/t1X12gB+BtMDfFxYxpC0OAYl2zQjrTm+NoetQW78JWCJw22zuA14DacH1BJVzReRn4rIZe5pjwJpIlIA3AHc6V6bDywBNgOvAreqqi9QsZqeq8nXzIrCcqumOoXc/vFEhYex2do5jJ9EBPLmqvoy8PIJ+37YYrsOuPIk194L3HuKe78LvOuPOE3Plb+/iur6Jk4baonjZCLDwxg5IMEayI3f9KjGcWNOtLzQbd+wxHFKztoclTg1wcZ0TbsSh4g8JyIXi4glGhNUlu8sY1h6H/olxngdSlAbm5HEkdpG9lfWeR2KCQHtTQQPAF8BdojIz0VkZABjMqZdGn3NrNpdbr2p2uH4GuTWQG78oV2JQ1XfVNVrgCnAbuBNEflIRG4UkchABmjMyWwoqqS2wcesoTbNSFtGD0gkTGzqEeMf7a56EpE04Abga8Ba4Lc4ieSNgERmTBs+dts3Thua6nEkwS82Kpxh6fE29Yjxi3b1qhKR54GRwFPApap6wD30rIjkBSo4Y05l+c4yRvZPIC0+2utQeoSxgxL5uLDc6zBMCGhvieNPqjpGVf/neNIQkWgAVZ0WsOiMOYmGpmby9pRbaaMDxmUkcbCqjpIqayA3XdPexPH/Wtm33J+BGNMR+fsrqWtsZoat9tdukwenALBm7xFvAzE93imrqkRkAM6stLEiMhk4PvVoIhAX4NiMOam83RUATM9O8TiSnmNcRiJR4WGs3VvB/HEDvA7H9GBttXHMw2kQzwT+r8X+auDuAMVkTJtW7S5nSFqcjd/ogOiIcMYMSmStlThMF50ycajqE8ATIvIlVf17N8VkzCmpKnl7KjhnZD+vQ+lxpgxO4a8r99DoayYy3Mbzms455U+OiByfeTZbRO448dUN8RnzOYWHj1J+tMGqqTphypBk6hqb2XLAxnOYzmvrT44+7td4IKGVlzHdLm+306V0Wrb1qOqo4w3kVl1luqKtqqqH3K8/6Z5wjGnbqt0VpPaJYlh6n7ZPNp8xKCmG/onRrNlbwfWnZ3sdjumh2jvJ4f+KSKKIRIrIWyJS2qIay5hulbe7nGlDUmx98U4QEaYMTmHN3gqvQzE9WHtbxy5Q1SrgEpy5qoYD3w1UUMacTEl1HbvLaplu1VSdNnlwMvvKj1FaXe91KKaHam/iOF6ldTHwN1W1CW+MJ1a74zemWcN4p035pJ3DSh2mc9qbOF4Ska3AVOAtEUkHbN4C0+1W7i4nJjKMse462qbjxmUkERkuNoLcdFp7p1W/EzgdmKaqjcBRYEEgAzOmNXm7K5iclUJUhI1B6KyYyHDGDEy0EofptI6sOT4KZzxHy2ue9HM8xpxUbUMTmw9UccvZw7wOpcebPDiFZ1fts4GAplPa26vqKeCXwGxguvuyWXFNt9pYVImvWT+pozedN3VICscafbawk+mU9pY4pgFj1Fa6Nx5au+8IABOzkj2NIxTMdKejX1FYxiR7nqaD2ltG3QTYdJrGU+v2HmFIWhypfaK8DqXH65cQw9D0PqzYZQs7mY5rb4mjL7BZRFYCn3T+VtXLAhKVMa1Yt+/IJ38pm66bmZPGS+v342tWwsNsMKVpv/aWOH4MfAH4GfCrFq9TEpH5IrJNRApE5M5WjkeLyLPu8RUikt3i2F3u/m0iMs/dFyMiK0VkvYjki4hNhdJLHKys42BVnVWr+NFpQ1Oprm9is7VzmA5qb3fc93BGjEe626uANae6RkTCgfuBC4ExwNUiMuaE024CKlR1OPBr4D732jHAQmAsMB94wL1fPXCuqk4EJgHzReS09nwG07Ot2+d0HbXE4T8z3dUTV+wq8zgS09O0t1fV14GlwEPurgzgH21cNgMoUNVCVW0AFvP5sR8LgCfc7aXAXHEmIFoALFbVelXdBRQAM9RR454f6b6swb4XWLv3CFHhYYwZlOh1KCFjQFIMOX378NFOSxymY9pbVXUrcAZQBaCqO4C2VtHJAPa1eF/k7mv1HFVtAiqBtFNdKyLhIrIOKAHeUNUVrX1zEblZRPJEJK+0tLStz2eC3Np9RxgzKJHoiHCvQwkpZwxP4+PCMhqamr0OxfQg7U0c9W6pAQB3EKAnf+mrqk9VJ+EsZztDRMad5LyHVXWaqk5LT0/v1hiNfzX5mtlYVGnVVAEwe3g6tQ0+1rldnY1pj/YmjvdE5G4gVkTOB/4GvNjGNcVAVov3me6+Vs9xk1ESUNaea1X1CPAOThuICWHbD9VwrNHH5MHJXocScmYNSyNM4F87rFRu2q+9ieNOoBTYCHwDeBn4QRvXrAJyRSRHRKJwGruXnXDOMuB6d/sK4G13kOEyYKHb6yoHyAVWiki6iCQDiEgscD6wtZ2fwfRQx/8athKH/yXFRjIhM5kPCg57HYrpQdo1jkNVm0XkH8A/VLVdf5qoapOI3Aa8BoQDj6lqvoj8FMhT1WXAo8BTIlIAlOMkF9zzlgCbgSbgVlX1ichA4Am3h1UYsERVX+rIBzY9z9q9zop/g1PjvA4lJJ2Z25f73ymg8lgjSbGRXodjeoBTJg63h9OPgNtwSyci4gN+r6o/bevmqvoyTumk5b4fttiuA648ybX3AveesG8DMLmt72tCy7p9R5iUlWwr/gXInBHp/P7tAv614zAXTxjodTimB2irquo/cHpTTVfVVFVNBWYCZ4jIfwQ8OtPrVdc1UlBaY9VUATQ5K5mk2Eje3lridSimh2grcXwVuNodSwGAqhYC1wLXBTIwYwA2FFWiau0bgRQRHsZZI9J5b3sJzc02LMq0ra3EEamqn2s1c9s5rDLUBNw6mxG3W5w7qh+HaxrYUGyrQpu2tZU4Gjp5zBi/WLv3CEPT+1ijbYCdNSKdMIG3txzyOhTTA7SVOCaKSFUrr2pgfHcEaHovVWXdvgqrpuoGKX2imDokhTe2WDuHadspE4eqhqtqYiuvBFW1PwFNQBVVHONwTQOTbcW/bjFv7AC2HKhiT9lRr0MxQc4WGzZB63j7xmQrcXSLeWOdtdpeyz/ocSQm2FniMEFr3b4jREeEMXJAgteh9ApZqXGMy0jk1U2WOMypWeIwQWvdviOMz0giMtx+TLvL/LEDWLP3CAcr67wOxQQx+400QamhqZmNxTYjbne7aLwzcvylDfs9jsQEM0scJihtPVhFQ1Mzk2xG3G41ND2ecRmJLFtvicOcnCUOE5Q+aRi3HlXdbsHEDDYUVbLrsPWuMq2zxGGC0rq9R0hPiGZQUozXofQ6l0wciAgsW2elDtM6SxwmKNmMuN4ZmBTLzJxUnltbZHNXmVZZ4jBB50htA4WHj1rDuIeump7FnrJaPt5V5nUoJghZ4jBBxwb+ee/CcQNJjIlg8cp9XodigpAlDhN01u07ggiMz0zyOpReKyYynC9OyeTVTQepOGrzmZrPssRhgs66fUcY0S+BhBibDs1LV03PosHXzPNri70OxQQZSxwmqKgq692GceOt0QMTmZiVzOJVe1G1RnLzKUscJqjsKaulorbRBv4FiYXTs9h+qIa1bruTMWCJwwSZtfsqAFsqNlhcOnEQcVHhLF651+tQTBCxxGGCyrq9R4iLCmdEf5sRNxjER0ewYFIGL6zbT7k1khuXJQ4TVNbuO8KEzCTCw2zgX7BYdEY29U3NPP3xHq9DMUHCEocJGnWNPjbvr2KKzU8VVHL7J3DWiHSe/HgP9U0+r8MxQSCgiUNE5ovINhEpEJE7WzkeLSLPusdXiEh2i2N3ufu3icg8d1+WiLwjIptFJF9Ebg9k/KZ7bSyupKlZbWLDIHTT7BxKq+t5af0Br0MxQSBgiUNEwoH7gQuBMcDVIjLmhNNuAipUdTjwa+A+99oxwEJgLDAfeMC9XxPwHVUdA5wG3NrKPU0PtWaP0zA+2XpUBZ0zc/syon88j/xrl3XNNQEtccwAClS1UFUbgMXAghPOWQA84W4vBeaKM6vdAmCxqtar6i6gAJihqgdUdQ2AqlYDW4CMAH4G043W7j3C4NQ4+sZHex2KOYGIsOiMHLYcqGJ5oc1f1dsFMnFkAC0nuini8//Jf3KOqjYBlUBae651q7UmAyta++YicrOI5IlIXmlpaec/hekWqsqavRVMsdJG0PrC5AzS+kTx6Ae7vA7FeKxHNo6LSDzwd+DfVbWqtXNU9WFVnaaq09LT07s3QNNh+yvrKKmuZ8oQa98IVjGR4Xx11hDe2lrC1oOt/tqZXiKQiaMYyGrxPtPd1+o5IhIBJAFlp7pWRCJxksbTqvpcQCI33e6T9o0sSxzB7IbTs4mLCufBd3d6HYrxUCATxyogV0RyRCQKp7F72QnnLAOud7evAN5Wp+VtGbDQ7XWVA+QCK932j0eBLar6fwGM3XSztXuPEBMZxqiBNvAvmCXHRXHtaUN4cf1+9pTZ0rK9VcASh9tmcRvwGk4j9hJVzReRn4rIZe5pjwJpIlIA3AHc6V6bDywBNgOvAreqqg84A/gqcK6IrHNfFwXqM5jus2ZvBRMykokM75G1p73K12bnEBEexh/fK/Q6FOORiEDeXFVfBl4+Yd8PW2zXAVee5Np7gXtP2PcvwIYUh5i6Rh/5+ytZNDvH61BMO/RLjOHL0zJZsqqI2+fmMsDWhe917M8747n8/VU0+tTaN3qQb8wZhk+VRz6wUkdvZInDeG7tXqdh3Lri9hxZqXEsmDiIp1fstRUCeyFLHMZza/ZWkJEcS79Eq/LoSb519jCONfp4/EMb19HbWOIwnlJVVu6qYHq2VVP1NLn9E5g3tj9//mg31XWNXodjupElDuOpwsNHOVxTz8yhaV6HYjrh1nOGU1XXxFM25XqvYonDeGpFYTkAM3NSPY7EdMaEzGTOGpHOIx/s4mh9k9fhmG5iicN4asWuMvrGR5PTt4/XoZhOuv28XMqPNvAXK3X0GpY4jGdUlRWF5cwcmoozKYDpiaYMTuHM3L48/H4htQ1W6ugNLHEYz+wrP8bBqjpOs2qqHu/2ubmUHW3gryv2eh2K6QYBHTluzKl8vMtZ16FbG8ZVoXg17P4AyndBcxP06QsDJ8GwcyE2uftiCSHTslM5Y3gaf3yvkGtmDiE2KtzrkEwAWeIwnllRWE5qnyhy+8UH/pupwsa/wfu/gMPbnX190iE8Go6WgK/B2Z58LZxxO6QMCXxMIeb2uSP48kPL+evKvdxk08eENEscxjMrdpUxPTsl8O0b1QfhH7fAzreg/3hY8ACMmA993JKOrxH2r4W1f4G1T8HqP8Ppt8HZd0OkDUpsrxk5qcwamsYf39vJNTMHExNppY5QZW0cxhPFR45RVHGMmTkBrqYqfBcemAV7PoKLfgnfeB8mX/Np0gAIj4SsGXDZ7+D29TDpK/Dhb+GPs6FodWDjCzG3n5dLaXU9z6y0to5QZonDeGJF4fH2jQA2jG95EZ6+EuL7OwljxtchrI0f+cRBsOAP8NV/QFMdPD4f8h53qrpMm04bmsbMnFT++N5O6hp9XodjAsQSh/HEyl3lJMZEMGpAYmC+wYYlsOQ6GDgRbnwZ0kd07Pph5zjJJmcOvPTvsOzb0FgXkFBDze1zczlUVc+SvH1eh2ICxBKH6Xaqyoc7DzMjJ43wsAC0b+x4A57/JmTPdkoOcZ0s1cSlwleWwJzvOm0fj8+HyiK/hhqKZg1LY3p2Cg++u5P6Jit1hCJLHKbb7S6rZV/5MeaM6Ov/mxetdkoaA8bBwr9CdBd7bIWFw7k/cO51uAAeOgt2feCfWEOUiHD73BEcqKxjSZ4l2lBkicN0u/e3lwIwJzfdvzeuLIJnrnK62X7lbxDtx/XLR10MX3/bKYU8uQCWP2DtHqdwxvA0pg5J4cF3CqzUEYIscZhu9/72UganxpHtz/mpGo/Bs9c67RDX/A0S+vvv3selj4CvvQUjL4TX7oLnvg4NR/3/fUKAU+rIZX9lHX9fXex1OMbPLHGYbtXQ1MzywjL/VlOpwj+/44zF+OLDkD7Sf/c+UUwifPkpOPe/YONSePAM2P1h4L5fD3Zmbl8mD07m/ncKaGhq9joc40eWOEy3yttdTm2Dj7NG9PPfTdc8CeuehrPuhFEX+e++JxMWBnP+E254CVD480VOr6tK+8u6JRHh388bQfGRYyxeZeM6QoklDtOt3thyiKiIME4f5qeBfyVb4JXvw9Cz4azv++ee7ZU9G771Ecy6DdY9A7+bDP/8Tziw3to/XHNy+zIzJ5XfvVVgM+eGEJtyxHQbVeWNzYeYPbwvfaL98KPXUAt/u8HpOXX5w20P7guEqD4w716YcTO897+w5glY9SdIHwVDToeMaZCcBX36OSPUm33Q3OjMjdVQ67SRNB512mjCoyAy1hmwmDoUYlOgh083LyJ8b/4ovvTgRzz+4W5uPWe41yEZP7DEYbrN1oPVFFUc4zZ//efx6vehdBt89bnANIZ3RMoQ+ML9cMF/Q/7zsGUZbPgb5D3W+XvG94ecs5xZe0df2vWuxR6ZOiSF80b3/2QOq+S4KK9DMl0U0MQhIvOB3wLhwCOq+vMTjkcDTwJTgTLgKlXd7R67C7gJ8AH/pqqvufsfAy4BSlR1XCDjN/71xuZDiMDc0X74T37jUqdtY/Ydzn+swSIuFabf5LyafVBeCNUHoKbEmcI9LMJ5hUdBVBxE9nG/xoKvCRprnfPLC6F4Dex8GzYugZe/CxO+7Ey+mDrU60/ZYd+dN5L5v32fB9/byV0XjvY6HNNFAUscIhIO3A+cDxQBq0RkmapubnHaTUCFqg4XkYXAfcBVIjIGWAiMBQYBb4rICFX1AX8G/oCTcEwP8lr+QSZnJZOeEN21G5XthBdvh6yZcM49/gkuEMLCoW+u8+qQSZ9uNjdD0Upnxt51TztVYVNvdNpz4v08DiaARg5I4PLJGfz5w91cPyubQcmxXodkuiCQlcIzgAJVLVTVBmAxsOCEcxYAT7jbS4G54syxvQBYrKr1qroLKHDvh6q+D5QHMG4TADtLa8jfX8XFEwZ17UZN9bD0Ruev9i89CuEhXtsaFgaDT4PL/wj/tg4mf9Wp/rp/Oqxf3KMa4e8435kv7Gcvb/E4EtNVgUwcGUDLWc6K3H2tnqOqTUAlkNbOa09JRG4WkTwRySstLe1g6MbfXli3HxG4dMLArt3ojR85vZYW3O80OvcmiQPh0t/ALcuh7wh4/hvwzEKo7Rl/R2WmxPGts4fx0oYDfOzOjmx6ppDtjquqD6vqNFWdlp7ec4r0oUhVWbaumNOHpdEvsQsLI21eBisehBnfgNGX+C/AniZ9JNz4Csz7H6cN5KE5UJTndVTt8s2zhpGRHMuPl+XT5LNBgT1VIBNHMdDyT8JMd1+r54hIBJCE00jenmtND7G+qJLdZbUsmNihQuNnle+CF26DQVOcnku9XVg4zLoFFr3mdNl9bD6seCjoq65iIsP5wcWj2Xqw2hZ76sECmThWAbkikiMiUTiN3ctOOGcZcL27fQXwtqqqu3+hiESLSA6QC6wMYKwmgJ5dtY+YyDDmjRvQuRs01TvjNQS48nGI6GLjeijJmOKsGzJ8LrzyPVh2GzQ1eB3VKc0fN4DTh6Xxy9e3U340uGM1rQtY4nDbLG4DXgO2AEtUNV9Efioil7mnPQqkiUgBcAdwp3ttPrAE2Ay8Ctzq9qhCRJ4BlgMjRaRIRG4K1GcwXVdd18gL64q5bOIgkmIjO3eT138AB9Y5a4WnZPszvNAQmwILn4E533PWTf/LF+FYhddRnZSI8OPLxlLb0MRPXsz3OhzTCQHtkqKqLwMvn7Dvhy2264ArT3LtvcC9rey/2s9hmgD6x9piaht8XDNzSOdukP88rHwYTru1d7drtCUsDM69B9KGOfNmPXKeswhV2jCvI2vViP4J3HZOLr9+czuXTBjE+WM8HsBpOiRkG8eN95qblac+3sP4jCQmZiV3/AaHC+CFbzvTdpz3Y3+HF5omLoTrXnB6Wj0yN6hn7v3W2cMYPTCRu57bQGl1vdfhmA6wxGEC5q2tJWw/VMONZ2R3/OK6SqeraUSU265h01S025DT4etvOQtaPbkA1j7tdUStiooI47cLJ1Fd18QdS9bR3BzcDfvmU5Y4TECoKn94p4Cs1Fgum9jBQX/NPlh6E1Tscta+SB4cmCBDWepQuOl1J4m8cAu8do8zpUmQGdE/gR9eOoYPdhzmd2/v8Doc006WOExA/KvgMOv3HeGbZw0jIryDP2Zv/QQK3oAL/xeyzwhMgL1BbApc+3dn5t7lf4C/fjkoG82/MmMwX5qSyW/e3MErGw94HY5pB0scxu+afM3c+88tZCTHcsXUzI5dvH4xfPhbmLbImSjQdE14JFz0C7j0t7DrffjTXCjd7nVUnyEi3Hv5OCYPTubfn11no8p7AEscxu8Wr9rH1oPV3HPxaKIjwtt/YcGb8MKtkH0mzL8vcAH2RlNvgOtfdNqOHpkL+f/wOqLPiIkM59Hrp5OVGsfXnsgjb3fPmEalt7LEYfyqpKqOX76+jZk5qVzYkQF/xavh2esgfTQsfNoawwNhyCy4+V1IGw5/u94Zid9w1OuoPpHaJ4q/3DST9IRornlkBW9uPuR1SOYkLHEYv2luVv5z6QbqGn3ce/l4pL2r1x0ugKevhD5pcO1SiEkKbKC9WXKW02g++w5nsOBDc5ykHSQGJMWw9JuzGDkgga8/lcevXt9mc1oFIUscxm8eer+Q97eXcs/FYxjer52r1VXsgacuBwS++g9I6OS0JKb9wiPhvB/B9cuc5WsfOQ9euRPqq72ODIC0+GievXkWV0zJ5PdvF7Dg/g9Zuzf4GvV7M0scxi9e2rCf+17dysUTBnLtzHZ2ny0vhMcvgvoqp/dPkI5yDlk5c5wp2qctghV/hN9PdRaMCoJuu7FR4fziyok8cM0USqvrufyBj/jaE3ks31lm4z2CgGiQz6bpD9OmTdO8vJ4x7XRP9MrGA9y+eB0TMpP4y9dmEhPZjgbxwwXwxKXQVOeMdB44IfCBmpMryoPX7oZ9K5y1Pk7/N2ep2iCYULK6rpHHP9zNIx8UUlXXRFZqLFdMyWLBpEFk9+3jdXghS0RWq+q0Vo9Z4jCdpao89uFu7v3nZiYPTuGx66eTFNeOiQwP5cNTX4TmRrhuGQywpeODgipsfQne/Tkc2gTx/WHG12HydZDg/VxSxxp8vJZ/kCV5+/hop9Nld9SABOaPG8CF4wYyon98+9vVTJsscVji8LuSqjru+ccm3th8iPNG9+d3V08iLqodc2bueNOZIj06Hr76PPQbHfBYTQepQuG7zqDBgjdBwmDoOTDhKhh1sfNv57Giilpeyz/Eq5sOkLenAlXI6duH+eMGcPnkDEb0T/A6xB7PEoclDr+pONrA0yv28NB7hdT7mvnevJHcNDunfX/prXoEXv4u9B8LVz8LSV1Y2Ml0j9LtsOFZ2LAEKvdCeDTknAkj5kPuBZDSyVmP/aikuo43Nh/i1U0HWb6zjKZm5fRhadx4Rg7njupHeJiVQjrDEoclji4rKKnhsQ938dyaIuoamzl3VD/+65Ix5LSnjrmxDl6/x0kcufPgiseC4q9W0wHNzU77x9aXYPurUFbg7E8fDSMucP5ds2Y4PbY8VH60gcWr9vLU8j0cqKxjWHofvjtvFPPG9rdqrA6yxGGJo1NUlQ92HOaxD3fx7rZSoiLC+OLkDBbNzml/VUDpdli6CA5thFm3wfk/dZY9NT3b4QLY8ZqTRPZ8BM1NEJ0Ew891SiLDz4f4dM/Ca/I188qmg/z6ze0Ulh5lYlYyd104itOGpnkWU09jicMSR4fUNfr4x9piHvtwF9sP1dA3PprrZg3hmpmDSYtvZy8bVWeA2Svfg8hY+MKDMGJeYAM33qirctpEdrwOO96AmoOAQMZUGHu58/KoWrLJ18xza4r59ZvbOVBZx7yx/bn7otEMSbPeWG2xxGGJo11Kquv4y/I9/GXFXsqPNjB6YCI3zc7h0okDOzbn1OEd8M87nEn1ss+EL/4JEgcGLnATPJqbndLl9tdh64twYL2zf/AsGPtFGLPAkx5adY0+/vR+IQ++t5Mmn3LDGdncdu5wEmO8rVoLZpY4LHGc0ub9VTz6r128uH4/jc3NzB3Vj0Wzc5g1NK1j9cINtfDhb+Bfv4aIWGd08tQbnWVNTe9UthM2PQf5z0HJZqeHVs4cp4fW6Eshunt7Px2qquOXr21j6ZoiUuOi+I/zR7BwelbHp/7vBSxxWOL4nEZfM29uPsSTy/ewvLCM2MhwrpyWyY1n5LSvwfszN6tzRhx/8Cs4WgLjvwzz7oX4fgGJ3fRQJVth099h49+cRboiYmHURU4SGXZutzasbyqu5KcvbWblrnJG9k/gB5eM5sxc79pkgpElDkscnyiqqGXxyn08m7eP0up6BiXFcN3p2Vw9fXD7Bu+1VFcF659x1s+oKoYhs+HcHzizsBpzMqpQtMrp5rvpOThWDnFpMO5LThLJmArd0ANKVXl100F+9soW9pUfY1JWMtfNGsJF4we2b/aDEGeJo5cnjiO1Dby++RAvbTjABztKEeCckf34yszBnD2yE/3cS7bAqkedpNFQA5kz4Nx7IOesbvmFNyGkqQF2vg0bFsO2V5wpaFJynAQy4cvdMn9ZXaOPZ1ft44nluyksPUpqnyjOH92fOSPSmT28b8f/oAoRljh6WeJo9DWzsbiS5TvLWL6zjI8LnUFRmSmxfHFyBlfNGExGcmzHblq+y6mn3vScMx1FeDSM+yJM/zpkTg3MBzG9S10VbHnRKYnseh9QyJjmlESGnwd9cwP6h4mq8mFBGc+s3Mv7O0qprmsiTGBwahw5ffuQ0zee9IRoEmMjSIyJpE90ODGRziv2M1/DiI0K71iHkiBkiSNEEkdzs3Ks0cfRhiaONfg4Wu+jtKaekqo6SqrrKSipYfuhagpKaqhvctYwGNE/nnNH9efi8QMZl5HY/sbuukqnf/7Od6DwHTjsLjeaOcNJGOOvhD59A/RJTa9XtR82LnWSyKFNzr6kLKctZPhcp5dWANvQmnzNrC86wgc7DrPjUA2Fh4+y63ANdY3tXxskMSaCAUkx9E+MYUBiDIOSY8nuG8fg1D5kp8WR2icqqAclepY4RGQ+8FsgHHhEVX9+wvFo4ElgKlAGXKWqu91jdwE3AT7g31T1tfbcszXBlDiafM2UHW2gtLqe0pp6ymsaKDtaT9nRBspqGig/2kBNXRPHGn3UNjRR19hMbYPzvq0f2oFJMeT2T2BEv3imDElhZk5q2+MufE3OVBJlO51utAfWQfEaKNvhHI+IhSGnO7+wYy6D5HZOmW6Mv1TsgZ1vQcFbUPgeNLjrhiQPhszpMGCCM6Nv31xIyQ5YI7uq84dbdV0TVccaqW3wcazRedW7X481NFPX6ONofRMl1fUcrKqjpKrO+VpdT8v/buOjIxicGveZZDI4LY4haX0YmBhDmMdTpXiSOEQkHNgOnA8UAauAq1V1c4tzbgEmqOo3RWQhcLmqXiUiY4BngBnAIOBNYIR72Snv2ZquJg5VpalZafIpTc3N+JqVRp+6X50flKq6Jmrqm6iua6SmronquiYqahsoqa6npLreSRTVdZQdbaC1Rx4VHkZafBSpfaJIjIkkNsop9rb8GhMZTp+ocOIiw4iLCqNPBKTFhdOvj/OKDVdQn1NP3HDUfdU4X+troPYw1JQ4r6MlUFkMFbudWWqPix8AGVNg0BRnComsmRAZ0+lnZ4xf+RqdFQuLVjlTwRflQVXRp8fDIpzkkTDQWRQsvr/zNTYFouKdqW6iEtyvfSA8yrnm+Cs88tNtP5cG6pt87Cs/xt7yo+w+XMve8lr2lB1lT1kt+ypqafR9+h9DVEQYWSmxZKf1YXBaHFkpcSTFRpIYG0liTAQJMZHERYUTFRFGZHgYURFhRIWHERkufutafKrE0Y7pTDttBlCgqoVuEIuBBUDL/+QXAD92t5cCfxCn7LYAWKyq9cAuESlw70c77uk3E3/yOjX1Tfg6uXBMRJiQnhBNekI0GckxTMpKIj0hxtkXH016QhRpfaJJi48iPjri1MXWZ6+Fba86UzvQhWQfEesU8eP7OTPTjr7EWYM6dZjTEGldaE0wC4+Ewac5r+PqKp0pUA5vd17lO6H6kJNcqg9B07HOfS8Jd5KHhMFVTztzcnVBdEQ4w/vFt7o6pq9Z2X/kGHvLa9lddpS9ZbXsKXO2lxeWUdvg69D3Cg8TwgTS46P56K65XYq7NYFMHBnAvhbvi4CZJztHVZtEpBJIc/d/fMK1x+csaOueAIjIzcDN7tsaEdnWic/QZTs/+7YvcNiLOD5VBRzyNoTPCoJnEnTsmXyet8/kh0E5XU6bz6QAkLs7ff+TTn0cyMThKVV9GHjY6zhaEpG8kxX9eit7Jp9nz+Tz7Jl8npfPJJDj7IuBrBbvM919rZ4jIhFAEk4j+cmubc89jTHGBFAgE8cqIFdEckQkClgILDvhnGXA9e72FcDb6rTWLwMWiki0iOQAucDKdt7TGGNMAAWsqspts7gNeA2n6+xjqpovIj8F8lR1GfAo8JTb+F2Okwhwz1uC0+jdBNyqqj6A1u4ZqM8QAEFVdRYk7Jl8nj2Tz7Nn8nmePZNeMQDQGGOM/9hcwsYYYzrEEocxxpgOscTRRSLymIiUiMimFvuuFJF8EWkWkWknnH+XiBSIyDYRmddi/3x3X4GI3Nmdn8HfTvJMfiEiW0Vkg4g8LyLJLY711mfy3+7zWCcir4vIIHe/iMjv3M+9QUSmtLjmehHZ4b6ub+179QStPY8Wx74jIioifd33If884KQ/Iz8WkWL3Z2SdiFzU4ph3vzeqaq8uvIA5wBRgU4t9o4GRwLvAtBb7xwDrgWggB2d8YLj72gkMBaLcc8Z4/dn8/EwuACLc7fuA++yZkNhi+9+AP7rbFwGvAAKcBqxw96cChe7XFHc7xevP5q/n4e7Pwun8sgfo21uexyl+Rn4M/Gcr53r6e2Mlji5S1fdxeoS13LdFVVsbqf7JVCqqugtnYOcMWkzPoqoNwPGpVHqkkzyT11W1yX37Mc4YHOjdz6Sqxds+fDqXzALgSXV8DCSLyEBgHvCGqparagXwBjA/8NH7X2vPw/Vr4Ht8dl6dkH8ecMpn0hpPf28scXSv1qZhyTjF/lC1COcvSOjlz0RE7hWRfcA1wA/d3b3ymYjIAqBYVdefcKhXPo8WbnOr6B4TkRR3n6fPxBKH6VYicg/O2JynvY4lGKjqPaqahfM8bvM6Hq+ISBxwN58mT+N4EBgGTAIOAL/yNBqXJY7u1aunUhGRG4BLgGvUraillz+TFp4GvuRu98ZnMgynrn69iOzG+WxrRGQAvfN5AKCqh1TVp6rNwJ/4dJZwT5+JJY7u1WunUhFnAa7vAZepam2LQ735meS2eLsA2OpuLwOuc3sTnQZUquoBnEbjC0Qkxa2yuMDd1+Op6kZV7aeq2aqajVPFMkVVD9ILn8dxblvOcZcDx3tceft743VPgp7+wllw6gDQiPPDfpP7D1wE1OPMYf5ai/Pvwen1sA24sMX+i3AWqdoJ3OP15wrAMynAqXtd577+aM+Ev+P8R7ABeBHIcM8V4H73c2/ksz3zFrnPsgC40evP5c/nccLx3Xzaqyrkn8cpfkaecj/zBpwEMLDF+Z793tiUI8YYYzrEqqqMMcZ0iCUOY4wxHWKJwxhjTIdY4jDGGNMhljiMMcZ0iCUOY/xMRHzuTKbrRWSNiJzexvnJInJLd8VnTFdZd1xj/ExEalQ13t2eB9ytqmed4vxs4CVVHddNIRrTJVbiMCawEoGK429E5LsissqdtO4n7u6fA8PcUsovRCReRN5ySysb3cn/jAkaEV4HYEwIihWRdUAMMBA4F0BELsCZGmIGzmjoZSIyB7gTGKeqk9zzIoDLVbXKXczoYxFZplY9YIKEJQ5j/O9YiyQwC3hSRMbhzKV0AbDWPS8eJ5HsPeF6AX7mJpVmnGmx+wMHAx+6MW2zxGFMAKnqcrfUkI6TEP5HVR9qeY7bxtHSNe75U1W10Z0tNqYbwjWmXayNw5gAEpFROMt5luHM3LpIRI43nGeISD+gGkhocVkSUOImjXOAId0ctjGnZCUOY/zveBsHOKWM61XVB7wuIqOB5SICUANcq6o7ReRDEdmEszLifcCLIrIRyOPT6daNCQrWHdcYY0yHWFWVMcaYDrHEYYwxpkMscRhjjOkQSxzGGGM6xBKHMcaYDrHEYYwxpkMscRhjjOmQ/w+SVT7grNkNtQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Beta distribution in train data\n",
    "sns.kdeplot(train['Beta'],data =train, hue =\"label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    71\n",
       "1    20\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# labels cooresponding to beat nan values\n",
    "train[\"label\"][train['Beta'].isnull()].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 602 entries, 0 to 601\n",
      "Data columns (total 14 columns):\n",
      " #   Column   Non-Null Count  Dtype  \n",
      "---  ------   --------------  -----  \n",
      " 0   label    602 non-null    int64  \n",
      " 1   id       602 non-null    int64  \n",
      " 2   Alpha    602 non-null    int64  \n",
      " 3   Beta     524 non-null    float64\n",
      " 4   Gamma    602 non-null    int64  \n",
      " 5   Delta    602 non-null    int64  \n",
      " 6   Epsilon  602 non-null    int64  \n",
      " 7   Zeta     602 non-null    int64  \n",
      " 8   Eta      602 non-null    int64  \n",
      " 9   Theta    602 non-null    int64  \n",
      " 10  Iota     602 non-null    int64  \n",
      " 11  Kappa    602 non-null    int64  \n",
      " 12  Lambda   602 non-null    float64\n",
      " 13  omikron  602 non-null    float64\n",
      "dtypes: float64(3), int64(11)\n",
      "memory usage: 66.0 KB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "based on the above info we found that only beta in train and test data have nan values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label        2\n",
      "id         602\n",
      "Alpha        4\n",
      "Beta       126\n",
      "Gamma        3\n",
      "Delta      177\n",
      "Epsilon      2\n",
      "Zeta         6\n",
      "Eta          9\n",
      "Theta       45\n",
      "Iota         4\n",
      "Kappa      153\n",
      "Lambda     155\n",
      "omikron    102\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "id          1012\n",
       "Alpha          6\n",
       "Beta         150\n",
       "Gamma          3\n",
       "Delta        207\n",
       "Epsilon        2\n",
       "Zeta           9\n",
       "Eta           10\n",
       "Theta          6\n",
       "Iota           5\n",
       "Kappa        236\n",
       "Lambda       237\n",
       "omikron      123\n",
       "Del_bins       3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2    564\n",
      "1     38\n",
      "Name: Epsilon, dtype: int64\n",
      "2    380\n",
      "1     30\n",
      "Name: Epsilon, dtype: int64\n",
      " 5     432\n",
      " 9     104\n",
      "-1      48\n",
      " 2       8\n",
      " 8       4\n",
      " 3       2\n",
      " 10      2\n",
      " 1       1\n",
      " 4       1\n",
      "Name: Eta, dtype: int64\n",
      " 5     254\n",
      " 9      70\n",
      "-1      67\n",
      " 2       6\n",
      " 8       4\n",
      " 3       3\n",
      " 7       3\n",
      " 4       2\n",
      " 10      1\n",
      "Name: Eta, dtype: int64\n",
      "8     234\n",
      "11    233\n",
      "15    130\n",
      "9       2\n",
      "12      2\n",
      "6       1\n",
      "Name: Zeta, dtype: int64\n",
      "11    180\n",
      "8     118\n",
      "15    103\n",
      "12      4\n",
      "4       2\n",
      "3       1\n",
      "5       1\n",
      "6       1\n",
      "Name: Zeta, dtype: int64\n",
      "5    285\n",
      "4    170\n",
      "3    144\n",
      "1      3\n",
      "Name: Iota, dtype: int64\n",
      "5    179\n",
      "4    115\n",
      "3    113\n",
      "1      2\n",
      "2      1\n",
      "Name: Iota, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "## Unique values of all\n",
    "print(train['Epsilon'].value_counts())\n",
    "print(test['Epsilon'].value_counts())\n",
    "print(train['Eta'].value_counts())\n",
    "print(test['Eta'].value_counts())\n",
    "print(train['Zeta'].value_counts())\n",
    "print(test['Zeta'].value_counts())\n",
    "print(train['Iota'].value_counts())\n",
    "print(test['Iota'].value_counts())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id                         1012\n",
      "Late_2                       21\n",
      "Late_1                       15\n",
      "Late_3                       33\n",
      "days_late_Sum               338\n",
      "normal_payment               37\n",
      "p_code                       21\n",
      "p_limit                     290\n",
      "last_update                2778\n",
      "curr_remaining             3470\n",
      "max_bal                    4478\n",
      "recent_payment_activity    1671\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "label       0\n",
       "id          0\n",
       "Alpha       0\n",
       "Beta       78\n",
       "Gamma       0\n",
       "Delta       0\n",
       "Epsilon     0\n",
       "Zeta        0\n",
       "Eta         0\n",
       "Theta       0\n",
       "Iota        0\n",
       "Kappa       0\n",
       "Lambda      0\n",
       "omikron     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train_meta.nunique())\n",
    "train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0\n",
       "Late_2                        0\n",
       "Late_1                        0\n",
       "Late_3                        0\n",
       "days_late_Sum                 0\n",
       "normal_payment                0\n",
       "p_code                        0\n",
       "p_limit                    5199\n",
       "last_update                  23\n",
       "curr_remaining                0\n",
       "max_bal                     338\n",
       "recent_payment_activity     934\n",
       "dtype: int64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our Obsevations on Demographics data (Observation no-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the three things that is \n",
    "a) Intutuon \n",
    "b) plotting featues\n",
    "c) Reading about different features on internet.\n",
    "\n",
    "We made some hypothesis about data and demographics features which are made anonymized:\n",
    "1) **Dataset is not of bank loan but about credit card default dataset** as it contains a feature p_limit which decribes the credit limit expenditure and it contains features regarding late repayments and max balance in history which explains that it is about spending from a card and then repaying it.\n",
    "\n",
    "2) Epsilon has 2 unique values so after studying from internet abount these type of data we hypothise that this feature must be one of *Sex -> male -0  and feamle-1 here female have low majority in data* or *must be about marital status married or not*\n",
    "\n",
    "3) Gamma - Which contains 3 unique value must be the location from where customer is belonging i.e., to Urban, rural or Suburban areas.\n",
    "\n",
    "4) Theta is a type of positive integer value which representing the age of person encoded in some style to make the maximum at 115 and min at 60\n",
    "\n",
    "5) And out of Delta or kappa one must be the salary of the person.\n",
    "\n",
    "6) Last but not least, Alpha which have 4 unique value seems to be random but actually in data it is modified ,if we divide Alpha by 523 we get **resultant in 1, 2, ,3 ,4 ,5 , 6, 7**, very expressive to see, it may represent the educational qualification of that customer whether they pass out high school, secondary and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    236\n",
       "7    235\n",
       "5    130\n",
       "6      1\n",
       "Name: Alpha, dtype: int64"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Explaination of 523 division converted into nice categories\n",
    "(train['Alpha']/523).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Alpha', ylabel='count'>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAV1ElEQVR4nO3dfZBddZ3n8fcHCBNHYBUILEvHCSgTedIoLTO7KDoyDMi6gA6ypECJwYkPYGmt5YhY69OWVVOjjosjMhVHBGYh6AQZLEtZLRZlXVHsIAPhaXgK0qkY2sCADIIkfvePPjm2oUNuQt8+/fB+Vd3qe37nnNuf7kr1J+d3zz0nVYUkSQA7dR1AkjR1WAqSpJalIElqWQqSpJalIElq7dJ1gOdi7733rgULFnQdQ5KmlVWrVv2iquaNt65vpZBkPnApsC9QwPKqOj/JnsBXgQXAGuDUqnokSYDzgROAJ4AlVXXTs32PBQsWMDQ01K8fQZJmpCQPbG1dP6ePNgIfqKpDgD8Gzk5yCHAucG1VHQRc2ywDvAE4qHksAy7sYzZJ0jj6VgpVtW7z//Sr6pfAHcD+wEnAJc1mlwAnN89PAi6tUT8CXpBkv37lkyQ906S80ZxkAfAK4MfAvlW1rln1c0anl2C0MB4cs9twM7blay1LMpRkaGRkpH+hJWkW6vsbzUl2A64E3l9Vj42+dTCqqirJdl1no6qWA8sBBgcHvUaHJsTTTz/N8PAwTz75ZNdRtmru3LkMDAwwZ86crqNoButrKSSZw2ghXFZVX2+G1yfZr6rWNdNDDzXja4H5Y3YfaMakvhseHmb33XdnwYIFjP2Py1RRVWzYsIHh4WEOOOCAruNoBuvb9FFzNtGXgTuq6m/GrPoGcGbz/Ezg6jHjb8uoPwYeHTPNJPXVk08+yV577TUlCwEgCXvttdeUPpLRzNDPI4WjgLcCtya5uRk7D/gr4GtJzgIeAE5t1n2L0dNR72H0lNS39zGb9AxTtRA2m+r5NDP0rRSq6gfA1v4VHzPO9gWc3a88kqRt8zIX0nbYbbfdnnX9mjVrOOyww7brNZcsWcLKlSufSyxpwkzry1z04ogPXtp1BFZ9+m1dR5CknnikIO2Axx9/nGOOOYZXvvKVHH744Vx99dXtuo0bN3L66adz8MEHc8opp/DEE08AsGrVKl772tdyxBFHcNxxx7FunedRaOqxFKQdMHfuXK666ipuuukmrrvuOj7wgQ+w+da2d911F+95z3u444472GOPPfjiF7/I008/zXvf+15WrlzJqlWrWLp0KR/5yEc6/imkZ5rx00dSP1QV5513Htdffz077bQTa9euZf369QDMnz+fo446CoAzzjiDz3/+8xx//PGsXr2aY489FoBNmzax335exUVTj6Ug7YDLLruMkZERVq1axZw5c1iwYEH7GYItTx1NQlVx6KGHcsMNN3QRV+qZ00fSDnj00UfZZ599mDNnDtdddx0PPPDbKxH/7Gc/a//4X3755bz61a9m4cKFjIyMtONPP/00t912WyfZpWdjKUg74PTTT2doaIjDDz+cSy+9lJe+9KXtuoULF3LBBRdw8MEH88gjj/Dud7+bXXfdlZUrV/KhD32Il7/85SxatIgf/vCHHf4E0vicPpK2w+OPPw7A3nvvvdWpoDvvvHPc8UWLFnH99dc/Y/ziiy+esHzSc2UpSJoR/EzSxHD6SJLUshQkSS1LQZLUshQkSS1LQZLU8uwjaRwTfSZLL2elXHPNNbzvfe9j06ZNvOMd7+Dcc8+d0AxSLzxSkKaATZs2cfbZZ/Ptb3+b22+/nRUrVnD77bd3HUuzkKUgTQE33ngjL3nJSzjwwAPZddddOe20037nctzSZOlbKSS5KMlDSVaPGftqkpubx5rN925OsiDJr8as+7t+5ZKmorVr1zJ//vx2eWBggLVr13aYSLNVP99TuBj4AtBOzlbVf938PMlngUfHbH9vVS3qYx5J0jb0rRSq6vokC8Zbl9FrC58KvL5f31+aTvbff38efPDBdnl4eJj999+/w0Sarbp6T+E1wPqqunvM2AFJfprk+0les7UdkyxLMpRkaGRkpP9JpUnwqle9irvvvpv777+fX//611xxxRWceOKJXcfSLNTVKamLgRVjltcBL6qqDUmOAP4pyaFV9diWO1bVcmA5wODgYE1KWs06k31hs1122YUvfOELHHfccWzatImlS5dy6KGHTmoGCToohSS7AG8Gjtg8VlVPAU81z1cluRf4Q2BosvNJXTnhhBM44YQTuo6hWa6L6aM/Be6squHNA0nmJdm5eX4gcBBwXwfZJGlW6+cpqSuAG4CFSYaTnNWsOo3fnToCOBq4pTlFdSXwrqp6uF/ZJEnj6+fZR4u3Mr5knLErgSv7lUWS1Bs/0SxJalkKkqSWpSBJannpbGkcP/vk4RP6ei/66K3b3Gbp0qV885vfZJ999mH16tXb3F7qB48UpCliyZIlXHPNNV3H0CxnKUhTxNFHH82ee+7ZdQzNcpaCJKllKUiSWpaCJKllKUiSWp6SKo2jl1NIJ9rixYv53ve+xy9+8QsGBgb4xCc+wVlnnbXtHaUJZClIU8SKFVteJ1KafE4fSZJaloIkqWUpSI2qqX1316meTzODpSABc+fOZcOGDVP2D29VsWHDBubOndt1FM1wvtEsAQMDAwwPDzMyMtJ1lK2aO3cuAwMDXcfQDGcpSMCcOXM44IADuo4hda6f92i+KMlDSVaPGft4krVJbm4eJ4xZ9+Ek9yS5K8lx/colSdq6fr6ncDFw/Djjn6uqRc3jWwBJDgFOAw5t9vlikp37mE2SNI6+lUJVXQ883OPmJwFXVNVTVXU/cA9wZL+ySZLG18XZR+ckuaWZXnphM7Y/8OCYbYabsWdIsizJUJKhqfymoCRNR5NdChcCLwYWAeuAz27vC1TV8qoarKrBefPmTXA8SZrdJrUUqmp9VW2qqt8AX+K3U0RrgfljNh1oxiRJk2hSSyHJfmMW3wRsPjPpG8BpSX4vyQHAQcCNk5lNktTHzykkWQG8Dtg7yTDwMeB1SRYBBawB3glQVbcl+RpwO7AROLuqNvUrmyRpfH0rhapaPM7wl59l+08Bn+pXHknStnntI0lSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSq2+lkOSiJA8lWT1m7NNJ7kxyS5KrkrygGV+Q5FdJbm4ef9evXJKkrevnkcLFwPFbjH0XOKyqXgb8C/DhMevurapFzeNdfcwlSdqKvpVCVV0PPLzF2HeqamOz+CNgoF/fX5K0/bp8T2Ep8O0xywck+WmS7yd5zdZ2SrIsyVCSoZGRkf6nlKRZpJNSSPIRYCNwWTO0DnhRVb0C+G/A5Un2GG/fqlpeVYNVNThv3rzJCSxJs8Skl0KSJcAbgdOrqgCq6qmq2tA8XwXcC/zhZGeTpNluUkshyfHAXwInVtUTY8bnJdm5eX4gcBBw32RmkyTBLv164SQrgNcBeycZBj7G6NlGvwd8NwnAj5ozjY4GPpnkaeA3wLuq6uFxX1iS1Dd9K4WqWjzO8Je3su2VwJX9yiJJ6o2faJYktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVKrp1JIcm0vY5Kk6e1ZP9GcZC7w+4xequKFQJpVewD79zmbJGmSbesyF+8E3g/8B2AVvy2Fx4Av9C+WJKkLz1oKVXU+cH6S91bV305SJklSR3q6IF5V/W2S/wQsGLtPVV3ap1ySpA70VApJ/gF4MXAzsKkZLsBSkKQZpNdLZw8Ch2y+U5okaWbqtRRWA/+e0Xspa5Y64oNT48Bw1aff1nUEacbqtRT2Bm5PciPw1ObBqjqxL6kkSZ3otRQ+3s8QkqSpodezj76/Iy+e5CLgjcBDVXVYM7Yn8FVGz2RaA5xaVY9k9KbN5wMnAE8AS6rqph35vpKkHdPrZS5+meSx5vFkkk1JHuth14uB47cYOxe4tqoOAq5tlgHeABzUPJYBF/aSTZI0cXoqharavar2qKo9gOcBfw58sYf9rgce3mL4JOCS5vklwMljxi+tUT8CXpBkv17ySZImxnZfJbX5o/1PwHE7+D33rarNZzH9HNi3eb4/8OCY7YYZ5/pKSZYlGUoyNDIysoMRJEnj6fXDa28es7gTo59bePK5fvOqqiTb9dmHqloOLAcYHBz0cxOSNIF6Pfvov4x5vpHRN4hP2sHvuT7JflW1rpkeeqgZXwvMH7PdQDMmSZokvZ599PYJ/J7fAM4E/qr5evWY8XOSXAH8EfDomGkmSdIk6PXso4EkVyV5qHlcmWSgh/1WADcAC5MMJzmL0TI4NsndwJ82ywDfAu4D7gG+BLxnB34eSdJz0Ov00VeAy4G3NMtnNGPHPttOVbV4K6uOGWfbAs7uMY8kqQ96PftoXlV9pao2No+LgXl9zCVJ6kCvpbAhyRlJdm4eZwAb+hlMkjT5ei2FpcCpjH6uYB1wCrCkT5kkSR3p9T2FTwJnVtUj0F6/6DOMloUkaYbo9UjhZZsLAaCqHgZe0Z9IkqSu9FoKOyV54eaF5kih16MMSdI00esf9s8CNyT5x2b5LcCn+hNJktSVXj/RfGmSIeD1zdCbq+r2/sWSJHWh5ymgpgQsAkmawbb70tmSpJnLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVJr0i9ql2Qh8NUxQwcCHwVeAPwFMNKMn1dV35rcdJI0u016KVTVXcAigCQ7A2uBq4C3A5+rqs9MdiZJ0qiup4+OAe6tqgc6ziFJovtSOA1YMWb5nCS3JLlo7P0bxkqyLMlQkqGRkZHxNpEk7aDOSiHJrsCJwOZ7NFwIvJjRqaV1jN7D4RmqanlVDVbV4Lx58yYjqiTNGl0eKbwBuKmq1gNU1fqq2lRVvwG+BBzZYTZJmpW6LIXFjJk6SrLfmHVvAlZPeiJJmuU6uc9ykucDxwLvHDP810kWAQWs2WKdJGkSdFIKVfVvwF5bjL21iyySpN/q+uwjSdIUYilIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSp1cmd1wCSrAF+CWwCNlbVYJI9ga8CCxi9JeepVfVIVxklabbp+kjhT6pqUVUNNsvnAtdW1UHAtc2yJGmSdF0KWzoJuKR5fglwcndRJGn26bIUCvhOklVJljVj+1bVuub5z4F9t9wpybIkQ0mGRkZGJiurJM0Knb2nALy6qtYm2Qf4bpI7x66sqkpSW+5UVcuB5QCDg4PPWC9JXfnZJw/vOgIAL/rorTu8b2dHClW1tvn6EHAVcCSwPsl+AM3Xh7rKJ0mzUSelkOT5SXbf/Bz4M2A18A3gzGazM4Gru8gnSbNVV9NH+wJXJdmc4fKquibJT4CvJTkLeAA4taN8kjQrdVIKVXUf8PJxxjcAx0x+IkkSTL1TUiVJHbIUJEktS0GS1Orycwqzxkw4d1nS7OCRgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqWgiSpZSlIklqTXgpJ5ie5LsntSW5L8r5m/ONJ1ia5uXmcMNnZJGm26+J+ChuBD1TVTUl2B1Yl+W6z7nNV9ZkOMkmS6KAUqmodsK55/sskdwD7T3YOSdIzdfqeQpIFwCuAHzdD5yS5JclFSV64lX2WJRlKMjQyMjJZUSVpVujsdpxJdgOuBN5fVY8luRD4H0A1Xz8LLN1yv6paDiwHGBwcrMlLrKliKtze1Fubaqbq5EghyRxGC+Gyqvo6QFWtr6pNVfUb4EvAkV1kk6TZrIuzjwJ8Gbijqv5mzPh+YzZ7E7B6srNJ0mzXxfTRUcBbgVuT3NyMnQcsTrKI0emjNcA7O8gmSbNaF2cf/QDIOKu+NdlZJEm/y080S5JaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqWUpSJJaloIkqTXlSiHJ8UnuSnJPknO7ziNJs8mUKoUkOwMXAG8ADgEWJzmk21SSNHvs0nWALRwJ3FNV9wEkuQI4Cbi901RSHxzxwUu7jgDAqk+/resImkJSVV1naCU5BTi+qt7RLL8V+KOqOmfMNsuAZc3iQuCuSQ+6/fYGftF1iBnE3+fE8vc5cabL7/IPqmreeCum2pHCNlXVcmB51zm2R5KhqhrsOsdM4e9zYvn7nDgz4Xc5pd5TANYC88csDzRjkqRJMNVK4SfAQUkOSLIrcBrwjY4zSdKsMaWmj6pqY5JzgP8N7AxcVFW3dRxrIkyr6a5pwN/nxPL3OXGm/e9ySr3RLEnq1lSbPpIkdchSkCS1LIU+S7Jzkp8m+WbXWaa7JGuS3Jrk5iRDXeeZ7pK8IMnKJHcmuSPJf+w603SVZGHz73Lz47Ek7+86146YUm80z1DvA+4A9ug6yAzxJ1U1HT4cNB2cD1xTVac0Z/v9fteBpququgtYBO3letYCV3WZaUd5pNBHSQaA/wz8fddZpLGS/DvgaODLAFX166r6105DzRzHAPdW1QNdB9kRlkJ//U/gL4HfdJxjpijgO0lWNZc70Y47ABgBvtJMb/59kud3HWqGOA1Y0XWIHWUp9EmSNwIPVdWqrrPMIK+uqlcyehXds5Mc3XWgaWwX4JXAhVX1CuDfAC9V/xw103AnAv/YdZYdZSn0z1HAiUnWAFcAr0/yv7qNNL1V1drm60OMztce2W2iaW0YGK6qHzfLKxktCT03bwBuqqr1XQfZUZZCn1TVh6tqoKoWMHo4+X+q6oyOY01bSZ6fZPfNz4E/A1Z3m2r6qqqfAw8mWdgMHYOXqJ8Ii5nGU0fg2UeaPvYFrkoCo/9uL6+qa7qNNO29F7ismfK4D3h7x3mmteY/K8cC7+w6y3PhZS4kSS2njyRJLUtBktSyFCRJLUtBktSyFCRJLUtB6lGSk5NUkpc2ywuSPOtnJXrZRppKLAWpd4uBHzRfpRnJUpB6kGQ34NXAWYx+Qn3L9UuSXJ3ke0nuTvKxMat3TvKlJLcl+U6S5zX7/EWSnyT55yRXJvHS1eqcpSD15iRG7z3wL8CGJEeMs82RwJ8DLwPekmSwGT8IuKCqDgX+tdkG4OtV9aqqejmj99w4q58/gNQLS0HqzWJGL2xI83W8KaTvVtWGqvoV8HVGjywA7q+qm5vnq4AFzfPDkvzfJLcCpwOH9iO4tD289pG0DUn2BF4PHJ6kgJ0ZvbfDBVtsuuU1YzYvPzVmbBPwvOb5xcDJVfXPSZYAr5u41NKO8UhB2rZTgH+oqj+oqgVVNR+4H5i/xXbHJtmzec/gZOD/beN1dwfWJZnD6JGC1DlLQdq2xTzzfrtXAh/eYuzGZvwW4MqqGtrG6/534MeMlsedE5BTes68Sqo0AZrpn8GqOqfrLNJz4ZGCJKnlkYIkqeWRgiSpZSlIklqWgiSpZSlIklqWgiSp9f8BUXIC3xkpJwsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# distributoon represent label 6 is less prone to taking or frauding\n",
    "#credit card as they are less in number, may be they are of early education years\n",
    "sns.countplot(x=(train['Alpha']//523), data= train,hue='label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Kappa', ylabel='Density'>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAERCAYAAACU1LsdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2xklEQVR4nO3dd3gc5bX48e9Z9S6rWpZkyw25N2ywsQGHZmMcSAhJIJCEQCAQIKTc3PRCbnJvfumQQBISEgidGAOhmG4w1ca9F9mWLcmyeu/l/f0xu0LYkqyyoxntns/zzKPd2dmZs+v1nn27GGNQSimlADxOB6CUUso9NCkopZTqoklBKaVUF00KSimlumhSUEop1UWTglJKqS6uSwoi8g8RKRWRnX4631gReVlE9ojIbhHJ8cd5lVIqELkuKQD3A8v9eL5/Ab82xkwFzgBK/XhupZQKKK5LCsaYdUBl930iMlFEXhSRTSLylohM6c+5RGQaEGqMecV77npjTKP/o1ZKqcDguqTQi3uB24wxpwP/BdzTz+edBlSLyGoR2SIivxaRENuiVEqpES7U6QBORURigbOAf4uIb3eE97HLgZ/18LQiY8wyrNd3NjAXOAo8DlwL3Gdv1EopNTK5PilglWaqjTFzTnzAGLMaWN3HcwuBrcaYQwAi8jSwEE0KSinVI9dXHxljaoHDIvJpALHM7ufTPwASRSTVe/88YLcNYSqlVEBwXVIQkUeB94BcESkUkeuBq4HrRWQbsAu4rD/nMsZ0YLVBvCYiOwAB/mZP5EopNfKJTp2tlFLKx3UlBaWUUs5xVUNzSkqKycnJcToMpZQaMTZt2lRujEk99ZH946qkkJOTw8aNG50OQymlRgwROeLP82n1kVJKqS6aFJRSSnXRpKCUUqqLq9oUlFLKCW1tbRQWFtLc3Ox0KL2KjIwkKyuLsLAwW6+jSUEpFfQKCwuJi4sjJyeHbnOsuYYxhoqKCgoLCxk/fryt19LqI6VU0GtubiY5OdmVCQFAREhOTh6WkoytJQURyQfqgA6g3Rgz387rKaXUYLk1IfgMV3zDUX30MWNM+TBcJ6h0dho8Hnd/iJVSI49WH40wRysa+fpjW5j2kxe589UDdHbq3FVKDZfY2Ng+H8/Pz2fGjBkDOue1117LqlWrhhKWX9mdFAzwsncZzRt7OkBEbhSRjSKysayszOZwRra2jk6uf+ADXt5dwqzMRH7/6n5++MxOp8NSSgUQu5PCEmPMPOBi4BYROefEA4wx9xpj5htj5qem+m36joD00PtHOFBazx8+O4fHv7KQa8/K4dENR9lfUud0aEoFlfr6es4//3zmzZvHzJkzeeaZZ7oea29v5+qrr2bq1KlcccUVNDZay8Jv2rSJc889l9NPP51ly5ZRXFzsVPh9sjUpGGOKvH9LgaeAM+y8XiCra27j96/s5+zJKVw4LR0R4fbzJxMTHsofXt3vdHhKBZXIyEieeuopNm/ezNq1a/nWt76FbxmCffv28dWvfpU9e/YQHx/PPffcQ1tbG7fddhurVq1i06ZNXHfddfzgBz9w+FX0zLaGZhGJATzGmDrv7YvoeT1l1Q+v7imhtrmd28+f3NULYVRMONctzuGu1/PIK61nUlrf9Z1KKf8wxvD973+fdevW4fF4KCoqoqSkBIDs7GwWL14MwDXXXMNdd93F8uXL2blzJxdeeCEAHR0dZGRkOBZ/X+zsfZQOPOX9AgsFHjHGvGjj9QLa89uLGZMQybyxoz6y/5qF4/jj2jye317M7RdMdig6pYLLww8/TFlZGZs2bSIsLIycnJyuMQQndh0VEYwxTJ8+nffee8+JcAfEtuojY8whY8xs7zbdGPMLu64V6Gqb21i3v5yLZ2ac1A01LT6SBeOSWLPTnfWTSgWimpoa0tLSCAsLY+3atRw58uHs1UePHu368n/kkUdYsmQJubm5lJWVde1va2tj165djsR+KtoldQR4dXcJrR2dXDKr5+LmxTNHs/d4HQfL6oc5MqWC09VXX83GjRuZOXMm//rXv5gyZUrXY7m5udx9991MnTqVqqoqbr75ZsLDw1m1ahXf+c53mD17NnPmzOHdd9918BX0Tuc+GgHe2FdGWlwEc7MTe3x8+YzR3PHsbtbsKObW87QKSSm71NdbP7xSUlJ6rQrau3dvj/vnzJnDunXrTtp///33+y0+f9CSgssZY/ggv5Izxif1Osw9IyGK2VkJvLlfx3kopYZGk4LLFVY1UVzTzBnjk/o8buHEZLYWVNPU2jFMkSmlApEmBZfbcLgSgAU5fSeFRROSaeswbDpSNRxhKaUClCYFl/sgv5L4yFBy0+P6PG5+ThIhHuG9Qzr3oFJq8DQpuNyG/Erm5ySdckbU2IhQZmUl8P6hymGKTCkViDQpuFhNYxuHyho4fdyoUx8MLJyQzLaCahpb222OTCkVqDQpuNju4loAZmQm9Ov4BTmjaO807CissTMspZRNXnzxRXJzc5k0aRK//OUvHYlBk4KL+ZLCtIz4fh0/KysRgG2F1TZFpJSyS0dHB7fccgtr1qxh9+7dPProo+zevXvY49Ck4GK7jtWQFhdBalxEv45PiY0ga1QU27SkoNSIs2HDBiZNmsSECRMIDw/nyiuv/MiU3MNFRzS72O5jtUwb079Sgs/s7ES2FVTbE5BSQeCOZ3ex+1itX885bUw8P/n49D6PKSoqIjs7u+t+VlYW69ev92sc/aElBZdqae8gr7S+31VHPnOyEimsaqK8vsWmyJRSgUxLCi51oKSe9k4zqJICwPbCas6bkm5DZEoFtlP9ordLZmYmBQUFXfcLCwvJzMwc9ji0pOBSvkbm6WP61/PIZ0ZmPB6BrQXarqDUSLJgwQIOHDjA4cOHaW1t5bHHHuPSSy8d9ji0pOBS+4/XERHqYWxS9ICeFx0eysTUWL/XiSql7BUaGsqf/vQnli1bRkdHB9dddx3Tpw9/qUWTgkvlldUzITWWkFOMZO7JtDHxbMzXOZCUGmlWrFjBihUrHI1Bq49c6mDZ4NdcnpYRT1F1E9WNrX6OSikV6DQpuFBzWweFVU1MSh1kUvA2TvvaJZRSqr80KbjQwbJ6jIGJaTGDev5UbzdWbVdQSg2UJgUXOljWADDo6qOU2AjS4iK0pKCUGjBNCi6UV1qPRyAneXAlBbCqkLSkoJQaKE0KLnSwtJ7spGgiw0IGfY6pGfHkldbT2t7px8iUUoFOk4IL5ZXWD7qR2WfK6DjaOw2Hyxv8FJVSyk7XXXcdaWlpzJgxw9E4NCm4TGenIb+igfEpg686ApicZi3fub+kzh9hKaVsdu211/Liiy86HYYmBbcpqWumpb2TnCEmhQmpMYR4RJOCUiPEOeecQ1JSktNh6Ihmt8kvbwSG1sgMEBkWwrjkaE0KSg3Umu/C8R3+PefomXCxMyupDZSWFFwmv8JqAxiXPLA5j3qSmx7H/pL6IZ9HKRU8tKTgMvkVDYSFCGMSo4Z8rsnpcby06zjNbR1D6smkVFAZIb/o7aIlBZc5Ut5IdlL0oCbCO9Fp6bF0GmuEtFJK9YcmBZfJr2gYcnuCT2669kBSaqS46qqrWLRoEfv27SMrK4v77rvPkTi0+shFjDEcqWhk0cRkv5wvJyWGsBDRdgWlRoBHH33U6RCAYSgpiEiIiGwRkefsvtZIV1bXQlNbh99KCmEhHiakxLL/uJYUlFL9MxzVR7cDe4bhOiNefoXVHdUfPY98JqfHsr9Uk4JSqn9sTQoikgVcAvzdzusEiiNd3VH9U1IAOC09joLKJhpa2v12TqUCkTHG6RD6NFzx2V1S+APw30Cvs7KJyI0islFENpaVldkcjrsVVDUhApl+6I7qc5q3sTmvVNsVlOpNZGQkFRUVrk0MxhgqKiqIjIy0/Vq2NTSLyEqg1BizSUSW9nacMeZe4F6A+fPnu/NfZJgUVjaSER9JeKj/cvVp6dbEevtK6pidnei38yoVSLKysigsLMTNP0wjIyPJysqy/Tp29j5aDFwqIiuASCBeRB4yxlxj4zVHtIKqRrKS/NeeAFZVVHiohwPaLVWpXoWFhTF+/Hinw3AF26qPjDHfM8ZkGWNygCuB1zUh9K2gsonsUf5NCiEeYVJqLPu0W6pSqh908JpLtLR3UFLXTHaS/9oTfE5Lj9WSglKqX4YlKRhj3jDGrByOa41URVVNGIPfSwpgzYFUXNNMbXOb38+tlAosWlJwiYKqJgCy/dymANoDSSnVf5oUXKKg0hq4ljXKnuojQKuQlFKnpEnBJQqrmggLEdLj/d8POWtUNBGhHg5oY7NS6hQ0KbhEQVUjmYlRfpky+0QhHmFSWiz7tfpIKXUKmhRcorCy0Zb2BJ/JabHkafWRUuoUNCm4RFF1ky3tCT6T0+M4VtNMnfZAUkr1QZOCCzS3dVBe38qYBPuSgq8H0gGtQlJK9UGTggscq7a6o2baWVJIs3og5Wljs1KqD5oUXKDImxTG+HF21BNlJ1k9kHRpTqVUXzQpuEBXScHGpBDiESamxmr1kVKqT5oUXKCoqgmPwOgEe+dK1zmQlFKnoknBBYqqm0mPjyQsxN5/Du2BpJQ6FU0KLlBU3Whr1ZFPV2OzViEppXqhScEFjlU329rI7NPVLVV7ICmleqFJwWGdnYbimiZbu6P6+HogHSjVdgWlVM80KTisrL6Ftg4zLCUFXw+k/VpSUEr1QpOCwwq96yhkDUNSAKsHkrYpKKV6o0nBYUXDMJq5u8npcRRVN1Hf0j4s11NKjSyaFBx2bBhGM3fn64Gk4xWUUj3RpOCwoqomEqLCiI0IHZbrTdaJ8ZRSfdCk4LBj1U3DVkoAGOvrgaQlBaVUDzQpOKyoumlYBq756BxISqm+aFJwmJUU7J3z6EST02N1AJtSqkeaFBxU29xGXXP7sPU88jlNeyAppXqhScFBw93zyGeS9kBSSvVCk4KDiqrsX0ehJ1NGWz2Q9h7XpKCU+ihNCg4a7oFrPtmjookJD2FPce2wXlcp5X6aFBxUVN1EeIiHlJiIYb2uxyNMyYjXpKCUOokmBQcVVTUxJjESj0eG/dpTM+LYW1yHMWbYr62Uci9NCg4a7oFr3U3NiKeupb1rQj6llAJNCo46Vt087I3MPlMz4gHYrVVISqluNCk4pK2jk5K64VlxrSdTRschgrYrKKU+QpOCQ47XNGPM8HdH9YkODyUnOUaTglLqI2xLCiISKSIbRGSbiOwSkTvsutZI5NTAte6mZsSxp1jHKiilPmRnSaEFOM8YMxuYAywXkYU2Xm9EKepKCsM771F3U0fHc7SykbrmNsdiUEq5S7+SgoisFpFLRKTfScRYfLOuhXk37f/o5Y6SgtXYvE9HNiulvPr7JX8P8DnggIj8UkRy+/MkEQkRka1AKfCKMWZ9D8fcKCIbRWRjWVlZf+Me8Yqqm0mOCScyLMSxGKaOsZKCtisopXz6lRSMMa8aY64G5gH5wKsi8q6IfElEwvp4XocxZg6QBZwhIjN6OOZeY8x8Y8z81NTUQb2IkcjJMQo+YxIiiY8MZbe2KyilvPpdHSQiycC1wJeBLcCdWEnilVM91xhTDawFlg8myEB0bJgX1+mJiDBVp7tQSnXT3zaFp4C3gGjg48aYS40xjxtjbgNie3lOqogkem9HARcCe/0S9QhnjHFFSQGsdoV9x+vo6NTmHqUU9He1+L8ZY17ovkNEIowxLcaY+b08JwN4QERCsJLPE8aY54YQa8CobWqnobXD0Z5HPtPGxNPU1kF+RQMTU3vM70qpINLfpPBz4IUT9r2HVX3UI2PMdmDuIOMKaF1TZrugpDArKwGA7YXVmhSUUn0nBREZDWQCUSIyF/BN5xmPVZWkBqHIBd1RfSanxREdHsK2gho+OTfL6XCUUg47VUlhGVbjchbwu27764Dv2xRTwHPDGAWfEI8wIzOBrQXVToeilHKBPpOCMeYBrHaBTxljnhymmALeseomwkM9JMeEOx0KAHOyE7n/nXxa2zsJD9XpsJQKZqeqPrrGGPMQkCMi3zzxcWPM73p4mjqFouomxiQ4s7hOT2ZnJdLa0cne47XMykp0OhyllINO9bMwxvs3FojrYVODcKy6adjXZe7L7GyrsXmbViEpFfROVX30V+9fneHUj45VN3P25BSnw+iSmRhFSmw4Wwtq+Pwip6NRSjmpv4PXfiUi8SISJiKviUiZiFxjd3CByOnFdXoiIszOSmRbYbXToSilHNbfVsWLjDG1wEqsuY8mAd+2K6hA5vTiOr2ZnZ3IwbJ6anUabaWCWn+Tgq+a6RLg38aYGpviCXhuGqPQ3ezsRIyBnYX6T6tUMOtvUnhORPYCpwOviUgq0GxfWIHrmAsW1+nJbO/I5q1ahaRUUOvv1NnfBc4C5htj2oAG4DI7AwtUbhq41l1idDg5ydHaA0mpINffuY8ApmCNV+j+nH/5OZ6AV1TdTEqss4vr9GZ2diLrD1U6HYZSykH97X30IPAbYAmwwLv1Njuq6oNbpszuyZzsRI7XNneVZpRSwae/JYX5wDRjjE66P0THqptcOxvpGeOTAFh/uEInx1MqSPW3oXknMNrOQIKBMcaa4sKlJYWpo+NJiArj/YNahaRUsOpvSSEF2C0iG4AW305jzKW2RBWgapraaHTJ4jo98XiEBTlJrD9c4XQoSimH9Dcp/NTOIIKFmxbX6c3CCUm8uqeEktpm0uPdmbyUUvbpb5fUN7FGMod5b38AbLYxroB0rNoa2uHW6iOAM8cnA/D+IS0tKBWM+tv76AZgFfBX765M4GmbYgpYbh2j0N20MfHERYTyvnZNVSoo9beh+RZgMVALYIw5AKTZFVSg8i2ukxLrjsV1ehLiERaM13YFpYJVf5NCizGm1XfHO4BNu6cOUFF1E5mJUYi4Y3Gd3pw5PolDZQ2U1ulMJkoFm/4mhTdF5PtAlIhcCPwbeNa+sAKT1R3V/Y23Z06w2hU2HNYqJKWCTX+TwneBMmAH8BXgBeCHdgUVqI5VNzEmwb3tCT4zxsQTEx6iU14oFYT61SXVGNMpIk8DTxtjyuwNKTC1tndSWtfi6kZmn9AQD6fnJGkPJKWCUJ8lBbH8VETKgX3APu+qaz8envACR3FNk7W4jovWZu7LWROTOVBaz/EabVdQKpicqvroG1i9jhYYY5KMMUnAmcBiEfmG7dEFkKOVjQCMTYp2OJL+WZqbCsCb+0sdjkQpNZxOlRQ+D1xljDns22GMOQRcA3zBzsACTUGlNUYhe4Qkhdz0OEbHR/LGPq0tVCqYnCophBljyk/c6W1XCLMnpMBUUNVIWIgweoRMHSEiLM1N5e0D5bR1dDodjlJqmJwqKbQO8jF1goLKRsYkRhHicfcYhe6W5qZS19LO5iNVToeilBomp+p9NFtEanvYL8DI+MnrEgVVTWSPGhlVRz5nTUoh1CO8sb+sa+yCUiqw9VlSMMaEGGPie9jijDFafTQABZWNI6Y9wSc+Mox540Zpu4JSQaS/g9fUEDS0tFPZ0Ep20sjojtrd0txU9hTXUlKrXVOVCga2JQURyRaRtSKyW0R2icjtdl3L7QqqrO6oI636CGDpada8h29qaUGpoGBnSaEd+JYxZhqwELhFRKbZeD3XGmndUbubmhFHenwEb+h4BaWCgm1JwRhTbIzZ7L1dB+zBWoch6PgGrmWPkNHM3YkIS09L46395bS2a9dUpQLdsLQpiEgOMBdY38NjN4rIRhHZWFYWmFUUBZWNxISHkBTj3nUU+nLR9HTqWtp5T+dCUirg2Z4URCQWeBL4ujHmpO6txph7jTHzjTHzU1NT7Q7HEYVVVs8jt6+j0JvFk1KIDg/hpV3HnQ5FKWUzW5OCiIRhJYSHjTGr7byWmxVUNpE1AhuZfSLDQliam8oru0vo7NS1lZQKZHb2PhLgPmCPMeZ3dl3H7YwxFFQ1jsjuqN0tmz6asroWthRUOx2KUspGdpYUFmNNqHeeiGz1bitsvJ4rVTS00tjaMSK7o3a3NDeNUI/w8m6tQlIqkNnZ++htY4wYY2YZY+Z4txfsup5bFfh6Ho3A7qjdJUSFsWhiMi/vKsEYrUJSKlDpiGabFVRZYxRGyjoKfblo+mgOlzeQV1rvdChKKZtoUrCZr6SQNQLHKJzoomnpANoLSakApknBZoVVjSTHhBMT0a/lsF0tPT6SuWMTeVGTglIBS5OCzY5WNpIVAFVHPhfPGM3OolqOVjQ6HYpSygaaFGxWUNk0Iqe36M3FMzIAeGFnscORKKXsoEnBRu0dnRyrbgqIRmaf7KRoZmcl8MIOTQpKBSJNCjYqrGqivdOQkxLjdCh+tWJmBtsLa7oa0ZVSgUOTgo0OVzQAMD4AkwKgpQWlApAmBRvll1tJISc5sJJCdlI0s7QKSamApEnBRvnlDcRGhJISOzKnzO7LipkZbNMqJKUCjiYFG+VXNJKTMnKnzO7LCm8vpDXaC0mpgKJJwUb5FQ0BV3XkMzY5mhmZ8Ty/QweyKRVINCnYpK2jk8KqpoBrZO5uxcwMthVUU1ilVUhKBQpNCjYpqGyko9MEbEkB4BJvL6Q1WlpQKmBoUrBJvrc7aqCNUehuXHIM08fE87z2QlIqYGhSsMnhcqtKJZCrj8CqQtpaUE1RdZPToSil/ECTgk3ySusZFR1GUkzgdUft7sMqJC0tKBUINCnY5GBZPZPSYp0Ow3Y5KTFMy9AqJKUChSYFmxwsrWdiauAnBYBLZmWw5Wg1x7QKSakRT5OCDaoaWqloaA2apKBzISkVODQp2OBgmbWGcTBUH4HVmD41I16TglIBQJOCDXxJIVhKCgArZoxms1YhKTXiaVKwQV5pPeGhHjIDaMW1U1kxyzcXkg5kU2ok06Rgg4NlDUxIiSHEE3gT4fVmYmosU0bHaRWSUiOcJgUb5JXWMzFI2hO6WzEzg01Hqiiu0SokpUYqTQp+1tDSztHKRnLT45wOZdit9FYhPb9dSwtKjVSaFPxsf0kdALmjgy8pTEiNZVpGPM9pUlBqxNKk4Gf7jltJYUoQJgWAlbOtuZB0RTalRiZNCn6293gd0eEhZI+KdjoUR6ycOQbQgWxKjVSaFPxs3/E6JqfH4QminkfdjU2OZlZWglYhKTVCaVLwI2MM+0rqmBKEjczdrZyVwY6iGvLLG5wORSk1QJoU/KisvoXKhtagbGTu7pJZVhWSzpyq1MhjW1IQkX+ISKmI7LTrGm7ja2QO9qSQmRjFvLGJWoWk1AhkZ0nhfmC5jed3nV3HagGYlhHvcCTOWzlrDHuKa7vmgVJKjQy2JQVjzDqg0q7zu9GOohoyE6MYFeCrrfXHipkZiOhANqVGGsfbFETkRhHZKCIby8rKnA5nSHYV1TAzM8HpMFxhdEIkC8Yl8dz2Y06HopQaAMeTgjHmXmPMfGPM/NTUVKfDGbTa5jbyKxqZmaVJwWfl7Az2l9R3jfJWSrmf40khUOwqstoTpo/R9gSfi2dkEOIRntla5HQoftXa3snmo1Ws3lzIU1sKeTevnIaWdqfDUsovQp0OIFDsLKoBYIZWH3VJjYtgyaQUnt5yjG9dmDviB/RVN7Zy99o8Vm0qpKqx7SOPhXqE86em8aXF41k4IdmhCJUaOtuSgog8CiwFUkSkEPiJMeY+u67ntJ3HashIiCQlNsLpUFzl8nmZ3P7YVj7Ir+TMEfxl+eLOYr63egc1TW1cPCODlbMyOG10HB4RjlQ08PaBclZvKeKlXSWce1oqd1w6nZyUGKfDVmrAbEsKxpir7Dq3G20tqNZG5h5cOC2d6PAQnt5aNCKTgjGG3768nz+tzWN2diK/vHwmU0/ocjw+JYaluWn817JcHnzvCHe9foCL73yL762YwjVnjhvxJSQVXLRNwQ/K61s4UtHI6eNGOR2K60SHh7J8+mie215Mc1uH0+EMiDGGO57dzZ/W5vHZ+dk88ZWFJyWE7iLDQrjhnAm8/I1zWDA+iR8/s4vP/2O9LjqkRhRNCn6w+UgVgCaFXnxyXiZ1ze2s3VvqdCgD8tuX93P/u/lcv2Q8v/zUTCJCQ/r1vIyEKB740gL+7/KZbDlazfI/vMUanfJDjRCaFPxg09EqwkJEG5l7cdbEFNLiIli9ZeT0QnrigwL+tDaPKxdk88NLpiIysCogEeGqM8by/NfOZlxyNDc/vJnvPrmdxlbtpaTcTZOCH2w+UsWMzAQiw/r3SzLYhHiEy+aM4Y19pVQ1tDodzintOlbDD5/eyZJJKfzPJ2YMOCF0Nz4lhidvPouvLp3I4xsLWHnX22w+WuXHaJXyL+2SOkSt7Z1sL6zh8wvHDc8Fq49C0WYoPwANpdDZDqGREJsGKbmQNd+67TKXz8vib28d5qktRVy3ZLzT4fSqoaWd2x7ZwqiYMO68cg5hIUP/3RQW4uG/l0/h7MmpfOuJrVx+z7t8dn42t18wmTGJUQM+X1ldC3uKazlS0UBlQxttHZ1EhnkYkxjF1Ix4coN4PQ81dJoUhmjnsRpa2juZZ2d7QlU+bH0Edq6GigMf7o9IgNBwaGuC1m4Tz2XOh9zlMOMKSHLHF/DUjHhmZyfyyIajfGlxzpB+fdvFGMMPn95JfkUDj9ywkGQ/dy9eNDGZl795Lne9doB/vmMlyJWzMrji9Czm5yQRHnpyAmpsbWd/ST1bj1axpaCazUerKKj8aMO1R6DTfHg/KSacFTNH87kzxjFNB1OqAdKkMETvHawA4MzxSf4/ecVBeP3nsPtpMAYmnAsLroexCyF1CoR1+5XZXAule+DwOtj3gvW8138BUy6BRbdaz3H4i/jqM8by309uZ+ORKhbk2PB+DdGTm4t4aksRX79gsm0D0GIjQvn+iql88awc/vrmQZ7aXMTqLUVEhnmYnBZHenwEYSEe6lvaOVLRyNFua12nx0cwb+wovrAwhxmZCUxIjSEpJpywEA/NbR0UVjWxraCatftK+ffGQh56/yjnT0njB5dMZUJqrC2vRwUeMcac+qhhMn/+fLNx40anwxiQz/3tfaoa21hz+9n+O2lLHaz7Nbx3j1U1tOB6OOMGSMjq/zlqimDjfbDxH9BUBTlnw7JfQMZs/8U5QI2t7Zz5i9e4YFo6v//sHMfi6ElxTRMX/W4dU8fE8+gNCwkZpuqXxtZ23smr4N2D5eSV1lNR30prRycxEaFkJUaROzqO09LjmJWVQEZCZL9LWDWNbTz4fj5/ffMQLe2d3LR0Il9dOlHbvQKQiGwyxsz32/k0KQxec1sHs+54mc8vHMePVk7zz0kPvArP3AL1x2HONXD+jyEuffDna22Ezf+CN/+flRxmXwUX/HRo5xyCHz29k8c3FrD+e+e7ZopxYwxffmAj7xws56Wvn8O45MAZiVxa18wvnt/DM1uPMT4lhj9eNVd7yQUYfycF7X00BJuPVNHa3sniSX6oamhvgRe/Dw9/CqKT4MuvwyfuHvqXd3g0LLwJvrYFzroNdq6CuxfA5getKqlhdtUZY2lt73RV99Rnth7jtb2lfHvZlIBKCABpcZHceeVcHrr+TJpaO7j8nnf513v5uOnHoHIXTQpD8M7BckI9whnjh5gUao/BfRfC+3fDghvghtch63T/BOkTlQgX/Q/c/C6kTYf/3Ar/uszqzTSMpo2JZ052Io+sP+KKL6ayuhZ++uwu5o1N5NqzcpwOxzZLJqfwwu1nc9akZH78zC5ufWQL9Tqzq+qBJoUheGNfGXPHJhIbMYT2+pJd8PcLoOIQXPkoXPKbjzYg+1vKZLj2eVj5e6tr658Xw/Yn7LteD64+cywHyxp419tI76Sf/Gcnja0d/OqK2cPWjuCUpJhw/vHFBXz34im8uOs4n7j7HfJKdblU9VGaFAapqLqJXcdquWDqEKp3Dr0J/1gOphOuWwNTVvgvwL54PDD/Orj5bUibBqtvgFXXQ1P1sFz+47PHkBIbzt/fOjQs1+vNmh3FvLDjOLefP5lJacHRO8fjEW46dyIPXn8GVQ2tfOLud3hxp07BoT6kSWGQXttTAlizgA7K9ifgoU9BfCZ8+VUYPdOP0fXTqByr1HDeD61ur39eDPlv237ZyLAQPr8wh7X7ysgrdWZVtqqGVn70zC5mZMZz4zkTHInBSWdNTOHZ25YwMS2Wmx7azC/X7KW9o9PpsJQLaFIYpFd2lzAhNWbg/b+Ngbd+Z/06H7sQrntxYF1N/S0kFM75Nlz/MoRGwP0r4ZWfQLu901F8ftE4IsM83LvOmdLC/zy3m+rGVn71qdl+GbU8Eo1JjOKJryzkc2eO5S9vHuSL/9xA5QiYhkTZKzj/NwxRbXMb7x+q4MKBVh11tMPz34LX7rBGG1/zpNUA7AaZp8NX1sG8L8A7f4C/nwele227XFJMOFcuGMvqzUUUdBugNRxe31vC6i1FfPVjk4J+xG9EaAj/+8mZ/OqKWXyQX8XH//g22wurnQ5LOUiTwiCs2VFMW4fh4pkZ/X9SawM8fo01oGzx1+Hyv1m/zN0kIhYuvQuufMTqEXXvubD+Xtu6rt507kQ8ItzzxkFbzt+TyoZWvvPkDqaMjuOWj00ctuu63WfmZ/PkTWcBcMWf3+Mvbx6ko9P53mFq+GlSGITVm4uYkBLD7Kx+DgKqL4MHPg4HXoIVv4EL77Aae91qyiVw83vWKOg134aHr4C6436/zOiESD67IJtVmwo4UtHg9/OfyBjDD57aQXVjK7/7zJx+r48QLGZmJfDsbUs4b0oav1yzl0//5V0OlWnvpGDj4m8mdyqsamT94Uo+OTezf1MOVBy0xiCU7IbPPmRNVzESxKXD1f+2klj+O3DPItjzrN8vc+t5kwj1ePjVS/v8fu4TPbWliDU7j/PNC3ODvtqoN0kx4fz5mnnceeUcDpY1sPzOt/jty/t0HYggoklhgJ72jsT9xNzMUx9c8IGVEFpq4drnrF/gI4mIlcS+sg4Ss63qr2dugeYav10iPT6SG84ez/Pbi9li4zoDhVWN/OSZXSzIGRWUvY0GQkS4bE4mL3/jHC6eMZo/vp7Heb95k9WbC7VKKQhoUhiAto5OHl5/lMWTkslOiu774D3PwgMrISIern/FWudgpEo9Da5/Fc7+ljWF991nwt7n/Xb6G8+dSGpcBD96Zqct3SKb2zq4+aHNGOC3n54T8IPU/CU93poiY9VNi0iJC+ebT2zjwt+9yZObCrX7agDTpDAAL+48TnFNM9ct7mONAl+X08evgfQZ1hiE5ABo0AwNtybn+/KrEJ0Mj30OnviCX9oaYiNC+cnHp7GzqJb7380feqzdWO0IO9lRVMPvPzuHscmnSObqJPNzkvjPLUv489XziAgL4Vv/3sa5v36Du9fmUV7f4nR4ys80KfSTMYZ/vnOYnORoPpbby8pm7S3w9M0fdjm99nmISRneQO2WeTrc+IaVIPa9CH86Azb8zepuOwSXzMzgY7mp/Pbl/X6deuHB94/w5OZCbj9/8uAHGio8HuHimRm88LUl/O0L8xmXHM2vX9rHov97jdsf28LG/EpXzGWlhk6nzu6ntw6U8fn7NvCzy6bzhUU5Jx/QUA6PXQ0F78PHfmANCHPh6mJ+VZ4Hz30d8t+yFv1Z9r8w6fxBn+54TTMX37mO0QlRPPXVs4Y89/9bB8r40j8/4NzTUvnbF+brEpV+lldax0PvH+XJTYXUtbQzITWGK07P4vK5WYxOiHQ6vKCh6yk4oLPTcNnd71DZ0Mrr/3XuyV0ZCzfCv79krZn8iT/DjMudCdQJxljtCy//EKoOw+SL4KKfQ2ruoE732p4Srn9gI5fPzeS3n5k96GU7Nx2p5Jq/b2BccjRP3LSI+MiwQZ1HnVpjazvPbStm1aZCNuRX4hFYMjmVK07P4qJp6bqwj800KTjgP9uO8bVHt/DbT8/mU6d3m5Kis9Oa7vrVn0L8GPj0A5A5z7E4HdXeAhvuhTd/Za0cN+NTcM5/QdrUAZ/qj68d4Lev7OerSyfy7WW5A04Mbx8o58YHN5IWF8G/bzqL1DiXDRIMYPnlDazeXMiTm4soqm4iLjKUj88ewxWnZzE3O9GVa3OPdJoUhllVQysX/v5NMhKiePqWxR/2XKkphGdvh7xXYerH4dI/uWfKCic1VMB7f7TaGVrrYeqlVlVaxqx+n8IYw/ef2sGjGwq4bvF4fnjJ1H5V/RhjeGTDUX76n11MSInlwevPIC1eqzGc0NlpeP9QBas2FfLCzmKa2zq1eskmmhSGkTGGbzy+lee2F/OfW5dYA546O2Hz/fDyj8F0wIU/gwVfDvz2g4FqrIT374H1f7XGaeScbY15yL3EmoTvFDo7DT97bjf3v5vPkkkp/PrTs8hI6H2dieKaJu74z25e3HWcc09L5a4r55IQ7ecqo7Zma0lT39bZBp0d1tTnnR3W6wqPhbBoCI+xuiNHJ4EnuKtP6prbWLPjuFYv2USTwjB68L18fvTMLr5+wWS+fsFp1mC0l38ABeth/LnWPEGjcpwO092aqmHjP6ytpgDiMmDmp2H2lZA+vc+nGmN47IMCfvbsbgyGq84Yy2VzMpmZmUCIR2ht72RrQTVPby1i9eZCjIHbL5jMTedMHFyjckc7VB6Csj3W3+qjUHXE+ltbBG2DmLhPPBCdArFp1haTZo0Wj8uwtvgxEDcaYkdb3X4DXH55A09uLuTJTYUcq2nW6iU/0KQwTN7YV8qXH9ho9Vy5JBHPG7+AXU9BbDqc9yOYe42WDgaiswP2vwSbH7Cq3DrbrQV+cldY25i5vc4HdbSikTtfO8DTW4vo6DR4xBrbUNfSjjEQHurhk3MyufW8SaceVAhWaa/6CJTu9m57oWwvlO+Hjm5TR0clQeJYa0vItn71R42y/kYmQki4VQqQECv2jnaryqytEVobrZHfDaVQXwoNZVBfYs2DVX/8o9fxiUm1EkTcGIjPsP7GjfYmDm8SiU4KiM9dZ6fhPW/10ppu1UufmJPJipkZQbPokT9oUhgGr+wu4ZaHNzNplIfHxjxG/P6nIDQSFn8NzvqaNZuoGryGctj5JOx+Bo6+Z1W/RCXBuLNg7CJr9HfaNIj86PxElQ2trNtfxsGyemqa2kiMDmf6mHgWTkgmIaqHqqLODqgrhvIDHyaAkt1WAuj+qz9hLKRNsbrVpk21/qZMhog4e16/MVb1Wt0xa/Bf7TErzlrv/bpjUFsMjeUnPzckwkoMEfEQmWC9Rx+5HQfhcdZnNNy7RZzwNzzWmqHXJcmlrrmNF3ZYvZc+yLemOjktPZYVMzNYPmM0uelxWoLogyYFG7V3dPLHl7Zz17pCZoUX8y/5CQlRobDgBjjzK1bxX/lXYyUceAUOr4Mj71jdWn0Sx0LSBIjPgoRM61d6RJz1peb7kuhst36RN1V/WNdfU2iVBKoLrHp/n5hU60s/bVq3bYp9X/5D1d5qlSpqi62k4UscTZXQXGu11TTXWLeba6z7PZVAeuLxtn9ExFntHx9JHifui7NGscemWtVfMam2tZUcr2lmzc5i1uw4zgdHKjEG0uIiWDI5hbMnp7B4UgppcdpI3Z0mBX/r7MSU7eWtd9/mf7dEsrc1mU953uTn2ZuImvdZmHOVe780AlFtMRRvg9JdULLLqtOvLfJOp3GKz2pIhNUDLD4TRo2DxHHW36QJkDbd+lILdO0t0FIPrXXWGh6+2y311v3WeqvLcGt9H493e35vSUY8VqKISfMmi9QPb8eme2+nWbejk/vVueBEJbXNvLGvlLcOlPNOXjlVjVaCzxoVxeysRGZmJTArK4Epo+MZFR0WtKWJEZUURGQ5cCcQAvzdGPPLvo63PSl0tFkNiKV7rK1oEweOFHBb/RfZa8aRGVLNj6aXs+y8C5DRfTeCqmHW0W79EvZ9oRljlRbEY1WdRI2CsN57J6lBam+13vPGCqt9pKHMahf5yG3v/foyaOtpXQyxpnvpShTdEkb3fTFpVlIPObkqsLPTsLu4lncPlrOtsIbthdUUVDZ1PR4fGcr41FjGJ0czNjmGtLgIa4uPJC0ugtS4iIBddnXEJAURCQH2AxcChcAHwFXGmN29PWfQSaF428nFaF+VQt0xqCmyfm3Wl1j111aEkDqF+jEL+WLeUj67cAKXLZquC68oNRStDVbDen2p9f+todvt+rKP7mtv7vkcYTHeNhLvFpX44e2IOCv5h0ZR1RnN9rpY8hoiyK8P43Cdh8M1hmP1nT2WKeMjQ4iPDCUuIpS4yFDio8KIjwwjPiqM2IhQIsNCiAwLISI8lMhQT9f9yDDv7dAQIsI8hHiEUI94/1r3Q7ruf/TvcJRe/J0UBl6m678zgDxjzCEAEXkMuAzoNSkM2n3LoL3p5P3hsVaPjYRMmHi+1YsjeaK3IfE0CI8mFnjS7wEpFaTCYyBpvLX1xRjrx9uJiaK5xrtVf3i7rtjqHOD74Wc6ABgFnOvdumuP8FBOAqUmkVIzyvpLItVtsdS2xlBLFLUmhiKi2GNiqCWaeqIwNs0PKgKCtU6FdN23dvrup8RG8PZ3zrPl+gNlZ1LIBAq63S8EzjzxIBG5EbjRe7deRPy4BFctcAzY5L9TDkwK0EMXkqCm78nJ9D052RDfk2rgiJ9Csd8+QL57ysN6e0/G+TMWO5NCvxhj7gXudToOO4jIRn8W6wKBvicn0/fkZPqenGy43hM7W16KgOxu97O8+5RSSrmUnUnhA2CyiIwXkXDgSuA/Nl5PKaXUENlWfWSMaReRW4GXsLqk/sMYs8uu67lUQFaLDZG+JyfT9+Rk+p6cbFjeE1cNXlNKKeWswBzNoZRSalA0KSillOqiScEmIrJcRPaJSJ5IP3ogu5yIZIvIWhHZLSK7ROR27/4kEXlFRA54/47y7hcRucv7+reLyLxu5/qi9/gDIvLFbvtPF5Ed3ufcJd7hoL1dwy1EJEREtojIc97740Vkvfd1PO7taIGIRHjv53kfz+l2ju959+8TkWXd9vf4OertGm4gIokiskpE9orIHhFZFOyfExH5hvf/zU4ReVREIl37OTHG6ObnDath/SAwAQgHtgHTnI5riK8pA5jnvR2HNYXJNOBXwHe9+78L/D/v7RXAGqxBmwuB9d79ScAh799R3tujvI9t8B4r3ude7N3f4zXcsgHfBB4BnvPefwK40nv7L8DN3ttfBf7ivX0l8Lj39jTvZyQCGO/97IT09Tnq7Rpu2IAHgC97b4cDicH8OcEayHsYiOr2b3etWz8njr9hgbgBi4CXut3/HvA9p+Py82t8Bmteq31AhndfBrDPe/uvWHNd+Y7f5338KuCv3fb/1bsvA9jbbX/Xcb1dww0b1vib14DzgOe8X1TlQOiJnwWsnniLvLdDvcfJiZ8P33G9fY76uobTG5Dg/QKUE/YH7eeED2d3SPL+uz8HLHPr50Srj+zR0xQfmQ7F4nfe4uxcYD2Qbowp9j50HEj33u7tPehrf2EP++njGm7wB+C/Ad9Mi8lAtTGm3Xu/++voeu3ex2u8xw/0verrGk4bD5QB//RWqf1dRGII4s+JMaYI+A1wFCjG+nffhEs/J5oU1ICIiG8Owa8bY2q7P2asnyO29nEejmv0l4isBEqNMY5NruVCocA84M/GmLlAA1ZVTpcg/JyMwpoMdDwwBogBljsaVB80KdgjIKf4EJEwrITwsDFmtXd3iYhkeB/PAEq9+3t7D/ran9XD/r6u4bTFwKUikg88hlWFdCeQKCK+gaHdX0fXa/c+ngBUMPD3qqKPazitECg0xqz33l+FlSSC+XNyAXDYGFNmjGkDVmN9dlz5OdGkYI+Am+LD28PjPmCPMeZ33R76D+DrGfJFrLYG3/4veHuXLARqvEX7l4CLRGSU9xfURVj1nMVArYgs9F7rCyecq6drOMoY8z1jTJYxJgfr3/h1Y8zVwFrgCu9hJ74nvtdxhfd4491/pbfXyXhgMlZjao+fI+9zeruGo4wxx4ECEcn17jofa7r8oP2cYFUbLRSRaG/MvvfEnZ8TpxthAnXD6lWxH6tXwA+cjscPr2cJVnF8O7DVu63Aqrd8DTgAvAokeY8X4G7v698BzO92ruuAPO/2pW775wM7vc/5Ex+OuO/xGm7agKV82Ptogvc/ax7wbyDCuz/Sez/P+/iEbs//gfd178Pbm6avz1Fv13DDBswBNno/K09j9R4K6s8JcAew1xv3g1g9iFz5OdFpLpRSSnXR6iOllFJdNCkopZTqoklBKaVUF00KSimlumhSUEop1UWTggoaIlLf7fYKEdkvIuOcjEkpt7FtOU6l3EpEzgfuApYZY444HY9SbqIlBRVUROQc4G/ASmPMQe++G0TkAxHZJiJPiki0d//9IvIXEdnoLVWs9O6/VkSeEZE3vHP3/6Tb+Z8WkU3eufNvdOI1KjUUOnhNBQ0RaQPqgKXGmO3d9icbYyq8t38OlBhj/igi9wOjsUaLTsSaMmAS1jQC/wfMABqxphm41hizUUSSjDGVIhLl3X+u79xKjQRaUlDBpA14F7j+hP0zROQtEdkBXA1M7/bYE8aYTmPMAayFXqZ4979ijKkwxjRhTXC2xLv/ayKyDXgfa5KyyTa9FqVsoUlBBZNO4DPAGSLy/W777wduNcbMxJqjJrLbYycWpU1v+0VkKdaMmIuMMbOBLSecSynX06SggooxphG4BLhaRHwlhjig2Ds1+NUnPOXTIuIRkYlYk4vt8+6/UKw1gaOATwDvYE1xXGWMaRSRKVhLRio1omjvIxV0vHX+y4F1IlIG/AhrFbky79+4bocfxZplMh64yRjTbM1+zAastSWygIe87Qk7gJtEZA9W8nh/uF6TUv6iDc1K9cLb0PycMWbVCfuvxZri+VYn4lLKTlp9pJRSqouWFJRSSnXRkoJSSqkumhSUUkp10aSglFKqiyYFpZRSXTQpKKWU6vL/Ad1QQitsgOMKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Distribution of kappa showing two peaks at different region which seems that there is something in it.\n",
    "sns.kdeplot(train[\"Kappa\"], data = train, hue = \"label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Delta', ylabel='Density'>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAERCAYAAACZystaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0F0lEQVR4nO3deXxcdbn48c8zk8m+NkuTNqVJ6Q7dy1KWIvsOLoggqAjKVZCf3OsG6PVevd6rXu/Pn3jdQFBQ2bQCKkoBEYWylLZ03+iSNEm37Ps6M9/fH+dMm5Y0SdM5c+bMPO/Xa14zc+ac830C0zz57mKMQSmlVHLzuR2AUkop92kyUEoppclAKaWUJgOllFJoMlBKKYUmA6WUUsRhMhCRX4hIvYhsitL9ThKRF0Vkq4hsEZGKaNxXKaUSSdwlA+AR4LIo3u9XwPeMMbOA04H6KN5bKaUSQtwlA2PMq0Dz4GMicrKILBeRNSLymojMHM29RGQ2kGKMecm+d6cxpjv6USullLfFXTI4hgeBu4wxi4AvAj8Z5XXTgVYReVpE1orI90TE71iUSinlUSluBzASEckGzgJ+JyKRw2n2Zx8EvjnEZXuNMZdi/XznAguAGuAp4BbgYWejVkopb4n7ZIBVe2k1xsw/+gNjzNPA08NcWwesM8bsBhCRZ4Ez0WSglFJHiPtmImNMO1AlIh8GEMu8UV6+CsgXkWL7/QXAFgfCVEopT4u7ZCAiTwBvAjNEpE5EbgNuAm4TkfXAZuDa0dzLGBPC6mN4WUQ2AgL83JnIlVLKu0SXsFZKKRV3NQOllFKxF1cdyEVFRaaiosLtMJRSyjPWrFnTaIwpHvnM4cVVMqioqGD16tVuh6GUUp4hInuicR9tJlJKKeVcMhCRGSKybtCjXUTudqo8pZRSY+dYM5ExZjswH8BeAmIv8IxT5SmllBq7WPUZXAjsMsZEpW1LKaWiaWBggLq6Onp7e90O5ZjS09MpLy8nEAg4cv9YJYMbgCdiVJZSSh2Xuro6cnJyqKioYNAaaHHDGENTUxN1dXVUVlY6UobjHcgikgpcA/zuGJ/fLiKrRWR1Q0OD0+EopdR79Pb2UlhYGJeJAEBEKCwsdLTmEovRRJcD7xhjDg71oTHmQWPMYmPM4uLiEx4qq5RSYxKviSDC6fhikQxuRJuIXBMOG3TJEaXUSBxNBiKSBVzM8MtMKwes2dPCB37yOrP/bTln/NfLfPv5rXT3B90OSylPy87OHvbz6upqTj311OO65y233MKyZctOJKyocDQZGGO6jDGFxpg2J8tRR3pqVQ3XP/Am9e193HTGZOaW5/PzV3fzkQfeor49fkdLKKXcozOQE8zamha+9uwmzjq5kOfvPpd/vWo2D31iMT//+GJ2NXRy66Or6AuG3A5TKU/r7OzkwgsvZOHChcyZM4c//OEPhz4LBoPcdNNNzJo1i+uuu47ubmvb9TVr1nDeeeexaNEiLr30Uvbv3+9W+EPSZJBAegdC3PXEWkpy0vnfGxeQm354PPKFs8Zz/w0L2LS3nf/881YXo1TK+9LT03nmmWd45513eOWVV/jCF75wqG9u+/bt3HHHHWzdupXc3Fx+8pOfMDAwwF133cWyZctYs2YNt956K1/96ldd/imOFFcL1akT89jKGupaenj8U2eQn5n6ns8vnj2e286p5OEVVVw7fwKLJo9zIUqlvM8Yw3333cerr76Kz+dj7969HDxoDZicNGkSZ599NgA333wzP/zhD7nsssvYtGkTF198MQChUIiysjLX4h+KJoME0d0f5Kd/38lZJxdy1tSiY573hUum89yGfXzzua0889mz8PniezidUvHoscceo6GhgTVr1hAIBKioqDg0B+DoIaAigjGGU045hTfffNONcEdFm4kSxFOramns7OcLl0wf9rzM1BS+dOlM1te28tzG+GqzVMor2traKCkpIRAI8Morr7Bnz+GVdmpqag790n/88cc555xzmDFjBg0NDYeODwwMsHnzZldiPxZNBgnAGMNTq2qZW543qqafDy6YyLSSbH7yyk6dg6DUGNx0002sXr2aOXPm8Ktf/YqZM2ce+mzGjBn8+Mc/ZtasWbS0tPDZz36W1NRUli1bxle+8hXmzZvH/PnzeeONN1z8Cd5Lm4kSwOZ97Ww70MF/vH9045t9PuHTS6fw5WUbeG1HI0un68xvpUajs7MTgKKiomM2+Wzbtm3I4/Pnz+fVV199z/FHHnkkavGdCK0ZJIDfrq4lNcXHNXMnjPqaa+dPoCQnjQdf3e1gZEopr9Bk4HGhsOHPG/Zz8ezx5GWOfmnbtBQ/H18ymRU7G6lu7HIwQqWUF2gy8Lj1da00dfVzyezxx33tdYsm4ROrZqGUSm6aDDzu5a0H8fuE900vOe5rS/PSOX9GCcvW1BEMhR2ITinlFZoMPO7lrfUsnlxwXE1Eg11/2iTqO/p4dYfuJaFUMtNk4GF7W3vYdqCDi2YdfxNRxPkzSsjLCPCn9TrnQKlkpsnAw17f2QjAeTPGPjQ0NcXH5aeW8uLmA/QO6AJ2Srlp+fLlzJgxg6lTp/Kd73wnpmVrMvCwlbubGZeVyrSS4ddYH8nV8ybQ1R/ilW31UYpMKXW8QqEQd955J88//zxbtmzhiSeeYMuWLTErX5OBh62sauL0inEnvB3emVMKKcpO47kN2lSklFvefvttpk6dypQpU0hNTeWGG244Ymlsp+kMZI/a29pDXUsPt51TecL38vuEi2eP54/r9tIXDJGW4o9ChEp50zf+tJkt+9qjes/ZE3L5t6tPGfacvXv3MmnSpEPvy8vLWblyZVTjGI7WDDxq5e4mAE6vjM4y1JfMHk9Xf4g3djVF5X5KKW/RmoFHvV3VTG56CjNLc6NyvyUnF5KZ6uelLQc5f8bxz1lQKlGM9Be8UyZOnEht7eEJoHV1dUycODFm5WvNwKPW1bay4KQC/FHajyA94Oe86cX8dctBwmFdyVSpWDvttNPYsWMHVVVV9Pf38+STT3LNNdfErHxNBh7U3R/k3YMdzCvPi+p9L5o1nvqOPjZHub1UKTWylJQUfvSjH3HppZcya9Ysrr/+ek45JXa1FG0m8qBNe9sJG5g3KT+q9z13urVD2ms7G5gT5USjlBrZFVdcwRVXXOFK2Y7WDEQkX0SWicg2EdkqIkucLC9ZbKhrBWBueX5U71uSk87M0hxW7GiM6n2VUvHP6Wai+4HlxpiZwDxgq8PlJYX1dW1MyEunOCct6vdeOr2Y1dUtdPcHo35vpVT8ciwZiEgesBR4GMAY02+MaXWqvGSyoa416rWCiHOnFdEfCrOyqtmR+yul4pOTNYNKoAH4pYisFZGHRCTr6JNE5HYRWS0iqxsadOXMkbR297OnqZu5k5xp0z+tYhypKT5ee1ebipRKJk4mgxRgIfBTY8wCoAu45+iTjDEPGmMWG2MWFxfrXrwj2bq/A4BTJziTDNIDfs6oHMeKnZqYlUomTiaDOqDOGBOZT70MKzmoE7DtgDXsc2ZZjmNlnDutiHcPdnKgrdexMpRS8cWxZGCMOQDUisgM+9CFQOyW4EtQW/e3U5iVSnF29DuPI86ZatXQXtMNb5SKmVtvvZWSkhJOPfVUV8p3ejTRXcBjIrIBmA/8l8PlJbxtBzqYWZZzwiuVDmdmaQ5F2Wms2Kn9BkrFyi233MLy5ctdK9/RSWfGmHXAYifLSCahsGH7gQ5uPnOyo+X4fMK504p49d0GjDGOJh6llGXp0qVUV1e7Vr7OQPaQ6qYu+oJhZpVFZ3G64SyZUsgza/eyq6GTqSXO9U8oFXeevwcObIzuPUvnwOWx3bnseOnaRB6ydb/deVzq/C/n0+ylsd+uanG8LKWU+7Rm4CHbD3Tg9wlTT3Cby9GoKMykKDuNVdXNfPSMkxwvT6m4Eed/wTtFawYesuNgJ5PHZZIecH4nMhHh9MoC3taZyEolBU0GHrKjviMmtYKI0yrGsbe1h32tPTErU6lkdeONN7JkyRK2b99OeXk5Dz/8cEzL12Yij+gPhtnT1M2lp5TGrMzTKqx+g1XVzVw7P3Y7LimVjJ544glXy9eagUfsaeoiGDZMGx+7msGsslyy01K0qUipJKDJwCN21ncCMLU4dsM8/T5h4eQCVlVrMlAq0Wky8IhIMji55D0Lvzrq9IoC3j3YSUtXf0zLVSrWjInvvb+djk+TgUfsqO9kYn4Gmamx7eYZ3G+gVKJKT0+nqakpbhOCMYampibS09MdK0M7kD1iZ31nTEcSRcyblE+q38eq6mYuiWHntVKxVF5eTl1dHfG8p0p6ejrl5eWO3V+TgQeEw4bdjZ2cOaUw5mWnB/zMLc9jVbXORFaJKxAIUFlZ6XYYrtJmIg842NFL70CYyuLY9hdELJxcwJZ97fQFQ66Ur5RyniYDD6hq7AKgstCdZDB/Uj79ofChXdaUUolHk4EHVDd2A1BRlOlK+fMn5QOwrkabipRKVJoMPGBPUxepfh9leRmulF+Wl05xThrr69pcKV8p5TxNBh5Q1djFSYWZ+H3ubDIjIsyflM+62lZXyldKOU+TgQdUN3VR4VJ/QcT8SflUNXbR2q2Tz5RKRJoM4lw4bNjT1E2lS/0FEZF+A20qUioxaTKIcwfae+kLhpnscs1gbnkeIrCuptXVOJRSztBkEOeqI8NKi9xNBjnpAaYWZ7OuVkcUKZWIHJ2BLCLVQAcQAoLGmMVOlpeIqpqsZFDhcjIAa2mKv22rxxiDiDud2UopZ8SiZnC+MWa+JoKx2dPUTWqKj7Jc5xaoGq35k/Jp7uqntll3PlMq0WgzUZyrauxi8rhMfC4NKx0s0om8VpuKlEo4TicDA7woImtE5PahThCR20VktYisjucVA91S3dgVF01EADNKc0hL8el8A6USkNPJ4BxjzELgcuBOEVl69AnGmAeNMYuNMYuLi4sdDsdbwmHDnuZu1zuPIwJ+H6dMyGXz3na3Q1FKRZmjycAYs9d+rgeeAU53srxEs7+9l/5g2PUJZ4OdOjGPzfvaCIfjcxMQpdTYOJYMRCRLRHIir4FLgE1OlZeIIsNKKwrdnXA22KkT8ujqD1Ftj3JSSiUGJ2sG44EVIrIeeBv4szFmuYPlJZzI0tXx0mcAcMrEXAA27dOmIqUSiWPzDIwxu4F5Tt0/Gexp6iItxUdpHAwrjZhWkkOq38fmfW1cM2+C2+EopaJEh5bGsarGbioKs+JiWGlEaoqP6aXZ2omsVILRZBDHapqtpavjzakT8ti0rw1jtBNZqUShySBOGWOobe7hpHHxlwxOmZhHa/cAe1t1JrJSiUKTQZxq7OynZyAUl8ng1AlWJ/Jm7URWKmFoMohTNc3WvsfxmAxmleXi9wmb9+reBkolCk0GcarWTgaTxrmz7/Fw0gN+Ti7O0uGlSiUQTQZxKpIMygvir2YAdiey1gyUShiaDOJUTXM343PTSA/43Q5lSKdMzKO+o4/6jl63Q1FKRYEmgzhV29LNpDitFYB2IiuVaDQZxKl4HVYaMTuSDLSpSKmEoMkgDvUHw+xr62FSHCeDnPQAJ43LZOuBDrdDUUpFgSaDOLSvtQdjiOtkADCzNIdt+7WZSKlEoMkgDsXzHIPBZpblUtXYRe9AyO1QlFInSJNBHPJKMphVmkPYwI6DnW6HopQ6QZoM4lBtSzepKT5KctLcDmVYM8usTuStB7SpSCmv02QQh2qbuykvyIirpauHctK4TDICfrbt105kpbxOk0EcqmmO7zkGEX6fML00h21aM1DK8zQZxKF4n2Mw2KzSHLbub9e9DZTyOE0GcaatZ4C2ngHPJIOZpTm0dA/Q0NHndihKqROgySDOxPNqpUOJdCJv0fkGSnnaqJKBiDwtIleKiCYPhx1OBt6oGcwqtZLBNp2JrJSnjfaX+0+AjwI7ROQ7IjJjtAWIiF9E1orIc2OKMMnUeCwZ5GUGmJCXrjORlfK4USUDY8xfjTE3AQuBauCvIvKGiHxSRAIjXP55YOuJhZk8alu6yc8MkJs+0n/W+DGzLFdrBkp53KibfUSkELgF+BSwFrgfKzm8NMw15cCVwEMnFGUSqfHQSKKImaU57KzvpD8YdjsUpdQYjbbP4BngNSATuNoYc40x5iljzF1A9jCX/gD4MnDM3xIicruIrBaR1Q0NDaOPPEHVNnd7pokoYmZZLsGwYVeDLkuhlFeNtmbwc2PMbGPMt40x+wFEJA3AGLN4qAtE5Cqg3hizZrgbG2MeNMYsNsYsLi4uPp7YE04obNjb0uOJCWeDzSrNAdDJZ0p52GiTwbeGOPbmCNecDVwjItXAk8AFIvKb44gt6Rxs76U/FPZcM1FlURapfp8uS6GUh6UM96GIlAITgQwRWQBEFsvJxWoyOiZjzL3AvfZ93gd80Rhz8wnGm9BqPDbHICLF72Pa+Gzd6EYpDxs2GQCXYnUalwPfH3S8A7jPoZiSVq1Hlq4eyszSXF7doX0+SnnVsMnAGPMo8KiIfMgY8/uxFmKM+Tvw97Fenyxqm7vxCUzI91bNAGBWWQ6/f6eOxs4+irLje+ltpdR7jdRMdLMx5jdAhYj8y9GfG2O+P8RlaoxqW3ooy8sg4PfeRO+Z9kzk7Qc6KJqqyUAprxnpt06W/ZwN5AzxUFFU09ztySYisGoGAFt1JrJSnjRSM9ED9vM3YhNOcqtp7ub8Gd4cXluYnUZxTprORFbKo0Y76ey/RSRXRAIi8rKINIiIjgyKop7+EA0dfZ6tGYA1E1lrBkp502gbpy8xxrQDV2GtTTQV+JJTQSWjuhZvLVA3lFllueyo7yQY0mUplPKa0SaDSHPSlcDvjDFtDsWTtLy2WulQZpbm0B8MU9XY5XYoSqnjNNpk8JyIbAMWAS+LSDHQ61xYycfLcwwiIiOKdPKZUt4z2iWs7wHOAhYbYwaALuBaJwNLNjXNPWQE/BRmpbodyphNLckmxSe6t4FSHjTSDOTBZmLNNxh8za+iHE/SigwrFZGRT45TqSk+ppZk64gipTxoVMlARH4NnAysA0L2YYMmg6ipa/He0tVDmVmaw9tVzW6HoZQ6TqOtGSwGZhtjjJPBJCtjDDXN3Sw5udDtUE7YzLJcnl23j9bufvIzvdvkpVSyGW0H8iag1MlAkllTVz/d/SFPdx5HzDy0t4E2FSnlJaOtGRQBW0TkbaAvctAYc40jUSWZyEgir21qM5TZZdaIom372zlzivdrOkoli9Emg393MohkF5ljcFKh95NBcU4a47JStWaglMeMKhkYY/4hIpOBacaYv4pIJuB3NrTkUdfSAyRGzUBEdFkKpTxotGsTfRpYBjxgH5oIPOtQTEmnpqmbouw0MlITI7/OLM1l+8EOQmEdb6CUV4y2A/lOrD2N2wGMMTuAEqeCSjbWHAPvbWhzLLPKcugdCLOnSZelUMorRpsM+owx/ZE39sQz/bMvSmoTZI5BxKxIJ7L2GyjlGaNNBv8QkfuADBG5GPgd8CfnwkoeA6Ew+1p7EmJYacTUkmx8gi5LoZSHjDYZ3AM0ABuBfwL+AnzNqaCSyb7WHsLG26uVHi094GdKcTZb9mvNQCmvGO1oorCIPAs8a4xpcDak5FLbnDgjiQabVZbL2poWt8NQSo3SsDUDsfy7iDQC24Ht9i5nXx/pxiKSLiJvi8h6EdksIrp15hASaY7BYDNLc6hr6aG9d8DtUJRSozBSM9E/Y40iOs0YM84YMw44AzhbRP55hGv7gAuMMfOA+cBlInLmiQacaGqauwn4hdLcdLdDiapZZdayFO9qJ7JSnjBSMvgYcKMxpipywBizG7gZ+PhwFxpLp/02YD90BNJRalu6mZifgd/n3aWrh3JooxvtRFbKE0ZKBgFjTOPRB+1+g8BINxcRv4isA+qBl4wxK4c453YRWS0iqxsakq87orY5sYaVRpTlpZOXEdBdz5TyiJGSQf8YPwPAGBMyxswHyoHTReTUIc550Biz2BizuLi4eKRbJpw9Td0JNaw0IrIshQ4vVcobRkoG80SkfYhHBzBntIUYY1qBV4DLTiDWhNPS1U9bzwCVRVluh+KIWWW5bD/QQViXpVAq7g2bDIwxfmNM7hCPHGPMsM1EIlIsIvn26wzgYmBb1CJPANX2cg2TCxMzGcyekEtXf4gqXZZCqbg32klnY1EGvCIiG4BVWH0GzzlYnudEkkFlUeI1EwHMmZgHwKa9bS5HopQayWj3MzhuxpgNwAKn7p8Iqhq78UlizT4ebFpJNmkpPjbUtXHt/Iluh6OUGoaTNQM1gurGLibkZ5CWkhhLVx8txe9j9oRcNmrNQKm4p8nARXuauqhI0P6CiDkT89i8t007kZWKc5oMXGKMoaqxi4oE7S+ImDMxj67+ELsbtRNZqXimycAlLd0DtPcGE79mUK6dyEp5gSYDl1Q1RkYSJXYymFqcTXrA6kRWSsUvTQYuiWwJWZHgySDF72N2Wa7WDJSKc5oMXFLd2GUNK02wfQyGMrc8n8372ghpJ7JScUuTgUuqmrqZWJBBakri/y841e5ErmrsHPlkpZQrEv83UZxKhmGlEZGZyDrfQKn4pcnABZFhpYneeRxxcnEWGQG/diIrFcc0Gbiguaufjt5gwi5Qd7TITGTtRFYqfmkycEGiL1A3lHnl+Wzc28ZAKOx2KEqpIWgycEF1YzdA0vQZACyaXEDvQFi3wVQqTmkycEF1Uxd+nyTsaqVDWTg5H4A1e1rcDUQpNSRNBi6oauyivCCDgD95/vOX5WUwIS9dk4FScSp5fhvFkV0NyTOSaLAFkwtYW9PqdhhKqSFoMoixUNiwu6GT6eNz3A4l5hadVMDe1h72t/W4HYpS6iiaDGKsrqWbvmCYqcXZbocSc4smFwCwulqbipSKN5oMYmzHQWtJhqnjky8ZnDIhl6xUPyurmtwORSl1FE0GMbaj3k4GJcmXDFL8PhZXjOOt3c1uh6KUOoomgxjbUd/B+Nw0ctMDbofiijOnFLKzvpPGzj63Q1FKDeJYMhCRSSLyiohsEZHNIvJ5p8rykl31nUwrSb7O44gzpowD4O0qrR0oFU+crBkEgS8YY2YDZwJ3ishsB8uLe8YYdtR3JmUTUcSciXlkpvp5a7f2GygVTxxLBsaY/caYd+zXHcBWYKJT5XnBvrZeuvtDTEvCzuOIgN/H6ZXjWLGz0e1QlFKDxKTPQEQqgAXAyiE+u11EVovI6oaGhliE45odBzsAknJY6WDnTitmd0MXdS3dboeilLI5ngxEJBv4PXC3MeY9q5QZYx40xiw2xiwuLi52OhxX7bRHEk1Lwglngy2dVgTAih1aO1AqXjiaDEQkgJUIHjPGPO1kWV6ws76TwqxUxmWluh2Kq6aWZFOWl86rOxK7JqiUlzg5mkiAh4GtxpjvO1WOlyR753GEiHDutCJW7GgkqPsbKBUXnKwZnA18DLhARNbZjyscLC+uGWPYcbBDk4HtvOkltPcGeUcXrlMqLqQ4dWNjzApAnLq/1zR09NHeG2SaJgMAlk4vIuAXXtpygNMrx7kdjlJJT2cgx8gO7Tw+Qk56gCUnF/HSloMYY9wOR6mkp8kgRnYm8ZpEx3Lx7PFUN3Wzq6HT7VCUSnqaDGJk+8EOctNTKMlJczuUuHHRrBIAXth80OVIlFKaDGJk6/52ZpblYg2yUmBthbngpHye27Df7VCUSnqaDGIgHDZsP9DB7LJct0OJO1fPncDW/e3srO9wOxSlkpomgxjY09xNd39Ik8EQrppbhgj8ab3WDpRyk2NDS9VhW/ZZq3DMOpFk0NUE2/4Ee9dAyx4QgezxUDoXZlwOhSdHKdrYKslN58zKQv60fh93XzRNm9GUconWDGJg6/52/D4Z22qlrbXwhzvhf6bBnz4P2/4M/V3Q3w3VK+DFr8L/LoRfXA47XwYPDtP8wIKJ7G7s4p0a3RtZKbdozSAGtu5v5+TiLNID/tFfZAy88ygsvxfCITj907DgZhh/qlUriGithc1Pw8oH4DcfhGmXwpX/A/knRf8HccgVc8v49z9t5rer6lg0WSegKeUGrRnEwNb97cfXRBTsh2fvsGoC5afBXavh8u9C6ZwjEwFA/iQ4+/Pwf9bCJf9p1RZ+ejZs8s66gNlpKVw1t4znNuyjqy/odjhKJSVNBg5r6uxjX1vv6DuP+7vh8Q/D+sfhfffBx54d3V/5KWlw1ufgjjeheAYs+yS89HUIe2MhuOsXT6KrP8Qf1+9zOxSlkpImA4dt3NsGwNzy/JFPHuiFp26C3f+Aa38C7/sK+I7zf1HBZPjk87Dok/D6/fDMP1k1jTi3aHIBs8ty+eXrVbo8hVIu0GTgsI11VjI4deIINYNwGJ79LOz6G1z7I1hw09gL9Qfgqv8HF/wrbPwtPHYd9L5nX6G4IiLcdk4l7x7s5DXd9EapmNNk4LANe9uYUpxFTnpg+BP/8R2rI/iib1gdxSdKBJZ+Ed7/M9jzupUQ+uJ7YtfV8yZQnJPGwyuq3A5FqaSjycBhG+vamDsxb/iTtv0F/vFdKwmc/fnoBjD/RrjuF1C3Gh7/iDUsNU6lpvj4+JmT+ce7DTojWakY02TgoPr2Xg609w7fX9BSDc9+Bsrmw5Xff+9ooWiYfS188EGoeROeuAEGeqJfRpTcdOZk0lJ8PLyi2u1QlEoqmgwcdLjz+Bg1g3AInr4dDHD9o9aIIKfMuQ7e/1Ooeg1+dwuE4nMI57isVD64cCJPv1NHU2ef2+EolTQ0GThobU0rfp8we8IxOo/f/BHUroQrvgcFFc4HNO8Ga0Lau8vhuc/H7Wzl286ZQn8ozEPad6BUzGgycNCaPS3MLsslM3WIid71W+Fv34KZV8Hc62MX1GmfgqVfhrW/scqPQ1NLsrl67gQefaNaawdKxYgmA4cEQ2HW1bayaHLBez8MDcAzn4G0HLjqB870Ewzn/Ptg4Sfgtf+Bt38e27JH6f9cOI2egRAPvrbb7VCUSgqaDByy7UAHPQMhFg6VDFb8APavsxJBdnGMI8NKPld+H2ZcAX/5Emx+NvYxjGBqSTbXzJvAr97Yo7UDpWLAsWQgIr8QkXoR2eRUGfEssgLne2oGTbvg1e/BKR+A2de4EJnNnwIfehgmnQ5Pf9rqWI4zd10wjb5giAdf1dqBUk5zsmbwCHCZg/ePa2v2tDA+N40JeemHDxoDf/4Xa9TQZd9xL7iI1Ey48UkYNwWe/Cgc2Oh2REc4VDt4cw/17b1uh6NUQnMsGRhjXgWanbp/PDPGsKqqmUWTC47crGXjMtj9d7jw65BT6lp8R8gcBzf/3uq/+M2HoDm+RvDcfdF0BkJhfvDyDrdDUSqhaZ+BA2qau9nX1suSKYWHD/a0wAv3woSFsPhW94IbSl45fOwZCPXDr98PHQfdjuiQiqIsbj5zMk+tqmVnfafb4SiVsFxPBiJyu4isFpHVDQ0NbocTFW/sagJgyclFhw++/E3oboKrfwC+49jkJlaKZ8BNy6Czwaoh9LS6HdEhd10wlYyAn/9evs3tUJRKWK4nA2PMg8aYxcaYxcXFLoysccCbu5oozknj5OIs60DtKlj9Szjjs1A2z93ghlO+GD7ya2jYBk/cGDfLVhRmp/GZ86bw4paDrKpOypZHpRznejJINMYY3tjVxJIphVZ/QSgIz90NuRPg/HvdDm9kUy88vI7R7z5pzYmIA7edM4XxuWn855+3Eg7H58xppbzMyaGlTwBvAjNEpE5EbnOqrHiyq6GTxs4+zjrZ7i946ydwcJM1eigtx93gRuvUD9rLVjwPv78tLhJCRqqfL14yg3W1rTyzdq/b4SiVcIZYJyE6jDE3OnXvePb37Va/x9lTi6BlD/z929bkrllXuxzZcTrtUxDsgxfus95/6GFr0xwXfWhhOY+trOHbz2/j4lPGkzvSHhFKqVHTZqIoe2V7PdPHZzOpIAP+8kVA4PL/jv2SE9Gw5E649L9gyx/ioobg8wnfvPYUmrr6+OFfdaipUtGkySCKOvuCvF3VzPkzSmDzM7DjRbjga5A/ye3Qxm7JnXDJf9oJ4VOuJ4S55fnccNokHnmjmh0HdQMcpaJFk0EUrdjRyEDIcH5lJiy/xxo5dPrtbod14s76HFzyLdjyLCz7pNV85KIvXjKDzFQ/X3t2k3YmKxUlmgyi6G/bDpKTlsKiHfdDVwNcfb+1BlAiOOsuuPTbsPVP1n7Kve2uhVKYncZ9V8xiZVUzv1m5x7U4lEokmgyiZCAU5sUtB7lwsp/AOw/DGZ+BCQvcDiu6ltwBH3gA9rwBj14FnfWuhfKR0yaxdHox3/7LNmqaul2LQ6lEockgSl7f2Uhr9wBXNT8KuROtPQMS0bwbrMXtGt6Fhy+GendmBYsI3/3QHFJ8wpeWrdfmIqVOkCaDKHluw35y/EHO7fgLXPND78wpGItpF8Mtz0F/Nzx0Ebz7oithlOVl8K9XzWZlVTO/fKPalRiUShSaDKKgLxjihY11XGzeIO30T8LUi9wOyXnli+H2V2BcJTx+Pbx+vyt7Kn94cTkXzRrPd57femgPCaXU8dNkEAUvrN1NRz+8v6AKLv6m2+HETl453LocZl8LL33dWs+oO7ZrB4kI//fD8yjNS+eO37zDQd33QKkx0WRwoozhiRdXUC4NnHPjl60NY5JJahZ8+BG47Luw86/ws3Oh5q2YhpCXGeCBmxfT0TvAJ3+5is6+YEzLVyoRaDI4QVUvP8SbHSXcOM3gK1/sdjjuEIEzPwO3vWgNpf3l5fDi12K66unsCbn8+KaFbD/YwS2/eFsTglLHSZPBiah6lV+9sgE/YT78wY+4HY37Ji6Ef3oNFn4c3vhf+OnZUP16zIp/34wSfnjDAtbWtnLTQyup79AmI6VGS5PBWLXW0PTU53gyfAHXziulJD/JmoeOJT3Xmmz38T9COAiPXGEthd0Sm8lhV84t46c3LeTdAx1c+6PXWbm7KSblKuV1mgzGorcNnvgoj/ScS69J5Y4LZ7odUfyZch7c8Sac9xXY/jz86DSrk7mr0fGiLzmllGWfXUJqio+PPPgW9z69kXrtWFZqWGJcGA54LIsXLzarV692O4zhDfTAb66jsWYz5wd/zNnTS/nZxxa5HVV8a9trbfu54SlISYdFt1jrHeWVO1psd3+Q772wnV+/uQefT7hqThlXz5/AkimFpAficOtRpcZARNYYY064w1KTwfHo74YnPwq7/849Ex9lWVWA5XcvZWpJttuReUPDdljxAyspAMy4HBZ+wtpdzcF9ofc0dfHQa1U8u3YvHX1B/D6hsiiLGeNzmDQuk4LMAHkZATLTUkhL8ZGa4iMtxUdait9+jhzzH/F5il8r1sp9mgxiracVnrwJ9rzO2nN+xgdfzuFT51Ty1Stnux2Z97TWwNsPwronoLsRcibA7GusDYBOWuJYYugdCPHW7iZWVTfz7sFOth/oYH9bDwOhsf0b8PuEVL+PtMCRCePIY0cnFB8+EfthzZOIvPb5Dr9OS/FzUmEGFYVZTC3JJkc38lHHoMkglpp3w+M3QPNuOq/6GVe+XMhAMMzyf16qu22diGA/vLsc1j8Bu/4GwV7IGGf1N1QuhYqlUHiyoxsDGWPo7g/R2jNAT3+Q3oEw/aEwfYeeQ/QFw/QHw/QFw/QFQ4de99vvj/X5ocdA6Ih7GmMIGwgbQzhsMAZCxljvjRXT4ATlE5g3KZ9zphZxwcwS5k/Kt/bXVgpNBrGz5Q/wh8+B+Ahf/2vueiub5zfu56l/WsJpFePcji5x9HVak9a2Pw9V/4CO/dbx9HyYMN9aAXbCAiidA/mTHW1Wigd9wRC1zd3sbuhi4942VuxsZH1tK2EDU4qz+NDCcj64cCJleRluh6pcpsnAaZ31sPxe2LQMJizEXPcLvrGim0feqOaey2fymfNOdjvCxGUMNO2C6tdg31rrUb/FGqoKVid04VQomgZF061H4VRrnaT0PHdjd1BbzwAvbDrAsnfqeLuqGZ/AhbPG89EzTmLptGL8Pq0tJCNNBk7pbYOVD1oLrwV7YemX6Dvz89z3x+38/p06bjunkq9dOUur6bE20Av1m+HAJmjaAY07rA7p1j1gwofPyxhnJYWCCiiotF/b73PKwJcYnb41Td08uaqG366upbGzn/KCDG48/SSuXzyJ4pw0t8NTMaTJIJqMgbpVsPF3sP5J6GuHGVfCxd9gfU8x9z69kS372/n8hdO4+6JpmgjiyUCv1afTtANaqqG5ClqqrOe2OjChw+empFtNTEMli/yTIJDu1k8xZv3BMC9uOcBjb9Xw5u4mUnzCkpMLuWT2eC6aPT7hm5HCYUNrzwANHX00dvbR3NVPXzDMQMh69AfDpAX85KSlkJ2WQll+OuUFmeRlJE5fnyeSgYhcBtwP+IGHjDHfGe78mCaD9v1Q+xbUrITtf7H+wkxJh5lXMXDG51jRNZHHVu7hr1vrKclJ41vvP5VLTimNTWwqOkID0FZ7OEEcShb280DXoJPFqjnklduPiZA3adD7SZBR4Ghn9ona1dDJ71bX8eLmA+xutH62srx05kzMY1ZZLmV56YzPTSc/M0DA77MfQorPh88HPhH8PkHEep2W4iMrNQVfjJufjv4F39jZR0NHHw2dfTR29B9639jZR1NXP6ExbGyUk55CeUEmk8dlUlmcxZSiLKYUZ1FZlE1BZsBTf/DFfTIQET/wLnAxUAesAm40xmw51jVjTgahARjotiaEDX4Ee6CnBboaMJ2NBFtqCDbtYaB5D8GuZrpMOq3+QlpKTqem+DyqA1PZ3tjHO3ta6OoPUZAZ4BNnVXDrOZU6aijRGGPNho7UIlqqrCGvbbVWjaJtL4T6jrzGF4CsIvtRbD0yiyAjH1KzrQ2N0rIh1X4OZIA/1brOH7Be++3XvsDhxHLo36A5Mr5Dr8NWf0mo334M2A/7dXjgyM+CfexsDvL3vYaNTcKG5hSqusb2/fURJtvXT46vjxxfH7m+PnJ9veT7esnz9ZLn67GepZc8fw8Z9BOQEAGCBAghGAaMj37jI2h89JgU2kLptJkMWkIZtIYzaDMZNIWzaAjl0BjOpimcRYj3DhAIEKLI30mxr4si/+FHsb+LIn83Rf4uCv3dpEuQgM8QkDABCdMn6XRKFu1ksT+UR10wn7pgDrX9OVT3ZVHTm0nQHG4+zAuEqMweYEpOmJOyDYWZfsZl+hmXEaAgK0BGahppaWmkpaWSlpZOaloaKYF0649JfwDEN+gh730fZV5IBkuAfzfGXGq/vxfAGPPtY10z5mTwrfFW+/4QvjLwaX4fOpcgI29Mnx7wUVmUzaLJ+bxveglLpxeTmpIYbczqOIXD1hyIQ8mhzhpU0N1oJZGuBvu58agaRnzqN37qKeCgKaDdl8+AL50BXxpBXxr9kobxBQj7AoR9KYQlQFhS6DUBOsKpdITTaA+l0h5OpSOUSpv9vi2URlf4xP5IyvYNkJ/Sz7hAP0UpfRQFeilO6aUo0EtRSi9FgT6KAz0U+3vI9fcjGA4lTTP4+ajXg5/DQTt59h2RMCMJNRgcoC6YS9XAOHaHitltJlBlSqkKl7GfwlH9HD7C+Anzrym/5uMpL41wtrw3UWSXwN0bj/8/IN5IBtcBlxljPmW//xhwhjHmc0eddztwu/12BrDdkYCipwhwfoEdZ3g5dtD43eTl2CGx459sjCk+0QJG/nPZYcaYB4EH3Y5jtERkdTSysBu8HDto/G7ycuyg8Y+Gk20ge4FJg96X28eUUkrFGSeTwSpgmohUikgqcAPwRwfLU0opNUaONRMZY4Ii8jngBayhpb8wxmx2qrwY8kyT1hC8HDto/G7ycuyg8Y8oriadKaWUcoeOm1RKKaXJQCmllCaDURORy0Rku4jsFJF7XCj/FyJSLyKbBh0bJyIvicgO+7nAPi4i8kM71g0isnDQNZ+wz98hIp8YdHyRiGy0r/mh2PPxj1XGccY+SUReEZEtIrJZRD7vsfjTReRtEVlvx/8N+3iliKy0y3zKHiiBiKTZ73fan1cMute99vHtInLpoONDfr+OVcYYfga/iKwVkec8GHu1/f92nYisto954rtj3ydfRJaJyDYR2SoiS+IyfmOMPkZ4YHWA7wKmAKnAemB2jGNYCiwENg069t/APfbre4Dv2q+vAJ4HBDgTWGkfHwfstp8L7NcF9mdv2+eKfe3lw5VxnLGXAQvt1zlYy5TM9lD8AmTbrwPASrus3wI32Md/BnzWfn0H8DP79Q3AU/br2fZ3Jw2otL9T/uG+X8cqYww/w78AjwPPDXffOI29Gig66pgnvjv2tY8Cn7JfpwL58Ri/679ovfAAlgAvDHp/L3CvC3FUcGQy2A6U2a/LgO326wew1oE64jzgRuCBQccfsI+VAdsGHT903rHKOMGf4w9Ya1Z5Ln4gE3gHOANrRmjK0d8RrBF0S+zXKfZ5cvT3JnLesb5f9jVDlnGcMZcDLwMXAM8Nd994i92+tpr3JgNPfHeAPKAKe7BOPMevzUSjMxGoHfS+zj7mtvHGGHtLMA4A4+3Xx4p3uON1QxwfrowxsZsdFmD9de2Z+O1mlnVAPfAS1l/DrcaY4BBlHorT/rwNKBzDz1U4TBnH4wfAl4HIxg/D3TfeYgdr4aEXRWSNWMvXgHe+O5VAA/BLu5nuIRHJisf4NRkkCGOlf0fHCZ9oGSKSDfweuNsY0x7Ne4/GiZRhjAkZY+Zj/ZV9OjAziqE5RkSuAuqNMWvcjuUEnGOMWQhcDtwpIksHfxjn350UrObdnxpjFgBdWE020bj3qI2mDE0GoxOvS2scFJEyAPu53j5+rHiHO14+xPHhyjguIhLASgSPGWOe9lr8EcaYVuAVrGaPfBGJTNwcXOahOO3P84CmMfxcTcOUMVpnA9eISDXwJFZT0f0eiR0AY8xe+7keeAYrGXvlu1MH1BljVtrvl2Elh7iLX5PB6MTr0hp/BCKjCj6B1RYfOf5xe2TCmUCbXV18AbhERArskQWXYLXj7gfaReRMeyTCx4+611BljJp9z4eBrcaY73sw/mIRybdfZ2D1d2zFSgrXHSP+SJnXAX+z/zL7I3CDWCN2KoFpWJ1/Q36/7GuOVcaoGGPuNcaUG2Mq7Pv+zRhzkxdiBxCRLBHJibzG+n++CY98d4wxB4BaEZlhH7oQ2BKX8Y+lQycZH1i9/O9itRV/1YXynwD2AwNYf23chtUu+zKwA/grMM4+V4Af27FuBBYPus+twE778clBxxdj/SPbBfyIw7PThyzjOGM/B6uKugFYZz+u8FD8c4G1dvybgK/bx6dg/ULcCfwOSLOPp9vvd9qfTxl0r6/aMW7HHvUx3PfrWGWM8Tv0Pg6PJvJE7PY91tuPzZH7e+W7Y99nPrDa/v48izUaKO7i1+UolFJKaTORUkopTQZKKaXQZKCUUgpNBkoppdBkoJRSCk0GKkmJSEisVTA3i7Ua6RdEZNh/DyJSIfaqsSIyX0SuiE20SjnPsW0vlYpzPcZaXgIRKcFa0TMX+LdRXj8fa3z3X5wITqlY03kGKimJSKcxJnvQ+ylYs2mLsGrM38GapJUG/NgY84C9yN5zWMsJ7AQysKb+fxtrZcr7sSZt9WBNCtoeq59HqROlNQOlAGPMbhHxAyXAtVjLAJwmImnA6yLyIvZCX8aYfhH5Otbs0M8BiEgucK4xJigiFwH/BXzIlR9GqTHQZKDUe10CzBWRyLo6eVhr8bw7zDV5wKMiMg0raQScDVGp6NJkoBSHmolCWCs7CnCXMeaFo86pGOYW/wG8Yoz5gH3e352JVCln6GgilfREpBhrW8YfGasT7QXgs/ay24jIdHvFzME6sLbwjMjj8NLBtzgbsVLRp8lAJauMyNBSrBUdXwS+YX/2ENYyw+/YQ0kf4L216FeA2fY9PoK13+y3RWTtEOcqFfd0NJFSSimtGSillNJkoJRSCk0GSiml0GSglFIKTQZKKaXQZKCUUgpNBkoppYD/D+DuHhgBb0HkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.kdeplot(train['Delta'], data = train, hue = \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above plot depicts that as the amount is getting incresed we found that the testing label fequency almost reaches 0 but there is quite a good amount of person which is not defaulters, means if this feature represent salary then we can hypothize that huge salary made the bank trust more on customer compared to low ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Beta', ylabel='Density'>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyFklEQVR4nO3deXwc1ZXo8d/RblmbtdiWJduysbxI3hFmSwhgsM0y+CUhiUl4kAnzyCSQDWbyILzHJMwjEyaTQCaBZAgwELKA2YKTEMy+BPAiG9vyJluWF0l4ka1d1tbd5/3RJSOELEuWqkvdfb6fT39cfetW6XRZ0lHdU3VLVBVjjDFmoGK8DsAYY0x4scRhjDFmUCxxGGOMGRRLHMYYYwbFEocxxphBifM6gFDIzs7WgoICr8MwxpiwsWHDhqOqmtPXuqhIHAUFBZSWlnodhjHGhA0R2X+ydTZUZYwxZlAscRhjjBkUSxzGGGMGxRKHMcaYQbHEYYwxZlAscRhjjBkUSxzGGGMGxRKHMSOIquIP2KMOzMgWFTcAGjOS+QPK85tq+MXrFew72kpiXCxfPHsSX7/wDLJSEr0Oz5iPscRhjIfau/zc/Pv3eWXHYYonpPH1C6dRXX+cR9/dxxvlR3jqH88jc3SC12Ea8xGWOIzxSHuXn+seWcf6fXXceWURXz6vgJgYAeCaRZO47pF1/P1/r+PJr55LUnysx9Ea8yGrcRjjAVXln5/ewvp9ddz3hfl85RNTTiQNgLOnZvGzFfPZXN3II+/s9TBSYz7OEocxHnjwrUr+tPkD/nnpDJbPz+uzz7LZuVxaNI5fvFbB4ab2EEdozMlZ4jAmxLZ90Mh/vFTOZbPH87VPndFv3zsun4XPr9z3yq4QRWfMqVniMCaE2rv8fOfJTYxJTuCHn56DiPTbvyB7NFeX5PPsxhrqWztDFKUx/bPEYUwI/eSlcnYdbuGeq+cyZoBXS1137mQ6fAGe2lDlcnTGDIwlDmNC5L09x3job3u59pxJXDRj7IC3mzk+jUUFmTy+Zr/dHGhGBEscxoRAc3sX//TUZiZnJvO9y2cNevtrz51MVV0bayqPuRCdMYNjicOYEPjXP2/nYGMbP/3CfJITBn/71KWzxjEqPpa/lB10ITpjBscShzEue3n7YVaWVvO1C89g4aQxp7WPUQmxXDxrLKu3HrLhKuM5VxOHiCwTkXIRqRCR2/pYnygiTzrr14pIQY91tzvt5SKy1GmbISKberyaROTbbn4GEx2a2rt4+G97+cGftvFvL+zgnYqj+PyBIe/3WEsHtz+7hVm5aXxr8fQh7evy2bkca+1k7V4brjLecm3KERGJBe4HLgWqgfUiskpVt/fodgNQr6rTRGQFcA/wBREpAlYAxcAE4BURma6q5cD8HvuvAZ5z6zOY6PD8phrufH4bjW1dpCbG0eEL8F9vVTJ9XAp3XlnMJwqzT2u/gYDy3ae30NTm47f/MI+EuKH9nXbRzByS4mN4oewg551xejEZMxzcPONYBFSoaqWqdgJPAMt79VkOPOYsPw0sluCF7cuBJ1S1Q1X3AhXO/npaDOxR1f2ufQIT8Z7fVMO3n9xE4dgU/vyNT1D2g6Vs+pdL+dmK+bR3Bbj24bXc8VwZbZ3+Qe/7F69X8OrOI9xxxSxmjk8bcqzJCXFcUJjD6ztrUbXhKuMdNxNHHtDzwvNqp63PPqrqAxqBrAFuuwL4w8m+uIjcKCKlIlJaW1t7Wh/ARLYt1Q3csnJz8FLXG85mdl46EPwFvXx+Hi/fcgE3XjCV3687wBU/f5uy6sYB73vV5g+495VdfGZBHtedO3nYYr5geg41DW1UHm0dtn0aM1hhWRwXkQTgKuCpk/VR1QdVtURVS3JyckIXnAkLXf4Atz1TRtboBB68roRRCR+ffTYxLpbvXT6L391wNm2dfj79wDvc/3rFKYvTL249yHee3MSigkx++JlT3x0+GBcUBr+X395lfwwZ77iZOGqAiT3e5zttffYRkTggHTg2gG0vAzaq6uFhjtlEiUff2cf2g03ctbyY9FHx/fY9b1o2L37rApbOHs+PV5dzzYNrqKo7/rF+Xf4A972yi6/9biNz8tJ5+MtnDft06JOykinISubt3UeHdb/GDIabz+NYDxSKyBSCv/RXAF/s1WcVcD3wHnA18JqqqoisAn4vIj8lWBwvBNb12O4a+hmmMqY/bZ1+fvnmHi6YnsPS4vED2iY9OZ5fXLOAxTPHcufz21h631tcOTeX86dlk5oUx85Dzfxh3QGq6tr4zII8fviZOa49Q+OThTk8s7GaTl9gyAV3Y06Ha4lDVX0icjOwGogFHlHVbSJyF1CqqquAh4HHRaQCqCOYXHD6rQS2Az7gJlX1A4jIaIJXan3VrdhNZHty/QHqWjv5xsXTBjWMJCJ8ZmE+ZxVk8p+v7uYvWw6ysrT6xPp5+en8y5XFLJ41dliHp3r7ZGE2j6/Zz8YD9ZwzNcu1r2PMyUg0XJ1RUlKipaWlXodhRoAuf4ALf/wGEzKSeOofzxvSvtq7/FTXH6exrYup2SkDnrRwqBqPdzH/X1/iO5dM55uLC0PyNU30EZENqlrS1zp7dKyJKq/uOEJNQxvfv6p4yPtKio9l2tjUYYhqcNKT45kxLpX1++pC/rWNgTC9qsqY0/X0hirGpiZy0YzwvtJu0ZRMNu6vH5a7240ZLEscJmrUNnfwenktn1mYT1xseH/rn1WQSWunn+0Hm7wOxUSh8P7pMWYQ/vh+Df6AcvWZfT/jO5wsmpIJwLq9NlxlQs8Sh4kaz2+uYV5+uid1ieE2Li2JSZnJljiMJyxxmKhQVXecrTVNXD4n1+tQhk1JwRg2HmiweatMyFniMFHhpe3BSQYGesNfOFgwMYOjLR3UNLR5HYqJMpY4TFRYvfUQM8enUpA92utQhs38icGHQm2qavA2EBN1LHGYiHe0pYP1++si6mwDYMb4VBLiYth0oMHrUEyUscRhIt4b5bWowqVF47wOZVglxMUwe0KanXGYkLPEYSLem7tqyUlNpHjC0B+mNNLMnziGsppGuuxGQBNCljhMRPMHlLd31/Kp6TmuTjzolfmTMujwBSg/1Ox1KCaKWOIwEW1LdQMNx7v41PTwnmLkZObnZwCwZRBPJzRmqCxxmIj25q5aYgQ+MS3b61BcMTFzFKlJcWz7wBKHCR1LHCaivbWrlrn5GSGb8jzURITZE9LZ+oHNWWVCxxKHiVitHT62VDdy3hmR/bCj4glp7DzYZDPlmpCxxGEi1ob99fgCGvFPyZudl06HL8Ce2lavQzFRwhKHiVhr9x4jNkY4c/IYr0Nx1ey84GXGW2uszmFCwxKHiVhrKuuYm5/O6MTIftDllOwURsXHstUK5CZEXE0cIrJMRMpFpEJEbutjfaKIPOmsXysiBT3W3e60l4vI0h7tGSLytIjsFJEdInKum5/BhKfjnT62VDdw9pTIHqYCiI0RZuWmsq3GCuQmNFxLHCISC9wPXAYUAdeISFGvbjcA9ao6DbgXuMfZtghYARQDy4AHnP0B/Ax4UVVnAvOAHW59BhO+Nu5voMuvnDM10+tQQmJ2XjrbDzYRCNgU68Z9bp5xLAIqVLVSVTuBJ4DlvfosBx5zlp8GFkvw9t7lwBOq2qGqe4EKYJGIpAMXAA8DqGqnqja4+BlMmOqub5QUREnimJBOS4eP/XXHvQ7FRAE3E0ceUNXjfbXT1mcfVfUBjUBWP9tOAWqB/xaR90XkIRHpc55sEblRREpFpLS2tnY4Po8JI2sqjzE7L52UCK9vdCuaYAVyEzrhVhyPAxYCv1TVBUAr8LHaCYCqPqiqJapakpMTmdNNmL61dfrZXNXIOVOi42wDYPq4VOJjhW12I6AJATcTRw0wscf7fKetzz4iEgekA8f62bYaqFbVtU770wQTiTEnvH+gnk5/gLOjpL4BwSnWZ4xPtalHTEi4mTjWA4UiMkVEEggWu1f16rMKuN5Zvhp4TYMPUF4FrHCuupoCFALrVPUQUCUiM5xtFgPbXfwMJgyt2VtHjBA19Y1uxbnpbK1ptGeQG9e5ljicmsXNwGqCVz6tVNVtInKXiFzldHsYyBKRCuAWnGEnVd0GrCSYFF4EblJVv7PNN4DficgWYD7wQ7c+gwlPayqPUTwhnbSkeK9DCanZeWnUH+/iYGO716GYCOdq5VBVXwBe6NV2Z4/lduBzJ9n2buDuPto3ASXDGqiJGJ2+AJurGvjS2ZO9DiXkivPSASiraWRCxiiPozGRLNyK48b0a/vBJjp8gYifZqQvM8enIgI7DlqB3LjLEoeJKBv31wOwcHKGt4F4IDkhjilZoy1xGNdZ4jARZcOBeiakJ5GbHp1DNbMmpLHdEodxmSUOE1He31/PwigcpupWlJtGVV0bze1dXodiIpglDhMxDja28UFje1TWN7rNyk0FYOehZo8jMZHMEoeJGBv3NwCwcFL0Jo6i3OCVVdvtDnLjIkscJmJs2F9PUnzMiXmbotG4tETGJMdbgdy4yhKHiRgbDtQzNy+D+Njo/bYWEWblplniMK6K3p8wE1Hau/xs/6Axqgvj3WblprHzUDM+f8DrUEyEssRhIkJZTSNdfo3qwni3otw0OnwB9h1r9ToUE6EscZiI0H3j34JJGd4GMgLMyg3WeLYftCurjDsscZiIsGF/PQVZyWSnJHodiuemjU0hPlaszmFcY4nDhD1VZeOBhqi+DLenhLgYpo1NtUtyjWsscZiw90FjO0dbOmyYqodZual2xmFcY4nDhL1NBxoAmDcxw9M4RpKi3DSONHdwtKXD61BMBLLEYcLe5uoGEuJimDk+em/8663IKZDbWYdxgyUOE/Y2HWigeEIaCXH27dxtliUO4yL7STNhzecPUFbTyLz8DK9DGVHGjE5gfFoSO+ySXOMCVxOHiCwTkXIRqRCR2/pYnygiTzrr14pIQY91tzvt5SKytEf7PhEpE5FNIlLqZvxm5Nt1uIW2Lr8VxvtgBXLjFtcSh4jEAvcDlwFFwDUiUtSr2w1AvapOA+4F7nG2LQJWAMXAMuABZ3/dLlLV+apqzx6PcpurGwDsjKMPRRPSqDjSQofP73UoJsK4ecaxCKhQ1UpV7QSeAJb36rMceMxZfhpYLCLitD+hqh2quheocPZnzEdsOtBARnI8k7OSvQ5lxJmVm4YvoOw+3OJ1KCbCuJk48oCqHu+rnbY++6iqD2gEsk6xrQIvicgGEbnxZF9cRG4UkVIRKa2trR3SBzEj1+bqBublZxD8e8P0ZAVy45ZwLI5/QlUXEhwCu0lELuirk6o+qKolqlqSk5MT2ghNSLR2+Nh1uJn5dv9GnwqyRjMqPtaeQW6GnZuJowaY2ON9vtPWZx8RiQPSgWP9bauq3f8eAZ7DhrCiVllNIwHFEsdJxMYIM8ZbgdwMPzcTx3qgUESmiEgCwWL3ql59VgHXO8tXA6+pqjrtK5yrrqYAhcA6ERktIqkAIjIaWAJsdfEzmBFsU1UDYHeM9yf4UKdmgj9WxgwP1xKHU7O4GVgN7ABWquo2EblLRK5yuj0MZIlIBXALcJuz7TZgJbAdeBG4SVX9wDjgbyKyGVgH/EVVX3TrM5iRbXNVA5Myk8kcneB1KCNWUW4qjW1dfNDY7nUoJoLEublzVX0BeKFX2509ltuBz51k27uBu3u1VQLzhj9SE442VTVwVkGm12GMaN3PX9/xQRN5GaM8jsZEinAsjhvD4aZ2Dja22zDVKcwYb1dWmeFnicOEpe76hhXG+5eSGMfkrGR2HLLEYYaPJQ4TljZXNRAXIxRPsBlxT6UoN80e6mSGlSUOE5Y2VTUwKzeNpPjYU3eOcrNy09hfd5zWDp/XoZgIYYnDhJ1AQNlS3ci8ielehxIWZuWmoQo7D9lMuWZ4WOIwYWdPbQstHT7mT7RnjA/ErNxUALuD3AwbSxwm7HxYGLczjoHIyxhFRnI822oavQ7FRIgBJQ4ReVZErhARSzTGc1uqG0lJjGNqdorXoYQFEWFOXjpbqi1xmOEx0ETwAPBFYLeI/EhEZrgYkzH9KqtppHhCGjExNiPuQM3NT2fX4Wbau+zZHGboBpQ4VPUVVf0SsBDYB7wiIu+KyN+LSLybARrTU5c/wPaDTczNt2GqwZiTl4EvoHYjoBkWAx56EpEs4MvAPwDvAz8jmEhediUyY/qw+3ALnb4As/MscQxGd6ItszqHGQYDmqtKRJ4DZgCPA3+nqgedVU/ac79NKG11fvHNscQxKLnpSWSnJFidwwyLgU5y+GtnwsITRCTRebSrPffbhMyWmgZSE+MoyBrtdShhpbtAXmaJwwyDgQ5V/b8+2t4bzkCMGYiymiaK86wwfjrm5Gew+0gzxzvtDnIzNP0mDhEZLyJnAqNEZIGILHReFwLJoQjQmG5d/gA7DjbZMNVpmpuXTkCxeavMkJ1qqGopwYJ4PvDTHu3NwPdcismYPu063EynL8Cc/AyvQwlL3QXyLdWNlNhzTMwQ9Js4VPUx4DER+ayqPhOimIzpkxXGh2ZsWhLj05LsyiozZP0mDhG5VlV/CxSIyC2916vqT/vYzBhXlNU0kpoYx+RMGyU9XXPy09lS3eB1GCbMnao43n3pSgqQ2sfLmJApq25kdl66FcaHYG5eOpVHW2lu7/I6FBPGTjVU9V/Ovz84nZ2LyDKCNwrGAg+p6o96rU8EfgOcCRwDvqCq+5x1twM3AH7gm6q6usd2sUApUKOqV55ObCa8dPkD7DjUzJfPK/A6lLA2Jz8dVdha08S5Z2R5HY4JUwOd5PDfRSRNROJF5FURqRWRa0+xTSxwP3AZUARcIyJFvbrdANSr6jTgXuAeZ9siYAVQDCwDHnD21+1bwI6BxG4iQ3dh3O4YH5ru+pANV5mhGOh9HEtUtQm4kuBcVdOAfz7FNouAClWtVNVO4Algea8+y4HHnOWngcUiIk77E84NhnuBCmd/iEg+cAXw0ABjNxGg+8a1uZY4hiQrJZGJmaPYbInDDMFAE0f3kNYVwFOqOpDLMvKAqh7vq522Pvuoqg9oBLJOse19wHeBQH9fXERuFJFSESmtra0dQLhmJCuraSQ1KY7JWVYYH6qFk8awYX89qup1KCZMDTRx/FlEdhKsRbwqIjlAu3th9U1ErgSOqOqGU/VV1QdVtURVS3JyckIQnXFTWU0jc/LSCZ6QmqE4c/IYDjd1UNPQ5nUoJkwNdFr124DzgBJV7QJa+fiwU281wMQe7/Odtj77iEgckE6wSH6ybc8HrhKRfQSHvi4Wkd8O5DOY8NXpC7DzYLPdvzFMFk4KPnJ344EGbwMxYWswT/SbCXxBRK4DrgaWnKL/eqBQRKaISALBYveqXn1WAdc7y1cDr2nw/HkVsEJEEkVkClAIrFPV21U1X1ULnP29pqr9FulN+Nt1uJlOvxXGh8vM8amMio9l4/56r0MxYWqg06o/DpwBbCJ4eSyAEryUtk+q6hORm4HVBC/HfURVt4nIXUCpqq4CHgYeF5EKoI5gMsDptxLYDviAm1TVHl0WpbrvdLaHNw2PuNgY5k/MYIMlDnOaBjqteglQpIOspjlTsb/Qq+3OHsvtwOdOsu3dwN397PsN4I3BxGPCU1lNI2lJcUyyO8aHzcLJGfzqzUqOd/pIThjorwFjggY6VLUVGO9mIMaczNaa4B3jVhgfPiUFmfgDysb9DV6HYsLQQBNHNrBdRFaLyKrul5uBGQM9CuM2TDWsSiaPIUZg3d5jXodiwtBAz1G/72YQxpxMd2HcrqgaXqlJ8czOS2fN3jqvQzFhaKCX475J8I7xeGd5PbDRxbiMAT4sjFviGH5nT8lkU1UD7V123YkZnIHOVfW/CE4J8l9OUx7wR5diMuaELdVWGHfLoilZdPoCbKpq8DoUE2YGWuO4ieDNd00AqrobGOtWUMZ021rTyJx8K4y7YVFBJiKwttKGq8zgDDRxdDgTFQIn7vK2iW6Mqzp8fnYearIb/1ySnhxPUW4a7+456nUoJswMNHG8KSLfA0aJyKXAU8Cf3AvLGNh1qIUuvzI3L8PrUCLWJ6Zls/FAPa0dPq9DMWFkoInjNqAWKAO+SvCmvv/jVlDGgBXGQ+EThdl0+ZV1dnWVGYQBXY6rqgER+SPwR1W1OcpNSJTVNJI+Kp6JmaO8DiVinVWQSUJcDG/vPspFM61saQam3zMOCfq+iBwFyoFy5+l/d/a3nTHDoaymwaZSd1lSfCyLCjL5W4X9PWgG7lRDVd8heDXVWaqaqaqZwNnA+SLyHdejM1Grw+en/FCzFcZD4JOF2ew63MKhxpA/YseEqVMljv8JXOM8vhUAVa0ErgWuczMwE93KDzUHC+M21YjrLpwRHKJ6vfyIx5GYcHGqxBGvqh+7Vs+pc8S7E5IxVhgPpenjUsjLGMWrOyxxmIE5VeLoPM11xgzJVqcwnj/GCuNuExEumTWWdyqO2vQjZkBOlTjmiUhTH69mYE4oAjTRaUt1I3PtjvGQuXjWONq6/Ly3x2bLNafWb+JQ1VhVTevjlaqqNlRlXNHh87PrsBXGQ+mcqZkkJ8Ty8o7DXodiwsBgnjluTEh0F8atvhE6iXGxXDgjh5e2HcYfsNmETP9cTRwiskxEykWkQkRu62N9oog86axfKyIFPdbd7rSXi8hSpy1JRNaJyGYR2SYiP3AzfuONLdVWGPfC5XNyOdrSYXeRm1NyLXGISCxwP3AZUARcIyJFvbrdANSr6jTgXuAeZ9siYAVQDCwDHnD21wFcrKrzgPnAMhE5x63PYLyxtaaRjGQrjIfaxTPHkhQfwwtlB70OxYxwbp5xLAIqVLXSmVn3CWB5rz7Lgcec5aeBxRKshi4HnlDVDucekgpgkQa1OP3jnZedV0eYsppGu2PcA8kJcVw8cyx/3XrIhqtMv9xMHHlAVY/31U5bn31U1Qc0Aln9bSsisSKyCTgCvKyqa/v64iJyo4iUikhpba1NpxAu2ruCd4zbMJU3rpgzgaMtHXZ1lelX2BXHVdWvqvOBfGCRiMw+Sb8HVbVEVUtycnJCGqM5feWHmvEFrDDulcWzxpKaFMezG6u9DsWMYG4mjhpgYo/3+U5bn32ch0OlA8cGsq2qNgCvE6yBmAhx4o5xm2rEE0nxsVw5N5e/bj1Eiz2jw5yEm4ljPVAoIlNEJIFgsXtVrz6rgOud5auB11RVnfYVzlVXU4BCYJ2I5IhIBoCIjAIuBXa6+BlMiJVVNzImOZ68DCuMe+WzC/Np6/JbkdyclGuJw6lZ3AysBnYAK1V1m4jcJSJXOd0eBrJEpAK4heADo1DVbcBKYDvwInCTqvqBXOB1EdlCMDG9rKp/duszmNDbUtPInPwMK4x76MzJYyjISuap0qpTdzZRaUAPcjpdqvoCwacF9my7s8dyO/C5k2x7N3B3r7YtwILhj9SMBG2dwTvGF9sDhTwlIlyzaBL/9ted7DzUxMzxaV6HZEaYsCuOm8i1/WAT/oBNpT4SfK5kIglxMfx2zX6vQzEjkCUOM2JsqW4AYG5+hqdxGMgcncCVc3N5bmMNze1dXodjRhhLHGbEKKtuZGxqIuPTk7wOxQDXnVtAa6eflaV2aa75KEscZsTYXN1gw1QjyPyJGZxVMIZH/raXLn/A63DMCGKJw4wIze1dVB5ttWGqEearF5xBTUObXZprPsIShxkRttY0oWo3/o00F88cy7SxKfzyjT0EbP4q47DEYUaEspoGAObaVCMjSkyMcPNF09h5qJkXttpZhwmyxGFGhM3VjeRljCIrJdHrUEwvfzdvAoVjU7j35V02a64BLHGYEaLMeca4GXliY4RbLp3OntpWnt/Ue7o5E40scRjP1bd2cqDuuBXGR7ClxeMpyk3jvld22xVWxhKH8V73jLh2xjFyxcQIty6ZzoG64zyzwe7riHaWOIznuu8Yn22F8RHt4pljmT8xg5+9upv2Lr/X4RgPWeIwnttc3ciU7NGkj4r3OhTTDxHhfy+bycHGdv77nX1eh2M8ZInDeEpVef9AAwsmZngdihmAc8/IYvHMsTzwegV1rZ1eh2M8YonDeKqqro2jLR0smDzG61DMAN122UxaO338/LXdXodiPGKJw3hq44F6AM6cZIkjXBSOS+ULZ03kt2v2s/9Yq9fhGA9Y4jCe2rC/ntEJscwYn+p1KGYQvnPJdOJiYvj31eVeh2I8YInDeGrD/nrmT8ogNsYeFRtOxqYl8b8umMpfthzkfees0UQPSxzGM60dPnYearJhqjB14wVTyU5J5N9e2ImqTUUSTVxNHCKyTETKRaRCRG7rY32iiDzprF8rIgU91t3utJeLyFKnbaKIvC4i20Vkm4h8y834jbs2VzUQUKwwHqZSEuP49iWFrNtXx8vbD3sdjgkh1xKHiMQC9wOXAUXANSJS1KvbDUC9qk4D7gXucbYtAlYAxcAy4AFnfz7gVlUtAs4BbupjnyZMdBfGF060xBGuVpw1kak5o/nRizttKpIo4uYZxyKgQlUrVbUTeAJY3qvPcuAxZ/lpYLGIiNP+hKp2qOpeoAJYpKoHVXUjgKo2AzuAPBc/g3HRhv31TBubQnqy3fgXruJiY7ht2Uwqa1t5cn2V1+GYEHEzceQBPb+Tqvn4L/kTfVTVBzQCWQPZ1hnWWgCs7euLi8iNIlIqIqW1tbWn/ymMKwIBZeOBBqtvRIBLi8axqCCT+17ZRUuHz+twTAiEZXFcRFKAZ4Bvq2pTX31U9UFVLVHVkpycnNAGaE6p8mgrjW1dnGn1jbAnItx++UyOtnTy4FuVXodjQsDNxFEDTOzxPt9p67OPiMQB6cCx/rYVkXiCSeN3qvqsK5Eb123c79Q3Jmd4G4gZFgsmjeGKubn8+q1KDje1ex2OcZmbiWM9UCgiU0QkgWCxe1WvPquA653lq4HXNHhd3ypghXPV1RSgEFjn1D8eBnao6k9djN24bOOBetJHxTM1O8XrUMww+e7SGfgCAe57ZZfXoRiXuZY4nJrFzcBqgkXslaq6TUTuEpGrnG4PA1kiUgHcAtzmbLsNWAlsB14EblJVP3A+8D+Bi0Vkk/O63K3PYNyzfl8dCydlEGM3/kWMyVmjufacyTy5vopdh5u9Dse4SKLhxp2SkhItLS31OgzjONLUzqIfvsrtl83kq586w+twzDCqb+3kgh+/zlkFmTzy5bO8DscMgYhsUNWSvtaFZXHchLc1e+sAOGdqlseRmOE2ZnQCX79wGq/tPML6fXVeh2NcYonDhNx7e46RmhhH8YQ0r0MxLvjyeQVkpyTyH6vLbSqSCGWJw4Tc2spjLJqSSVysfftFolEJsdx80Rms3VvHOxXHvA7HuMB+ck1IHW5qp/Joqw1TRbhrzp7EhPQk/uMlO+uIRJY4TEi9U3EUCD6C1ESuxLhYvrG4kE1VDby284jX4ZhhZonDhNRbu2rJGp1AUa7VNyLd1WfmMzkrmZ+8tItAwM46IoklDhMygYDy9u6jfLIw2+7fiALxsTF8+5JCth9s4q9bD3kdjhlGljhMyGw/2MSx1k4umG5zh0WLq+blMW1sCj99uRy/nXVEDEscJmTe3BWcpfiThZY4okVsjHDLpdPZU9vK85t6T1VnwpUlDhMyb+6qpXhCGjmpiV6HYkJoWfF4iiekcd8ru+1hTxHCEocJibrWTkr31XHxzLFeh2JCLCZGuHXJdA7UHeep0mqvwzHDwBKHCYlXdhwmoLC0eLzXoRgPXDRjLAsmZfDz13bT3uX3OhwzRJY4TEi8tO0weRmjbJqRKCUi/NOSGRxsbOcP6w54HY4ZIkscxnXHO328vbuWS4vGEXykiolG50/L5typWdz/+h6Od9ojZsOZJQ7jujfKa+nwBVhSPM7rUIzHbl0ynaMtHTz27n6vQzFDYInDuO75TTVkpySyqCDT61CMx0oKMrlwRg6/enMPTe1dXodjTpMlDuOqhuOdvL6zluXzJ9hsuAaAWy+dQWNbF4/8ba/XoZjTZD/JxlUvlB2i0x/g0wvyvA7FjBBz8tNZVjyeh97eS11rp9fhmNPgauIQkWUiUi4iFSJyWx/rE0XkSWf9WhEp6LHudqe9XESW9mh/RESOiMhWN2M3w+O596uZNjbFrqYyH3Hrkum0dfn58epyr0Mxp8G1xCEiscD9wGVAEXCNiBT16nYDUK+q04B7gXucbYuAFUAxsAx4wNkfwKNOmxnhyg81s35fPVefmW9XU5mPKByXyvXnFvDE+gNsqW7wOhwzSG6ecSwCKlS1UlU7gSeA5b36LAcec5afBhZL8DfMcuAJVe1Q1b1AhbM/VPUtwB5mHAYeX7OPhLgYPl8y0etQzAj07UsLyRqdyP99fptNux5m3EwceUBVj/fVTluffVTVBzQCWQPctl8icqOIlIpIaW1t7SBDN0PV1N7FsxtruGreBDJHJ3gdjhmB0pLi+d7lM9lc1cDK0qpTb2BGjIgtjqvqg6paoqolOTk2G2uorVxfxfFOP9edO9nrUMwI9ukFeZxVMIZ7XtxJvRXKw4abiaMG6DlGke+09dlHROKAdODYALc1I1Rbp59fvVnJ+dOymJuf4XU4ZgQTEe5aPpvmdh/f/9M2r8MxA+Rm4lgPFIrIFBFJIFjsXtWrzyrgemf5auA1DT7ZfhWwwrnqagpQCKxzMVYzjH63dj9HWzr41uLpXodiwsCs3DS+cXEhz2/6gBfKDnodjhkA1xKHU7O4GVgN7ABWquo2EblLRK5yuj0MZIlIBXALcJuz7TZgJbAdeBG4SVX9ACLyB+A9YIaIVIvIDW59BjN4ze1dJ842Fk2xO8XNwHz9ojOYk5fO7c+WUVV33OtwzClI8A/8yFZSUqKlpaVehxEV/vXP23nknb388evnM29ihtfhmDCy72grf/fzvzE1ZzQr//FcEuNiT72RcY2IbFDVkr7WRWxx3IRe+aFmHn13HyvOmmRJwwxaQfZofvy5uWyubuT2Z8qIhj9qw5UlDjMsOnx+vvPkJtKS4vju0hleh2PC1LLZudxy6XSefb+Gn768y+twzEnEeR2AiQz3/LWc7QebeOi6EsbYfRtmCL5x8TRq6tv4+WsVxMYI377ELrIYaSxxmCF7qrSKR97Zy/XnTuaSInvmhhkaEeGHn5mDX5X7XtlNw/Eu/s8Vs2x25RHEEocZkjfKj3D7s2V8Ylo2d1zReyoyY05PbIzw75+dS1pSPI+8s5eKIy385PPzGJeW5HVoBqtxmCFYve0QN/5mAzPGp/LLaxeSEGffTmb4xMQId/5dEf/+2bmU7q9jyb1v8fu1B/DbvFaes590M2iqyq/fquRrv91AcV4av/+Hc0hNivc6LBOhPn/WRF745ieZMS6V7z1XxtL7ggmkrdPvdWhRy+7jMINS39rJHX8s44WyQywrHs9PPj+P0Yk24mncp6r8deshfvFaBdsPNpE+Kp7PnZnPZ8/MZ1auPe9luPV3H4clDjMgqsqfthzk//15O/XHO7l1yQy+esFUe86GCTlVpXR/PY++s4/V2w7hCygzx6fy6QV5XDV/Arnpo7wOMSJY4rDEMSS7DjfzL89v473KY8zOS+NHn5nL7Lx0r8MyhrrWTv6y5QOee7+GjQcaEIFzp2bxPxbksWz2eNJsCPW0WeKwxHFaDjW287NXd7OytIqUxDj+aekMvrhoErExdpZhRp59R1t5ftMHPPd+NfuOHScxLoYr507gHz81lcJxqV6HF3YscVjiGJT61k5++eYeHnt3HwFVrlk0iW8tLiQrJdHr0Iw5JVVlc3Ujz2yo5ukN1bR1+VlaPI6bLypkTr6dKQ+UJQ5LHAPS0uHjkb/t5ddvVdLS6ePTC/L4ziXTmZiZ7HVoxpyWutZOHn1nL4++u4+mdh+XzR7PrUumM22snYGciiUOSxz9am7v4rdrDvDQ25Uca+1kafE4bl0yg+l2em8iRHN7Fw+9vZeH3q6krcvPZxfm861LCskfY38UnYwlDkscfTrW0sGj7+7j0Xf30dzu45OF2dy6ZAbzbWZbE6GOtXTwwBt7eHzNflD44tmTuOmiaeSk2jBsb5Y4LHF8RMWRFn67Zj9Prq+i3ednWfF4vn7hNBv/NVHjg4Y2/vPV3Ty1oZrEuBi+cv4Uvnx+AdlWxzvBEoclDo53+nhlxxH+sPYA71UeIz5WuGpeHl+7cKqN95qoVVnbwk9f3sWftxwkPlZYWjyeL549iXOmZBET5VcPWuKI0sRR29zBe5XHeHHrQV7beYT2rgD5Y0bxxbMn8fmSifbXlTGOiiPN/G7tAZ7ZUE1Tu48xyfGcNy2bT07LpmhCGpMzR5OeHF33hFjiiPDE4Q8ox1o6qKhtoeJICzsPNbN+bx27j7QAkJ2SyGWzx3PF3FzOKsi0+zCMOYm2Tj8vbT/Em7tqeafiKIebOk6sy0iOJ3N0AskJsSQnxJEYF0NcjBAbE0N8rBAbI8THxjj/CnExMYxOjCMnNZGc1ETGOv/mZYwiKX7kPxa3v8Th6iRDIrIM+BkQCzykqj/qtT4R+A1wJnAM+IKq7nPW3Q7cAPiBb6rq6oHsM1z4/AEa27qoP95Fw/HOE/82HO+i/ngnTe1ddPoCdPmVTl+ATn+ALn/AaQvQ6Vc6uvwca+3kWEsHPScMTU2K48zJY/jsmfmcPSWTufkZliyMGYBRCbEsn5/H8vl5qCqVR1upONLC/mOt7D92nMa2Lo53+mnt8NHS4cMfULr8ij8QwOdXfAHF5w8E/w0oLe0+Ov2Bj32d7JRE8seMIn/MKCZmJjvLwX/DIbG4ljhEJBa4H7gUqAbWi8gqVd3eo9sNQL2qThORFcA9wBdEpAhYARQDE4BXRKT7MWCn2uewU1X8zjeCP6DON0gg+E0TUNq7/DS3+2hp99HS0RVc7vDR2NZFQ6/E0P1vU7vvpF8vLkZIGxVPYlwM8bExJHT/GysnlpMTYkhMTWTBpAxyUoJ/yUzJTqFwXApjUxNtDiljhkhEOCMnhTNyUk57H6pKY1sXR5o7qG3u4HBTOx80tFFdH3yV1TSyetshuvwfHfnJSU08kUwmjhlFbsYo0kfFk5oYR2pSHKlJ8SQnxBIXGzzLiY+J+XA5Vlz/+XfzjGMRUKGqlQAi8gSwHOj5S3458H1n+WngFxL8xMuBJ1S1A9grIhXO/hjAPofN/LteoqXdh28I8/+nJsUxJjmBjOR4MpITKMgefeJ9z/YxPd6nJMbZL35jIoCIkJGcQEZywknvi/IHlCPN7U4yOU51nZNYGo6zpbqBv5YdHPTvoNgYIVaEnNRE3rnt4uH4KB/hZuLIA6p6vK8Gzj5ZH1X1iUgjkOW0r+m1bZ6zfKp9AiAiNwI3Om9bRKT8ND7DcMoGjnocw0hkx+Xj7Jh8nB2TvvV7XHYDcvtp73vyyVZE7IMUVPVB4EGv4+gmIqUnKzRFMzsuH2fH5OPsmPTNq+Pi5hMAa4CJPd7nO2199hGROCCdYJH8ZNsOZJ/GGGNc5GbiWA8UisgUEUkgWOxe1avPKuB6Z/lq4DUNXh+8ClghIokiMgUoBNYNcJ/GGGNc5NpQlVOzuBlYTfDS2UdUdZuI3AWUquoq4GHgcaf4XUcwEeD0W0mw6O0DblJVP0Bf+3TrMwyzETNsNsLYcfk4OyYfZ8ekb54cl6i4AdAYY8zwcXOoyhhjTASyxGGMMWZQLHEMgYg8IiJHRGRrj7bPicg2EQmISEmv/reLSIWIlIvI0h7ty5y2ChG5LZSfYbid5Jj8WER2isgWEXlORDJ6rIv4YwInPS7/6hyTTSLykohMcNpFRP7T+exbRGRhj22uF5Hdzuv6vr5WuOjrmPRYd6uIqIhkO++j9piIyPdFpMb5PtkkIpf3WOfNz4+q2us0X8AFwEJga4+2WcAM4A2gpEd7EbAZSASmAHsIFvhjneWpQILTp8jrzzbMx2QJEOcs3wPcE03HpJ/jktZj+ZvAr5zly4G/AgKcA6x12jOBSuffMc7yGK8/23AeE6d9IsELYPYD2dF+TAjOrvFPffT17OfHzjiGQFXfIng1WM+2Hara113qJ6ZRUdW9QPc0KiemZlHVTqB7GpWwdJJj8pKqdk/OtYbg/TcQJccETnpcmnq8HQ10X6myHPiNBq0BMkQkF1gKvKyqdapaD7wMLHM/enf0dUwc9wLf5cPjAXZM+uLZz48ljtDpawqWvH7aI9VXCP7lCHZMEJG7RaQK+BJwp9MctcdFRJYDNaq6udeqqD0mjpudIbpHRGSM0+bZMbHEYUJGRO4geF/O77yOZaRQ1TtUdSLBY3Kz1/F4SUSSge/xYQI1Qb8EzgDmAweBn3gaDZY4Qimqp1ERkS8DVwJfUmeAlig/Jr38Dvissxytx+UMgmP1m0VkH8HPt1FExhO9xwRVPayqflUNAL/mw5nCPTsmljhCJ2qnUZHgw7e+C1ylqsd7rIraYwIgIoU93i4HdjrLq4DrnCuJzgEaVfUgwYLxEhEZ4wxXLHHaIoKqlqnqWFUtUNUCgkMsC1X1EFF6TACcWk63TwPdV1x59/Pj9VUE4fwC/kDw1LGL4Df5Dc5/bDXQARwGVvfofwfBqx3Kgct6tF8O7HLW3eH153LhmFQQHHPd5Lx+FU3HpJ/j8gzBXwJbgD8BeU5fIfjAsj1AGR+9Ou8rzvGsAP7e68813Mek1/p9fHhVVdQeE+Bx5zNvIZgAcnv09+Tnx6YcMcYYMyg2VGWMMWZQLHEYY4wZFEscxhhjBsUShzHGmEGxxGGMMWZQLHEYM8xExO/MYrpZRDaKyHmn6J8hIl8PVXzGDJVdjmvMMBORFlVNcZaXAt9T1U/1078A+LOqzg5RiMYMiZ1xGOOuNKC++42I/LOIrHcmrPuB0/wj4AznLOXHIpIiIq86ZytlzsR/xowYcV4HYEwEGiUim4AkIBe4GEBElhCcFmIRwTuhV4nIBcBtwGxVne/0iwM+rapNzoOM1ojIKrXhATNCWOIwZvi19UgC5wK/EZHZBOdRWgK87/RLIZhIDvTaXoAfOkklQHBK7HHAIfdDN+bULHEY4yJVfc85a8ghmBD+TVX/q2cfp8bR05ec/meqapczU2xSCMI1ZkCsxmGMi0RkJsFHeR4jOGvrV0Sku3CeJyJjgWYgtcdm6cARJ2lcBEwOcdjG9MvOOIwZft01DgieZVyvqn7gJRGZBbwnIgAtwLWqukdE3hGRrQSfjngP8CcRKQNK+XC6dWNGBLsc1xhjzKDYUJUxxphBscRhjDFmUCxxGGOMGRRLHMYYYwbFEocxxphBscRhjDFmUCxxGGOMGZT/D6szkoKy6XTwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Beta distribution showing a good approximate normal distribution which made us beleieve that \n",
    "# the nan values can be filled by median without any problem\n",
    "sns.kdeplot(train['Beta'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3    383\n",
      "1    145\n",
      "2     74\n",
      "Name: Gamma, dtype: int64\n",
      "3    247\n",
      "1    116\n",
      "2     47\n",
      "Name: Gamma, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "## Gamma feature visualisation\n",
    "print(train['Gamma'].value_counts())\n",
    "print(test['Gamma'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Gamma', ylabel='count'>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAUbklEQVR4nO3df4xd5X3n8fcXGDJtgfLDg9fxmB0neAk4LgYclpYooSAWx1Ix6VIWxM/YrKPEYWHFZkOJlF/aSFmlCQoJYeUsFFyBKTWhUERoWerUmw0JeByKf4FwweAZOfZgCMFFBnvy3T/umcMNjM0de849M573S7q65zznOed+rSv74+ecc58TmYkkSQAH1V2AJGnsMBQkSSVDQZJUMhQkSSVDQZJUOqTuAvbHpEmTsqenp+4yJGlc6e3tfTkzu4bbNq5Doaenh1WrVtVdhiSNKxHx4p62efpIklQyFCRJJUNBklQa19cUJKkuu3btoq+vj507d9Zdyh51dnbS3d1NR0dHy/sYCpK0D/r6+jj88MPp6ekhIuou510yk+3bt9PX18f06dNb3s/TR5K0D3bu3MkxxxwzJgMBICI45phjRjySMRQkaR+N1UAYsi/1GQqSpJKhIEkVOuyww/a6fdOmTXz4wx8e0TGvuuoqli9fvj9l7ZEXmiVV7rTPL627hBHp/eYVdZdQG0cKktQGO3bs4JxzzuHUU09l1qxZPPDAA+W23bt3c+mll3LiiSdy4YUX8sYbbwDQ29vLxz/+cU477TTOO+88tmzZUnmdhoIktUFnZyf3338/q1evZsWKFVx//fUMPQ752Wef5bOf/SwbNmzgiCOO4Pvf/z67du3immuuYfny5fT29rJgwQK++MUvVl6np48kqQ0ykxtvvJGVK1dy0EEH0d/fz9atWwGYNm0aZ555JgCXXXYZN998M3PnzmXt2rWce+65AAwODjJlypTK66wsFCKiE1gJvK/4nOWZ+eWImA7cAxwD9AKXZ+ZbEfE+YClwGrAd+E+Zuamq+iSpne666y4GBgbo7e2lo6ODnp6e8jcE77x1NCLITGbOnMnjjz/e1jqrPH30JnB2Zp4MzAbmRsQZwP8EbsrM44FXgYVF/4XAq0X7TUU/STogvPbaaxx77LF0dHSwYsUKXnzx7dmrX3rppfIf/7vvvpuPfvSjnHDCCQwMDJTtu3btYt26dZXXWVkoZMOOYrWjeCVwNjB0L9WdwAXF8vxinWL7OTHWfxkiSS269NJLWbVqFbNmzWLp0qV86EMfKredcMIJ3HLLLZx44om8+uqrfOYzn+HQQw9l+fLlfOELX+Dkk09m9uzZ/PSnP628zkqvKUTEwTROER0P3AL8C/CrzNxddOkDphbLU4HNAJm5OyJeo3GK6eV3HHMRsAjguOOOq7J8SdpvO3Y0/m88adKkPZ4KeuaZZ4Ztnz17NitXrnxX+x133DFq9b1TpXcfZeZgZs4GuoHTgQ/tfY+WjrkkM+dk5pyurmGfJidJ2kdtuSU1M38FrAD+EDgyIoZGKN1Af7HcD0wDKLb/Po0LzpKkNqksFCKiKyKOLJZ/BzgX2EAjHC4sul0JDP2C48FinWL7P+bQTbySpLao8prCFODO4rrCQcC9mflQRKwH7omI/wH8Arit6H8b8FcRsRF4Bbi4wtokScOoLBQy82nglGHan6dxfeGd7TuBP6uqHknSe3OaC0lSyWkuJGkUjPZMsK3M1PrII49w7bXXMjg4yNVXX80NN9yw35/rSEGSxqHBwUEWL17Mj370I9avX8+yZctYv379fh/XUJCkceiJJ57g+OOP5wMf+ACHHnooF1988W9Nx72vDAVJGof6+/uZNm1aud7d3U1/f/9e9miNoSBJKhkKkjQOTZ06lc2bN5frfX19TJ06dS97tMZQkKRx6CMf+QjPPfccL7zwAm+99Rb33HMP559//n4f11tSJWkUtHIL6Wg65JBD+N73vsd5553H4OAgCxYsYObMmft/3FGoTZJUg3nz5jFv3rxRPaanjyRJJUNBklQyFCRJJUNBklQyFCRJJUNBklTyllRJGgUvfW3WqB7vuC+tec8+CxYs4KGHHuLYY49l7dq1o/K5jhQkaZy66qqreOSRR0b1mIaCJI1TH/vYxzj66KNH9ZiGgiSpZChIkkqGgiSpZChIkkrekipJo6CVW0hH2yWXXMKPf/xjXn75Zbq7u/nqV7/KwoUL9+uYlYVCREwDlgKTgQSWZOZ3IuIrwH8GBoquN2bmw8U+fw4sBAaB/5KZf19VfZI03i1btmzUj1nlSGE3cH1mro6Iw4HeiHi02HZTZv5Fc+eIOAm4GJgJvB/4PxHx7zJzsMIaJUlNKrumkJlbMnN1sfw6sAHY2wNE5wP3ZOabmfkCsBE4var6JEnv1pYLzRHRA5wC/Lxo+lxEPB0Rt0fEUUXbVGBz0259DBMiEbEoIlZFxKqBgYF3bpaktsnMukvYq32pr/JQiIjDgPuA6zLz18CtwAeB2cAW4FsjOV5mLsnMOZk5p6ura7TLlaSWdHZ2sn379jEbDJnJ9u3b6ezsHNF+ld59FBEdNALhrsz8IUBmbm3a/gPgoWK1H5jWtHt30SZJY053dzd9fX2M5TMWnZ2ddHd3j2ifKu8+CuA2YENmfrupfUpmbilWPwkMTe33IHB3RHybxoXmGcATVdUnSfujo6OD6dOn113GqKtypHAmcDmwJiKeKtpuBC6JiNk0blPdBHwaIDPXRcS9wHoady4t9s4jSWqvykIhM38CxDCbHt7LPl8Hvl5VTZKkvXOaC0lSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUMBUlSyVCQJJUqC4WImBYRKyJifUSsi4hri/ajI+LRiHiueD+qaI+IuDkiNkbE0xFxalW1SZKGV+VIYTdwfWaeBJwBLI6Ik4AbgMcycwbwWLEO8AlgRvFaBNxaYW2SpGFUFgqZuSUzVxfLrwMbgKnAfODOotudwAXF8nxgaTb8DDgyIqZUVZ8k6d3ack0hInqAU4CfA5Mzc0ux6ZfA5GJ5KrC5abe+ok2S1CaVh0JEHAbcB1yXmb9u3paZCeQIj7coIlZFxKqBgYFRrFSSVGkoREQHjUC4KzN/WDRvHTotVLxvK9r7gWlNu3cXbb8lM5dk5pzMnNPV1VVd8ZI0AVV591EAtwEbMvPbTZseBK4slq8EHmhqv6K4C+kM4LWm00ySpDY4pMJjnwlcDqyJiKeKthuBbwD3RsRC4EXgomLbw8A8YCPwBvCpCmuTJA2jslDIzJ8AsYfN5wzTP4HFVdUjSXpv/qJZklQyFCRJJUNBklQyFCRJJUNBklQyFCRJJUNBklQyFCRJpZZCISIea6VNkjS+7fUXzRHRCfwuMKl4QtrQL5SPwGmtJemA817TXHwauA54P9DL26Hwa+B71ZUlSarDXkMhM78DfCcirsnM77apJklSTVqaEC8zvxsRfwT0NO+TmUsrqkuSVIOWQiEi/gr4IPAUMFg0J2AoSNIBpNWps+cAJxXTW0uSDlCt/k5hLfBvqixEklS/VkcKk4D1EfEE8OZQY2aeX0lVkqRatBoKX6myCEnS2NDq3Uf/VHUhkqT6tXr30es07jYCOBToAP41M4+oqjBJUvu1OlI4fGg5IgKYD5xRVVGSpHqMeJbUbPhb4LzRL0eSVKdWTx/9adPqQTR+t7CzkookSbVp9e6jP2la3g1sonEKSZJ0AGn1msKnqi5EklS/Vh+y0x0R90fEtuJ1X0R0v8c+txd91za1fSUi+iPiqeI1r2nbn0fExoh4NiK8XiFJNWj1QvNfAg/SeK7C+4G/K9r25g5g7jDtN2Xm7OL1MEBEnARcDMws9vl+RBzcYm2SpFHSaih0ZeZfZubu4nUH0LW3HTJzJfBKi8efD9yTmW9m5gvARuD0FveVJI2SVkNhe0RcFhEHF6/LgO37+Jmfi4ini9NLRxVtU4HNTX362MPjPiNiUUSsiohVAwMD+1iCJGk4rYbCAuAi4JfAFuBC4Kp9+LxbaTyXYXZxnG+N9ACZuSQz52TmnK6uvQ5WJEkj1OotqV8DrszMVwEi4mjgL2iERcsyc+vQckT8AHioWO0HpjV17S7aJElt1OpI4Q+GAgEgM18BThnph0XElKbVT9J4TgM0LmJfHBHvi4jpwAzgiZEeX5K0f1odKRwUEUe9Y6Sw130jYhlwFjApIvqALwNnRcRsGpPrbQI+DZCZ6yLiXmA9jR/HLc7MwWEOK0mqUKuh8C3g8Yj4m2L9z4Cv722HzLxkmObb9tL/6+91TElStVr9RfPSiFgFnF00/Wlmrq+uLElSHVodKVCEgEEgSQewEU+dLUk6cBkKkqSSoSBJKhkKkqSSoSBJKhkKkqSSoSBJKhkKkqSSoSBJKhkKkqSSoSBJKrU899F4d9rnl9Zdwoj1fvOKukuQNME4UpAklQwFSVLJUJAklQwFSVLJUJAklQwFSVLJUJAklQwFSVLJUJAklQwFSVKpslCIiNsjYltErG1qOzoiHo2I54r3o4r2iIibI2JjRDwdEadWVZckac+qHCncAcx9R9sNwGOZOQN4rFgH+AQwo3gtAm6tsC5J0h5UFgqZuRJ45R3N84E7i+U7gQua2pdmw8+AIyNiSlW1SZKG1+5rCpMzc0ux/EtgcrE8Fdjc1K+vaHuXiFgUEasiYtXAwEB1lUrSBFTbhebMTCD3Yb8lmTknM+d0dXVVUJkkTVztDoWtQ6eFivdtRXs/MK2pX3fRJklqo3aHwoPAlcXylcADTe1XFHchnQG81nSaSZLUJpU9eS0ilgFnAZMiog/4MvAN4N6IWAi8CFxUdH8YmAdsBN4APlVVXZKkPassFDLzkj1sOmeYvgksrqoWSVJr/EWzJKlkKEiSSoaCJKlkKEiSSoaCJKlkKEiSSoaCJKlkKEiSSoaCJKlkKEiSSoaCJKlkKEiSSoaCJKlU2SypkjRevfS1WXWXMGLHfWnNqBzHkYIkqWQoSJJKhoIkqWQoSJJKhoIkqWQoSJJKhoIkqWQoSJJKhoIkqWQoSJJKtUxzERGbgNeBQWB3Zs6JiKOBvwZ6gE3ARZn5ah31aew57fNL6y5hxHq/eUXdJUgjVudI4Y8zc3ZmzinWbwAey8wZwGPFuiSpjcbS6aP5wJ3F8p3ABfWVIkkTU12hkMA/RERvRCwq2iZn5pZi+ZfA5HpKk6SJq66psz+amf0RcSzwaEQ807wxMzMicrgdixBZBHDcccdVX6kkTSC1jBQys7943wbcD5wObI2IKQDF+7Y97LskM+dk5pyurq52lSxJE0LbQyEifi8iDh9aBv4DsBZ4ELiy6HYl8EC7a5Okia6O00eTgfsjYujz787MRyLiSeDeiFgIvAhcVENtkjShtT0UMvN54ORh2rcD57S7HknS28bSLamSpJoZCpKkkqEgSSoZCpKkkqEgSSoZCpKkkqEgSSoZCpKkkqEgSSoZCpKkkqEgSSoZCpKkUl0P2VELXvrarLpLGLHjvrSm7hIk7QdHCpKkkqEgSSp5+kiqyHg7/eepP4EjBUlSE0NBklQyFCRJJUNBklQyFCRJJUNBklQyFCRJJUNBklQyFCRJpTEXChExNyKejYiNEXFD3fVI0kQypkIhIg4GbgE+AZwEXBIRJ9VblSRNHGMqFIDTgY2Z+XxmvgXcA8yvuSZJmjAiM+uuoRQRFwJzM/PqYv1y4N9n5uea+iwCFhWrJwDPtr3Q9pkEvFx3Edpnfn/j14H+3f3bzOwabsO4myU1M5cAS+quox0iYlVmzqm7Du0bv7/xayJ/d2Pt9FE/MK1pvbtokyS1wVgLhSeBGRExPSIOBS4GHqy5JkmaMMbU6aPM3B0RnwP+HjgYuD0z19VcVp0mxGmyA5jf3/g1Yb+7MXWhWZJUr7F2+kiSVCNDQZJUMhTGoIi4PSK2RcTaumvRyETEtIhYERHrI2JdRFxbd01qXUR0RsQTEfHPxff31bprajevKYxBEfExYAewNDM/XHc9al1ETAGmZObqiDgc6AUuyMz1NZemFkREAL+XmTsiogP4CXBtZv6s5tLaxpHCGJSZK4FX6q5DI5eZWzJzdbH8OrABmFpvVWpVNuwoVjuK14T6n7OhIFUkInqAU4Cf11yKRiAiDo6Ip4BtwKOZOaG+P0NBqkBEHAbcB1yXmb+uux61LjMHM3M2jRkVTo+ICXUK11CQRllxLvo+4K7M/GHd9WjfZOavgBXA3JpLaStDQRpFxYXK24ANmfntuuvRyEREV0QcWSz/DnAu8EytRbWZoTAGRcQy4HHghIjoi4iFddeklp0JXA6cHRFPFa95dRellk0BVkTE0zTmYns0Mx+quaa28pZUSVLJkYIkqWQoSJJKhoIkqWQoSJJKhoIkqWQoSE0iYnJE3B0Rz0dEb0Q8HhGfrLsuqV0MBalQ/PDsb4GVmfmBzDyNxnPCu2stTGojQ0F629nAW5n5v4YaMvPFzPxuRPRExP+NiNXF648AIuKsiPiniHigGF18IyIuLebkXxMRHyz63RERt0bEz4p+ZxXPzdgQEXcMfV7RZ9VEnctf9Tuk7gKkMWQmsHoP27YB52bmzoiYASwD5hTbTgZOpDHd+fPA/87M04sH7FwDXFf0Owr4Q+B84EEav36+GngyImZn5lPAFzPzlYg4GHgsIv4gM58e5T+ntEeOFKQ9iIhbiidwPUljXv0fRMQa4G+Ak5q6Plk8R+FN4F+Afyja1wA9Tf3+LhtTCKwBtmbmmsz8DbCuqd9FEbEa+AWNkGr+HKlyjhSkt60D/uPQSmYujohJwCrgvwJbaYwKDgJ2Nu33ZtPyb5rWf8Nv/x17c5g+Zb+ImA78N+AjmflqcVqpcz//TNKIOFKQ3vaPQGdEfKap7XeL998HthT/s78cOLiCzz8C+FfgtYiYDHyigs+Q9sqRglTIzIyIC4CbIuK/AwM0/pH+Ao1rDfdFxBXAI0X7aH/+P0fEL2hM1bwZ+H+j/RnSe3GWVElSydNHkqSSoSBJKhkKkqSSoSBJKhkKkqSSoSBJKhkKkqTS/wcgup8CnixV3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x='Gamma', data= train,hue='label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Eta', ylabel='count'>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXS0lEQVR4nO3df7BfdZ3f8ecLCEYFqkBgY25oUCi/pESILF2sP6AsmG7FdVwHxx8g2FiNih3Xijij4tSpjr+6rOgMWxTYIkhRCnWUSjEus1XBBCOEIDWVXzcbSQw/hKVgEt/943vu8QqX5Ca5556be5+Pme/c8/2cc77fVyY331fOz2+qCkmSAHbrO4AkaeqwFCRJLUtBktSyFCRJLUtBktTao+8AO2P//fevBQsW9B1DknYpK1as+HVVzRlr3i5dCgsWLGD58uV9x5CkXUqS+55tnruPJEktS0GS1LIUJEmtXfqYwlg2bdrE8PAwTz75ZN9Rtmr27NkMDQ0xa9asvqNIUmvalcLw8DB77703CxYsIEnfccZUVWzcuJHh4WEOPvjgvuNIUmva7T568skn2W+//aZsIQAkYb/99pvyWzOSZp5pVwrAlC6EEbtCRkkzz7QsBUnSjpmxpbDXXnttdf69997LS1/60u16zbPOOotrrrlmZ2JJUq+m3YFmabo77kOX7/C6Kz779glMoumosy2FJLOT3JrkZ0nuTHJBM35pknuSrGweC5vxJLkwyZoktyc5tqtsoz3++OOcfPLJHHvssRx99NFcd9117bzNmzfzlre8hSOOOII3vvGNPPHEEwCsWLGCV73qVRx33HGceuqprFu3bjKiSlLnutx99BRwUlUdAywETktyQjPvQ1W1sHmsbMZeCxzaPJYAX+kwW2v27Nlce+213HbbbSxbtowPfvCDjHxF6d1338173vMe7rrrLvbZZx++/OUvs2nTJt73vvdxzTXXsGLFCs4++2w++tGPTkZUSepcZ7uPavDJ+njzdFbz2NoXQp8OXN6s9+MkL0gyt6o6/W94VXH++edz8803s9tuu7F27VoefPBBAObPn8+JJ54IwFvf+lYuvPBCTjvtNFatWsUpp5wCwJYtW5g7d26XESVp0nR6TCHJ7sAK4BDgoqq6Jcm7gU8l+RhwE3BeVT0FzAMeGLX6cDO27mmvuYTBlgQHHXTQTme84oor2LBhAytWrGDWrFksWLCgvX7g6aeNJqGqOOqoo/jRj3600+8tSVNNp2cfVdWWqloIDAHHJ3kp8BHgcODlwL7Ah7fzNS+uqkVVtWjOnDFvB75dHn30UQ444ABmzZrFsmXLuO++399R9v77728//L/+9a/zile8gsMOO4wNGza045s2beLOO+/c6RySNBVMyimpVfUIsAw4rarW1cBTwNeA45vF1gLzR6021Ix16i1veQvLly/n6KOP5vLLL+fwww9v5x122GFcdNFFHHHEETz88MO8+93vZs899+Saa67hwx/+MMcccwwLFy7khz/8YdcxJWlSdLb7KMkcYFNVPZLkucApwGdGjhNksG/m9cCqZpXrgfcmuQr4Y+DRLo8nPP744HDH/vvv/6y7gn7+85+POb5w4UJuvvnmZ4xfeumlE5ZPkvrQ5TGFucBlzXGF3YCrq+rbSb7fFEaAlcC/a5b/DrAYWAM8Abyjw2ySpDF0efbR7cDLxhg/6VmWL2BpV3kkSds2Y29zIUl6JktBktSyFCRJLUtBktSa9ndJ3Zk7So5lvHeZvOGGGzj33HPZsmUL73znOznvvPMmNIckdcEthQ5s2bKFpUuX8t3vfpfVq1dz5ZVXsnr16r5jSdI2WQoduPXWWznkkEN48YtfzJ577skZZ5zxB7fklqSpylLowNq1a5k///d37BgaGmLt2s7v2CFJO81SkCS1LIUOzJs3jwce+P1dwIeHh5k3b16PiSRpfCyFDrz85S/nF7/4Bffccw+//e1vueqqq3jd617XdyxJ2qZpf0pqH19Uvscee/ClL32JU089lS1btnD22Wdz1FFHTXoOSdpe074U+rJ48WIWL17cdwxJ2i7uPpIktSwFSVLLUpAktSwFSVLLUpAktSwFSVJr2p+Sev8nj57Q1zvoY3dsc5mzzz6bb3/72xxwwAGsWrVqQt9fkrrU2ZZCktlJbk3ysyR3JrmgGT84yS1J1iT5RpI9m/HnNM/XNPMXdJWta2eddRY33HBD3zEkabt1ufvoKeCkqjoGWAicluQE4DPAF6vqEOBh4Jxm+XOAh5vxLzbL7ZJe+cpXsu+++/YdQ5K2W2elUAOPN09nNY8CTgKuacYvA17fTJ/ePKeZf3KSdJVPkvRMnR5oTrJ7kpXAeuBG4P8Cj1TV5maRYWDk9qHzgAcAmvmPAvuN8ZpLkixPsnzDhg1dxpekGafTUqiqLVW1EBgCjgcOn4DXvLiqFlXVojlz5uzsy0mSRpmUU1Kr6hFgGfAvgBckGTnraQgY+UqytcB8gGb+PwE2TkY+SdJAZ6ekJpkDbKqqR5I8FziFwcHjZcAbgauAM4GRLy++vnn+o2b+96uqdjbHeE4hnWhvfvOb+cEPfsCvf/1rhoaGuOCCCzjnnHO2vaIk9azL6xTmApcl2Z3BFsnVVfXtJKuBq5L8R+CnwCXN8pcAf5tkDfAQcEaH2Tp15ZVX9h1BknZIZ6VQVbcDLxtj/JcMji88ffxJ4C+6yiNJ2jZvcyFJak3LUpiAQxGd2xUySpp5pl0pzJ49m40bN07pD92qYuPGjcyePbvvKJL0B6bdDfGGhoYYHh5mql/YNnv2bIaGhvqOIUl/YNqVwqxZszj44IP7jiFJu6Rpt/tIkrTjLAVJUstSkCS1LAVJUstSkCS1LAVJUstSkCS1LAVJUstSkCS1LAVJUstSkCS1LAVJUstSkCS1LAVJUstSkCS1OiuFJPOTLEuyOsmdSc5txj+RZG2Slc1j8ah1PpJkTZK7k5zaVTZJ0ti6/JKdzcAHq+q2JHsDK5Lc2Mz7YlV9bvTCSY4EzgCOAl4E/K8k/6yqtnSYUZI0SmdbClW1rqpua6YfA+4C5m1lldOBq6rqqaq6B1gDHN9VPknSM03KMYUkC4CXAbc0Q+9NcnuSryZ5YTM2D3hg1GrDjFEiSZYkWZ5k+VT/HmZJ2tV0XgpJ9gK+CXygqn4DfAV4CbAQWAd8fnter6ourqpFVbVozpw5Ex1Xkma0TkshySwGhXBFVX0LoKoerKotVfU74G/4/S6itcD8UasPNWOSpEnS5dlHAS4B7qqqL4wanztqsT8HVjXT1wNnJHlOkoOBQ4Fbu8onSXqmLs8+OhF4G3BHkpXN2PnAm5MsBAq4F3gXQFXdmeRqYDWDM5eWeuaRJE2uzkqhqv4eyBizvrOVdT4FfKqrTJKkrfOKZklSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSy1KQJLUsBUlSa1ylkOSm8YxJknZtWy2FJLOT7Avsn+SFSfZtHguAedtYd36SZUlWJ7kzybnN+L5Jbkzyi+bnC5vxJLkwyZoktyc5doL+jJKkcdrWlsK7gBXA4c3Pkcd1wJe2se5m4INVdSRwArA0yZHAecBNVXUocFPzHOC1wKHNYwnwle3+00iSdsoeW5tZVX8F/FWS91XVX2/PC1fVOmBdM/1YkrsYbF2cDry6Wewy4AfAh5vxy6uqgB8neUGSuc3rSJImwVZLYURV/XWSPwEWjF6nqi4fz/rN7qaXAbcAB476oP8VcGAzPQ94YNRqw83YH5RCkiUMtiQ46KCDxvP2kqRxGlcpJPlb4CXASmBLM1zANkshyV7AN4EPVNVvkrTzqqqS1PYErqqLgYsBFi1atF3rSpK2blylACwCjmx27YxbklkMCuGKqvpWM/zgyG6hJHOB9c34WmD+qNWHmjFJ0iQZ73UKq4A/2p4XzmCT4BLgrqr6wqhZ1wNnNtNnMjhoPTL+9uYspBOARz2eIEmTa7xbCvsDq5PcCjw1MlhVr9vKOicCbwPuSLKyGTsf+DRwdZJzgPuANzXzvgMsBtYATwDvGGc2SdIEGW8pfGJ7X7iq/h7Is8w+eYzlC1i6ve8jSZo44z376O+6DiJJ6t94zz56jMHZRgB7ArOAf6yqfboKJkmafOPdUth7ZLo5gHw6g6uUJUnTyHbfJbUG/jtw6sTHkST1aby7j94w6uluDK5beLKTRJKk3oz37KN/M2p6M3Avg11IkqRpZLzHFLxmQJJmgPF+yc5QkmuTrG8e30wy1HU4SdLkGu+B5q8xuA3Fi5rH/2jGJEnTyHhLYU5Vfa2qNjePS4E5HeaSJPVgvKWwMclbk+zePN4KbOwymCRp8o23FM5mcOO6XzH40ps3Amd1lEmS1JPxnpL6SeDMqnoYIMm+wOcYlIUkaZoY75bCPx8pBICqeojB12tKkqaR8ZbCbkleOPKk2VIY71aGJGkXMd4P9s8DP0ry35rnfwF8qptIkqS+jPeK5suTLAdOaobeUFWru4slSerDuHcBNSVgEUjSNLbdt86WJE1floIkqdVZKST5anPzvFWjxj6RZG2Slc1j8ah5H0myJsndSfwCH0nqQZdbCpcCp40x/sWqWtg8vgOQ5EjgDOCoZp0vJ9m9w2ySpDF0VgpVdTPw0DgXPx24qqqeqqp7gDXA8V1lkySNrY9jCu9Ncnuze2nkgrh5wAOjlhluxp4hyZIky5Ms37BhQ9dZJWlGmexS+ArwEmAhgxvrfX57X6CqLq6qRVW1aM4c794tSRNpUkuhqh6sqi1V9Tvgb/j9LqK1wPxRiw41Y5KkSTSppZBk7qinfw6MnJl0PXBGkuckORg4FLh1MrNJkjq8qV2SK4FXA/snGQY+Drw6yUKggHuBdwFU1Z1JrmZwxfRmYGlVbekqmyRpbJ2VQlW9eYzhS7ay/KfwJnuS1CuvaJYktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVLLUpAktSwFSVKrs1JI8tUk65OsGjW2b5Ibk/yi+fnCZjxJLkyyJsntSY7tKpck6dl1uaVwKXDa08bOA26qqkOBm5rnAK8FDm0eS4CvdJhLkvQsOiuFqroZeOhpw6cDlzXTlwGvHzV+eQ38GHhBkrldZZMkjW2yjykcWFXrmulfAQc20/OAB0YtN9yMPUOSJUmWJ1m+YcOG7pJK0gzU24HmqiqgdmC9i6tqUVUtmjNnTgfJJGnmmuxSeHBkt1Dzc30zvhaYP2q5oWZMkjSJJrsUrgfObKbPBK4bNf725iykE4BHR+1mkiRNkj26euEkVwKvBvZPMgx8HPg0cHWSc4D7gDc1i38HWAysAZ4A3tFVLknSs+usFKrqzc8y6+Qxli1gaVdZJEnj4xXNkqSWpSBJanW2+0iS+nTchy7f4XVXfPbtE5hk1+KWgiSpZSlIklqWgiSpZSlIklqWgiSp5dlH0gxy/yeP3uF1D/rYHROYRFOVWwqSpJalIElqWQqSpJalIElqWQqSpJalIElqWQqSpJalIElqWQqSpJalIElqWQqSpFYv9z5Kci/wGLAF2FxVi5LsC3wDWADcC7ypqh7uI58kzVR9bim8pqoWVtWi5vl5wE1VdShwU/NckjSJptLuo9OBy5rpy4DX9xdFkmamvkqhgO8lWZFkSTN2YFWta6Z/BRw41opJliRZnmT5hg0bJiOrJM0YfX2fwiuqam2SA4Abk/x89MyqqiQ11opVdTFwMcCiRYvGXEaStGN62VKoqrXNz/XAtcDxwINJ5gI0P9f3kU2SZrJJL4Ukz0+y98g08KfAKuB64MxmsTOB6yY7myTNdH3sPjoQuDbJyPt/vapuSPIT4Ook5wD3AW/qIZskzWiTXgpV9UvgmDHGNwInT3YeSdLvTaVTUiVJPbMUJEktS0GS1LIUJEktS0GS1OrriuZOHPehy3d43RWfffsEJpGkXZNbCpKklqUgSWpZCpKklqUgSWpZCpKklqUgSWpNq1NSJWki3P/Jo3d43YM+dscEJpl8lkJjJv8SSNIIdx9JklqWgiSpZSlIklqWgiSpZSlIklqefTQDeTdZSc9mym0pJDktyd1J1iQ5r+88kjSTTKkthSS7AxcBpwDDwE+SXF9Vq/tNJmlr3PqcPqZUKQDHA2uq6pcASa4CTgd22VKYbv9YpuJFfl1mmm5/f1PRVPyd6tKO/k5N1u9TqmpS3mg8krwROK2q3tk8fxvwx1X13lHLLAGWNE8PA+6eoLffH/j1BL3WRDHT+EzFTDA1c5lpfKZ7pn9aVXPGmjHVthS2qaouBi6e6NdNsryqFk306+4MM43PVMwEUzOXmcZnJmeaagea1wLzRz0fasYkSZNgqpXCT4BDkxycZE/gDOD6njNJ0owxpXYfVdXmJO8F/iewO/DVqrpzkt5+wndJTQAzjc9UzARTM5eZxmfGZppSB5olSf2aaruPJEk9shQkSa0ZXwpJDk/yoyRPJfnLvvMAJPlqkvVJVvWdZUSS+UmWJVmd5M4k506BTLOT3JrkZ02mC/rONCLJ7kl+muTbfWcBSHJvkjuSrEyyvO88AEn+ffP3tirJlUlm950JIMm5TaY7k3ygpwzP+AxIsm+SG5P8ovn5wi7ee8aXAvAQ8H7gc30HGeVS4LS+QzzNZuCDVXUkcAKwNMmRPWd6Cjipqo4BFgKnJTmh30itc4G7+g7xNK+pqoVT4fz7JPMY/LtbVFUvZXBiyRn9poIkLwX+LYO7KxwD/FmSQ3qIcinP/Aw4D7ipqg4FbmqeT7gZXwpVtb6qfgJs6jvLiKq6mUFZTRlVta6qbmumH2PwgTev50xVVY83T2c1j97PnEgyBPxr4L/0nWWK2wN4bpI9gOcB/9BzHoAjgFuq6omq2gz8HfCGyQ7xLJ8BpwOXNdOXAa/v4r1nfClo+yVZALwMuKXnKCO7aVYC64Ebq6r3TMB/Bv4D8Luec4xWwPeSrGhuFdNvmKq1DLbO7wfWAY9W1ff6TQXAKuBfJtkvyfOAxfzhBbV9OrCq1jXTvwIO7OJNLAVtlyR7Ad8EPlBVv+k7T1VtqaqFDK5+P77Z/O9Nkj8D1lfVij5zjOEVVXUs8FoGu/5e2WeYZn/46cDBwIuA5yd5a5+ZAKrqLuAzwPeAG4CVwJY+M42lBtcSdLJVPCNLIcnS5oDbyiQv6jvPriLJLAaFcEVVfavvPKNV1SPAMvo/FnMi8Lok9wJXAScl+a/9Rmr/Z05VrQeuZbDPvE//CrinqjZU1SbgW8Cf9JwJgKq6pKqOq6pXAg8D/6fvTI0Hk8wFaH6u7+JNZmQpVNVFzQG3hVU1FfZjTnlJAlwC3FVVX+g7D0CSOUle0Ew/l8H3cPy8z0xV9ZGqGqqqBQwOnH6/qnr9H3CS5yfZe2Qa+FMGu0n6dD9wQpLnNb9bJzNFDswnOaD5eRCD4wlf7zdR63rgzGb6TOC6Lt5kSt3mog9J/ghYDuwD/K45Be3IPneNJLkSeDWwf5Jh4ONVdUlfeRonAm8D7mj24QOcX1Xf6S8Sc4HLmi9n2g24uqqmxCmgU8yBwLWDz172AL5eVTf0GaiqbklyDXAbgzPbfsrUubXEN5Psx+Dkk6XNVuikGuszAPg0cHWSc4D7gDd18t7e5kKSNGJG7j6SJI3NUpAktSwFSVLLUpAktSwFSVLLUpB2UJItoy6CXJnkvGb8A80tEqRdjqekSjsoyeNVtdcY4/cyuPvnryc/lbRz3FKQJlCS9zO4l8+yJMuasa8kWT7VvvNBGotbCtIOSrIFuGPU0H+qqm88fUshyb5V9VBz5fVNwPur6vbJTyxt24y/zYW0E/5fc4fWbXlTc7vqPRjcmuNIwFLQlGQpSB1KcjDwl8DLq+rhJJcCU+JrJ6WxeExBmniPAXs30/sA/wg8muRABt9nIE1ZbilIO+65o+4YC3BDVZ3H4G6fNyT5h6p6TZKfMril9wPA/+4hpzRuHmiWJLXcfSRJalkKkqSWpSBJalkKkqSWpSBJalkKkqSWpSBJav1/+XPceTHtI90AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Eta visualisation\n",
    "sns.countplot(x='Eta', data= train,hue='label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Zeta', ylabel='count'>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEJCAYAAAB7UTvrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAW+UlEQVR4nO3de7RedX3n8fcHCWYQLJcEJiak4WaURIklWC3KwrEKZjmkOC2FBQLGNqLg6MiyKq6l1tGWGbQuFWUmFoyZhYgDIrQLqQyLkXYENUEWhFsF5XIyIaQBudRBk/CdP56d7SGcwElynmefnLxfa+2VvX/78nw3Cedzfvvye1JVSJIEsEvXBUiSxg9DQZLUMhQkSS1DQZLUMhQkSS1DQZLU6lsoJDkgyQ1J7kxyR5IPNO37JLkuyc+aP/du2pPkS0nuTXJbkt/rV22SpJH1s6ewATinqg4DXgecleQw4KPA9VV1KHB9swzwNuDQZloMXNjH2iRJI9i1XweuqtXA6mb+ySR3AdOBhcAxzWbfAP438JGmfVn13qa7OcleSaY1xxnRlClTatasWf06BUmakFasWPEvVTV1pHV9C4XhkswCXgP8CNh/2A/6h4H9m/npwEPDdhtq2rYYCrNmzWL58uVjXq8kTWRJHtjSur7faE6yB3AF8MGqemL4uqZXsFXjbCRZnGR5kuVr164dw0olSX0NhSST6AXCJVX1naZ5TZJpzfppwCNN+yrggGG7z2janqWqllTV/KqaP3XqiL0fSdI26ufTRwEuAu6qqr8Ztupq4PRm/nTgqmHtpzVPIb0OePz57idIksZeP+8pHAW8E7g9ya1N27nAecC3k7wbeAA4sVl3DbAAuBf4FfCuPtYmSdtl/fr1DA0N8fTTT3ddyhZNnjyZGTNmMGnSpFHv08+nj/4JyBZWv3mE7Qs4q1/1SNJYGhoaYs8992TWrFn0LoyML1XFunXrGBoa4sADDxz1fr7RLEnb4Omnn2bfffcdl4EAkIR99913q3syhoIkbaPxGgibbEt9hoIk9dEee+zxvOvvv/9+5s6du1XHPOOMM7j88su3p6wtMhQkSa2BvNGssXXEh5f15bgrzj+tL8eVBE899RQLFy7kscceY/369XzmM59h4cKFAGzYsIFTTjmFW265hTlz5rBs2TJ23313VqxYwYc+9CGeeuoppkyZwtKlS5k2bVpf67SnIEkDMHnyZK688kpuueUWbrjhBs455xx6D13CPffcw/ve9z7uuusuXvrSl/LVr36V9evX8/73v5/LL7+cFStWsGjRIj7+8Y/3vU57CpI0AFXFueeey4033sguu+zCqlWrWLNmDQAHHHAARx11FACnnnoqX/rSlzjuuONYuXIlb3nLWwDYuHFj33sJYChI0kBccsklrF27lhUrVjBp0iRmzZrVPi66+VNCSagq5syZw0033TTQOr18JEkD8Pjjj7PffvsxadIkbrjhBh544LcDlT744IPtD/9vfvObvOENb2D27NmsXbu2bV+/fj133HFH3+s0FCRpAE455RSWL1/Oq171KpYtW8YrXvGKdt3s2bP5yle+witf+Uoee+wx3vve97Lbbrtx+eWX85GPfITDDz+cefPm8cMf/rDvdXr5SJL66KmnngJgypQpW7wUdPfdd4/YPm/ePG688cbntC9dunTM6tucPQVJUsuegjRA/XrHBHzPRGPDnoIkqWUoSJJahoIkqWUoSJJahoIk7aCuvfZaZs+ezSGHHMJ55503Jsfs29NHSS4G3g48UlVzm7bLgNnNJnsBv6yqeUlmAXcB9zTrbq6qM/tVmySNtbF+suyFnibbuHEjZ511Ftdddx0zZszgyCOP5Pjjj+ewww7brs/t5yOpS4ELgPa/VFX96ab5JJ8HHh+2/X1VNa+P9UjShPHjH/+YQw45hIMOOgiAk046iauuumq7Q6Fvl4+q6kbg0ZHWpTf604nApf36fEmayFatWsUBBxzQLs+YMYNVq1Zt93G7uqfwRmBNVf1sWNuBSX6a5AdJ3thRXZK0U+vqjeaTeXYvYTUws6rWJTkC+G6SOVX1xOY7JlkMLAaYOXPmQIqVpPFm+vTpPPTQQ+3y0NAQ06dP3+7jDrynkGRX4B3AZZvaqurXVbWumV8B3Ae8fKT9q2pJVc2vqvlTp04dRMmSNO4ceeSR/OxnP+MXv/gFv/nNb/jWt77F8ccfv93H7aKn8IfA3VU1tKkhyVTg0aramOQg4FDg5x3UJkk7hF133ZULLriAY489lo0bN7Jo0SLmzJmz/ccdg9pGlORS4BhgSpIh4JNVdRFwEs+9wXw08Okk64FngDOrasSb1JI0HnUxIOGCBQtYsGDBmB6zb6FQVSdvof2MEdquAK7oVy2SpNHxjWZJUstQkCS1DAVJUstQkCS1DAVJUstQkKQd1KJFi9hvv/2YO3fumB2zq2EuJGlCefDTrxrT4838xO0vuM0ZZ5zB2WefzWmnjd07EvYUJGkHdfTRR7PPPvuM6TENBUlSy1CQJLUMBUlSy1CQJLUMBUnaQZ188sm8/vWv55577mHGjBlcdNFF231MH0mVpDEwmkdIx9qll47919zbU5AktQwFSVLLUJAktQwFSdpGVdV1Cc9rW+rrWygkuTjJI0lWDmv7VJJVSW5tpgXD1n0syb1J7klybL/qkqSxMHnyZNatWzdug6GqWLduHZMnT96q/fr59NFS4AJg2WbtX6iqzw1vSHIYcBIwB3gZ8L+SvLyqNvaxPknaZjNmzGBoaIi1a9d2XcoWTZ48mRkzZmzVPn0Lhaq6McmsUW6+EPhWVf0a+EWSe4HXAjf1qz5J2h6TJk3iwAMP7LqMMdfFPYWzk9zWXF7au2mbDjw0bJuhpk2SNECDDoULgYOBecBq4PNbe4Aki5MsT7J8PHfbJGlHNNBQqKo1VbWxqp4BvkbvEhHAKuCAYZvOaNpGOsaSqppfVfOnTp3a34IlaScz0FBIMm3Y4gnApieTrgZOSvLiJAcChwI/HmRtkqQ+3mhOcilwDDAlyRDwSeCYJPOAAu4H3gNQVXck+TZwJ7ABOMsnjyRp8Pr59NHJIzRvcQi/qvos8Nl+1SNJemG+0SxJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqSWoSBJahkKkqRW30IhycVJHkmycljb+UnuTnJbkiuT7NW0z0ry/5Lc2kz/rV91SZK2rJ89haXAcZu1XQfMrapXA/8MfGzYuvuqal4zndnHuiRJW9C3UKiqG4FHN2v7flVtaBZvBmb06/MlSVuvy3sKi4DvDVs+MMlPk/wgyRu7KkqSdma7dvGhST4ObAAuaZpWAzOral2SI4DvJplTVU+MsO9iYDHAzJkzB1WyJO0UBt5TSHIG8HbglKoqgKr6dVWta+ZXAPcBLx9p/6paUlXzq2r+1KlTB1S1JO0cBhoKSY4D/gI4vqp+Nax9apIXNfMHAYcCPx9kbZKkPl4+SnIpcAwwJckQ8El6Txu9GLguCcDNzZNGRwOfTrIeeAY4s6oeHfHAkqS+6VsoVNXJIzRftIVtrwCu6FctkqTR8Y1mSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVJrVKGQ5PrRtEmSdmzPO8xFksnA7vTGL9obSLPqpcD0PtcmSRqwFxr76D3AB4GXASv4bSg8AVzQv7IkSV143lCoqi8CX0zy/qr68oBqkiR1ZFSjpFbVl5P8ATBr+D5VtaxPdUmSOjCqUEjyP4CDgVuBjU1zAYaCJE0go/0+hfnAYZu+PlOSdkZHfLg/vwevOP+0vhx3W4z2PYWVwL/tZyGSpO6NtqcwBbgzyY+BX29qrKrj+1KVJKkTow2FT23LwZNcDLwdeKSq5jZt+wCX0btpfT9wYlU9lt6XNn8RWAD8Cjijqm7Zls+VJG2bUV0+qqofjDSNYtelwHGbtX0UuL6qDgWub5YB3gYc2kyLgQtHU5skaeyMdpiLJ5M80UxPJ9mY5IkX2q+qbgQe3ax5IfCNZv4bwB8Na19WPTcDeyWZNqqzkCSNidG+p7DnpvnmMs9C4HXb+Jn7V9XqZv5hYP9mfjrw0LDthpq21UiSBmKrR0ltfpP/LnDs9n5484jrVj3mmmRxkuVJlq9du3Z7S5AkDTPal9feMWxxF3rvLTy9jZ+5Jsm0qlrdXB56pGlfBRwwbLsZTduzVNUSYAnA/PnzfW9CksbQaHsK/37YdCzwJL1LSNviauD0Zv504Kph7ael53XA48MuM0mSBmC09xTetS0HT3IpcAy9obeHgE8C5wHfTvJu4AHgxGbza+g9jnovvUdSt+kzJUnbbrSXj2YAXwaOapr+EfhAVQ09335VdfIWVr15hG0LOGs09UiS+mO0l4++Tu/yzsua6e+aNknSBDLaUJhaVV+vqg3NtBSY2se6JEkdGG0orEtyapIXNdOpwLp+FiZJGrzRhsIiejeEH6b3MtkfA2f0qSZJUkdGOyDep4HTq+oxaAe1+xy9sJAkTRCj7Sm8elMgAFTVo8Br+lOSJKkrow2FXZLsvWmh6SmMtpchSdpBjPYH++eBm5L8z2b5T4DP9qckSVJXRvtG87Iky4F/1zS9o6ru7F9ZkqQujPoSUBMCBoEkTWBbPXS2JGniMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSa2Bj1+UZDZw2bCmg4BPAHsBfw6sbdrPraprBludJO3cBh4KVXUPMA8gyYuAVcCVwLuAL1TV5wZdkySpp+vLR28G7quqBzquQ5JE96FwEnDpsOWzk9yW5OLhQ3VLkgajs1BIshtwPLBpOO4LgYPpXVpaTW+47pH2W5xkeZLla9euHWkTSdI26rKn8DbglqpaA1BVa6pqY1U9A3wNeO1IO1XVkqqaX1Xzp06dOsByJWni6zIUTmbYpaMk04atOwFYOfCKJGkn18lXaiZ5CfAW4D3Dmv9rknlAAfdvtk6SNACdhEJV/Suw72Zt7+yiFknSb3X99JEkaRwxFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJrU6+jhMgyf3Ak8BGYENVzU+yD3AZMIve9zSfWFWPdVWjJO1suu4pvKmq5lXV/Gb5o8D1VXUocH2zLEkakK5DYXMLgW80898A/qi7UiRp59NlKBTw/SQrkixu2vavqtXN/MPA/t2UJkk7p87uKQBvqKpVSfYDrkty9/CVVVVJavOdmgBZDDBz5szBVCpJO4nOQqGqVjV/PpLkSuC1wJok06pqdZJpwCMj7LcEWAIwf/7854SGtLN68NOv6stxZ37i9r4cV+NTJ5ePkrwkyZ6b5oG3AiuBq4HTm81OB67qoj5J2ll11VPYH7gyyaYavllV1yb5CfDtJO8GHgBO7Kg+SdopdRIKVfVz4PAR2tcBbx58RZIkGH+PpEqSOmQoSJJahoIkqWUoSJJaXb68pnHG59wl2VOQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUGHgpJDkhyQ5I7k9yR5ANN+6eSrEpyazMtGHRtkrSz62Lo7A3AOVV1S5I9gRVJrmvWfaGqPtdBTZIkOgiFqloNrG7mn0xyFzB90HVIkp6r03sKSWYBrwF+1DSdneS2JBcn2bu7yiRp59RZKCTZA7gC+GBVPQFcCBwMzKPXk/j8FvZbnGR5kuVr164dVLmStFPo5Os4k0yiFwiXVNV3AKpqzbD1XwP+fqR9q2oJsARg/vz51f9qJam/+vVVuLD1X4fbxdNHAS4C7qqqvxnWPm3YZicAKwddmyTt7LroKRwFvBO4PcmtTdu5wMlJ5gEF3A+8p4PaJGmn1sXTR/8EZIRV1wy6FknSs/lGsySpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklrjLhSSHJfkniT3Jvlo1/VI0s5kXIVCkhcBXwHeBhwGnJzksG6rkqSdx65dF7CZ1wL3VtXPAZJ8C1gI3NlpVRqoIz68rC/HXXH+aX05rn7Lv7sd33gLhenAQ8OWh4Df39qD+A9TkrZNqqrrGlpJ/hg4rqr+rFl+J/D7VXX2sG0WA4ubxdnAPQMscQrwLwP8vEHz/HZsE/n8JvK5weDP73eraupIK8ZbT2EVcMCw5RlNW6uqlgBLBlnUJkmWV9X8Lj57EDy/HdtEPr+JfG4wvs5vXN1oBn4CHJrkwCS7AScBV3dckyTtNMZVT6GqNiQ5G/gH4EXAxVV1R8dlSdJOY1yFAkBVXQNc03UdW9DJZasB8vx2bBP5/CbyucE4Or9xdaNZktSt8XZPQZLUIUNhlJLsleTyJHcnuSvJ67uuaSwl+U9J7kiyMsmlSSZ3XdNYSfKB5rzuSPLBruvZXkkuTvJIkpXD2v6kOb9nkoyLp1i21RbO7/zm/73bklyZZK8OS9wuWzi/TyVZleTWZlrQVX2Gwuh9Ebi2ql4BHA7c1XE9YybJdOA/AvOrai69m/wndVvV2EgyF/hzem/LHw68Pckh3Va13ZYCx23WthJ4B3DjwKsZe0t57vldB8ytqlcD/wx8bNBFjaGlPPf8AL5QVfOaqbP7qobCKCT5HeBo4CKAqvpNVf2y06LG3q7Av0myK7A78H87rmesvBL4UVX9qqo2AD+g98Nzh1VVNwKPbtZ2V1UN8kXOvtnC+X2/+fsDuJneO0w7pJHObzwxFEbnQGAt8PUkP03yt0le0nVRY6WqVgGfAx4EVgOPV9X3u61qzKwE3phk3yS7Awt49guS2vEsAr7XdRF9cHZzeeziJHt3VYShMDq7Ar8HXFhVrwH+FZgww3o3/wAX0gu/lwEvSXJqt1WNjaq6C/gvwPeBa4FbgY1d1qRtl+TjwAbgkq5rGWMXAgcD8+j9Yvb5rgoxFEZnCBiqqh81y5fTC4mJ4g+BX1TV2qpaD3wH+IOOaxozVXVRVR1RVUcDj9G7Jq0dTJIzgLcDp9QEe5a+qtZU1caqegb4Gr17YJ0wFEahqh4GHkoyu2l6MxNrOO8Hgdcl2T1J6J3fRLqRvl/z50x69xO+2W1F2lpJjgP+Aji+qn7VdT1jLcm0YYsn0Lvs2U0tEyxw+ybJPOBvgd2AnwPvqqrHOi1qDCX5S+BP6XXNfwr8WVX9utuqxkaSfwT2BdYDH6qq6zsuabskuRQ4ht7ImmuAT9K7cfllYCrwS+DWqjq2oxK3yxbO72PAi4F1zWY3V9WZnRS4nbZwfsfQu3RUwP3Ae6pqdSf1GQqSpE28fCRJahkKkqSWoSBJahkKkqSWoSBJahkK0lZIcsKwkSw3Tc8kedsWtt8ryfsGXae0rXwkVdoOSRYDpwBvat5G3Xz9LODvm9FnpXHPnoK0jZK8HPgE8M6qeibJh5P8pBnU7C+bzc4DDm56FOcn2SPJ9UluSXJ7koXdnYH0XOPuO5qlHUGSSfSGyzinqh5M8lbgUHpj1gS4OsnR9AZOnFtV85r9dgVOqKonkkwBbk5y9UQby0c7LkNB2jb/Gbijqi5rlt/aTD9tlvegFxIPbrZfgL9qAuMZYDqwP/Bw3yuWRsFQkLZSkmOA/8CzR8oN8NdV9d8323bWZrufQm98oiOqan2S+4EJ89Wn2vF5T0HaCs13T3wdOK2qnhy26h+ARUn2aLab3ozO+iSw57Dtfgd4pAmENwG/O6DSpVGxpyBtnTOB/YALe6OMt/6a3j2Gm5r2p4BTq+q+JP+n+ZL279H7wp+/S3I7sBy4e5DFSy/ER1IlSS0vH0mSWoaCJKllKEiSWoaCJKllKEiSWoaCJKllKEiSWoaCJKn1/wGimJ87zAhJlgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Zeta visualisation\n",
    "sns.countplot(x='Zeta', data= train,hue='label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Some cells which we ran for annalysing and taking intutition about the aggregated feature me made**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta[(train_meta.last_update.isnull())|(train_meta.recent_payment_activity.isnull())][\"normal_payment\"].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta[train_meta['last_update'].isnull()].groupby(\"id\").size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta[train_meta.id==3730406]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train[train.id==3730406]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta.lu_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta.sort_values([\"id\",\"lu_year\",\"lu_month\", \"lu_date\"])[train_meta.sort_values([\"id\",\"lu_year\" ,\"lu_month\", \"lu_date\"]).id ==]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta['p_limit'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta.sort_values([\"id\",\"lu_year\",\"lu_month\", \"lu_date\"]).groupby(\"id\")[\"p_limit\"].apply(lambda x: x.bfill()+1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing and Transformation (Actual and Advanced preprocessing start here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In this section we had done three things**\n",
    "\n",
    "1. Flled nan values of payment.csv and other datas by analysing those features, the description on how we done it can make the comments ver long for this notebook to address. we can describe if needed before second round selection.\n",
    "\n",
    "2. Aggregating from payment.csv and concateing into original data, Now how we made those features is a combination of outputs we got from above cells and some other analysis which can be described in a well expained manner later.\n",
    "\n",
    "3. Now made aggregations within the data resulted from part-2 on some basis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Converting date features to pd.to_datetime\n",
    "train_meta['last_update'] = pd.to_datetime(train_meta['last_update'], dayfirst=True)\n",
    "train_meta['recent_payment_activity'] = pd.to_datetime(train_meta['recent_payment_activity'], dayfirst=True)\n",
    "train_meta['lu_month'] = train_meta['last_update'].dt.month\n",
    "train_meta['lu_date'] = train_meta['last_update'].dt.day\n",
    "train_meta['lu_year'] = train_meta['last_update'].dt.year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filling nans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta = train_meta.sort_values([\"id\",\"lu_year\",\"lu_month\", \"lu_date\"]).reset_index(drop = True) # sorted\n",
    "# drooping these lu columns\n",
    "train_meta.drop(columns=  [\"lu_month\", \"lu_year\", \"lu_date\"], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n",
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  if __name__ == '__main__':\n",
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:11: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # This is added back by InteractiveShellApp.init_path()\n"
     ]
    }
   ],
   "source": [
    "# fillings values in train_meta\n",
    "dt1 = pd.to_datetime(\"30/12/2016\", dayfirst = True) # for above 30/12/2016\n",
    "dt2 = pd.to_datetime(\"28/02/2017\", dayfirst = True) # for recent activity greater than this\n",
    "dt3 = pd.to_datetime(\"01/08/2015\", dayfirst = True) # for update onlt 2 of this type\n",
    "dt4 = pd.to_datetime(\"01/01/2021\", dayfirst = True) # for update onlt 2 of this type\n",
    "train_meta['max_bal'].fillna(0, inplace = True)### Filling max_balance with 0 in meta_data\n",
    "train_meta[\"p_limit\"] = train_meta.groupby(\"id\")[\"p_limit\"].ffill() #using ffill for filling 66 % data\n",
    "train_meta[\"last_update\"][(train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())] = train_meta[\"last_update\"][(train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())].fillna(dt1)\n",
    "train_meta[\"recent_payment_activity\"][(train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())] = train_meta[\"recent_payment_activity\"][(train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())].fillna(dt1)\n",
    "train_meta[\"last_update\"][(train_meta.last_update.isnull())&(~train_meta.recent_payment_activity.isnull())] = train_meta[\"last_update\"][(train_meta.last_update.isnull())&(~train_meta.recent_payment_activity.isnull())].fillna(dt3)\n",
    "train_meta[\"recent_payment_activity\"][(~train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())] = train_meta[\"recent_payment_activity\"][(~train_meta.last_update.isnull())&(train_meta.recent_payment_activity.isnull())].fillna(dt4)\n",
    "train_meta[\"p_limit\"] = train_meta.groupby(\"id\")[\"p_limit\"].apply(lambda x: x.bfill().ffill()+1000)\n",
    "train_meta[\"p_limit\"].fillna(400000, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                         0\n",
       "Late_2                     0\n",
       "Late_1                     0\n",
       "Late_3                     0\n",
       "days_late_Sum              0\n",
       "normal_payment             0\n",
       "p_code                     0\n",
       "p_limit                    0\n",
       "last_update                0\n",
       "curr_remaining             0\n",
       "max_bal                    0\n",
       "recent_payment_activity    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying original demographics data and fill nans of it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenated both dataset train and test\n",
    "Y_train = train['label']\n",
    "dataset = pd.concat([train.drop(columns = ['label']), test]).reset_index(drop = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id           0\n",
       "Alpha        0\n",
       "Beta       149\n",
       "Gamma        0\n",
       "Delta        0\n",
       "Epsilon      0\n",
       "Zeta         0\n",
       "Eta          0\n",
       "Theta        0\n",
       "Iota         0\n",
       "Kappa        0\n",
       "Lambda       0\n",
       "omikron      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it sounds reasonable to replace the nans with mean value of it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filling beta feature with median which we described earlier why we do it\n",
    "med = dataset['Beta'].median()\n",
    "dataset['Beta'].fillna(med, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying Train data features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['Alpha'] = dataset['Alpha']//523  # as already mentioned divided by 523\n",
    "dataset[\"Del_bins\"] = (dataset['Delta']//1000).map(lambda x: 1 if x< 70 else 2 if x<250 else 3)  # biining the delta function into three based on its distribition\n",
    "dataset[\"Theta\"]  = dataset['Theta'].map(lambda x: 1 if x< 70 else 2 if x <80 else 3 if x<90 else 4 if x<100 else 5 if x<110 else 6) # Similarly Binning theta function which we hypothised as ages into discrete age group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aggregation type1\n",
    "\n",
    "**Within payment_history.csv**\n",
    "\n",
    "1. curr_remaining-max_ball>0 feature agg in terms of everyhting(both as a category label and numerical label)\n",
    "2. variation of product limits how freuently changes(montonocity in this as well)\n",
    "3. monotnicity in curr_balance and max_bal\n",
    "4. days late sum aggregation\n",
    "5. amount of each late type aggregate\n",
    "6. ratios of 4th, 5th\n",
    "7. number of normal_payments/per_unit_time\n",
    "8. binning of days_late_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregation1 starts from here (for making features from payment history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta['days_per_session'] = (train_meta['recent_payment_activity'].dt.year-train_meta['last_update'].dt.year)*12 + (train_meta['recent_payment_activity'].dt.month-train_meta['last_update'].dt.month)+1\n",
    "train_meta['late_per_session'] = train_meta['days_late_Sum']/train_meta['days_per_session']\n",
    "train_meta[\"diff_bw_upd_pay\"] = train_meta['recent_payment_activity'].dt.year-train_meta['last_update'].dt.year\n",
    "agg_trans1 = train_meta.groupby('id')[[\"Late_2\", \"Late_3\", \"Late_1\", \"days_late_Sum\", \"normal_payment\"]].agg(['mean','std'])\n",
    "agg_trans1.columns = ['_'.join(col).strip() for col in agg_trans1.columns.values]\n",
    "agg_trans1.reset_index(inplace=True)\n",
    "\n",
    "df = train_meta.groupby('id').size().reset_index(name='transactions_count')\n",
    "agg_trans1 = pd.merge(df, agg_trans1, on='id', how='left')\n",
    "\n",
    "\n",
    "train_meta[\"p_limit_decr\"] = (train_meta.groupby('id')['p_limit'].diff().reset_index(drop=True)<0).map(lambda x:1 if x else 0 if np.isnan(x) else 0)\n",
    "\n",
    "train_meta[\"curr_greater_prev_max_bal\"] = (train_meta.curr_remaining- train_meta.max_bal>0).map(lambda x:1 if x else 0)\n",
    "agg_trans2 = train_meta.groupby('id')[[\"p_limit_decr\", \"curr_greater_prev_max_bal\", \"late_per_session\", \"diff_bw_upd_pay\"]].agg(['mean','std'])\n",
    "\n",
    "agg_trans2.columns = ['_'.join(col).strip() for col in agg_trans2.columns.values]\n",
    "agg_trans2.reset_index(inplace=True)\n",
    "agg_trans = pd.merge(agg_trans1, agg_trans2, on='id', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    111\n",
       "1     20\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[train.id.isin(dataset.sort_values(\"id\")[(train_meta.groupby(\"id\").tail(1).sort_values(\"id\")[\"curr_remaining\"]>100000).values].id)][\"label\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>Kappa</th>\n",
       "      <th>Lambda</th>\n",
       "      <th>omikron</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1489815797</td>\n",
       "      <td>2092</td>\n",
       "      <td>1244.0</td>\n",
       "      <td>3</td>\n",
       "      <td>107000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>112</td>\n",
       "      <td>5</td>\n",
       "      <td>60017</td>\n",
       "      <td>3.162278</td>\n",
       "      <td>0.026786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1489759034</td>\n",
       "      <td>2092</td>\n",
       "      <td>1364.0</td>\n",
       "      <td>3</td>\n",
       "      <td>194000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>110</td>\n",
       "      <td>5</td>\n",
       "      <td>151004</td>\n",
       "      <td>223.606798</td>\n",
       "      <td>0.027273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1486529479</td>\n",
       "      <td>3661</td>\n",
       "      <td>1289.0</td>\n",
       "      <td>3</td>\n",
       "      <td>305000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>110</td>\n",
       "      <td>5</td>\n",
       "      <td>60033</td>\n",
       "      <td>236.643191</td>\n",
       "      <td>0.027273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1487734487</td>\n",
       "      <td>3661</td>\n",
       "      <td>1382.0</td>\n",
       "      <td>3</td>\n",
       "      <td>115000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>100</td>\n",
       "      <td>5</td>\n",
       "      <td>60058</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>2233421</td>\n",
       "      <td>3661</td>\n",
       "      <td>1295.0</td>\n",
       "      <td>1</td>\n",
       "      <td>116000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>9</td>\n",
       "      <td>108</td>\n",
       "      <td>4</td>\n",
       "      <td>151300</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.009259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>710</th>\n",
       "      <td>0</td>\n",
       "      <td>1485631659</td>\n",
       "      <td>2615</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>82000</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>106</td>\n",
       "      <td>5</td>\n",
       "      <td>60058</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.018868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>711</th>\n",
       "      <td>0</td>\n",
       "      <td>1487505580</td>\n",
       "      <td>3661</td>\n",
       "      <td>1326.5</td>\n",
       "      <td>1</td>\n",
       "      <td>141000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>67</td>\n",
       "      <td>5</td>\n",
       "      <td>60007</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.014925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>712</th>\n",
       "      <td>0</td>\n",
       "      <td>1492854287</td>\n",
       "      <td>2092</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>111000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>110</td>\n",
       "      <td>4</td>\n",
       "      <td>60091</td>\n",
       "      <td>220.868739</td>\n",
       "      <td>0.018182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>0</td>\n",
       "      <td>1492283318</td>\n",
       "      <td>2092</td>\n",
       "      <td>1370.0</td>\n",
       "      <td>1</td>\n",
       "      <td>273000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>110</td>\n",
       "      <td>5</td>\n",
       "      <td>60015</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.009091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714</th>\n",
       "      <td>0</td>\n",
       "      <td>1488721718</td>\n",
       "      <td>3661</td>\n",
       "      <td>1235.0</td>\n",
       "      <td>1</td>\n",
       "      <td>75000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>91</td>\n",
       "      <td>5</td>\n",
       "      <td>350092</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>0.010989</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>549 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     label          id  Alpha    Beta  Gamma   Delta  Epsilon  Zeta  Eta  \\\n",
       "2        0  1489815797   2092  1244.0      3  107000        2     8    5   \n",
       "3        0  1489759034   2092  1364.0      3  194000        2     8    5   \n",
       "4        0  1486529479   3661  1289.0      3  305000        1    11    5   \n",
       "5        1  1487734487   3661  1382.0      3  115000        2    11    5   \n",
       "9        0     2233421   3661  1295.0      1  116000        2    11    9   \n",
       "..     ...         ...    ...     ...    ...     ...      ...   ...  ...   \n",
       "710      0  1485631659   2615     NaN      2   82000        2    15    5   \n",
       "711      0  1487505580   3661  1326.5      1  141000        2    11   -1   \n",
       "712      0  1492854287   2092     NaN      2  111000        2     8    5   \n",
       "713      0  1492283318   2092  1370.0      1  273000        2     8    9   \n",
       "714      0  1488721718   3661  1235.0      1   75000        2    11    5   \n",
       "\n",
       "     Theta  Iota   Kappa      Lambda   omikron  \n",
       "2      112     5   60017    3.162278  0.026786  \n",
       "3      110     5  151004  223.606798  0.027273  \n",
       "4      110     5   60033  236.643191  0.027273  \n",
       "5      100     5   60058    1.000000  0.030000  \n",
       "9      108     4  151300    1.000000  0.009259  \n",
       "..     ...   ...     ...         ...       ...  \n",
       "710    106     5   60058    1.000000  0.018868  \n",
       "711     67     5   60007    1.000000  0.014925  \n",
       "712    110     4   60091  220.868739  0.018182  \n",
       "713    110     5   60015    1.000000  0.009091  \n",
       "714     91     5  350092  200.000000  0.010989  \n",
       "\n",
       "[549 rows x 14 columns]"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[train.id.isin(dataset.sort_values(\"id\")[(train_meta.groupby(\"id\").tail(1).sort_values(\"id\")[\"curr_remaining\"]>1000).values].id)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Late_2</th>\n",
       "      <th>Late_1</th>\n",
       "      <th>Late_3</th>\n",
       "      <th>days_late_Sum</th>\n",
       "      <th>normal_payment</th>\n",
       "      <th>p_code</th>\n",
       "      <th>p_limit</th>\n",
       "      <th>last_update</th>\n",
       "      <th>curr_remaining</th>\n",
       "      <th>max_bal</th>\n",
       "      <th>recent_payment_activity</th>\n",
       "      <th>days_per_session</th>\n",
       "      <th>late_per_session</th>\n",
       "      <th>diff_bw_upd_pay</th>\n",
       "      <th>p_limit_decr</th>\n",
       "      <th>curr_greater_prev_max_bal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>10</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>1998-06-17</td>\n",
       "      <td>6.0</td>\n",
       "      <td>81139.0</td>\n",
       "      <td>2013-05-10</td>\n",
       "      <td>180</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>12</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>1998-10-02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31666.0</td>\n",
       "      <td>2006-06-28</td>\n",
       "      <td>93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>2001-01-14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18000.0</td>\n",
       "      <td>2021-01-01</td>\n",
       "      <td>241</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>2001-01-14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16000.0</td>\n",
       "      <td>2021-01-01</td>\n",
       "      <td>241</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>10</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>2006-03-30</td>\n",
       "      <td>15938.4</td>\n",
       "      <td>28740.0</td>\n",
       "      <td>2015-11-26</td>\n",
       "      <td>117</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>2006-06-19</td>\n",
       "      <td>0.0</td>\n",
       "      <td>600500.0</td>\n",
       "      <td>2010-10-26</td>\n",
       "      <td>53</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1114</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>27</td>\n",
       "      <td>331000.0</td>\n",
       "      <td>2006-06-19</td>\n",
       "      <td>0.0</td>\n",
       "      <td>600500.0</td>\n",
       "      <td>2021-01-01</td>\n",
       "      <td>176</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  Late_2  Late_1  Late_3  days_late_Sum  normal_payment  p_code  \\\n",
       "18  1114       0       0       0              0              36      10   \n",
       "19  1114       0       0       0              0              18      12   \n",
       "20  1114       0       0       0              0               0       6   \n",
       "21  1114       0       0       0              0              10       6   \n",
       "22  1114       0       0       0              0              36      10   \n",
       "23  1114       0       0       0              0               1      27   \n",
       "24  1114       0       0       0              0              16      27   \n",
       "\n",
       "     p_limit last_update  curr_remaining   max_bal recent_payment_activity  \\\n",
       "18  331000.0  1998-06-17             6.0   81139.0              2013-05-10   \n",
       "19  331000.0  1998-10-02             0.0   31666.0              2006-06-28   \n",
       "20  331000.0  2001-01-14             0.0   18000.0              2021-01-01   \n",
       "21  331000.0  2001-01-14             0.0   16000.0              2021-01-01   \n",
       "22  331000.0  2006-03-30         15938.4   28740.0              2015-11-26   \n",
       "23  331000.0  2006-06-19             0.0  600500.0              2010-10-26   \n",
       "24  331000.0  2006-06-19             0.0  600500.0              2021-01-01   \n",
       "\n",
       "    days_per_session  late_per_session  diff_bw_upd_pay  p_limit_decr  \\\n",
       "18               180               0.0               15             0   \n",
       "19                93               0.0                8             0   \n",
       "20               241               0.0               20             0   \n",
       "21               241               0.0               20             0   \n",
       "22               117               0.0                9             0   \n",
       "23                53               0.0                4             0   \n",
       "24               176               0.0               15             0   \n",
       "\n",
       "    curr_greater_prev_max_bal  \n",
       "18                          0  \n",
       "19                          0  \n",
       "20                          0  \n",
       "21                          0  \n",
       "22                          0  \n",
       "23                          0  \n",
       "24                          0  "
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta[train_meta.id==1114]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17       True\n",
       "28      False\n",
       "30      False\n",
       "34       True\n",
       "44      False\n",
       "        ...  \n",
       "7051    False\n",
       "7055    False\n",
       "7063     True\n",
       "7066    False\n",
       "7069    False\n",
       "Name: curr_remaining, Length: 1012, dtype: bool"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta.groupby(\"id\").tail(1).sort_values(\"id\")[\"curr_remaining\"]<=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Combining both data payment history and original based on agg features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df = pd.merge(dataset, agg_trans, on='id', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df.fillna(-1, inplace = True)  # Filling some nan values if there is any obtained from mathematical \n",
    "                                    # faults like divide by 0 likewise ith -1 to make it explicit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>...</th>\n",
       "      <th>normal_payment_mean</th>\n",
       "      <th>normal_payment_std</th>\n",
       "      <th>p_limit_decr_mean</th>\n",
       "      <th>p_limit_decr_std</th>\n",
       "      <th>curr_greater_prev_max_bal_mean</th>\n",
       "      <th>curr_greater_prev_max_bal_std</th>\n",
       "      <th>late_per_session_mean</th>\n",
       "      <th>late_per_session_std</th>\n",
       "      <th>diff_bw_upd_pay_mean</th>\n",
       "      <th>diff_bw_upd_pay_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1484006679</td>\n",
       "      <td>1</td>\n",
       "      <td>1281.5</td>\n",
       "      <td>2</td>\n",
       "      <td>500000</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>13.333333</td>\n",
       "      <td>14.449494</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>12.166667</td>\n",
       "      <td>6.336522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1489825072</td>\n",
       "      <td>4</td>\n",
       "      <td>1250.0</td>\n",
       "      <td>3</td>\n",
       "      <td>137000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>25.125000</td>\n",
       "      <td>14.156649</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.353553</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.002976</td>\n",
       "      <td>0.008418</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.817254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1489815797</td>\n",
       "      <td>4</td>\n",
       "      <td>1244.0</td>\n",
       "      <td>3</td>\n",
       "      <td>107000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>16.142857</td>\n",
       "      <td>7.335498</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.377964</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.377964</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.899735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1489759034</td>\n",
       "      <td>4</td>\n",
       "      <td>1364.0</td>\n",
       "      <td>3</td>\n",
       "      <td>194000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>15.750000</td>\n",
       "      <td>10.180514</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.353553</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.375000</td>\n",
       "      <td>2.722263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1486529479</td>\n",
       "      <td>7</td>\n",
       "      <td>1289.0</td>\n",
       "      <td>3</td>\n",
       "      <td>305000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>12.727273</td>\n",
       "      <td>11.849818</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.301511</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.404520</td>\n",
       "      <td>8.253961</td>\n",
       "      <td>17.825034</td>\n",
       "      <td>2.727273</td>\n",
       "      <td>2.760105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1120</th>\n",
       "      <td>1489702642</td>\n",
       "      <td>4</td>\n",
       "      <td>1313.0</td>\n",
       "      <td>3</td>\n",
       "      <td>134000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>22.909091</td>\n",
       "      <td>14.046028</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.467099</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.010250</td>\n",
       "      <td>0.031172</td>\n",
       "      <td>8.272727</td>\n",
       "      <td>5.728716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1121</th>\n",
       "      <td>1835338</td>\n",
       "      <td>7</td>\n",
       "      <td>1125.5</td>\n",
       "      <td>3</td>\n",
       "      <td>79000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>16.750000</td>\n",
       "      <td>12.685293</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.395427</td>\n",
       "      <td>0.523094</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1122</th>\n",
       "      <td>1236915</td>\n",
       "      <td>7</td>\n",
       "      <td>1188.5</td>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>11.500000</td>\n",
       "      <td>7.852813</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.785714</td>\n",
       "      <td>1.571429</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.258306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1123</th>\n",
       "      <td>1489208099</td>\n",
       "      <td>7</td>\n",
       "      <td>1386.5</td>\n",
       "      <td>1</td>\n",
       "      <td>165000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>9.643651</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.447214</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.447214</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.200000</td>\n",
       "      <td>1.095445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1124</th>\n",
       "      <td>3788282</td>\n",
       "      <td>7</td>\n",
       "      <td>1350.5</td>\n",
       "      <td>3</td>\n",
       "      <td>130000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>16.400000</td>\n",
       "      <td>11.193748</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.447214</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1125 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id  Alpha    Beta  Gamma   Delta  Epsilon  Zeta  Eta  Theta  \\\n",
       "0     1484006679      1  1281.5      2  500000        2     5   -1      2   \n",
       "1     1489825072      4  1250.0      3  137000        2     8    5      4   \n",
       "2     1489815797      4  1244.0      3  107000        2     8    5      6   \n",
       "3     1489759034      4  1364.0      3  194000        2     8    5      6   \n",
       "4     1486529479      7  1289.0      3  305000        1    11    5      6   \n",
       "...          ...    ...     ...    ...     ...      ...   ...  ...    ...   \n",
       "1120  1489702642      4  1313.0      3  134000        2     8    5      5   \n",
       "1121     1835338      7  1125.5      3   79000        2    11   -1      5   \n",
       "1122     1236915      7  1188.5      3   90000        2    11    5      4   \n",
       "1123  1489208099      7  1386.5      1  165000        2    11   -1      5   \n",
       "1124     3788282      7  1350.5      3  130000        1    11    5      4   \n",
       "\n",
       "      Iota  ...  normal_payment_mean  normal_payment_std  p_limit_decr_mean  \\\n",
       "0        5  ...            13.333333           14.449494           0.000000   \n",
       "1        5  ...            25.125000           14.156649           0.125000   \n",
       "2        5  ...            16.142857            7.335498           0.142857   \n",
       "3        5  ...            15.750000           10.180514           0.000000   \n",
       "4        5  ...            12.727273           11.849818           0.090909   \n",
       "...    ...  ...                  ...                 ...                ...   \n",
       "1120     3  ...            22.909091           14.046028           0.272727   \n",
       "1121     5  ...            16.750000           12.685293           0.000000   \n",
       "1122     5  ...            11.500000            7.852813           0.000000   \n",
       "1123     3  ...            23.000000            9.643651           0.200000   \n",
       "1124     5  ...            16.400000           11.193748           0.000000   \n",
       "\n",
       "      p_limit_decr_std  curr_greater_prev_max_bal_mean  \\\n",
       "0             0.000000                        0.000000   \n",
       "1             0.353553                        0.000000   \n",
       "2             0.377964                        0.142857   \n",
       "3             0.000000                        0.125000   \n",
       "4             0.301511                        0.181818   \n",
       "...                ...                             ...   \n",
       "1120          0.467099                        0.000000   \n",
       "1121          0.000000                        0.250000   \n",
       "1122          0.000000                        0.250000   \n",
       "1123          0.447214                        0.200000   \n",
       "1124          0.000000                        0.200000   \n",
       "\n",
       "      curr_greater_prev_max_bal_std  late_per_session_mean  \\\n",
       "0                          0.000000               0.000000   \n",
       "1                          0.000000               0.002976   \n",
       "2                          0.377964               0.000000   \n",
       "3                          0.353553               0.000000   \n",
       "4                          0.404520               8.253961   \n",
       "...                             ...                    ...   \n",
       "1120                       0.000000               0.010250   \n",
       "1121                       0.500000               0.395427   \n",
       "1122                       0.500000               0.785714   \n",
       "1123                       0.447214               0.000000   \n",
       "1124                       0.447214               0.000000   \n",
       "\n",
       "      late_per_session_std  diff_bw_upd_pay_mean  diff_bw_upd_pay_std  \n",
       "0                 0.000000             12.166667             6.336522  \n",
       "1                 0.008418              4.000000             3.817254  \n",
       "2                 0.000000              0.857143             0.899735  \n",
       "3                 0.000000              1.375000             2.722263  \n",
       "4                17.825034              2.727273             2.760105  \n",
       "...                    ...                   ...                  ...  \n",
       "1120              0.031172              8.272727             5.728716  \n",
       "1121              0.523094              1.250000             1.500000  \n",
       "1122              1.571429              1.250000             1.258306  \n",
       "1123              0.000000              1.200000             1.095445  \n",
       "1124              0.000000              1.000000             0.000000  \n",
       "\n",
       "[1125 rows x 33 columns]"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AGGREGATION TYPE2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let'us use some of the categorical datas to generate features which can be used by tree based models.\n",
    "Here used groupby on the above generated data to made features which can be benefical in making best split for a model like xgboost of LGBM with more depth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df[\"mean_late_days_per_delta_week\"] = full_df.groupby([\"Del_bins\", \"Alpha\"])[\"late_per_session_mean\"].transform(\"mean\")\n",
    "full_df[\"late_vs_epsilon\"] = full_df.groupby([\"Epsilon\"])[\"late_per_session_mean\"].transform(\"mean\")\n",
    "#full_df[\"beta_bins\"] = full_df.groupby([\"Del_bins\"])[\"Beta\"].transform(\"mean\")\n",
    "full_df[\"payments_regions\"] = full_df.groupby([\"Gamma\", \"Del_bins\", \"Epsilon\"])[\"normal_payment_mean\"].transform(\"sum\")\n",
    "#full_df[\"bal_incre_region\"] = full_df.groupby([\"Gamma\"])[\"max_bal_incr_mean\"].transform(\"sum\")\n",
    "full_df[\"limit_dec_region\"] = full_df.groupby([\"Gamma\"])[\"p_limit_decr_mean\"].transform(\"sum\")\n",
    "#full_df[\"total_days_per_days_update\"] = full_df.groupby([\"Alpha\"])[\"days_per_session_mean\"].transform(\"mean\")\n",
    "#full_df[\"kappa_per_bin\"] = full_df.groupby([\"Del_bins\"])[\"Kappa\"].transform(\"mean\")\n",
    "\n",
    "full_df[\"delta_per_theta\"] = full_df.groupby([\"Theta\"])[\"Delta\"].transform(\"mean\")\n",
    "#full_df[\"late_per_theta\"] = full_df.groupby([\"Theta\"])[\"Late_3_mean\"].transform(\"mean\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>...</th>\n",
       "      <th>diff_bw_upd_pay_mean</th>\n",
       "      <th>diff_bw_upd_pay_std</th>\n",
       "      <th>mean_late_days_per_delta_week</th>\n",
       "      <th>late_vs_epsilon</th>\n",
       "      <th>beta_bins</th>\n",
       "      <th>payments_regions</th>\n",
       "      <th>bal_incre_region</th>\n",
       "      <th>limit_dec_region</th>\n",
       "      <th>total_days_per_days_update</th>\n",
       "      <th>delta_per_theta</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1489825072</td>\n",
       "      <td>4</td>\n",
       "      <td>1250.0</td>\n",
       "      <td>3</td>\n",
       "      <td>137000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.817254</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1489815797</td>\n",
       "      <td>4</td>\n",
       "      <td>1244.0</td>\n",
       "      <td>3</td>\n",
       "      <td>107000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.899735</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1489759034</td>\n",
       "      <td>4</td>\n",
       "      <td>1364.0</td>\n",
       "      <td>3</td>\n",
       "      <td>194000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.375000</td>\n",
       "      <td>2.722263</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1487734487</td>\n",
       "      <td>7</td>\n",
       "      <td>1382.0</td>\n",
       "      <td>3</td>\n",
       "      <td>115000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>4.166667</td>\n",
       "      <td>2.136976</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2959839</td>\n",
       "      <td>4</td>\n",
       "      <td>1314.5</td>\n",
       "      <td>1</td>\n",
       "      <td>177000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>-1</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>1.914854</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>3021.218784</td>\n",
       "      <td>91.594112</td>\n",
       "      <td>12.963645</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1007</th>\n",
       "      <td>1489702642</td>\n",
       "      <td>4</td>\n",
       "      <td>1313.0</td>\n",
       "      <td>3</td>\n",
       "      <td>134000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>8.272727</td>\n",
       "      <td>5.728716</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1008</th>\n",
       "      <td>1835338</td>\n",
       "      <td>7</td>\n",
       "      <td>1125.5</td>\n",
       "      <td>3</td>\n",
       "      <td>79000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1009</th>\n",
       "      <td>1236915</td>\n",
       "      <td>7</td>\n",
       "      <td>1188.5</td>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.258306</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1010</th>\n",
       "      <td>1489208099</td>\n",
       "      <td>7</td>\n",
       "      <td>1386.5</td>\n",
       "      <td>1</td>\n",
       "      <td>165000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>1.200000</td>\n",
       "      <td>1.095445</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>3021.218784</td>\n",
       "      <td>91.594112</td>\n",
       "      <td>12.963645</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>3788282</td>\n",
       "      <td>7</td>\n",
       "      <td>1350.5</td>\n",
       "      <td>3</td>\n",
       "      <td>130000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>1.358755</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>494.057824</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1012 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id  Alpha    Beta  Gamma   Delta  Epsilon  Zeta  Eta  Theta  \\\n",
       "0     1489825072      4  1250.0      3  137000        2     8    5      4   \n",
       "1     1489815797      4  1244.0      3  107000        2     8    5      6   \n",
       "2     1489759034      4  1364.0      3  194000        2     8    5      6   \n",
       "3     1487734487      7  1382.0      3  115000        2    11    5      5   \n",
       "4        2959839      4  1314.5      1  177000        2     8   -1      6   \n",
       "...          ...    ...     ...    ...     ...      ...   ...  ...    ...   \n",
       "1007  1489702642      4  1313.0      3  134000        2     8    5      5   \n",
       "1008     1835338      7  1125.5      3   79000        2    11   -1      5   \n",
       "1009     1236915      7  1188.5      3   90000        2    11    5      4   \n",
       "1010  1489208099      7  1386.5      1  165000        2    11   -1      5   \n",
       "1011     3788282      7  1350.5      3  130000        1    11    5      4   \n",
       "\n",
       "      Iota  ...  diff_bw_upd_pay_mean  diff_bw_upd_pay_std  \\\n",
       "0        5  ...              4.000000             3.817254   \n",
       "1        5  ...              0.857143             0.899735   \n",
       "2        5  ...              1.375000             2.722263   \n",
       "3        5  ...              4.166667             2.136976   \n",
       "4        5  ...              2.500000             1.914854   \n",
       "...    ...  ...                   ...                  ...   \n",
       "1007     3  ...              8.272727             5.728716   \n",
       "1008     5  ...              1.250000             1.500000   \n",
       "1009     5  ...              1.250000             1.258306   \n",
       "1010     3  ...              1.200000             1.095445   \n",
       "1011     5  ...              1.000000             0.000000   \n",
       "\n",
       "      mean_late_days_per_delta_week  late_vs_epsilon    beta_bins  \\\n",
       "0                         -0.012682         3.949673  1289.836376   \n",
       "1                         -0.012682         3.949673  1289.836376   \n",
       "2                         -0.012682         3.949673  1289.836376   \n",
       "3                          5.878214         3.949673  1289.836376   \n",
       "4                         -0.012682         3.949673  1289.836376   \n",
       "...                             ...              ...          ...   \n",
       "1007                      -0.012682         3.949673  1289.836376   \n",
       "1008                       5.878214         3.949673  1289.836376   \n",
       "1009                       5.878214         3.949673  1289.836376   \n",
       "1010                       5.878214         3.949673  1289.836376   \n",
       "1011                       5.878214         1.358755  1289.836376   \n",
       "\n",
       "      payments_regions  bal_incre_region  limit_dec_region  \\\n",
       "0          5747.087156        229.968746         28.650335   \n",
       "1          5747.087156        229.968746         28.650335   \n",
       "2          5747.087156        229.968746         28.650335   \n",
       "3          5747.087156        229.968746         28.650335   \n",
       "4          3021.218784         91.594112         12.963645   \n",
       "...                ...               ...               ...   \n",
       "1007       5747.087156        229.968746         28.650335   \n",
       "1008       5747.087156        229.968746         28.650335   \n",
       "1009       5747.087156        229.968746         28.650335   \n",
       "1010       3021.218784         91.594112         12.963645   \n",
       "1011        494.057824        229.968746         28.650335   \n",
       "\n",
       "      total_days_per_days_update  delta_per_theta  \n",
       "0                      29.693415    115893.129771  \n",
       "1                      29.693415    107848.837209  \n",
       "2                      29.693415    107848.837209  \n",
       "3                      30.964916    114717.460317  \n",
       "4                      29.693415    107848.837209  \n",
       "...                          ...              ...  \n",
       "1007                   29.693415    114717.460317  \n",
       "1008                   30.964916    114717.460317  \n",
       "1009                   30.964916    115893.129771  \n",
       "1010                   30.964916    114717.460317  \n",
       "1011                   30.964916    115893.129771  \n",
       "\n",
       "[1012 rows x 45 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Late_2</th>\n",
       "      <th>Late_1</th>\n",
       "      <th>Late_3</th>\n",
       "      <th>days_late_Sum</th>\n",
       "      <th>normal_payment</th>\n",
       "      <th>p_code</th>\n",
       "      <th>p_limit</th>\n",
       "      <th>last_update</th>\n",
       "      <th>curr_remaining</th>\n",
       "      <th>max_bal</th>\n",
       "      <th>recent_payment_activity</th>\n",
       "      <th>days_per_session</th>\n",
       "      <th>late_per_session</th>\n",
       "      <th>diff_bw_upd_pay</th>\n",
       "      <th>p_limit_decr</th>\n",
       "      <th>max_bal_incr</th>\n",
       "      <th>curr_greater_prev_max_bal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>276000.0</td>\n",
       "      <td>1994-12-18</td>\n",
       "      <td>13909.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2021-01-01</td>\n",
       "      <td>314</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>35</td>\n",
       "      <td>10</td>\n",
       "      <td>276000.0</td>\n",
       "      <td>1997-04-21</td>\n",
       "      <td>-1197.6</td>\n",
       "      <td>333140.0</td>\n",
       "      <td>2015-07-21</td>\n",
       "      <td>220</td>\n",
       "      <td>0.068182</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>276000.0</td>\n",
       "      <td>2001-02-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>248874.0</td>\n",
       "      <td>2007-06-22</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>10</td>\n",
       "      <td>6500.0</td>\n",
       "      <td>2003-01-05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4043.0</td>\n",
       "      <td>2003-11-25</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>6500.0</td>\n",
       "      <td>2003-09-17</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4458.0</td>\n",
       "      <td>2006-03-28</td>\n",
       "      <td>31</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7065</th>\n",
       "      <td>1492854287</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>35</td>\n",
       "      <td>10</td>\n",
       "      <td>34000.0</td>\n",
       "      <td>2013-01-28</td>\n",
       "      <td>9956.4</td>\n",
       "      <td>22591.0</td>\n",
       "      <td>2016-02-04</td>\n",
       "      <td>38</td>\n",
       "      <td>0.736842</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7066</th>\n",
       "      <td>1492854287</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>34000.0</td>\n",
       "      <td>2015-04-05</td>\n",
       "      <td>528427.2</td>\n",
       "      <td>500500.0</td>\n",
       "      <td>2016-02-21</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7067</th>\n",
       "      <td>1492861707</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>10</td>\n",
       "      <td>16400.0</td>\n",
       "      <td>2014-02-07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13901.0</td>\n",
       "      <td>2015-11-14</td>\n",
       "      <td>22</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7068</th>\n",
       "      <td>1492861707</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>16400.0</td>\n",
       "      <td>2015-02-14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21000.0</td>\n",
       "      <td>2015-10-22</td>\n",
       "      <td>9</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7069</th>\n",
       "      <td>1492861707</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>10</td>\n",
       "      <td>16400.0</td>\n",
       "      <td>2015-03-23</td>\n",
       "      <td>2492.4</td>\n",
       "      <td>15440.0</td>\n",
       "      <td>2016-02-19</td>\n",
       "      <td>12</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7070 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id  Late_2  Late_1  Late_3  days_late_Sum  normal_payment  \\\n",
       "0              1       0       0       0              0               0   \n",
       "1              1       1       0       0             15              35   \n",
       "2              1       0       0       0              0              11   \n",
       "3              1       0       0       0              0              35   \n",
       "4              1       0       0       0              0               4   \n",
       "...          ...     ...     ...     ...            ...             ...   \n",
       "7065  1492854287       1       0       0             28              35   \n",
       "7066  1492854287       0       0       0              0              10   \n",
       "7067  1492861707       0       0       0              0              19   \n",
       "7068  1492861707       0       0       0              0               1   \n",
       "7069  1492861707       0       0       0              0              12   \n",
       "\n",
       "      p_code   p_limit last_update  curr_remaining   max_bal  \\\n",
       "0         10  276000.0  1994-12-18         13909.2       0.0   \n",
       "1         10  276000.0  1997-04-21         -1197.6  333140.0   \n",
       "2          2  276000.0  2001-02-21             0.0  248874.0   \n",
       "3         10    6500.0  2003-01-05             0.0    4043.0   \n",
       "4         10    6500.0  2003-09-17             0.0    4458.0   \n",
       "...      ...       ...         ...             ...       ...   \n",
       "7065      10   34000.0  2013-01-28          9956.4   22591.0   \n",
       "7066       5   34000.0  2015-04-05        528427.2  500500.0   \n",
       "7067      10   16400.0  2014-02-07             0.0   13901.0   \n",
       "7068       6   16400.0  2015-02-14             0.0   21000.0   \n",
       "7069      10   16400.0  2015-03-23          2492.4   15440.0   \n",
       "\n",
       "     recent_payment_activity  days_per_session  late_per_session  \\\n",
       "0                 2021-01-01               314          0.000000   \n",
       "1                 2015-07-21               220          0.068182   \n",
       "2                 2007-06-22                77          0.000000   \n",
       "3                 2003-11-25                11          0.000000   \n",
       "4                 2006-03-28                31          0.000000   \n",
       "...                      ...               ...               ...   \n",
       "7065              2016-02-04                38          0.736842   \n",
       "7066              2016-02-21                11          0.000000   \n",
       "7067              2015-11-14                22          0.000000   \n",
       "7068              2015-10-22                 9          0.000000   \n",
       "7069              2016-02-19                12          0.000000   \n",
       "\n",
       "      diff_bw_upd_pay  p_limit_decr  max_bal_incr  curr_greater_prev_max_bal  \n",
       "0                  27             0             0                          1  \n",
       "1                  18             0             1                          0  \n",
       "2                   6             0             0                          0  \n",
       "3                   0             1             0                          0  \n",
       "4                   3             0             1                          0  \n",
       "...               ...           ...           ...                        ...  \n",
       "7065                3             0             0                          0  \n",
       "7066                1             0             1                          1  \n",
       "7067                1             0             0                          0  \n",
       "7068                0             0             1                          0  \n",
       "7069                1             0             0                          0  \n",
       "\n",
       "[7070 rows x 18 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>...</th>\n",
       "      <th>diff_bw_upd_pay_mean</th>\n",
       "      <th>diff_bw_upd_pay_std</th>\n",
       "      <th>mean_late_days_per_delta_week</th>\n",
       "      <th>late_vs_epsilon</th>\n",
       "      <th>beta_bins</th>\n",
       "      <th>payments_regions</th>\n",
       "      <th>bal_incre_region</th>\n",
       "      <th>limit_dec_region</th>\n",
       "      <th>total_days_per_days_update</th>\n",
       "      <th>delta_per_theta</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1489825072</td>\n",
       "      <td>4</td>\n",
       "      <td>1250.0</td>\n",
       "      <td>3</td>\n",
       "      <td>137000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.817254</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1489815797</td>\n",
       "      <td>4</td>\n",
       "      <td>1244.0</td>\n",
       "      <td>3</td>\n",
       "      <td>107000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.899735</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1489759034</td>\n",
       "      <td>4</td>\n",
       "      <td>1364.0</td>\n",
       "      <td>3</td>\n",
       "      <td>194000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.375000</td>\n",
       "      <td>2.722263</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1487734487</td>\n",
       "      <td>7</td>\n",
       "      <td>1382.0</td>\n",
       "      <td>3</td>\n",
       "      <td>115000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>4.166667</td>\n",
       "      <td>2.136976</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2959839</td>\n",
       "      <td>4</td>\n",
       "      <td>1314.5</td>\n",
       "      <td>1</td>\n",
       "      <td>177000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>-1</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>1.914854</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>3021.218784</td>\n",
       "      <td>91.594112</td>\n",
       "      <td>12.963645</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>107848.837209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1007</th>\n",
       "      <td>1489702642</td>\n",
       "      <td>4</td>\n",
       "      <td>1313.0</td>\n",
       "      <td>3</td>\n",
       "      <td>134000</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>8.272727</td>\n",
       "      <td>5.728716</td>\n",
       "      <td>-0.012682</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>29.693415</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1008</th>\n",
       "      <td>1835338</td>\n",
       "      <td>7</td>\n",
       "      <td>1125.5</td>\n",
       "      <td>3</td>\n",
       "      <td>79000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1009</th>\n",
       "      <td>1236915</td>\n",
       "      <td>7</td>\n",
       "      <td>1188.5</td>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.258306</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>5747.087156</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1010</th>\n",
       "      <td>1489208099</td>\n",
       "      <td>7</td>\n",
       "      <td>1386.5</td>\n",
       "      <td>1</td>\n",
       "      <td>165000</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>-1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>1.200000</td>\n",
       "      <td>1.095445</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>3.949673</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>3021.218784</td>\n",
       "      <td>91.594112</td>\n",
       "      <td>12.963645</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>114717.460317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>3788282</td>\n",
       "      <td>7</td>\n",
       "      <td>1350.5</td>\n",
       "      <td>3</td>\n",
       "      <td>130000</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.878214</td>\n",
       "      <td>1.358755</td>\n",
       "      <td>1289.836376</td>\n",
       "      <td>494.057824</td>\n",
       "      <td>229.968746</td>\n",
       "      <td>28.650335</td>\n",
       "      <td>30.964916</td>\n",
       "      <td>115893.129771</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1012 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id  Alpha    Beta  Gamma   Delta  Epsilon  Zeta  Eta  Theta  \\\n",
       "0     1489825072      4  1250.0      3  137000        2     8    5      4   \n",
       "1     1489815797      4  1244.0      3  107000        2     8    5      6   \n",
       "2     1489759034      4  1364.0      3  194000        2     8    5      6   \n",
       "3     1487734487      7  1382.0      3  115000        2    11    5      5   \n",
       "4        2959839      4  1314.5      1  177000        2     8   -1      6   \n",
       "...          ...    ...     ...    ...     ...      ...   ...  ...    ...   \n",
       "1007  1489702642      4  1313.0      3  134000        2     8    5      5   \n",
       "1008     1835338      7  1125.5      3   79000        2    11   -1      5   \n",
       "1009     1236915      7  1188.5      3   90000        2    11    5      4   \n",
       "1010  1489208099      7  1386.5      1  165000        2    11   -1      5   \n",
       "1011     3788282      7  1350.5      3  130000        1    11    5      4   \n",
       "\n",
       "      Iota  ...  diff_bw_upd_pay_mean  diff_bw_upd_pay_std  \\\n",
       "0        5  ...              4.000000             3.817254   \n",
       "1        5  ...              0.857143             0.899735   \n",
       "2        5  ...              1.375000             2.722263   \n",
       "3        5  ...              4.166667             2.136976   \n",
       "4        5  ...              2.500000             1.914854   \n",
       "...    ...  ...                   ...                  ...   \n",
       "1007     3  ...              8.272727             5.728716   \n",
       "1008     5  ...              1.250000             1.500000   \n",
       "1009     5  ...              1.250000             1.258306   \n",
       "1010     3  ...              1.200000             1.095445   \n",
       "1011     5  ...              1.000000             0.000000   \n",
       "\n",
       "      mean_late_days_per_delta_week  late_vs_epsilon    beta_bins  \\\n",
       "0                         -0.012682         3.949673  1289.836376   \n",
       "1                         -0.012682         3.949673  1289.836376   \n",
       "2                         -0.012682         3.949673  1289.836376   \n",
       "3                          5.878214         3.949673  1289.836376   \n",
       "4                         -0.012682         3.949673  1289.836376   \n",
       "...                             ...              ...          ...   \n",
       "1007                      -0.012682         3.949673  1289.836376   \n",
       "1008                       5.878214         3.949673  1289.836376   \n",
       "1009                       5.878214         3.949673  1289.836376   \n",
       "1010                       5.878214         3.949673  1289.836376   \n",
       "1011                       5.878214         1.358755  1289.836376   \n",
       "\n",
       "      payments_regions  bal_incre_region  limit_dec_region  \\\n",
       "0          5747.087156        229.968746         28.650335   \n",
       "1          5747.087156        229.968746         28.650335   \n",
       "2          5747.087156        229.968746         28.650335   \n",
       "3          5747.087156        229.968746         28.650335   \n",
       "4          3021.218784         91.594112         12.963645   \n",
       "...                ...               ...               ...   \n",
       "1007       5747.087156        229.968746         28.650335   \n",
       "1008       5747.087156        229.968746         28.650335   \n",
       "1009       5747.087156        229.968746         28.650335   \n",
       "1010       3021.218784         91.594112         12.963645   \n",
       "1011        494.057824        229.968746         28.650335   \n",
       "\n",
       "      total_days_per_days_update  delta_per_theta  \n",
       "0                      29.693415    115893.129771  \n",
       "1                      29.693415    107848.837209  \n",
       "2                      29.693415    107848.837209  \n",
       "3                      30.964916    114717.460317  \n",
       "4                      29.693415    107848.837209  \n",
       "...                          ...              ...  \n",
       "1007                   29.693415    114717.460317  \n",
       "1008                   30.964916    114717.460317  \n",
       "1009                   30.964916    115893.129771  \n",
       "1010                   30.964916    114717.460317  \n",
       "1011                   30.964916    115893.129771  \n",
       "\n",
       "[1012 rows x 45 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df = pd.merge(full_df, train_meta.groupby(\"id\").tail(1)[[\"id\", \"curr_remaining\"]], how = \"left\", on= \"id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df.drop(columns = [\"id\"], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Late_2</th>\n",
       "      <th>Late_1</th>\n",
       "      <th>Late_3</th>\n",
       "      <th>days_late_Sum</th>\n",
       "      <th>normal_payment</th>\n",
       "      <th>p_code</th>\n",
       "      <th>p_limit</th>\n",
       "      <th>last_update</th>\n",
       "      <th>curr_remaining</th>\n",
       "      <th>max_bal</th>\n",
       "      <th>recent_payment_activity</th>\n",
       "      <th>days_per_session</th>\n",
       "      <th>late_per_session</th>\n",
       "      <th>diff_bw_upd_pay</th>\n",
       "      <th>p_limit_decr</th>\n",
       "      <th>max_bal_incr</th>\n",
       "      <th>curr_greater_prev_max_bal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [id, Late_2, Late_1, Late_3, days_late_Sum, normal_payment, p_code, p_limit, last_update, curr_remaining, max_bal, recent_payment_activity, days_per_session, late_per_session, diff_bw_upd_pay, p_limit_decr, max_bal_incr, curr_greater_prev_max_bal]\n",
       "Index: []"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_meta.sort_values(\"id\")[(train_meta.sort_values(\"id\").max_bal.isnull())&(train_meta.sort_values(\"id\").curr_remaining>0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>Alpha</th>\n",
       "      <th>Beta</th>\n",
       "      <th>Gamma</th>\n",
       "      <th>Delta</th>\n",
       "      <th>Epsilon</th>\n",
       "      <th>Zeta</th>\n",
       "      <th>Eta</th>\n",
       "      <th>Theta</th>\n",
       "      <th>Iota</th>\n",
       "      <th>Kappa</th>\n",
       "      <th>Lambda</th>\n",
       "      <th>omikron</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [label, id, Alpha, Beta, Gamma, Delta, Epsilon, Zeta, Eta, Theta, Iota, Kappa, Lambda, omikron]\n",
       "Index: []"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[train.id.isin(train_meta.sort_values(\"id\")[(train_meta.sort_values(\"id\").max_bal.isnull())&(train_meta.sort_values(\"id\").curr_remaining>0)].groupby(\"id\").size().reset_index()['id'].values)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_meta[x]['curr_remaining']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Alpha                               6\n",
       "Beta                              158\n",
       "Gamma                               3\n",
       "Delta                             229\n",
       "Epsilon                             2\n",
       "Eta                                10\n",
       "Theta                               6\n",
       "Kappa                             280\n",
       "Lambda                            266\n",
       "omikron                           125\n",
       "Del_bins                            3\n",
       "transactions_count                 37\n",
       "Late_2_mean                       124\n",
       "Late_2_std                        237\n",
       "Late_3_mean                       108\n",
       "Late_3_std                        136\n",
       "Late_1_mean                        95\n",
       "Late_1_std                        146\n",
       "days_late_Sum_mean                366\n",
       "days_late_Sum_std                 398\n",
       "days_per_session_mean             730\n",
       "days_per_session_std              981\n",
       "normal_payment_mean               538\n",
       "normal_payment_std                918\n",
       "p_limit_decr_mean                  48\n",
       "p_limit_decr_std                   93\n",
       "max_bal_incr_mean                  82\n",
       "max_bal_incr_std                   96\n",
       "curr_greater_prev_max_bal_mean     73\n",
       "curr_greater_prev_max_bal_std     108\n",
       "late_per_session_mean             417\n",
       "late_per_session_std              421\n",
       "diff_bw_upd_pay_mean              302\n",
       "diff_bw_upd_pay_std               623\n",
       "mean_late_days_per_delta_week      12\n",
       "late_vs_epsilon                     2\n",
       "beta_bins                           3\n",
       "payments_regions                   18\n",
       "bal_incre_region                    3\n",
       "limit_dec_region                    3\n",
       "total_days_per_days_update          6\n",
       "kappa_per_bin                       3\n",
       "delta_per_theta                     6\n",
       "late_per_theta                      6\n",
       "curr_remaining                    896\n",
       "dtype: int64"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now some tree based models like lgbm and xgboost accept features of as type categorical and treates those features specifically compared to numerical and give them treatment like categories and we not need to do seprate one hot encodings for them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we selected these features to treat it as categorical\n",
    "categorical_cols = ['Del_bins',  'Gamma', \"Eta\", \"Theta\", \"Epsilon\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting to astype category\n",
    "for i in categorical_cols:\n",
    "    full_df[i] = full_df[i].astype(\"category\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Alpha                             0\n",
       "Beta                              0\n",
       "Gamma                             0\n",
       "Delta                             0\n",
       "Epsilon                           0\n",
       "Eta                               0\n",
       "Theta                             0\n",
       "Kappa                             0\n",
       "Lambda                            0\n",
       "omikron                           0\n",
       "Del_bins                          0\n",
       "transactions_count                0\n",
       "Late_2_mean                       0\n",
       "Late_2_std                        0\n",
       "Late_3_mean                       0\n",
       "Late_3_std                        0\n",
       "Late_1_mean                       0\n",
       "Late_1_std                        0\n",
       "days_late_Sum_mean                0\n",
       "days_late_Sum_std                 0\n",
       "days_per_session_mean             0\n",
       "days_per_session_std              0\n",
       "normal_payment_mean               0\n",
       "normal_payment_std                0\n",
       "p_limit_decr_mean                 0\n",
       "p_limit_decr_std                  0\n",
       "max_bal_incr_mean                 0\n",
       "max_bal_incr_std                  0\n",
       "curr_greater_prev_max_bal_mean    0\n",
       "curr_greater_prev_max_bal_std     0\n",
       "late_per_session_mean             0\n",
       "late_per_session_std              0\n",
       "diff_bw_upd_pay_mean              0\n",
       "diff_bw_upd_pay_std               0\n",
       "mean_late_days_per_delta_week     0\n",
       "late_vs_epsilon                   0\n",
       "beta_bins                         0\n",
       "payments_regions                  0\n",
       "bal_incre_region                  0\n",
       "limit_dec_region                  0\n",
       "total_days_per_days_update        0\n",
       "kappa_per_bin                     0\n",
       "delta_per_theta                   0\n",
       "late_per_theta                    0\n",
       "curr_remaining                    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MODEL BUILDING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we preformed several techniques for hyperparameter tuning and tested different models\n",
    "\n",
    "We found **Optuna** to be the best tool for yperparameter optimization instead of Random and Grid Search CV in terms of transversing the hyperparameter space.\n",
    "\n",
    "For model we tried different models but as the training and testing have no correlation we found that the model performing well on cross validation on training giving bad on leaderboard, it signifies there is less correlation on the distribution and we had done cross validation so there is no point of overfitting type of thing for justifying why we move to the below in leaderboard it is just the lack of trust in crossvalidation\n",
    "\n",
    "So on basis of consistent result we used **LGBM(Light Gradient Boosting Method)**\n",
    "**Even we adopted SMOTE, oversampling techinque to make equal distribution but still problem of correlation between train and test data is there**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting the dataset back\n",
    "x_train, x_test = full_df[0:len(train)], full_df[len(train):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using optuna\n",
    "from optuna import Trial\n",
    "import gc\n",
    "import optuna\n",
    "from sklearn.model_selection import train_test_split\n",
    "from lightgbm import LGBMClassifier\n",
    "\n",
    "#category_cols = ['disrict', 'client_catg', 'region', 'region_group']\n",
    "\n",
    "def objective(trial:Trial):\n",
    "    \n",
    "    gc.collect()\n",
    "    models=[]\n",
    "    validScore=0\n",
    "   \n",
    "    model,log = fitLGBM(trial,x_train,Y_train)\n",
    "    \n",
    "    models.append(model)\n",
    "    gc.collect()\n",
    "    validScore+=log\n",
    "    validScore/=len(models)\n",
    "    \n",
    "    return validScore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.under_sampling import RandomUnderSampler\n",
    "smote = RandomUnderSampler(random_state = 44)\n",
    "def fitLGBM(trial,X, y):\n",
    "    \n",
    "    params={\n",
    "      'n_estimators':trial.suggest_int('n_estimators', 0, 1000), \n",
    "      'num_leaves':trial.suggest_int('num_leaves', 2, 512),\n",
    "      'max_depth':trial.suggest_int('max_depth', 2, 128),\n",
    "      'learning_rate': trial.suggest_loguniform('learning_rate', 0.001, 0.15),\n",
    "      'min_split_gain': trial.suggest_loguniform('min_split_gain', 0.001, 0.1),\n",
    "      'feature_fraction':trial.suggest_uniform('feature_fraction',0.1, 1.0),\n",
    "      'bagging_freq':trial.suggest_int('bagging_freq',0.1,10),\n",
    "      'verbosity': -1,\n",
    "      'random_state':44\n",
    "            }\n",
    "    stkfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=44)\n",
    "    model = LGBMClassifier(**params, categorical_features = \"auto\")\n",
    "    \n",
    "    res=[]\n",
    "    for i, (tdx, vdx) in enumerate(stkfold.split(X, y)):\n",
    "        X_train, X_valid, y_train, y_valid = X.iloc[tdx], X.iloc[vdx], y[tdx], y[vdx]\n",
    "        #X_train, y_train = smote.fit_resample(X_train, y_train)\n",
    "        model.fit(X_train, y_train,\n",
    "                 eval_set=[(X_train, y_train), (X_valid, y_valid)],\n",
    "                 early_stopping_rounds=30, verbose=False)\n",
    "        preds = model.predict_proba(X_valid)\n",
    "        false_positive_rate, true_positive_rate, thresholds = roc_curve(y_valid, preds[:,1])\n",
    "        # may differ\n",
    "        print (auc(false_positive_rate, true_positive_rate))\n",
    "        res.append(auc(false_positive_rate, true_positive_rate))\n",
    "    err = np.mean(res)\n",
    "    \n",
    "    return model, err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:43,193]\u001b[0m A new study created in memory with name: no-name-ca9b7ec4-6df1-44c3-a343-9cd5107bc7c3\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7739130434782608\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39309985076157594, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39309985076157594\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7380434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39309985076157594, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39309985076157594\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6402173913043478\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39309985076157594, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39309985076157594\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6413043478260869\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:43,826]\u001b[0m Trial 0 finished with value: 0.6990579710144929 and parameters: {'n_estimators': 265, 'num_leaves': 64, 'max_depth': 19, 'learning_rate': 0.039032243371170866, 'min_split_gain': 0.03869279854056714, 'feature_fraction': 0.39309985076157594, 'bagging_freq': 7}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39309985076157594, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39309985076157594\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7018115942028985\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5104922737149502, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5104922737149502\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7579710144927536\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5104922737149502, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5104922737149502\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7376811594202899\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5104922737149502, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5104922737149502\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6253623188405797\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5104922737149502, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5104922737149502\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6092391304347827\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5104922737149502, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5104922737149502\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6945652173913044\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:44,376]\u001b[0m Trial 1 finished with value: 0.6849637681159421 and parameters: {'n_estimators': 884, 'num_leaves': 13, 'max_depth': 17, 'learning_rate': 0.030532097318654722, 'min_split_gain': 0.0013875330158273126, 'feature_fraction': 0.5104922737149502, 'bagging_freq': 6}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.700711189656274, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.700711189656274\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7641304347826088\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.700711189656274, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.700711189656274\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7141304347826087\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.700711189656274, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.700711189656274\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6340579710144927\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.700711189656274, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.700711189656274\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6021739130434783\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.700711189656274, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.700711189656274\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:46,163]\u001b[0m Trial 2 finished with value: 0.6789855072463767 and parameters: {'n_estimators': 320, 'num_leaves': 234, 'max_depth': 55, 'learning_rate': 0.0033418627409586396, 'min_split_gain': 0.0017849151436314073, 'feature_fraction': 0.700711189656274, 'bagging_freq': 0}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6804347826086957\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14176012403928095, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14176012403928095\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7702898550724638\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14176012403928095, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14176012403928095\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7246376811594203\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14176012403928095, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14176012403928095\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6076086956521739\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14176012403928095, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14176012403928095\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:46,667]\u001b[0m Trial 3 finished with value: 0.6514492753623188 and parameters: {'n_estimators': 377, 'num_leaves': 429, 'max_depth': 47, 'learning_rate': 0.060330950619222744, 'min_split_gain': 0.0039177665309687365, 'feature_fraction': 0.14176012403928095, 'bagging_freq': 0}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5721014492753622\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14176012403928095, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14176012403928095\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.5826086956521739\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.573791774650503, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.573791774650503\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7677536231884058\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.573791774650503, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.573791774650503\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7170289855072465\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.573791774650503, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.573791774650503\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6228260869565218\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.573791774650503, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.573791774650503\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5934782608695652\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.573791774650503, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.573791774650503\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7010869565217391\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:47,294]\u001b[0m Trial 4 finished with value: 0.6804347826086957 and parameters: {'n_estimators': 466, 'num_leaves': 466, 'max_depth': 92, 'learning_rate': 0.02188643923448543, 'min_split_gain': 0.08429098242286057, 'feature_fraction': 0.573791774650503, 'bagging_freq': 9}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.408311226781658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.408311226781658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7576086956521739\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.408311226781658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.408311226781658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.743840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.408311226781658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.408311226781658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6322463768115941\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.408311226781658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.408311226781658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6094202898550725\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.408311226781658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.408311226781658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6916666666666667\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:47,881]\u001b[0m Trial 5 finished with value: 0.6869565217391305 and parameters: {'n_estimators': 947, 'num_leaves': 245, 'max_depth': 72, 'learning_rate': 0.030741553519713832, 'min_split_gain': 0.0010376142090080887, 'feature_fraction': 0.408311226781658, 'bagging_freq': 9}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22026823913982937, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22026823913982937\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7782608695652173\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22026823913982937, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22026823913982937\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7427536231884058\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22026823913982937, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22026823913982937\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.5800724637681159\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22026823913982937, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22026823913982937\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:48,387]\u001b[0m Trial 6 finished with value: 0.6736956521739129 and parameters: {'n_estimators': 57, 'num_leaves': 89, 'max_depth': 92, 'learning_rate': 0.007153069181724517, 'min_split_gain': 0.0033864190909085527, 'feature_fraction': 0.22026823913982937, 'bagging_freq': 0}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6054347826086957\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22026823913982937, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22026823913982937\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6619565217391303\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.8375878487283657, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.8375878487283657\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7318840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.8375878487283657, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.8375878487283657\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.736231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.8375878487283657, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.8375878487283657\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.5894927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.8375878487283657, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.8375878487283657\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6376811594202898\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:48,882]\u001b[0m Trial 7 finished with value: 0.6719565217391306 and parameters: {'n_estimators': 653, 'num_leaves': 341, 'max_depth': 115, 'learning_rate': 0.14851703729250715, 'min_split_gain': 0.004836628830991963, 'feature_fraction': 0.8375878487283657, 'bagging_freq': 7}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.8375878487283657, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.8375878487283657\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6644927536231884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.21894732205957929, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.21894732205957929\n",
      "[LightGBM] [Warning] bagging_freq is set=3, subsample_freq=0 will be ignored. Current value: bagging_freq=3\n",
      "0.7815217391304348\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.21894732205957929, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.21894732205957929\n",
      "[LightGBM] [Warning] bagging_freq is set=3, subsample_freq=0 will be ignored. Current value: bagging_freq=3\n",
      "0.7278985507246377\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.21894732205957929, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.21894732205957929\n",
      "[LightGBM] [Warning] bagging_freq is set=3, subsample_freq=0 will be ignored. Current value: bagging_freq=3\n",
      "0.5945652173913043\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.21894732205957929, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.21894732205957929\n",
      "[LightGBM] [Warning] bagging_freq is set=3, subsample_freq=0 will be ignored. Current value: bagging_freq=3\n",
      "0.5865942028985507\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.21894732205957929, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.21894732205957929\n",
      "[LightGBM] [Warning] bagging_freq is set=3, subsample_freq=0 will be ignored. Current value: bagging_freq=3\n",
      "0.6659420289855073\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:49,773]\u001b[0m Trial 8 finished with value: 0.671304347826087 and parameters: {'n_estimators': 189, 'num_leaves': 112, 'max_depth': 78, 'learning_rate': 0.002736935753977978, 'min_split_gain': 0.00637280976974754, 'feature_fraction': 0.21894732205957929, 'bagging_freq': 3}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7350966184718217, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7350966184718217\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7514492753623189\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7350966184718217, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7350966184718217\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7170289855072465\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7350966184718217, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7350966184718217\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.6268115942028986\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7350966184718217, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7350966184718217\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.5996376811594203\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7350966184718217, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7350966184718217\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:50,549]\u001b[0m Trial 9 finished with value: 0.6758695652173914 and parameters: {'n_estimators': 346, 'num_leaves': 409, 'max_depth': 22, 'learning_rate': 0.01587872858931728, 'min_split_gain': 0.0025943030427176147, 'feature_fraction': 0.7350966184718217, 'bagging_freq': 4}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6844202898550725\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38106362109202985, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38106362109202985\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7387681159420291\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38106362109202985, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38106362109202985\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7164855072463768\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38106362109202985, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38106362109202985\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6175724637681159\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38106362109202985, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38106362109202985\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.592572463768116\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38106362109202985, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38106362109202985\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6532608695652173\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:50,933]\u001b[0m Trial 10 finished with value: 0.6637318840579711 and parameters: {'n_estimators': 20, 'num_leaves': 4, 'max_depth': 3, 'learning_rate': 0.13538922816381466, 'min_split_gain': 0.031806987463719336, 'feature_fraction': 0.38106362109202985, 'bagging_freq': 7}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3579219104289127, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3579219104289127\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.761231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3579219104289127, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3579219104289127\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.744927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3579219104289127, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3579219104289127\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6594202898550725\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3579219104289127, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3579219104289127\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:51,490]\u001b[0m Trial 11 finished with value: 0.6989130434782609 and parameters: {'n_estimators': 917, 'num_leaves': 221, 'max_depth': 41, 'learning_rate': 0.05641666686452752, 'min_split_gain': 0.02028900262593613, 'feature_fraction': 0.3579219104289127, 'bagging_freq': 9}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.625\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3579219104289127, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3579219104289127\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7039855072463768\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35848317815796954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35848317815796954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7405797101449275\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35848317815796954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35848317815796954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7275362318840579\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35848317815796954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35848317815796954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.625\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35848317815796954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35848317815796954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:52,013]\u001b[0m Trial 12 finished with value: 0.6872463768115942 and parameters: {'n_estimators': 670, 'num_leaves': 133, 'max_depth': 33, 'learning_rate': 0.06843239705631642, 'min_split_gain': 0.022627524676648414, 'feature_fraction': 0.35848317815796954, 'bagging_freq': 9}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6413043478260869\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35848317815796954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35848317815796954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7018115942028986\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2881808424449254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2881808424449254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7746376811594202\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2881808424449254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2881808424449254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7163043478260869\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2881808424449254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2881808424449254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6159420289855073\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2881808424449254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2881808424449254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:52,560]\u001b[0m Trial 13 finished with value: 0.6860144927536233 and parameters: {'n_estimators': 686, 'num_leaves': 190, 'max_depth': 37, 'learning_rate': 0.06193286589049885, 'min_split_gain': 0.01884364578205177, 'feature_fraction': 0.2881808424449254, 'bagging_freq': 7}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6318840579710144\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2881808424449254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2881808424449254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.691304347826087\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5217259179867012, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5217259179867012\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7681159420289856\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5217259179867012, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5217259179867012\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7054347826086956\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5217259179867012, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5217259179867012\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6322463768115941\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5217259179867012, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5217259179867012\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6047101449275362\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5217259179867012, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5217259179867012\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6822463768115943\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:53,450]\u001b[0m Trial 14 finished with value: 0.6785507246376812 and parameters: {'n_estimators': 166, 'num_leaves': 330, 'max_depth': 8, 'learning_rate': 0.00672474302446209, 'min_split_gain': 0.06753956265948292, 'feature_fraction': 0.5217259179867012, 'bagging_freq': 5}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4565027883170921, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4565027883170921\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.826086956521739\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4565027883170921, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4565027883170921\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7492753623188406\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4565027883170921, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4565027883170921\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6117753623188406\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4565027883170921, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4565027883170921\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5880434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4565027883170921, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4565027883170921\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:53,943]\u001b[0m Trial 15 finished with value: 0.6906884057971014 and parameters: {'n_estimators': 809, 'num_leaves': 42, 'max_depth': 28, 'learning_rate': 0.09298250805477247, 'min_split_gain': 0.011720428158777453, 'feature_fraction': 0.4565027883170921, 'bagging_freq': 8}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6782608695652174\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.10356157996702031, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.10356157996702031\n",
      "[LightGBM] [Warning] bagging_freq is set=2, subsample_freq=0 will be ignored. Current value: bagging_freq=2\n",
      "0.7670289855072464\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.10356157996702031, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.10356157996702031\n",
      "[LightGBM] [Warning] bagging_freq is set=2, subsample_freq=0 will be ignored. Current value: bagging_freq=2\n",
      "0.6974637681159419\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.10356157996702031, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.10356157996702031\n",
      "[LightGBM] [Warning] bagging_freq is set=2, subsample_freq=0 will be ignored. Current value: bagging_freq=2\n",
      "0.5826086956521739\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.10356157996702031, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.10356157996702031\n",
      "[LightGBM] [Warning] bagging_freq is set=2, subsample_freq=0 will be ignored. Current value: bagging_freq=2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:54,483]\u001b[0m Trial 16 finished with value: 0.655072463768116 and parameters: {'n_estimators': 198, 'num_leaves': 189, 'max_depth': 53, 'learning_rate': 0.03542268814459767, 'min_split_gain': 0.042974656412146144, 'feature_fraction': 0.10356157996702031, 'bagging_freq': 2}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5971014492753625\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.10356157996702031, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.10356157996702031\n",
      "[LightGBM] [Warning] bagging_freq is set=2, subsample_freq=0 will be ignored. Current value: bagging_freq=2\n",
      "0.631159420289855\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.6367947659482023, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6367947659482023\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7554347826086957\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.6367947659482023, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6367947659482023\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7409420289855072\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.6367947659482023, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6367947659482023\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6494565217391304\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.6367947659482023, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6367947659482023\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5929347826086957\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.6367947659482023, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6367947659482023\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6791666666666667\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:55,184]\u001b[0m Trial 17 finished with value: 0.6835869565217391 and parameters: {'n_estimators': 562, 'num_leaves': 318, 'max_depth': 2, 'learning_rate': 0.010253385888437123, 'min_split_gain': 0.010526260652958946, 'feature_fraction': 0.6367947659482023, 'bagging_freq': 8}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.9801641990656074, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9801641990656074\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7721014492753624\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.9801641990656074, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9801641990656074\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7329710144927537\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.9801641990656074, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9801641990656074\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.5927536231884057\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.9801641990656074, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9801641990656074\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6164855072463769\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.9801641990656074, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9801641990656074\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6731884057971015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:55,785]\u001b[0m Trial 18 finished with value: 0.6775 and parameters: {'n_estimators': 505, 'num_leaves': 165, 'max_depth': 40, 'learning_rate': 0.04072414856359005, 'min_split_gain': 0.04765706534607614, 'feature_fraction': 0.9801641990656074, 'bagging_freq': 5}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.26423752164864267, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.26423752164864267\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.8021739130434783\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.26423752164864267, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.26423752164864267\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.716304347826087\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.26423752164864267, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.26423752164864267\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6039855072463767\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.26423752164864267, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.26423752164864267\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5804347826086957\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.26423752164864267, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.26423752164864267\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:58,654]\u001b[0m Trial 19 finished with value: 0.6778260869565218 and parameters: {'n_estimators': 789, 'num_leaves': 61, 'max_depth': 16, 'learning_rate': 0.0013871399823037385, 'min_split_gain': 0.01670082038874214, 'feature_fraction': 0.26423752164864267, 'bagging_freq': 8}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.686231884057971\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3346149097566225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3346149097566225\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7782608695652173\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3346149097566225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3346149097566225\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.688768115942029\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3346149097566225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3346149097566225\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6293478260869565\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3346149097566225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3346149097566225\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6485507246376812\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3346149097566225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3346149097566225\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:59,172]\u001b[0m Trial 20 finished with value: 0.6815942028985507 and parameters: {'n_estimators': 115, 'num_leaves': 290, 'max_depth': 70, 'learning_rate': 0.09533892717768488, 'min_split_gain': 0.03553567076121872, 'feature_fraction': 0.3346149097566225, 'bagging_freq': 6}. Best is trial 0 with value: 0.6990579710144929.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6630434782608696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4739033556097208, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4739033556097208\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.8170289855072463\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4739033556097208, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4739033556097208\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7510869565217392\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4739033556097208, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4739033556097208\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6530797101449276\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4739033556097208, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4739033556097208\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6396739130434783\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4739033556097208, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4739033556097208\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:20:59,683]\u001b[0m Trial 21 finished with value: 0.7090579710144927 and parameters: {'n_estimators': 966, 'num_leaves': 52, 'max_depth': 28, 'learning_rate': 0.10515998674942041, 'min_split_gain': 0.010137122280832698, 'feature_fraction': 0.4739033556097208, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6844202898550724\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4516711655092751, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4516711655092751\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.8054347826086956\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4516711655092751, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4516711655092751\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7028985507246377\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4516711655092751, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4516711655092751\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5873188405797101\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4516711655092751, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4516711655092751\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6391304347826088\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4516711655092751, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4516711655092751\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6773550724637681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:00,161]\u001b[0m Trial 22 finished with value: 0.6824275362318841 and parameters: {'n_estimators': 975, 'num_leaves': 79, 'max_depth': 26, 'learning_rate': 0.1488693561403073, 'min_split_gain': 0.00873459361095549, 'feature_fraction': 0.4516711655092751, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44255507407955197, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44255507407955197\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7684782608695652\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44255507407955197, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44255507407955197\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7304347826086957\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44255507407955197, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44255507407955197\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6692028985507246\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44255507407955197, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44255507407955197\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:00,722]\u001b[0m Trial 23 finished with value: 0.6877536231884058 and parameters: {'n_estimators': 989, 'num_leaves': 149, 'max_depth': 45, 'learning_rate': 0.04739941352263922, 'min_split_gain': 0.026279827210564426, 'feature_fraction': 0.44255507407955197, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6025362318840579\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44255507407955197, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44255507407955197\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6681159420289855\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5788877652726364, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5788877652726364\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7757246376811595\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5788877652726364, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5788877652726364\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7173913043478262\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5788877652726364, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5788877652726364\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6297101449275362\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5788877652726364, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5788877652726364\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:01,245]\u001b[0m Trial 24 finished with value: 0.668804347826087 and parameters: {'n_estimators': 847, 'num_leaves': 43, 'max_depth': 58, 'learning_rate': 0.10055054408437171, 'min_split_gain': 0.013762176556976586, 'feature_fraction': 0.5788877652726364, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5728260869565217\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5788877652726364, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5788877652726364\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6483695652173913\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2969850420662254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2969850420662254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7876811594202898\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2969850420662254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2969850420662254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7405797101449275\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2969850420662254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2969850420662254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6626811594202899\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2969850420662254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2969850420662254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6043478260869566\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2969850420662254, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2969850420662254\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7032608695652174\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:01,835]\u001b[0m Trial 25 finished with value: 0.6997101449275362 and parameters: {'n_estimators': 278, 'num_leaves': 5, 'max_depth': 10, 'learning_rate': 0.0174521295499316, 'min_split_gain': 0.007488637481560341, 'feature_fraction': 0.2969850420662254, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.19354626034885214, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.19354626034885214\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7655797101449275\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.19354626034885214, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.19354626034885214\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7297101449275363\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.19354626034885214, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.19354626034885214\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:02,420]\u001b[0m Trial 26 finished with value: 0.683695652173913 and parameters: {'n_estimators': 229, 'num_leaves': 8, 'max_depth': 12, 'learning_rate': 0.024473192046112015, 'min_split_gain': 0.00811032636423595, 'feature_fraction': 0.19354626034885214, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6380434782608695\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.19354626034885214, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.19354626034885214\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6010869565217392\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.19354626034885214, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.19354626034885214\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6840579710144927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.30227893853431564, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.30227893853431564\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7855072463768116\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.30227893853431564, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.30227893853431564\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.721376811594203\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.30227893853431564, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.30227893853431564\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6268115942028986\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.30227893853431564, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.30227893853431564\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.5981884057971014\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.30227893853431564, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.30227893853431564\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:03,144]\u001b[0m Trial 27 finished with value: 0.6858695652173913 and parameters: {'n_estimators': 263, 'num_leaves': 100, 'max_depth': 7, 'learning_rate': 0.015477544563126954, 'min_split_gain': 0.006049242225016753, 'feature_fraction': 0.30227893853431564, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.697463768115942\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4889721774182526, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4889721774182526\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7634057971014493\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4889721774182526, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4889721774182526\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7253623188405798\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4889721774182526, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4889721774182526\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.636231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4889721774182526, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4889721774182526\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6123188405797102\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4889721774182526, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4889721774182526\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6902173913043479\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:03,817]\u001b[0m Trial 28 finished with value: 0.6855072463768115 and parameters: {'n_estimators': 402, 'num_leaves': 23, 'max_depth': 26, 'learning_rate': 0.019979390618657753, 'min_split_gain': 0.09768228049065934, 'feature_fraction': 0.4889721774182526, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.613916917299529, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.613916917299529\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.777536231884058\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.613916917299529, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.613916917299529\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7297101449275363\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.613916917299529, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.613916917299529\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.6393115942028985\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.613916917299529, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.613916917299529\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.5992753623188406\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.613916917299529, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.613916917299529\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.6827898550724638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:04,482]\u001b[0m Trial 29 finished with value: 0.6857246376811595 and parameters: {'n_estimators': 273, 'num_leaves': 4, 'max_depth': 15, 'learning_rate': 0.010004010493649376, 'min_split_gain': 0.06290587085683172, 'feature_fraction': 0.613916917299529, 'bagging_freq': 4}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5195191923558611, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5195191923558611\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7547101449275362\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5195191923558611, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5195191923558611\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.7192028985507247\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5195191923558611, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5195191923558611\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6275362318840579\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5195191923558611, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5195191923558611\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6192028985507246\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5195191923558611, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5195191923558611\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "0.6840579710144927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:05,095]\u001b[0m Trial 30 finished with value: 0.6809420289855073 and parameters: {'n_estimators': 108, 'num_leaves': 51, 'max_depth': 20, 'learning_rate': 0.0413163274260148, 'min_split_gain': 0.002340719402531238, 'feature_fraction': 0.5195191923558611, 'bagging_freq': 5}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38994584335283167, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38994584335283167\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7358695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38994584335283167, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38994584335283167\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7181159420289855\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38994584335283167, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38994584335283167\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6463768115942029\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38994584335283167, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38994584335283167\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:05,621]\u001b[0m Trial 31 finished with value: 0.6868478260869566 and parameters: {'n_estimators': 929, 'num_leaves': 205, 'max_depth': 36, 'learning_rate': 0.0750571825893164, 'min_split_gain': 0.015250379636829537, 'feature_fraction': 0.38994584335283167, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6405797101449274\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.38994584335283167, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.38994584335283167\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6932971014492755\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33790761195981844, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33790761195981844\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7880434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33790761195981844, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33790761195981844\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7322463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33790761195981844, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33790761195981844\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6503623188405797\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33790761195981844, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33790761195981844\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6210144927536232\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33790761195981844, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33790761195981844\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7068840579710145\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:06,191]\u001b[0m Trial 32 finished with value: 0.6997101449275362 and parameters: {'n_estimators': 442, 'num_leaves': 132, 'max_depth': 33, 'learning_rate': 0.04949686103246903, 'min_split_gain': 0.021972734656059756, 'feature_fraction': 0.33790761195981844, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14771805651439543, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14771805651439543\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7894927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14771805651439543, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14771805651439543\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7347826086956522\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14771805651439543, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14771805651439543\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6467391304347826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:06,736]\u001b[0m Trial 33 finished with value: 0.6922101449275362 and parameters: {'n_estimators': 459, 'num_leaves': 117, 'max_depth': 2, 'learning_rate': 0.02937122616161339, 'min_split_gain': 0.007101093789815526, 'feature_fraction': 0.14771805651439543, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14771805651439543, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14771805651439543\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5976449275362319\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.14771805651439543, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.14771805651439543\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6923913043478263\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.32696799537623533, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.32696799537623533\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7847826086956522\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.32696799537623533, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.32696799537623533\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7177536231884057\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.32696799537623533, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.32696799537623533\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6242753623188405\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.32696799537623533, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.32696799537623533\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.5996376811594203\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.32696799537623533, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.32696799537623533\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:07,680]\u001b[0m Trial 34 finished with value: 0.6828985507246377 and parameters: {'n_estimators': 295, 'num_leaves': 76, 'max_depth': 31, 'learning_rate': 0.014940491122085769, 'min_split_gain': 0.02740988465302248, 'feature_fraction': 0.32696799537623533, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6880434782608696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.25736407830870117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.25736407830870117\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7572463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.25736407830870117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.25736407830870117\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7170289855072464\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.25736407830870117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.25736407830870117\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6411231884057972\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.25736407830870117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.25736407830870117\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5942028985507247\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.25736407830870117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.25736407830870117\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7068840579710145\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:08,345]\u001b[0m Trial 35 finished with value: 0.6832971014492755 and parameters: {'n_estimators': 380, 'num_leaves': 4, 'max_depth': 51, 'learning_rate': 0.028356690182087318, 'min_split_gain': 0.011797416483965376, 'feature_fraction': 0.25736407830870117, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4319193666047928, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4319193666047928\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.7862318840579711\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4319193666047928, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4319193666047928\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.7521739130434782\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4319193666047928, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4319193666047928\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.6612318840579712\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4319193666047928, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4319193666047928\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.6047101449275362\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4319193666047928, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4319193666047928\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.7068840579710144\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:08,960]\u001b[0m Trial 36 finished with value: 0.7022463768115942 and parameters: {'n_estimators': 429, 'num_leaves': 31, 'max_depth': 18, 'learning_rate': 0.04798868086298652, 'min_split_gain': 0.004693382652349537, 'feature_fraction': 0.4319193666047928, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.40906898687446863, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.40906898687446863\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.8039855072463769\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.40906898687446863, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.40906898687446863\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7253623188405797\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.40906898687446863, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.40906898687446863\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6246376811594203\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.40906898687446863, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.40906898687446863\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.5909420289855072\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.40906898687446863, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.40906898687446863\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:09,462]\u001b[0m Trial 37 finished with value: 0.677608695652174 and parameters: {'n_estimators': 582, 'num_leaves': 40, 'max_depth': 11, 'learning_rate': 0.10812041671309419, 'min_split_gain': 0.00498937250795626, 'feature_fraction': 0.40906898687446863, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6431159420289856\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.47455946267778615, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.47455946267778615\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7742753623188405\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.47455946267778615, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.47455946267778615\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7130434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.47455946267778615, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.47455946267778615\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6478260869565218\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.47455946267778615, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.47455946267778615\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.588768115942029\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.47455946267778615, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.47455946267778615\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:10,514]\u001b[0m Trial 38 finished with value: 0.6818840579710145 and parameters: {'n_estimators': 444, 'num_leaves': 160, 'max_depth': 21, 'learning_rate': 0.005502489916219746, 'min_split_gain': 0.0033539334868087044, 'feature_fraction': 0.47455946267778615, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6855072463768116\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5462091488511102, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5462091488511102\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.769927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5462091488511102, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5462091488511102\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7144927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5462091488511102, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5462091488511102\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6202898550724638\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5462091488511102, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5462091488511102\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5952898550724638\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5462091488511102, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5462091488511102\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7039855072463768\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:11,175]\u001b[0m Trial 39 finished with value: 0.6807971014492754 and parameters: {'n_estimators': 510, 'num_leaves': 90, 'max_depth': 61, 'learning_rate': 0.019768904183680755, 'min_split_gain': 0.004745067432406305, 'feature_fraction': 0.5462091488511102, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.665016062738184, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.665016062738184\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7666666666666667\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.665016062738184, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.665016062738184\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7239130434782608\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.665016062738184, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.665016062738184\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6641304347826087\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.665016062738184, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.665016062738184\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:11,714]\u001b[0m Trial 40 finished with value: 0.6947826086956521 and parameters: {'n_estimators': 336, 'num_leaves': 25, 'max_depth': 47, 'learning_rate': 0.04580558958445308, 'min_split_gain': 0.008890583410756754, 'feature_fraction': 0.665016062738184, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6192028985507246\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.665016062738184, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.665016062738184\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.41993977037019575, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.41993977037019575\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7840579710144928\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.41993977037019575, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.41993977037019575\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7130434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.41993977037019575, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.41993977037019575\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6141304347826086\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.41993977037019575, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.41993977037019575\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6217391304347826\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.41993977037019575, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.41993977037019575\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6713768115942029\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:12,284]\u001b[0m Trial 41 finished with value: 0.6808695652173913 and parameters: {'n_estimators': 432, 'num_leaves': 120, 'max_depth': 127, 'learning_rate': 0.0522773244543474, 'min_split_gain': 0.0013232655367245432, 'feature_fraction': 0.41993977037019575, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1960575311763729, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1960575311763729\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7728260869565218\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1960575311763729, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1960575311763729\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7119565217391304\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1960575311763729, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1960575311763729\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.586231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1960575311763729, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1960575311763729\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.597463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1960575311763729, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1960575311763729\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:12,761]\u001b[0m Trial 42 finished with value: 0.6679710144927536 and parameters: {'n_estimators': 357, 'num_leaves': 68, 'max_depth': 20, 'learning_rate': 0.07821321788279112, 'min_split_gain': 0.0029153872916464285, 'feature_fraction': 0.1960575311763729, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6713768115942029\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33385179533298265, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33385179533298265\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7710144927536232\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33385179533298265, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33385179533298265\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7188405797101449\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33385179533298265, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33385179533298265\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6463768115942029\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33385179533298265, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33385179533298265\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6195652173913043\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33385179533298265, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33385179533298265\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7206521739130436\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:13,332]\u001b[0m Trial 43 finished with value: 0.6952898550724639 and parameters: {'n_estimators': 561, 'num_leaves': 97, 'max_depth': 26, 'learning_rate': 0.03580551547540386, 'min_split_gain': 0.00414721199268062, 'feature_fraction': 0.33385179533298265, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.37544792834117346, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.37544792834117346\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.744927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.37544792834117346, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.37544792834117346\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7057971014492753\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.37544792834117346, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.37544792834117346\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5838768115942029\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.37544792834117346, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.37544792834117346\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.667391304347826\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.37544792834117346, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.37544792834117346\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6978260869565218\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:13,758]\u001b[0m Trial 44 finished with value: 0.679963768115942 and parameters: {'n_estimators': 311, 'num_leaves': 22, 'max_depth': 6, 'learning_rate': 0.13084295701593734, 'min_split_gain': 0.006096582067866867, 'feature_fraction': 0.37544792834117346, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2580937513316698, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2580937513316698\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7923913043478261\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2580937513316698, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2580937513316698\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6931159420289856\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2580937513316698, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2580937513316698\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.605072463768116\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2580937513316698, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2580937513316698\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.586231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2580937513316698, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2580937513316698\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6960144927536231\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:14,361]\u001b[0m Trial 45 finished with value: 0.6745652173913044 and parameters: {'n_estimators': 402, 'num_leaves': 61, 'max_depth': 14, 'learning_rate': 0.024347797982058415, 'min_split_gain': 0.03813575396172622, 'feature_fraction': 0.2580937513316698, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4147412309510684, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4147412309510684\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7416666666666667\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4147412309510684, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4147412309510684\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7217391304347827\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4147412309510684, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4147412309510684\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6413043478260869\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4147412309510684, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4147412309510684\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6239130434782608\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4147412309510684, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4147412309510684\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:15,684]\u001b[0m Trial 46 finished with value: 0.6777536231884058 and parameters: {'n_estimators': 618, 'num_leaves': 139, 'max_depth': 32, 'learning_rate': 0.06034582450480878, 'min_split_gain': 0.051825255794841936, 'feature_fraction': 0.4147412309510684, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6601449275362319\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7458663626885033, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7458663626885033\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7539855072463768\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7458663626885033, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7458663626885033\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7384057971014493\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7458663626885033, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7458663626885033\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6353260869565218\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7458663626885033, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7458663626885033\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:16,257]\u001b[0m Trial 47 finished with value: 0.6892028985507246 and parameters: {'n_estimators': 720, 'num_leaves': 387, 'max_depth': 103, 'learning_rate': 0.08140531046554514, 'min_split_gain': 0.002000602465946932, 'feature_fraction': 0.7458663626885033, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6215579710144928\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7458663626885033, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7458663626885033\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6967391304347826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5550577846607748, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5550577846607748\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.7615942028985507\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5550577846607748, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5550577846607748\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.7119565217391304\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5550577846607748, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5550577846607748\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.6315217391304349\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5550577846607748, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5550577846607748\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.6025362318840579\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5550577846607748, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5550577846607748\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).\n",
      "0.6894927536231883\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:16,840]\u001b[0m Trial 48 finished with value: 0.6794202898550725 and parameters: {'n_estimators': 148, 'num_leaves': 31, 'max_depth': 22, 'learning_rate': 0.03163726021494225, 'min_split_gain': 0.023585772555049137, 'feature_fraction': 0.5550577846607748, 'bagging_freq': 5}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.501604395946151, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.501604395946151\n",
      "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n",
      "0.7742753623188405\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.501604395946151, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.501604395946151\n",
      "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n",
      "0.719927536231884\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.501604395946151, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.501604395946151\n",
      "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n",
      "0.6322463768115941\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.501604395946151, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.501604395946151\n",
      "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n",
      "0.6047101449275363\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.501604395946151, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.501604395946151\n",
      "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:17,636]\u001b[0m Trial 49 finished with value: 0.6834782608695653 and parameters: {'n_estimators': 237, 'num_leaves': 248, 'max_depth': 39, 'learning_rate': 0.01243392308433927, 'min_split_gain': 0.013020736747617516, 'feature_fraction': 0.501604395946151, 'bagging_freq': 1}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6862318840579711\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.317333710793954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.317333710793954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7347826086956522\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.317333710793954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.317333710793954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7539855072463768\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.317333710793954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.317333710793954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6528985507246376\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.317333710793954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.317333710793954\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5942028985507246\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.317333710793954, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.317333710793954\n",
      "[LightGBM] [Warning] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:18,146]\u001b[0m Trial 50 finished with value: 0.6793115942028984 and parameters: {'n_estimators': 496, 'num_leaves': 2, 'max_depth': 9, 'learning_rate': 0.05660447215708559, 'min_split_gain': 0.018365229567314754, 'feature_fraction': 0.317333710793954, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6606884057971014\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.36431118671703655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.36431118671703655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.772463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.36431118671703655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.36431118671703655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7362318840579709\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.36431118671703655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.36431118671703655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5809782608695653\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.36431118671703655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.36431118671703655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6257246376811595\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.36431118671703655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.36431118671703655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6894927536231884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:19,128]\u001b[0m Trial 51 finished with value: 0.6809782608695653 and parameters: {'n_estimators': 919, 'num_leaves': 220, 'max_depth': 43, 'learning_rate': 0.11900859359780942, 'min_split_gain': 0.027407488427334772, 'feature_fraction': 0.36431118671703655, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2857495520155243, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2857495520155243\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7757246376811594\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2857495520155243, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2857495520155243\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7181159420289855\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2857495520155243, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2857495520155243\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6159420289855072\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2857495520155243, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2857495520155243\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6289855072463768\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2857495520155243, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2857495520155243\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6717391304347826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:19,713]\u001b[0m Trial 52 finished with value: 0.6821014492753623 and parameters: {'n_estimators': 210, 'num_leaves': 489, 'max_depth': 33, 'learning_rate': 0.06691883056994974, 'min_split_gain': 0.01892460574590184, 'feature_fraction': 0.2857495520155243, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3555444158836644, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3555444158836644\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7663043478260869\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3555444158836644, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3555444158836644\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7380434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3555444158836644, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3555444158836644\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6572463768115943\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3555444158836644, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3555444158836644\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6456521739130435\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3555444158836644, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3555444158836644\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6992753623188406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:20,351]\u001b[0m Trial 53 finished with value: 0.7013043478260871 and parameters: {'n_estimators': 866, 'num_leaves': 176, 'max_depth': 28, 'learning_rate': 0.04026433979385358, 'min_split_gain': 0.021638496419626677, 'feature_fraction': 0.3555444158836644, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22792415394435664, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22792415394435664\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7318840579710144\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22792415394435664, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22792415394435664\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7264492753623188\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22792415394435664, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22792415394435664\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.618840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22792415394435664, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22792415394435664\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:20,912]\u001b[0m Trial 54 finished with value: 0.6698913043478261 and parameters: {'n_estimators': 737, 'num_leaves': 174, 'max_depth': 18, 'learning_rate': 0.0407023606096809, 'min_split_gain': 0.03367887921606027, 'feature_fraction': 0.22792415394435664, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6128623188405797\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.22792415394435664, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.22792415394435664\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6594202898550724\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44281526540977717, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44281526540977717\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7800724637681159\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44281526540977717, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44281526540977717\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7358695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44281526540977717, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44281526540977717\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6394927536231885\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44281526540977717, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44281526540977717\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5978260869565217\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.44281526540977717, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.44281526540977717\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6829710144927537\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:21,555]\u001b[0m Trial 55 finished with value: 0.6872463768115942 and parameters: {'n_estimators': 534, 'num_leaves': 130, 'max_depth': 28, 'learning_rate': 0.034733501301547655, 'min_split_gain': 0.007389139193511516, 'feature_fraction': 0.44281526540977717, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35287490335349225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35287490335349225\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.756159420289855\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35287490335349225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35287490335349225\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7202898550724637\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35287490335349225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35287490335349225\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6043478260869565\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35287490335349225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35287490335349225\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.5797101449275361\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.35287490335349225, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.35287490335349225\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7050724637681159\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:22,023]\u001b[0m Trial 56 finished with value: 0.6731159420289854 and parameters: {'n_estimators': 28, 'num_leaves': 96, 'max_depth': 86, 'learning_rate': 0.0225872705600354, 'min_split_gain': 0.010427128675494552, 'feature_fraction': 0.35287490335349225, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3888522608114387, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3888522608114387\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7576086956521739\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3888522608114387, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3888522608114387\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7373188405797102\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3888522608114387, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3888522608114387\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6483695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3888522608114387, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3888522608114387\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.5766304347826087\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3888522608114387, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3888522608114387\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:22,531]\u001b[0m Trial 57 finished with value: 0.6760144927536231 and parameters: {'n_estimators': 411, 'num_leaves': 56, 'max_depth': 2, 'learning_rate': 0.04868865656140099, 'min_split_gain': 0.015234360598756663, 'feature_fraction': 0.3888522608114387, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6601449275362319\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.46497743734029084, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.46497743734029084\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7797101449275363\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.46497743734029084, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.46497743734029084\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.743840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.46497743734029084, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.46497743734029084\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6420289855072464\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.46497743734029084, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.46497743734029084\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6173913043478261\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.46497743734029084, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.46497743734029084\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6847826086956521\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:23,233]\u001b[0m Trial 58 finished with value: 0.6935507246376812 and parameters: {'n_estimators': 875, 'num_leaves': 267, 'max_depth': 24, 'learning_rate': 0.02696767918379265, 'min_split_gain': 0.005320821785683025, 'feature_fraction': 0.46497743734029084, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4351842538287887, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4351842538287887\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7757246376811594\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4351842538287887, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4351842538287887\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7307971014492753\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4351842538287887, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4351842538287887\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.6163043478260869\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4351842538287887, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4351842538287887\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.5931159420289854\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4351842538287887, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4351842538287887\n",
      "[LightGBM] [Warning] bagging_freq is set=4, subsample_freq=0 will be ignored. Current value: bagging_freq=4\n",
      "0.7014492753623188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:23,926]\u001b[0m Trial 59 finished with value: 0.6834782608695653 and parameters: {'n_estimators': 330, 'num_leaves': 111, 'max_depth': 36, 'learning_rate': 0.019158730741304957, 'min_split_gain': 0.022539679002088454, 'feature_fraction': 0.4351842538287887, 'bagging_freq': 4}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.29114772302238645, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.29114772302238645\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.8202898550724638\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.29114772302238645, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.29114772302238645\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7155797101449275\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.29114772302238645, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.29114772302238645\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6148550724637681\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.29114772302238645, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.29114772302238645\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:24,475]\u001b[0m Trial 60 finished with value: 0.6941304347826086 and parameters: {'n_estimators': 272, 'num_leaves': 178, 'max_depth': 16, 'learning_rate': 0.08592879634410566, 'min_split_gain': 0.003959405681296534, 'feature_fraction': 0.29114772302238645, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6373188405797101\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.29114772302238645, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.29114772302238645\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6826086956521739\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33598956814484826, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33598956814484826\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7855072463768116\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33598956814484826, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33598956814484826\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7130434782608696\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33598956814484826, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33598956814484826\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.633695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33598956814484826, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33598956814484826\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:25,026]\u001b[0m Trial 61 finished with value: 0.6848188405797102 and parameters: {'n_estimators': 961, 'num_leaves': 237, 'max_depth': 29, 'learning_rate': 0.06572729766509484, 'min_split_gain': 0.020356592475747542, 'feature_fraction': 0.33598956814484826, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6224637681159421\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.33598956814484826, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.33598956814484826\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6693840579710145\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3730079929974963, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3730079929974963\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7597826086956522\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3730079929974963, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3730079929974963\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7336956521739131\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3730079929974963, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3730079929974963\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.661231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3730079929974963, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3730079929974963\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:25,569]\u001b[0m Trial 62 finished with value: 0.695072463768116 and parameters: {'n_estimators': 826, 'num_leaves': 207, 'max_depth': 52, 'learning_rate': 0.05485031019636577, 'min_split_gain': 0.04017514995004878, 'feature_fraction': 0.3730079929974963, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6293478260869566\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3730079929974963, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3730079929974963\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.691304347826087\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.23286942888400658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.23286942888400658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7452898550724638\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.23286942888400658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.23286942888400658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.722463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.23286942888400658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.23286942888400658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.618840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.23286942888400658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.23286942888400658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:26,126]\u001b[0m Trial 63 finished with value: 0.6717753623188406 and parameters: {'n_estimators': 908, 'num_leaves': 273, 'max_depth': 40, 'learning_rate': 0.04073338016369715, 'min_split_gain': 0.03195248501654857, 'feature_fraction': 0.23286942888400658, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6128623188405797\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.23286942888400658, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.23286942888400658\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6594202898550724\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39587330929823655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39587330929823655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.786231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39587330929823655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39587330929823655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7307971014492753\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39587330929823655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39587330929823655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.633695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39587330929823655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39587330929823655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.635144927536232\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.39587330929823655, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.39587330929823655\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:26,709]\u001b[0m Trial 64 finished with value: 0.6971739130434782 and parameters: {'n_estimators': 1000, 'num_leaves': 193, 'max_depth': 35, 'learning_rate': 0.03518831245765867, 'min_split_gain': 0.05949627867652389, 'feature_fraction': 0.39587330929823655, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1475554426676829, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1475554426676829\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7869565217391304\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1475554426676829, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1475554426676829\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7333333333333334\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1475554426676829, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1475554426676829\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.618840579710145\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1475554426676829, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1475554426676829\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:27,248]\u001b[0m Trial 65 finished with value: 0.6853623188405797 and parameters: {'n_estimators': 871, 'num_leaves': 156, 'max_depth': 43, 'learning_rate': 0.047343311411381116, 'min_split_gain': 0.013087486493389281, 'feature_fraction': 0.1475554426676829, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6307971014492755\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.1475554426676829, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.1475554426676829\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6568840579710145\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3047646431179061, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3047646431179061\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7826086956521738\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3047646431179061, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3047646431179061\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7036231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3047646431179061, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3047646431179061\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.5969202898550725\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3047646431179061, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3047646431179061\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6739130434782608\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:27,770]\u001b[0m Trial 66 finished with value: 0.6908333333333332 and parameters: {'n_estimators': 785, 'num_leaves': 83, 'max_depth': 13, 'learning_rate': 0.0693134420618684, 'min_split_gain': 0.00913712110094735, 'feature_fraction': 0.3047646431179061, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3047646431179061, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3047646431179061\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6971014492753623\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4767248835354545, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4767248835354545\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7916666666666666\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4767248835354545, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4767248835354545\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.7344202898550725\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4767248835354545, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4767248835354545\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.633695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4767248835354545, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4767248835354545\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n",
      "0.6396739130434783\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.4767248835354545, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.4767248835354545\n",
      "[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:28,277]\u001b[0m Trial 67 finished with value: 0.6967753623188406 and parameters: {'n_estimators': 377, 'num_leaves': 136, 'max_depth': 23, 'learning_rate': 0.08797595243508512, 'min_split_gain': 0.07677762243493089, 'feature_fraction': 0.4767248835354545, 'bagging_freq': 7}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6844202898550725\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3513000710205091, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3513000710205091\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7793478260869565\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3513000710205091, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3513000710205091\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.7072463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3513000710205091, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3513000710205091\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6210144927536232\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3513000710205091, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3513000710205091\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:29,000]\u001b[0m Trial 68 finished with value: 0.6781884057971015 and parameters: {'n_estimators': 486, 'num_leaves': 40, 'max_depth': 49, 'learning_rate': 0.016893813485540154, 'min_split_gain': 0.0167006522841689, 'feature_fraction': 0.3513000710205091, 'bagging_freq': 6}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5931159420289855\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3513000710205091, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3513000710205091\n",
      "[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6\n",
      "0.6902173913043479\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2757438346687468, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2757438346687468\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7942028985507246\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2757438346687468, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2757438346687468\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.717391304347826\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2757438346687468, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2757438346687468\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6202898550724637\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2757438346687468, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2757438346687468\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.586231884057971\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.2757438346687468, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.2757438346687468\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6797101449275362\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:29,996]\u001b[0m Trial 69 finished with value: 0.6795652173913043 and parameters: {'n_estimators': 163, 'num_leaves': 220, 'max_depth': 6, 'learning_rate': 0.011943304017520797, 'min_split_gain': 0.007148723564115653, 'feature_fraction': 0.2757438346687468, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.414348798293691, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.414348798293691\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.8137681159420289\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.414348798293691, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.414348798293691\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.7141304347826087\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.414348798293691, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.414348798293691\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.6231884057971014\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.414348798293691, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.414348798293691\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "0.5909420289855072\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.414348798293691, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.414348798293691\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:30,503]\u001b[0m Trial 70 finished with value: 0.6778623188405797 and parameters: {'n_estimators': 949, 'num_leaves': 76, 'max_depth': 32, 'learning_rate': 0.11257126920142455, 'min_split_gain': 0.030384470710198737, 'feature_fraction': 0.414348798293691, 'bagging_freq': 8}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6472826086956521\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3934559901469813, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3934559901469813\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7797101449275363\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3934559901469813, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3934559901469813\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7358695652173913\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3934559901469813, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3934559901469813\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.647463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3934559901469813, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3934559901469813\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6373188405797102\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3934559901469813, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3934559901469813\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7097826086956521\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:31,107]\u001b[0m Trial 71 finished with value: 0.7020289855072465 and parameters: {'n_estimators': 986, 'num_leaves': 202, 'max_depth': 36, 'learning_rate': 0.032603097286425776, 'min_split_gain': 0.06695907269268744, 'feature_fraction': 0.3934559901469813, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5021645970523259, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5021645970523259\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7764492753623189\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5021645970523259, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5021645970523259\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7340579710144928\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5021645970523259, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5021645970523259\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6481884057971015\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5021645970523259, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5021645970523259\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6105072463768116\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.5021645970523259, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.5021645970523259\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6920289855072463\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:31,697]\u001b[0m Trial 72 finished with value: 0.6922463768115941 and parameters: {'n_estimators': 993, 'num_leaves': 174, 'max_depth': 18, 'learning_rate': 0.04033032116139169, 'min_split_gain': 0.053123330316614946, 'feature_fraction': 0.5021645970523259, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n",
      "/opt/conda/lib/python3.7/site-packages/optuna/distributions.py:565: UserWarning: The distribution is specified by [0.1, 10] and step=1, but the range is not divisible by `step`. It will be replaced by [0.1, 9.1].\n",
      "  low=low, old_high=old_high, high=high, step=step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.43588322163048976, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.43588322163048976\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7634057971014493\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.43588322163048976, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.43588322163048976\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7427536231884058\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.43588322163048976, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.43588322163048976\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.6195652173913043\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.43588322163048976, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.43588322163048976\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.597463768115942\n",
      "[LightGBM] [Warning] Unknown parameter: categorical_features\n",
      "[LightGBM] [Warning] feature_fraction is set=0.43588322163048976, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.43588322163048976\n",
      "[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9\n",
      "0.7003623188405798\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-04-12 13:21:32,312]\u001b[0m Trial 73 finished with value: 0.6847101449275362 and parameters: {'n_estimators': 917, 'num_leaves': 222, 'max_depth': 27, 'learning_rate': 0.031889210119505206, 'min_split_gain': 0.08624590232834174, 'feature_fraction': 0.43588322163048976, 'bagging_freq': 9}. Best is trial 21 with value: 0.7090579710144927.\u001b[0m\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-126-680830e85fdc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mstudy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moptuna\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_study\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirection\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'maximize'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpruner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptuna\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpruners\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMedianPruner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_warmup_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mstudy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobjective\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;32mNone\u001b[0m \u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/optuna/study.py\u001b[0m in \u001b[0;36moptimize\u001b[0;34m(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m    392\u001b[0m             \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    393\u001b[0m             \u001b[0mgc_after_trial\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgc_after_trial\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 394\u001b[0;31m             \u001b[0mshow_progress_bar\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshow_progress_bar\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    395\u001b[0m         )\n\u001b[1;32m    396\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/optuna/_optimize.py\u001b[0m in \u001b[0;36m_optimize\u001b[0;34m(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m     74\u001b[0m                 \u001b[0mreseed_sampler_rng\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m                 \u001b[0mtime_start\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 76\u001b[0;31m                 \u001b[0mprogress_bar\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprogress_bar\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     77\u001b[0m             )\n\u001b[1;32m     78\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/optuna/_optimize.py\u001b[0m in \u001b[0;36m_optimize_sequential\u001b[0;34m(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)\u001b[0m\n\u001b[1;32m    161\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 163\u001b[0;31m             \u001b[0mtrial\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_run_trial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstudy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    164\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m             \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/optuna/_optimize.py\u001b[0m in \u001b[0;36m_run_trial\u001b[0;34m(study, func, catch)\u001b[0m\n\u001b[1;32m    215\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    216\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 217\u001b[0;31m         \u001b[0mvalue_or_values\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    218\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTrialPruned\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m         \u001b[0;31m# TODO(mamu): Handle multi-objective cases.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-123-c111a157056e>\u001b[0m in \u001b[0;36mobjective\u001b[0;34m(trial)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mobjective\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mTrial\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mgc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     \u001b[0mmodels\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0mvalidScore\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "study = optuna.create_study(direction='maximize', pruner=optuna.pruners.MedianPruner(n_warmup_steps=5))\n",
    "study.optimize(objective,None , timeout=60*60*2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we selected the best feature space from the above optuna method and used it for final training\n",
    "\n",
    "def lgb_f1_score(y_true, y_hat):\n",
    "\n",
    "    y_hat= np.where(y_hat < 0.5, 0, 1)   # scikits f1 doesn't like probabilities\n",
    "    return f1_score(y_true, y_hat, average = \"micro\")\n",
    "\n",
    "opt_params =  {'n_estimators': 876, 'num_leaves': 70, 'scale_pos_weight': 15, 'max_depth': 50, 'learning_rate': 0.07767353478949224, 'min_split_gain': 0.00870738026281286, 'feature_fraction': 0.3561368369391315, 'bagging_freq': 0}\n",
    "model = LGBMClassifier(**opt_params, random_state=44, metric = \"auc\")\n",
    "\n",
    "stkfold = StratifiedKFold(n_splits=5, shuffle=True, random_state = 44)\n",
    "\n",
    "def calc(X, y, model, cv):\n",
    "    res=[]\n",
    "    local_probs=pd.DataFrame()\n",
    "    probs = pd.DataFrame()\n",
    "\n",
    "    for i, (tdx, vdx) in tn(enumerate(cv.split(X, y))):\n",
    "        X_train, X_valid, y_train, y_valid = X.iloc[tdx], X.iloc[vdx], y[tdx], y[vdx]\n",
    "      #  X_train, y_train = smote.fit_resample(X_train, y_train)\n",
    "        model.fit(X_train, y_train,\n",
    "                 eval_set=[(X_train, y_train), (X_valid, y_valid)],\n",
    "                 #eval_metric = evaluate_macroF1_lgb,\n",
    "                 early_stopping_rounds=30, verbose=0)\n",
    "        \n",
    "        preds = model.predict_proba(X_valid)\n",
    "       # preds1 = np.array(list(map(lambda x: 1 if x>0.5 else x, preds[:,1])))\n",
    "       # print(preds1.shape)\n",
    "        oof_predict = model.predict_proba(x_test)\n",
    "        local_probs['fold_%i'%i] = oof_predict[:,1]\n",
    "        # may differ\n",
    "        false_positive_rate, true_positive_rate, thresholds = roc_curve(y_valid, preds[:,1])\n",
    "        # may differ\n",
    "        print (auc(false_positive_rate, true_positive_rate))\n",
    "        res.append(auc(false_positive_rate, true_positive_rate))\n",
    "        #print (lgb_f1_score(y_valid, preds[:,1]))\n",
    "        #res.append(lgb_f1_score(y_valid, preds[:,1]))\n",
    "\n",
    "    print('f1_Score:', round(np.mean(res), 6))    \n",
    "    local_probs['res'] = local_probs.mean(axis=1)\n",
    "    probs['target'] = local_probs['res']\n",
    "    return probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75b23c554a4048c4a094cda28c45ecc0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] feature_fraction is set=0.3561368369391315, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3561368369391315\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7623188405797101\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3561368369391315, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3561368369391315\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.7427536231884058\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3561368369391315, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3561368369391315\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6413043478260869\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3561368369391315, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3561368369391315\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6054347826086957\n",
      "[LightGBM] [Warning] feature_fraction is set=0.3561368369391315, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.3561368369391315\n",
      "[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0\n",
      "0.6463768115942028\n",
      "f1_Score: 0.679638\n",
      "CPU times: user 535 ms, sys: 33.9 ms, total: 569 ms\n",
      "Wall time: 315 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "probs = calc(x_train, Y_train, model, stkfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(probs['target']>0.55)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPqklEQVR4nO3da4xc9XnH8e8THBSXJRgCWVnGytLigChOSD2lVJGqXUgqGlpwFYRANLIlJ6tWSRoprhS36Yv0pphGBEUqb9wQxZWSLIgmwgWRljpsUao6jR0uG4MSLjUtFjUNNTRLaVrTpy/2uNps1jtn7vs/fD/Syuc2/3ken5mfj8+csxOZiSSpPG8YdQGSpO4Y4JJUKANckgplgEtSoQxwSSrUmmE+2bnnnpsTExMDGfuVV17hjDPOGMjYw9aUXprSBzSnl6b0Aa+vXg4dOvSDzDxv6fKhBvjExAQHDx4cyNizs7NMTk4OZOxha0ovTekDmtNLU/qA11cvEfHscss9hSJJhTLAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUa6p2Y0mo1seu+Fdfv3HyC7W226daR3dcMZFw1n0fgklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQtX6dbEQcAX4IvAacyMxWRJwD3AlMAEeAGzLz+GDKlCQt1ckR+FRmXpaZrWp+F7A/MzcB+6t5SdKQ9HIK5TpgbzW9F9jaczWSpNrqBngCfxMRhyJiulo2npnPV9P/Coz3vTpJ0ilFZrbfKGJDZh6NiLcCDwAfBfZl5rpF2xzPzLOXeew0MA0wPj6+ZWZmpl+1/5j5+XnGxsYGMvawjbqXuaMv92Wc8bVw7NXOHrN5w1l9ee5Oteu5m17qGmbPi19b/drP3ehHz6N+n/RTu16mpqYOLTp9/f9qBfiPPSDiU8A88CFgMjOfj4j1wGxmXrTSY1utVh48eLCj56trdnaWycnJgYw9bKPupd33Q9a1c/MJbp3r7GtXR/X9kHW+E7PTXuoaZs+LX1v92s/d6EfPo36f9FO7XiJi2QBvewolIs6IiDNPTgO/DHwX2AdsqzbbBtzTedmSpG7VOaQYB74WESe3/3Jmfj0ivg3cFRE7gGeBGwZX5uiM4ihlkN+ALqk52gZ4Zj4DvHOZ5S8CVw2iKElSe96JKUmFMsAlqVCD+Vhd6tIor4wYlWH27OcrzeIRuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpELVDvCIOC0iHo6Ie6v5CyLiWxHxVETcGRGnD65MSdJSnRyBfwx4YtH8LcBtmXkhcBzY0c/CJEkrqxXgEXE+cA3w+Wo+gCuBu6tN9gJbB1CfJOkUIjPbbxRxN/Bp4Ezgd4DtwIHq6JuI2Ajcn5mXLvPYaWAaYHx8fMvMzEzfil9sfn6esbGxvo87d/Tlvo/ZzvhaOPbq0J+275rSBzSnl9XSx+YNZ/U8xqDe86PQrpepqalDmdlaunxNu4Ej4leBFzLzUERMdlpYZu4B9gC0Wq2cnOx4iFpmZ2cZxNjbd93X9zHb2bn5BLfOtd01q15T+oDm9LJa+jhy82TPYwzqPT8K3fZSZ0++G7g2It4HvAl4M/A5YF1ErMnME8D5wNGOn12S1LW258Az83cz8/zMnABuBL6RmTcDDwLXV5ttA+4ZWJWSpJ/Qy3XgnwA+HhFPAW8B7uhPSZKkOjo6GZaZs8BsNf0McHn/S5Ik1eGdmJJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhWob4BHxpoj4x4h4NCIOR8QfVMsviIhvRcRTEXFnRJw++HIlSSfVOQL/EXBlZr4TuAy4OiKuAG4BbsvMC4HjwI6BVSlJ+gltAzwXzFezb6x+ErgSuLtavhfYOogCJUnLi8xsv1HEacAh4ELgduAzwIHq6JuI2Ajcn5mXLvPYaWAaYHx8fMvMzEz/ql9kfn6esbGxvo87d/Tlvo/ZzvhaOPbq0J+275rSBzSnl9XSx+YNZ/U8xqDe86PQrpepqalDmdlaunxNncEz8zXgsohYB3wNuLhuYZm5B9gD0Gq1cnJysu5DOzI7O8sgxt6+676+j9nOzs0nuHWu1q5Z1ZrSBzSnl9XSx5GbJ3seY1Dv+VHotpeOrkLJzJeAB4FfBNZFxMlXwvnA0Y6fXZLUtTpXoZxXHXkTEWuB9wJPsBDk11ebbQPuGVCNkqRl1Pm/1Hpgb3Ue/A3AXZl5b0Q8DsxExB8DDwN3DLBOSdISbQM8Mx8D3rXM8meAywdRlCSpPe/ElKRCjf7jaEmvOxN9uLpr5+YTHV8ldmT3NT0/72riEbgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIV85Vq7b6CqZuvV5KkknkELkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSodoGeERsjIgHI+LxiDgcER+rlp8TEQ9ExJPVn2cPvlxJ0kl1jsBPADsz8xLgCuDDEXEJsAvYn5mbgP3VvCRpSNoGeGY+n5nfqaZ/CDwBbACuA/ZWm+0Ftg6oRknSMiIz628cMQE8BFwK/HNmrquWB3D85PySx0wD0wDj4+NbZmZmuip07ujLK64fXwvHXu1q6FWnKb00pQ9oTi9N6QO662XzhrMGU0yP5ufnGRsbO+X6qampQ5nZWrq8doBHxBjwd8CfZOZXI+KlxYEdEcczc8Xz4K1WKw8ePFjr+Zaq87tQbp0r5le7rKgpvTSlD2hOL03pA7rr5cjuawZUTW9mZ2eZnJw85fqIWDbAa12FEhFvBP4S+FJmfrVafCwi1lfr1wMvdFq0JKl7da5CCeAO4InM/OyiVfuAbdX0NuCe/pcnSTqVOv//eDfwAWAuIh6plv0esBu4KyJ2AM8CNwykQknSstoGeGZ+E4hTrL6qv+VIkuryTkxJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEK1DfCI+EJEvBAR31207JyIeCAinqz+PHuwZUqSlqpzBP5F4Ooly3YB+zNzE7C/mpckDVHbAM/Mh4B/X7L4OmBvNb0X2NrfsiRJ7URmtt8oYgK4NzMvreZfysx11XQAx0/OL/PYaWAaYHx8fMvMzExXhc4dfXnF9eNr4dirXQ296jSll6b0Ac3ppSl9QFm9bN5w1orr5+fnGRsbO+X6qampQ5nZWrp8Ta+FZWZGxCn/FcjMPcAegFarlZOTk109z/Zd9624fufmE9w613M7q0JTemlKH9CcXprSB5TVy5GbJ1dcPzs7SzfZ2O1VKMciYj1A9ecLXY4jSepStwG+D9hWTW8D7ulPOZKkuupcRvgV4B+AiyLiuYjYAewG3hsRTwLvqeYlSUPU9gRSZt50ilVX9bkWSVIHvBNTkgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQPQV4RFwdEd+LiKciYle/ipIktdd1gEfEacDtwK8AlwA3RcQl/SpMkrSyXo7ALweeysxnMvO/gRnguv6UJUlqJzKzuwdGXA9cnZkfrOY/APxCZn5kyXbTwHQ1exHwve7LXdG5wA8GNPawNaWXpvQBzemlKX3A66uXt2XmeUsXrhlcPQsycw+wZ9DPExEHM7M16OcZhqb00pQ+oDm9NKUPsBfo7RTKUWDjovnzq2WSpCHoJcC/DWyKiAsi4nTgRmBff8qSJLXT9SmUzDwRER8B/ho4DfhCZh7uW2WdG/hpmiFqSi9N6QOa00tT+gB76f5DTEnSaHknpiQVygCXpEIVF+Dtbt+PiF+KiO9ExInqWvVVqUYfH4+IxyPisYjYHxFvG0WdddTo5TcjYi4iHomIb67mO3br/nqIiHh/RGRErMrL2Grsk+0R8W/VPnkkIj44ijrrqLNPIuKG6v1yOCK+POwa66ixT25btD++HxEvtR00M4v5YeHD0qeBnwZOBx4FLlmyzQTwDuAvgOtHXXMPfUwBP1VN/xZw56jr7qGXNy+avhb4+qjr7raXarszgYeAA0Br1HV3uU+2A3826lr71Msm4GHg7Gr+raOuu9vX1qLtP8rChSErjlvaEXjb2/cz80hmPgb87ygKrKlOHw9m5n9WswdYuM5+NarTy38smj0DWK2fnNf99RB/BNwC/Ncwi+tAk37NRZ1ePgTcnpnHATLzhSHXWEen++Qm4CvtBi0twDcA/7Jo/rlqWWk67WMHcP9AK+perV4i4sMR8TTwp8BvD6m2TrXtJSJ+DtiYmfcNs7AO1X19vb86RXd3RGxcZv1qUKeXtwNvj4i/j4gDEXH10Kqrr/Z7vjpdegHwjXaDlhbgrzsR8RtAC/jMqGvpRWbenpk/A3wC+P1R19ONiHgD8Flg56hr6YO/AiYy8x3AA8DeEdfTizUsnEaZZOHI9c8jYt0oC+rRjcDdmflauw1LC/Cm3L5fq4+IeA/wSeDazPzRkGrrVKf7ZAbYOsiCetCulzOBS4HZiDgCXAHsW4UfZLbdJ5n54qLX1OeBLUOqrVN1Xl/PAfsy838y85+A77MQ6KtJJ++TG6lx+gQo7kPMNcAzLPz34uQHAT97im2/yOr9ELNtH8C7WPjQY9Oo6+1DL5sWTf8acHDUdff6+qq2n2V1fohZZ5+sXzT968CBUdfdQy9XA3ur6XNZOFXxllHX3s1rC7gYOEJ1k2XbcUfdWBd/Ee9j4V/Yp4FPVsv+kIWjVICfZ+Ff5FeAF4HDo665yz7+FjgGPFL97Bt1zT308jngcNXHgyuF4qh/2vWyZNtVGeA198mnq33yaLVPLh51zT30Eiyc2nocmANuHHXN3b62gE8Bu+uO6a30klSo0s6BS5IqBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkq1P8BdGsXCegV7wAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "probs['target'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submission\n",
    "# Used 0.55 as threshold because we selected a higher penalty on positive weight false prediction, hence tried to \n",
    "# Compensate it\n",
    "submission = pd.DataFrame({\n",
    "        \"label\": np.where(probs['target']<0.55, 0,1),\n",
    "        \"id\": sample_Submission[\"id\"],\n",
    "        \n",
    "    })\n",
    "submission.to_csv('lg_pos_weight.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Oversampling used but not given trustworthy results\n",
    "# from imblearn.over_sampling import SMOTE\n",
    "# smote = SMOTE()\n",
    "# X_traino,  y_traino = smote.fit_resample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**XGBM also not gave good results, LGBM best out of all still not good to avoid this correlation problem**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from imblearn.over_sampling import SMOTE\n",
    "# smote = SMOTE()\n",
    "\n",
    "# import gc\n",
    "# import xgboost as xgb\n",
    "# oof = np.zeros(len(x_train))\n",
    "# preds = np.zeros(len(x_test))\n",
    "\n",
    "# skf = StratifiedKFold(n_splits=5, shuffle = True, random_state = 44)\n",
    "\n",
    "# for i, (idxT, idxV) in enumerate( skf.split(x_train, Y_train)):\n",
    "                                 \n",
    "#     print(' rows of train =',len(idxT),'rows of holdout =',len(idxV))\n",
    "        \n",
    "#     xgb_params = {\n",
    "#               'booster' : 'gbtree',\n",
    "#               'max_depth' : 25,\n",
    "#               'gamma' : 12,\n",
    "#               'eta' : 0.08,\n",
    "#               'objective' : 'binary:logistic',\n",
    "#                \"scale_pos_weight\":1,\n",
    "#               'eval_metric': 'auc',\n",
    "#               'subsample' : 0.8,\n",
    "#               'colsample_bytree' : 0.85,\n",
    "#               'min_child_weight' : 3,\n",
    "#               'max_delta_step' : 2,\n",
    "#               \"random_state\":42\n",
    "#               }\n",
    "\n",
    "                                 \n",
    "#     ################ Fitting ################               \n",
    "#   #  X_traino,  y_traino = smote.fit_resample(x_train.iloc[idxT], Y_train.iloc[idxT])\n",
    "#     train = xgb.DMatrix(data = x_train.iloc[idxT],label=Y_train.iloc[idxT], enable_categorical = True)\n",
    "#     valid = xgb.DMatrix(data=x_train.iloc[idxV],label=Y_train.iloc[idxV], enable_categorical = True)\n",
    "#     clf = xgb.train(xgb_params, dtrain=train,\n",
    "#         num_boost_round=5000,evals=[(valid,'valid')],\n",
    "#         early_stopping_rounds=200,maximize=True,\n",
    "#         verbose_eval=100)   \n",
    "    \n",
    "#     oof[idxV] += clf.predict(valid)                             \n",
    "#     ####################### Test data pred ##################                             \n",
    "#     test = xgb.DMatrix(data=x_test, enable_categorical = True)\n",
    "#     preds += clf.predict(test)/skf.n_splits\n",
    "#     del clf; x=gc.collect()\n",
    "                              \n",
    "# print('#'*20)\n",
    "# false_positive_rate, true_positive_rate, thresholds = roc_curve(Y_train.values,oof)\n",
    "# # may differ\n",
    "# print (auc(false_positive_rate, true_positive_rate))                                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What actually is Coorelation problem or say Train/Test Similarity problem\n",
    "\n",
    "We used the methid described in [this](https://www.kaggle.com/shikhar1/train-test-similarity) kernel and able to find actiually that the train and test data have good amount of dissimilarity in  their distributions which made the cross validation of train to fail on test data + the submission swap we saw on private leaderboard is not due to overfitting but lack of data similarity."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}