Getting-Started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "<hr>\n",
    "\n",
    "## An introduction to scikit-learn\n",
    "\n",
    "Since its release in 2007, scikit-learn has become one of the most popular open\n",
    "source machine learning libraries for Python. scikit-learn provides algorithms for\n",
    "machine learning tasks including classification, regression, dimensionality reduction,\n",
    "and clustering. It also provides modules for extracting features, processing data,\n",
    "and evaluating models.\n",
    "\n",
    "Conceived as an extension to the SciPy library, scikit-learn is built on the popular\n",
    "Python libraries NumPy and matplotlib. NumPy extends Python to support efficient\n",
    "operations on large arrays and multidimensional matrices. matplotlib provides\n",
    "visualization tools, and SciPy provides modules for scientific computing.\n",
    "scikit-learn is popular for academic research because it has a well-documented,\n",
    "easy-to-use, and versatile API. Developers can use scikit-learn to experiment with\n",
    "different algorithms by changing only a few lines of the code. scikit-learn wraps\n",
    "some popular implementations of machine learning algorithms, such as LIBSVM\n",
    "and LIBLINEAR. Other Python libraries, including NLTK, include wrappers for\n",
    "scikit-learn. scikit-learn also includes a variety of datasets, allowing developers to\n",
    "focus on algorithms rather than obtaining and cleaning data.\n",
    "\n",
    "Licensed under the permissive BSD license, scikit-learn can be used in commercial\n",
    "applications without restrictions. Many of scikit-learn's algorithms are fast and\n",
    "scalable to all but massive datasets. Finally, scikit-learn is noted for its reliability;\n",
    "much of the library is covered by automated tests.\n",
    "\n",
    "## Estimators\n",
    "\n",
    "Given a scikit-learn *estimator* object named `model`, the following methods are available:\n",
    "\n",
    "- Available in **all Estimators**\n",
    "  + `model.fit()` : fit training data. For supervised learning applications,\n",
    "    this accepts two arguments: the data `X` and the labels `y` (e.g. `model.fit(X, y)`).\n",
    "    For unsupervised learning applications, this accepts only a single argument,\n",
    "    the data `X` (e.g. `model.fit(X)`).\n",
    "- Available in **supervised estimators**\n",
    "  + `model.predict()` : given a trained model, predict the label of a new set of data.\n",
    "    This method accepts one argument, the new data `X_new` (e.g. `model.predict(X_new)`),\n",
    "    and returns the learned label for each object in the array.\n",
    "  + `model.predict_proba()` : For classification problems, some estimators also provide\n",
    "    this method, which returns the probability that a new observation has each categorical label.\n",
    "    In this case, the label with the highest probability is returned by `model.predict()`.\n",
    "  + `model.score()` : for classification or regression problems, most (all?) estimators implement\n",
    "    a score method.  Scores are between 0 and 1, with a larger score indicating a better fit.\n",
    "- Available in **unsupervised estimators**\n",
    "  + `model.predict()` : predict labels in clustering algorithms.\n",
    "  + `model.transform()` : given an unsupervised model, transform new data into the new basis.\n",
    "    This also accepts one argument `X_new`, and returns the new representation of the data based\n",
    "    on the unsupervised model.\n",
    "  + `model.fit_transform()` : some estimators implement this method,\n",
    "    which more efficiently performs a fit and a transform on the same input data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started\n",
    "Assuming that you have Python already installed (everything at least as recent as 2.7\n",
    "should be fine), we need to install NumPy and SciPy for numerical operations, as well\n",
    "as matplotlib for visualization.\n",
    "\n",
    "## Introduction to NumPy, SciPy, and matplotlib\n",
    "\n",
    "Before we can talk about concrete machine learning algorithms, we have to talk\n",
    "about how best to store the data we will chew through. This is important as the most\n",
    "advanced learning algorithm will not be of any help to us if it will never finish. This\n",
    "may be simply because accessing the data is too slow. Or maybe its representation\n",
    "forces the operating system to swap all day. Add to this that Python is an interpreted\n",
    "language (a highly optimized one, though) that is slow for many numerically\n",
    "heavy algorithms compared to C or FORTRAN. So we might ask why on earth so\n",
    "many scientists and companies are betting their fortune on Python even in highly\n",
    "computation-intensive areas?\n",
    "\n",
    "The answer is that, in Python, it is very easy to off-load number crunching tasks to\n",
    "the lower layer in the form of C or FORTRAN extensions. And that is exactly what\n",
    "NumPy and SciPy do (http://scipy.org/Download). In this tandem, NumPy\n",
    "provides the support of highly optimized multidimensional arrays, which are the\n",
    "basic data structure of most state-of-the-art algorithms. SciPy uses those arrays to\n",
    "provide a set of fast numerical recipes. Finally, matplotlib (http://matplotlib.\n",
    "org/) is probably the most convenient and feature-rich library to plot high-quality\n",
    "graphs using Python.\n",
    "\n",
    "### * Modules  : Numpy,Scipy,Matplotlib... *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our first machine learning method – \n",
    "\n",
    "# Supervised Learning: Classification\n",
    "\n",
    "To get a grip on the problem of machine learning in scikit-learn, we will start with a\n",
    "very simple machine learning problem: we will try to predict the Iris flower species\n",
    "using only two attributes: sepal width and sepal length. This is an instance of a\n",
    "classification problem, where we want to assign a label (a value taken from a discrete\n",
    "set) to an item according to its features.\n",
    "\n",
    "Let's first build our training dataset—a subset of the original sample, represented by\n",
    "the two attributes we selected and their respective target values. After importing the\n",
    "dataset, we will randomly select about 75 percent of the instances, and reserve the\n",
    "remaining ones (the evaluation dataset) for evaluation purposes (we will see later\n",
    "why we should always do that):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(112L, 2L) (112L,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Get dataset with only the first two attributes\n",
    "X, y = X_iris[:,:2], y_iris\n",
    "# Split the dataset into a trainig and a testing set\n",
    "# Test set will be the 25% taken randomly\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=33)\n",
    "print X_train.shape, y_train.shape\n",
    "# Standarize the features\n",
    "scaler = StandardScaler().fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train_test_split function automatically builds the training and evaluation\n",
    "datasets, randomly selecting the samples. Why not just select the first 112 examples?\n",
    "This is because it could happen that the instance ordering within the sample could\n",
    "matter and that the first instances could be different to the last ones. In fact, if you\n",
    "look at the Iris datasets, the instances are ordered by their target class, and this\n",
    "implies that the proportion of 0 and 1 classes will be higher in the new training set,\n",
    "compared with that of the original dataset. We always want our training data to be a\n",
    "representative sample of the population they represent.\n",
    "The last three lines of the previous code modify the training set in a process usually\n",
    "called feature scaling. For each feature, calculate the average, subtract the mean\n",
    "value from the feature value, and divide the result by their standard deviation. After\n",
    "scaling, each feature will have a zero average, with a standard deviation of one. This\n",
    "standardization of values (which does not change their distribution, as you could\n",
    "verify by plotting the X values before and after scaling) is a common requirement of\n",
    "machine learning methods, to avoid that features with large values may weight too\n",
    "much on the final results.\n",
    "\n",
    "Now, let's take a look at how our training instances are distributed in the twodimensional\n",
    "space generated by the learning feature. pyplot, from the matplotlib\n",
    "library, will help us with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xaf127f0>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEPCAYAAABP1MOPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VNX18PHvGkhgIAYICQgEJ0hAEq0K1oiCElSC2uKl\n1QpYrZXSooaL9fVW60/a+vSnWN8qtmpR0GolYL2htiAqBKu24gsqyK0USUQiEIqIl0BCst4/ZhiS\nkGQmmZmcMzPr8zznITNzLmtOJrPZZ6+ztqgqxhhjkpfH6QCMMcY4yxoCY4xJctYQGGNMkrOGwBhj\nkpw1BMYYk+SsITDGmCTnioZARDwislpEXnI6FmOMSTauaAiA6cB6p4Mwxphk5HhDICLZwAXAY07H\nYowxycjxhgD4PXATYLc4G2OMAxxtCETkO8BOVf0AkMBijDGmHYmTtYZE5LfAD4GDgBc4CnheVa9q\ntJ71Fowxpg1UNeR/sB3tEajqL1T1GFU9FhgPLGvcCNRb1xZV7rzzTsdjcMti58LOhZ2LlpdwuWGM\nwBhjjIM6Oh3AIaq6AljhdBzGGJNsrEcQZwoLC50OwTXsXBxm5+IwOxet5+hgcbhEROMhTmOMcRMR\nQcMYLHbNpSFjjPvl5ORQXl7udBimEZ/PR1lZWZu3tx6BMSZsgf9hOh2GaaS530u4PQIbIzDGmCRn\nDYExxiQ5awiMMSbJWUNgjDFJzhoCY0zSKi8vx+PxUFdX53QojrL0UWNMuygtLWX5smVk9erFNddc\nQ5cuXZwOCVW1TCisR2CMiYKamhr+8pe/8Lvf/Y533nnniNfnPfYYV37nO9TddRdv3Hwzo779baqq\nqqIexz333EN2djbp6enk5eWxfPlyVJW7776b3NxcsrKyGD9+PHv37gVg1KhRAHTv3p309HTeffdd\nVJW77rqLnJwcjj76aK6++mr27dsHwIEDB7jyyivJzMykR48enHbaaVRWVgLwxBNPkJ+fT3p6Orm5\nucyZMyfq7y9mnK6OF2YFPTXGOK+pv8WamhodO3KkntW1q85ISdG+Xq8++sgjDdbJTEvTNaAKWgda\n1LWrPvnkkw3WmfPww3pMz56amZam10+apAcOHGhVbJs2bdL+/fvrjh07VFW1vLxcP/74Y73//vv1\n9NNP14qKCq2urtYpU6bohAkTVFW1rKxMPR6P1tXVBfczd+5cHTRokJaVlenXX3+t3/ve9/Sqq65S\nVdU//elPeuGFF+r+/fu1rq5OV69erV9++aWqqv7973/XrVu3qqrqm2++qV26dNH333+/Ve+hrZr7\njgw8H/o7NpyVnF6sITDGHZr6W1y0aJGempamBwNf9BtBj+rcOfjlWldXp6kdOuiXgdcV9GedO+uD\nDz4Y3Mcrr7yivi5ddDXoNtAir1dvmjq1VbH95z//0d69e+vrr7+uNTU1wefz8vJ02bJlwccVFRWa\nkpKitbW1unXrVvV4PFpbWxt8/ZxzztGHH344+HjTpk2ampqqtbW1Om/ePB0xYoSuWbMmZDwXX3yx\nzp49u1Xvoa0ibQjs0pAxJiJ79uzhOFU6BB7nAvurq6murgb8d7eOGzOG4k6d2AYsBp7zeDjnnHOC\n+1iyaBHTv/mGoUA2cHdVFUteeqlVcQwcOJD777+fmTNn0qtXLyZOnMhnn31GeXk5l1xyCRkZGWRk\nZJCfn09KSgo7d+5E5MibbisqKvD5fMHHPp+Pmpoadu7cyZVXXsnYsWMZP3482dnZ3HrrrdTW1gKw\nePFiTj/9dHr27EmPHj1YvHgxu3fvbtV7cIo1BMaYiIwcOZIlqrwOfAHc1rEjZwwdSqdOnYLrzFu4\nkIMXXMBp3bpxS04OJYsWkZeXF3y9R1YWmzsezl3ZjP+6fWuNHz+ef/zjH3zyyScA3HLLLRxzzDEs\nXryYPXv2sGfPHj7//HO+/vpr+vTp02RD0Ldv3wb1lMrLy0lJSaF379507NiRO+64g3Xr1vHOO+/w\n8ssv8+STT1JdXc2ll17KzTffTGVlJZ9//jnnn39+3AxCW0NgjIlIbm4uT7/wAtf26UO/1FTWDh/O\nwr/9rcE66enp/OX556nYu5c1W7dy7rnnNni9ePp0lvTsyRWdO3NDx45c36ULd82e3ao4/v3vf7N8\n+XKqq6tJTU3F6/XSoUMHpkyZwi9+8Ytg41BZWclLgd5GVlYWHo+HLVu2BPczYcIEfv/731NWVsZX\nX33F7bffzvjx4/F4PJSWlvLRRx9RV1dHWloaKSkpdOjQgepADygzMxOPx8PixYtZunRpW06nIyx9\n1BgTsaKiIjZXVLR5+169evHeunU8/fTTVFVVsWLcOPLz81u1jwMHDnDrrbeyceNGUlJSOOOMM5gz\nZw69e/dGVSkqKuKzzz6jV69eXH755Vx44YV4vV5uv/12RowYwcGDB1myZAnXXHMNn332GWeddRYH\nDhzgvPPOY3agUdqxYwdTpkxh+/btpKWlMX78eH74wx/i8XiYPXs2l112GdXV1YwbN46LLrqozeej\nvVn1UWNM2Czn3p2s+qgxxpiIWENgjDFJzhoCY4xJctYQGGNMkrOGwBhjkpyj6aMi0gl4E0gNxPKs\nqv7KyZiMMSbZOJ4+KiJdVPUbEekAvA1MU9WVjdax9FFjXMDSR90p7tNHVfWbwI+d8PcK7FNmjDHt\nyPGGQEQ8IvI+sAN4TVXfczomY4xpzlFHHUVZWVlE+xgwYADLli2LTkBR4HiJCVWtA4aKSDrwoojk\nq+r6xuvNnDkz+HNhYSGFhYXtFqMxxhzy5ZdfOh1Cs0pLSyktLW31do6PEdQnIncAX6vq/230vI0R\nGOMCkYwRlJaWsmz5MnpluWeqyqbU1tbSoUOH0CtGYMCAAcydO5ezzz67Vds1F1tcjxGISKaIdAv8\n7AXGABudjMm4R2VlJe+9915wKkDjXqGmqpw77zEuv/JCNtQ9yvw3fsOIUQVRn6py1qxZXHbZZQ2e\nmz59OjNmzGDfvn1MmjSJvn370r9/f+64447gF+ef//xnRo4cyc9//nMyMzP51a9+xZYtWygsLKR7\n9+706tWLCRMmBPfp8Xj4+OOPAdi/fz833ngjOTk59OjRI1ioDuCll17ihBNOICMjg7PPPpuNG5v+\naquurmbGjBn069eP7OxsbrjhBmpqagBYsWIF/fv3Z9asWfTp04drrrkmqucsKJzZa2K1AN8CVgMf\nAGuA25tZL5xJekwCWTB/vmZ4vTqsWzfN8Hp1wfz5TodktPmpKs8Ze5aeeFaWjpvRR7P6HqV/erTh\nVJUZmen6f9cM1uf0JH227kQ9pajXEVNVPjLnYe13TJZmZB6lU67/SaunqiwvL9euXbvqV199paqq\ntbW12qdPH3333Xf1kksu0WuvvVarqqq0srJSTzvtNJ0zZ46qqj7xxBPasWNH/eMf/6i1tbVaVVWl\nEyZM0N/+9reqqnrgwAF9++23g8fxeDy6ZcsWVVW97rrrdPTo0frZZ59pXV2d/vOf/9Tq6mrdtGmT\ndu3aVd944w09ePCgzpo1S3Nzc4Mzp+Xk5Ogbb7yhqqp33HGHnn766bp7927dvXu3nnHGGfo///M/\nqqpaWlqqHTt21Ntuu02rq6t1//79Yf9e6j1vU1Wa+LNr1y7N8Hr1w8C0hh+CZni9umvXLqdDS3pN\n/S0uWrRIh5zaU585eKI+pyfp7I3HadejvA2mqkxJ7ah/+fIEfU5P0uf0JD3vZ32OmKryaF+63rt6\nkM7ZlqfDirL05zdNa3V8Z555pj711FOqqrp06VLNzc3VnTt3aqdOnRp8iZaUlOjo0aNV1d8Q+Hy+\nBvu56qqr9Gc/+5l++umnRxxDRHTLli1aV1enXq9X165de8Q6v/nNb/Tyyy8PPq6rq9N+/frpihUr\nVLVhQzBw4EBdsmRJcN1XX31VBwwYoKr+hqBTp05aXV3d4vuOtCFwPGvImMbKysrISU3lxMDjEwFf\nSkrEmRomNvbs2UPf4zrRoYP/UvTRuZ04sL/hVJUXjCtiXnElu7dVs3rxPt597ssGU1X+bclLnDc9\njWOHdqFndioT7u7O4iUvtzqWCRMmUFJSAkBJSQkTJ06kvLycmpoa+vTpQ0ZGBj169GDKlCkNppHs\n379/g/3ce++91NXVUVBQwLe+9S0ef/zxI461e/duDhw4wLHHHnvEa42nuxQR+vfvz/bt25tc95hj\njgk+9vl8VNSb2yErK4uUlJRWnIXWs4bAuE5OTg5l1dWsCTxeA5TX1JCTk+NgVKY5I0eO5P0l+/jw\n9S/5+ota5t9WyWlnDGswVeWT80rIOjiCO07bwfO3eHim5IUGU1X27JHJzs21wcefba6mW/cerY7l\nsssuo7S0lO3bt/PCCy9wxRVX0L9/fzp37sx///vf4FSVe/fuZc2aNcHtGk9Z2atXL+bMmcP27dt5\n5JFHuO6664LjAodkZmbSuXPnBrObHdJ4ukuAbdu2kZ2dHXLd8vJy+vbt22xsMRFOt8HpBbs0lHQO\njREMTU+3MQIXae5v8dVXX9Wc3H7apWsnPfe8s3THjh2t2u/OnTv1mAF9dNTE3jpuxtHaIzMteBml\ntc4//3wdM2aMDhs2LPjcxRdfrNOnT9d9+/ZpXV2dbtmyJbj/J554Qs8888wG+/jrX/8avCz00Ucf\naZcuXXTr1q2qevjSkKrq9ddfr+eee65WVFRobW1tgzGCtLQ0XbZsmdbU1Oi9996rAwcObHKM4Je/\n/KWOGDFCKysrtbKyUkeOHNlgjKB///4h33NzvxdsjMDEu127dunKlSttbMBFYvm3uHv3bn3ggQf0\n7rvv1nXr1rV5P0899ZR6PB697777gs/t27dPr732Ws3Oztbu3bvrsGHDdOHCharadENw8803a79+\n/fSoo47S3Nxcfeyxx4Kv1R8srqqq0htuuEH79eun3bt311GjRgXHIl588UXNz8/X7t27a2Fhoa5f\nvz64jwEDBgQbgv379+v06dO1T58+2rdvX50xY0ZwoLy9GgJX3UfQHLuPwBh3sFpD7hTX9xEYY4xx\nnjUExhiT5KwhMMaYJGcNgTHGJDlrCIwxJslZQ2CMMUnOGgITU1ZB1Bj3s4bAxMzCkhKG+HxMGTOG\nIT4fCwM1YIwx7mINgYmJyspKrps0ieVVVaz64guWV1Vx3aRJ1jMwjrjgggt46qmn2mX7SI/lBLuz\n2MTEe++9x5QxY1j1xRfB54alp/On11/n1FNPdTAyEwm7s9idIr2z2PE5i01iql9B9ESsgqgJTFW5\nbDm9emW5aqrK9pia0u3s0pCJiaysLB6aO5fRXi/D0tMZ7fXy0Ny5ZGVlOR2aiYFQU1U+9tg8vvOd\nK7nrrjpuvvkNvv3tUe0yVeWMGTOYPn06o0ePZt68eUDTU1PW1dVx4403kpWVxcCBA/njH/+Ix+Oh\nrq4O4IjtzzzzTG666SYyMjIYOHAgS5YsCR6z/roAjz76KPn5+aSnp3PCCSfwwQcfAHDPPfeQm5sb\nfP7FF1+M6vlolXAq0zm9YNVH45ZVEE0sTf0t1tTU6MiRY7Vr17M0JWWGer199ZFHHm2wTlpapsIa\n9U86V6dduxYdMVXlww/P0Z49j9G0tEydNOn6qE5VWVhYqHPnzlXVI6em3L9/vz788MN6/PHHa0VF\nhe7du1fPPfdc9Xg8Wltbq6p6xPapqak6d+5craur04cfflj79u0bjKP+us8884xmZ2frqlWrVFV1\ny5Yt+sknn6iq6rPPPhss1/3MM89o165dW12++5DmviOxMtQmHPZFbVqjqb/FRYsWaVraqQoHA1/0\nG7Vz56MaTFXZoUOqwpeB11U7d/7ZEVNVduniU1itsE293iKdOvWmVsfX1FSVqkd+kTeemvLss88O\nzmGsqvr666+32BAMGjQouO4333yjIqI7d+48Yt2xY8fq7Nmzw4r95JNP1pdeeqm1b1lVI28I7NJQ\nErP0ThMNe/bsQfU44NB19lyqq/c3mKpyzJhxdOpUDGwDFuPxPNdgqspFi5bwzTfTgaFANlVVd/PS\nS0torcZTVV5xxRVNrtd4asqKiooGzzV+vbGjjz46+LPX6wXgq6++OmK9bdu2MXDgwCb38eSTTzJ0\n6FB69OhBjx49WLduXYPpM9uTNQRJytI7TbSMHDkS1SXA68AXdOx4G0OHntFgqsqFC+dxwQUH6dbt\nNHJybmHRopIGU1VmZfWgY8fN9fa6me7du7c6lsZTVU6cOLHJ9RpP/9inTx8+/fTT4ONPPvmk1cdu\nSv/+/ZucyvKTTz7hpz/9KQ899BCff/45n3/+Occff7xjGVnWECQpmyDeREtubi4vvPA0ffpcS2pq\nP4YPX8vf/rawwTrp6ek8//xf2Lu3gq1b13Duuec2eH369GJ69lxC585X0LHjDXTpcj2zZ9/V6lgy\nMzMZNWoUP/7xjzn22GMZPHhwWNv94Ac/4IEHHqCiooK9e/cya9asVh+7KT/5yU/43e9+x+rVqwHY\nsmUL27Zt4+uvv8bj8ZCZmUldXR2PP/44H330UVSO2RaWPpqkLL3TRFNRUREVFZtDr9iMXr16sW7d\nezz99NNUVVUxbtwK8vPz27SviRMn8qMf/Yh77703+FyoCeAnT57M5s2bOfHEE+nWrRvTpk1jxYoV\neDyesLav/3r9ny+99FL27NnDxIkTqaioICcnh6eeeoqTTjqJG2+8keHDh9OhQweuuuoqRo4c2Za3\nGxWO3lAmItnAk0BvoA54VFVnN7GeOhlnolpYUsJ1kybhS0mhvKaGh+bO5fIJE5wOy7hYstxQtmTJ\nEq699lq2bt3qdChhifSGMqcbgqOBo1X1AxFJA1YBF6nqxkbrWUMQI5WVlf7LRDk5cZfjH8+xx6tE\nbQj279/P8uXLKSoqYseOHVx66aWcccYZ3HfffU6HFpa4bggaE5EXgQdV9Y1Gz1tDYBo41JvJSU2l\nrLraejPtJFEbgqqqKkaNGsWmTZvwer1897vf5f777yctLc3p0MKSMA2BiOQApcAJqvpVo9esITBB\nlZWVDPH5WF5VFRzfGO31srG83HoGMZaoDUG8S4haQ4HLQs8C0xs3AofMnDkz+HNhYSGFhYXtEptx\nn2DGU6BEQf2MJ2sITDIrLS2ltLS01ds53iMQkY7AK8BiVX2gmXWsR2CCrEfgHOsRuFOkPQI33Ecw\nD1jfXCNgTGNW0M6Y6HI6a2gE8CawFtDA8gv136ZYfz3rEZgjWNZQ+8vJyaG8vNzpMEwjPp+vyZtB\n426wuCXWELjXhg0bWLlyJQUFBQ1KBhhjnBdPl4ZMnJoxdSqn5Ofz26uv5pT8fKZPnep0SMaYNrAe\ngWmTDRs2cEp+Pv/icImK4cCq9eutZ2CMS1iPwMTUypUr6Q8NitZlB543xsQXawhMmxQUFLANf0+A\nwL+fBp43xsQXawhMm+Tl5TG5uJjhwGD8l4UmFxfbZSFj4pCNEZiIWNaQMe5l6aMm6O2332bp0qUU\nFRUxYsQIp8MxxpUivS/Fjfe12GCxAeCCoiLGjBxJya9/zZiRI7lg7FinQzLGdUpKFuLzDWHMmCn4\nfEMoKVkYeqMobu806xEksLfffpsxI0cekeL52ltvWc/AmIDKykp8viFUVS3n0F+K1zua8vKNYf3P\nPtLtY8l6BIalS5eSTcMUz36B540xfmVlZaSm5lD/LyUlpemSDbHY3g2sIUhgRUVFfErDFM/tgeeN\nMX45OTlUV5dR/y+lpqY87Pm7I93eDawhSGAjRoygsKiI4cAg/JeFCm3A2JgGsrKymDv3Ibze0aSn\nD8PrHc3cuQ+FfVkn0u3dwMYIkoBlDRkTWjJnDVlDkAQi+YCG2taNH35jjJ8NFhvAP8n7EJ+PKWPG\nMMTnY2FJSdS2jWTfxhj3sB5BAotkSsdQ29p0kca4X9QmrxeRTsD3gZz666vqryMJ0MReJJO8h9rW\nJpA3JnGEc2loEXARcBD4ut5iXC4nJ4ey6uoG6aPlNTVhpbWF2jaSfRtj3CVkjwDIVtXzYh6Jibrg\nJO+TJuFLSaG8pibsSd5DbRvJvo0x7hJyjEBE5gAPqura9gmpyRhcP0bgZPZMLDN7LGvIRJN9XtpX\nuGMEqGqTC7AWf49/PVADbAo8XgusaW67WCz+MN1rwfz5muH16rBu3TTD69UF8+cnxbGNaY358xeo\n15uh3boNU683Q+fPX+B0SAkv8N0Z8ju22R6BiPhCNCDl4bdLkXFzj8DJ7BnL3DHxws2F2RJZxPcR\nqGp54Mv+rkM/138umsHGs2D2TOBx/eyZRD62Ma2RCIXZElk4WUPH138gIh2AU6IVgIjMFZGdIrIm\n9Nru42T2jGXumHiRCIXZElmzDYGI3CYiXwInisi+wPIlsAt/Smm0PA7E7WwpwewZr5dh6emM9nrb\nLXvGyWMb0xqJUJgtkYWTNfS/qnpbTIPwj0e8rKonNvO6a8cIDnFz1pAxbmGf1fYV8Z3FIjIs8ONf\n6/0cpKqrI4gv4WRlZTn2wd69ezfr168nLS3NdSme9odv6nPy78Q0r6Ubyu4L/NsZ+DbwISD4R3v+\nH3B6bENraObMmcGfCwsLKSwsbM/Du9aMqVOZ84c/0B/YBkwuLuaBBx8Mvr6wpITrJk0iJzWVsupq\nHpo7l8snTGiX2Jw8tjHJqLS0lNLS0lZvF86loeeBOzVwQ5mInADMVNVL2xBnc8eI+0tDTtiwYQOn\n5OcfMSfxqvXrycvLs9RWY5JcNMtQH6f17ipW1Y+AvEiCa4IEFtMKK1eupD8N5yTODjwPltpqjAlP\nOA3BGhF5TEQKA8ujHM4Bi5iIzAfeAQaLyCci8uNo7TvRFRQUsI2GcxJ/GngeLLXVGBOecBqCHwPr\ngOmBZX3guahQ1Ymq2ldVO6nqMar6eLT2nejy8vKYXFzMcGAw/stCk4uLycvzd9gstdUYEw6bmCYB\nbNiwgZUrV1JQUBBsBOqzrCFjklM00kefUdUfiMha4Ihv4eYGdpNVLCt8xlKkxw7VCLk5XTBef2fG\nRF1z1eiAPoF/fU0t4VS0i9ZCAlcfjbR66PTiYvWCDgb1gk4rLg57/7E+tpuVLJiv3TK66nHDsrRb\nRlctWRD+e7cqmiZeEGb10XC+hCcBg8LZWawWNzcEu3bt0gyvVz8EVdAPQTO8Xt21a1dMt1VVXb9+\nvXoD2x3a3gu6fv36kPuP9bHdbNeuXdoto6ve9+FgfU5P0vs+HKzdMrqG/TvzejMUPlT/W/9Qvd6M\nsM+bMe0p3IYgnMHiY4A/icjHIvJXEZkqIidHt18SvyJJk4w0xTKS9NFYH9vNysrKODqnCzknegHI\nOdFLb1+XsH9nVkXTJJqQDYGq3qmqZ+OvQvoP4CZgVawDixexnBc4lEjSR2N9bDfLyclhR9k3lK2p\nAqBsTRU7y78J+3dmVTRNwgnVZQB+CSzG3wjMBn5AYPygvRZcfGlI9fC19qHp6W0eI2jLtqqq0wLX\n6QeFGCNoav+xPrabHRojGDy07WME6elDbYzAuBqRzlB2iIisBg4CfwNWAP9U1QMxa5majkFDxek0\nJzNQIkkfjfWx3cyyhkyiCzd9NKz7CEQkHRgBjAQuA3ap6siIowxTPDQETrIJ5k198dw4m+iKWq2h\nQJG5K4AfAZcD24FlEUdoomJhSQlDfD6mjBnDEJ+PhSUlrXrdJJapU2eQn38KV1/9W/LzT2Hq1OlO\nh2TiQDiXhl4B3gTeAt5T1Zr2CKxRDNYjaEKoCp9WATS5bNiwgfz8U6BRPdr161dZzyBJRa1HoKrf\nVdVZqvqOE42AaV6oFFCrAJpc/Km7Ryb1xkNKr3FWOPcRGJcKlQJqFUCTiz9198ik3nhI6TXOsoYg\njoWq8GkVQJNLXl4excWToV492uLiyXZZyIRk1UcDIs2scTJzx83VR53k5vcdy89LLLOG3HxOzZHC\nHSNo6Saul4GXmlvCuUkhWgsxvqEs0uJrobaPdP+xjD1RRVJULtZCxebWonZuPqemaURadA4Y1dIS\nzs6jtcSyIYi0+Fqo7SPdfyxjT1SRFJVzOja3FrVz8zk1zQu3IWh2jEBVV7S0tKmf4kKRZtY4mblj\nWUFNi6SoXKyFis2tRe3cfE5N5MK5oWyQiDwrIusDFUg/FpGP2yO49hBpZo2TmTuWFdS0SIrKxVqo\n2Nxa1M7N59REQaguA/4byc7B/8n0ATOBX4fT3YjWQjuNEbS1+Fqo7SPdfyxjT1SRFJWLtVCxubWo\nnZvPqWkaUSw6t0pVTxGRtar6rfrPxa55OiIGDRVnpOI5a8gyOZrm5vMSr/Wh3BqXaVrUis6JyDv4\ni809i7/G0HbgblU9LhqBhiMeSkzEspKlFREz7ck+b4kj4vRRPXxZ5lQgDf8EVI8DzwPDw+luRGsh\nTuYjiMWcxfE8L7CJP8VTpyl41ePJVfBq8dSpTodkIkC05izWw1/G6cBR4a7fiv2eB2wE/g3c0sw6\nMTlJ0RDLOYvjeV5gE3/Wr1+v4G2Qugpe+7zFsXAbgnCyhr4tImvxDxavFZEPRSQq4wMi4gH+AIzF\nPxXmBBEZEo19t5dYzlkcz/MCm/izcuVKPJ5+1P/EeTx97fOWBMKpNTQPuE5Vc1Q1B7ge/yWiaCgA\nNqtquformy4ALorSvttFLOcsjud5gU38KSgooK5uO/U/cXV1FfZ5SwLhNAS1qvqPQw9U9S38U1dG\nQz/85RIP+TTwXNyIpLBbqG3z8vKYXFxcr4QYTC4utgE8ExN5eXkUT/0JMByPJxcYTvHUn9jnLQmE\nkzV0P+AFSgDFP0vZfuAvAKq6us0HF/k+MFZVfxp4/EOgQFWnNVpP77zzzuDjwsJCCgsL23rYmLCs\nIZMo7PMWv0pLSyktLQ0+/tWvfhW19NHlLbysqnp2uEE2se/hwExVPS/w+NbAPu9ptJ6GijMUN+c/\nR5pT7uR7i/TYbq2U6fTnpaXju/nzEs+fVad/57EQtfTRWC5AB+A/+O9YTgU+APKaWC+ikXM3V+iM\ntHKpk+8t0mqU06YXa6rXo30Hd9JUr0enTYteamwksTld/bOl44d6X05WNg21byfPq5tjiyWilT4K\n9AbmAotJgY/MAAAQT0lEQVQDj/OBSeHsPKwA/Omjm4DNwK3NrNPmE+HmCp2RVi518r1FWo1y/fr1\nmur1NNg+1euJSqpiJLE5Xf2zpeOHU7nUqcqmofbt5Hl1c2yxFm5DEM5g8RPAq0DfwON/AzPC2C4s\nqrpEVY9T1UGqene09nuImyt0Rlq51Mn3Fmk1ypUrV5LZP6XB9j2zU6KSqhhJbE5X/2zp+OFULnWq\nsmmofTt5Xt0cm1uE0xBkquozQB2Aqh4EamMaVRS5uUJnpJVLnXxvkVajLCgoYPe2mgbb//fTmqik\nKkYSm9PVP1s6fjiVS52qbBpq306eVzfH5hqhugxAKdATWB14PBxYEU53I1oLURojcGOFzkgrlzr5\n3iKtRjltmn+MoM+g2I0RtCU2p6t/tnT8UO/Lycqmofbt5Hl1c2yxRBSrjw4DHgROAD4CsoBLVXVN\nixtGkWUNuTfbwbKGYsOyhqLPzbHFStSqjwZ21hE4DhBgk/rvAm438VCG2rSNm78YItnWyo63TSK/\nNydEY/L6U4Gj6z2+ClgEzAYywuluRGvB5ZPXm7ZxcjL0SI7tZIpmIk8gn6gpnE4iCpPXrz70hQ+c\nBVQA3wd+Azwbzs6jtcSyIXBzemkic3Iy9EjTS51M0UzUCeQTOYXTSeE2BC1lDXVQ1T2Bny8H5qjq\nc6p6B5Dbmu6Jm7k5vTSRRZp+6tSxnU7RTNQJ5C2F01ktNgSBsQHwz1m8rN5rHZtYPy65Ob00kUWa\nfurUsZ1O0UzUCeQthdNhzXUVgNuBt/GPC7zP4YHlXODtcLob0Vpw+eT1pm2cnAw9kmM7maKZyBPI\nJ2oKp5OIRvpooChcH2Cpqn4deG4wkKYRVB1tLcsaSlwtnfdQqaWR/s5imaIZKvZI0mZj+b6dZtlY\n0RUXRefCXXDxVJUmNkIVpHNzZk6o2IqLpwemhBzsnxe4eFrUYg8lnrOOnCyYF6+I9pzFTi7WECSX\nUAXp3JyZEyo2J+cFjuesIycL5sWzcBuCcGoNGdOuQhWkc3NmTqjY/O/hyJmo22Ne4HjOOor17zzZ\nM5asITCuE6ognZszc0LF5n8PR85E3R7zAsdz1lGsf+dJn7EUTrfB6QW7NJR0QhWkc3NmTqjYioun\nBS4PDXJsjCAes46cLJgXr4hW0Tk3aI+sIeM+sc4aakmsM3OcnBc4nrNj3Pw7d6OoFp1zmjUEsRPP\nH/63336bpUuXUlRUxIgRI1q9fbJW4TTJw9JHTUjxnEo4duxYBa+KZ6CCV8eOHduq7WP53pN1flzj\nPlj6qGlJPKcSvvXWW02mYL711lthbR/L957M8+Ma9wm3IbCsoSQVz6mES5cuRTx9qZ/uJ56+LF26\nNKztY/neQ6UiWqqicSNrCJJUPKcSFhUVoXUV1E/307oKioqKwto+lu89VCqipSoaVwqn2+D0gl0a\niol4TiWM1hhBLN57ss6Pa9wHSx814Yjn7BXLGjKmZa5PHxWRS4GZQB5wqrZQzdQagvgVr/MCm+ST\niJ+ncBsCJ8cI1gKXACscjMHE0IKFJQwaMoArp3yHQUMGsGBhSdS2jWTfxjRWUrIQn28IY8ZMwecb\nQknJQqdDaleOXxoSkeXAjdYjSCyVlZUMGjKA/1nej5wTvZStqeLXo7ezeePWkP/bCrVtJPs2prHK\nykp8viFUVS3Hn821Bq93NOXlG+P+8xQPPQKTwGI9L3C8pr4a97GU3hjPPSwirwG96z8FKHC7qr7c\nmn3NnDkz+HNhYSGFhYVRiNDESv0UzUP/a2/LvMBNbRvJvo1prGFKr79HEK8pvaWlpZSWlrZ+w3BS\ni2K5AMuBYSHWiTyPyrS7WM4LHM+pr8Z9EjWll3hJHw2MEfwfVV3VwjrqdJymbSxryMSLRPw8hTtG\nENNLQy0RkYuBB4FM4BUR+UBVz3cqHjdL5snKW5KVleW693OIlZmOP27+PMVcON0GpxeS+NJQpJUq\nnawwGurYLb0ezxU6i6f6J57xeHL9E89Mndpux47n82aiD6s+Gv8irVTpZIXRUMdu6fV4rtDp9OT0\n8XreTGyE2xBY+qiLRZrW5mSaZSQpoPGczrdy5Uo8nn7Uj93j6dtuk9PH63kzzrKGwMUirVTpZIXR\nUMdu6fV4rtBZUFBAXd126sdeV1fRbpPTx+t5Mw4Lp9vg9EKSXhpSjTytzck0y0hSQOM5na946tTA\nGMFAx8YI4vG8megjXtJHw5Hs6aORZqA4mUUSqkJoS7HFc/aLZQ0ZN3B99dHWSOaGYMHCEqZcN5mj\nc7qwo+wbHnnoUcZfPsHpsMIyfcZUHpnzEJn9U9i9rYYpk6/jgQcedDosY5KGNQQJIJ6Lq23YsIGT\nTzmB//1XbjD224b/hw9WfdTu/0M2JllZ0bkEEM/F1VauXElm/5QGsffMTmmX7BljTOtYQ+Bi8Tyv\ncEFBAbu31TSI/b+f1rRL9owxpnUcKzFhQsvKyuKRhx5lyujJ9PZ1YWe5f4zA7ZeFAPLy8pgy+Tpu\nG/4QPbNT+O+n/jECuyxkjPvYGEEciOcsECezZ4xJdjZYHGVu/jK22NwnWd+3cRcbLI6ihSUlDPH5\nmDJmDEN8PhaWuGd+XDfP3evm2GIpWd+3iV/WIwihsrKSIT4fy6uqAnMXwWivl43l5Y7/T8/N6aVu\nji2WkvV9G3eyHkGUlJWVkZOaWq+MF/hSUlyRwunm9FI3xxZLyfq+TXyzhiCEnJwcyqqr65XxgvKa\nGlekcLo5vdTNscVSsr5vE+fCKUjk9ILDRecWzJ+vGV6vDk1P1wyvVxfMd8/8uG6eu9fNscVSsr5v\n4z5Y0bnocnMWiMXmPsn6vo27WPqoMSFE8mVtX/QmHthgsTEtiCTF09JDTaKxHoFJOpGkeFp6qIkn\n1iMwphmRpHhaeqhJRNYQmKQTSYqnpYeaRORY9VERmQWMAw4AW4Afq+o+p+IxySOSqq7xXBHWmOY4\nNkYgIucCy1S1TkTuxp/velsz69oYgYk6yxoyiS6u0kdF5GLg+6p6ZTOvW0NgjDGtFG+DxdcAi50O\nwhhjklFMxwhE5DWgd/2nAAVuV9WXA+vcDtSo6vyW9jVz5szgz4WFhRQWFkY7XGOMiWulpaWUlpa2\nejtHLw2JyNXAZOBsVT3Qwnp2acgYY1op3EtDTmYNnQfcBJzVUiNgjDEmtpzMGtoMpAL/DTz1L1W9\nrpl1rUdgjDGtFFdZQ6FYQ9AyS2U0xjQl3rKGTBtZATRjTKSsRxDHrACaMaYl1iNIAlYAzRgTDdYQ\nxDErgGaMiQbH0kdN5KwAmjEmGmyMIAFY1pAxpimWPmqMMUnOBouNMcaExRoCY4xJctYQGGNMkrOG\nwBhjkpw1BMYYk+SsITDGmCRnDYExxiQ5awiMMSbJWUNgjDFJzhoCY4xJctYQGGNMkrOGwBhjkpw1\nBMYYk+SsITDGmCRnDYExxiQ5xxoCEfm1iHwoIu+LyBIROdqpWIwxJpk52SOYpaonqepQ4G/AnQ7G\nEjdKS0udDsE17FwcZufiMDsXredYQ6CqX9V72BWocyqWeGIf8sPsXBxm5+IwOxet5+jk9SJyF3AV\nsBcY7WQsxhiTrGLaIxCR10RkTb1lbeDfcQCq+ktVPQZ4Gpgay1iMMcY0zRWT14tIf+DvqvqtZl53\nPkhjjIlD4Uxe79ilIRHJVdX/BB5eDGxobt1w3ogxxpi2caxHICLPAoPxDxKXA1NU9TNHgjHGmCTm\niktDxhhjnBM3dxaLyCwR2SAiH4jIcyKS7nRMThGRS0XkIxGpFZFhTsfT3kTkPBHZKCL/FpFbnI7H\nSSIyV0R2isgap2Nxkohki8gyEVkXSEqZ5nRMThGRTiLybuBm3bUiEvIerbhpCIClwPGqejKwGbjN\n4XictBa4BFjhdCDtTUQ8wB+AscDxwAQRGeJsVI56HP+5SHYHgZ+r6vHA6cD1yfq5UNUDwOjAzbon\nA+eLSEFL28RNQ6Cqr6vqoZvO/gVkOxmPk1R1k6puBpJxEL0A2Kyq5apaAywALnI4Jseo6lvA507H\n4TRV3aGqHwR+/gp/8kk/Z6Nyjqp+E/ixE/6koBbHAOKmIWjkGmCx00EYR/QDttV7/ClJ/AdvjiQi\nOfj/J/yus5E4R0Q8IvI+sAN4TVXfa2l9R+8sbkxEXgN6138Kf0t2u6q+HFjndqBGVec7EGK7Cedc\nGGMaEpE04FlgeqMyNkklcPVkaGAs9UURyVfV9c2t76qGQFXHtPS6iFwNXACc3S4BOSjUuUhi24Fj\n6j3ODjxnkpyIdMTfCDylqoucjscNVHWfiCwHzgOabQji5tKQiJwH3ARcGBgMMX7JNk7wHpArIj4R\nSQXGAy85HJPThOT7HDRlHrBeVR9wOhAniUimiHQL/OwFxgAbW9ombhoC4EEgDXhNRFaLyENOB+QU\nEblYRLYBw4FXRCRpxktUtRYoxp9Ftg5YoKrN3pWe6ERkPvAOMFhEPhGRHzsdkxNEZARwBXB2IG1y\ndeA/j8moD7BcRD7AP07yqqr+vaUN7IYyY4xJcvHUIzDGGBMD1hAYY0ySs4bAGGOSnDUExhiT5Kwh\nMMaYJGcNgTHGJDlrCExCEJHbA6W5PwzkkJ8a5f2PEpEjSns093wUjndR/eqZIrI8GUuOm/bhqhIT\nxrSFiAzHX3rkZFU9KCIZQGoMDtXcTTexuBnnYuAVQtwRakw0WI/AJII+wG5VPQigqntUdQeAiAwT\nkVIReU9EFotI78Dzy0Xk/sBdqGtE5NuB508VkXdEZJWIvCUig8INQkS6BCaK+Vdg+3GB538UmExp\nsYhsEpF76m0zKfDcv0Rkjog8KCKnAxcCswK9m2MDq/8gMOHIxsCdtMZEhTUEJhEsBY4JfEH+UUTO\ngmARsgeB76vqqfgncfltve28gck7rg+8Bv469iNV9RTgTuB/WxHH7cAbqjocf2HE3wVqvQCcBFwG\nnAhcLiL9RKQP8Ev8cyyMAIYAqqr/xF8/6SZVHaaqHwf20UFVTwNuAGa2Ii5jWmSXhkzcU9WvA9fP\nz8T/BbxARG4FVgEn4K9PJfj/41NRb9OSwPb/EJGjAiV704EnAz0BpXV/I0XAOBG5KfA4lcOVUt84\nVBZZRNYBPiALKFXVLwLP/xVoqQfyfODfVYHtjYkKawhMQlB/0aw3gTdFZC1wFbAa+EhVm7uM0vja\nvgK/AZap6vdExAcsb0UYgr/3sbnBk/4xjPoVc+s4/LfXmqqhh/ZRi/3tmiiyS0Mm7onIYBHJrffU\nyUA5sAnICnwRIyIdRSS/3nqXB54fCXyhql8C3Tg8v0FrK3m+CgQnTReRk0Os/x5wloh0C1zG+n69\n177E3ztpjpWdNlFjDYFJBGnAnwPpox8AecDMwJzGlwL3BJ5/H//E5ofsF5HVwEP4pz8FmAXcLSKr\naP3fx2+AlMDg80fAr5tZTwFUtQL/mMVK4B/AVuCLwDoLgJsCg87H0nTvxZiosDLUJikFZm26UVVX\nOxxH18AYRwfgBWCuza5l2pv1CEyycsv/gGYGJhlfC3xsjYBxgvUIjDEmyVmPwBhjkpw1BMYYk+Ss\nITDGmCRnDYExxiQ5awiMMSbJWUNgjDFJ7v8DMse49XEikycAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd773cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "colors = ['red', 'greenyellow', 'blue']\n",
    "for i in xrange(len(colors)):\n",
    "    px = X_train[:, 0][y_train == i]\n",
    "    py = X_train[:, 1][y_train == i]\n",
    "    plt.scatter(px, py, c=colors[i])\n",
    "\n",
    "plt.legend(iris.target_names)\n",
    "plt.xlabel('Sepal length')\n",
    "plt.ylabel('Sepal width')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scatter function simply plots the first feature value (sepal width) for each\n",
    "instance versus its second feature value (sepal length) and uses the target class\n",
    "values to assign a different color for each class. This way, we can have a pretty good\n",
    "idea of how these attributes contribute to determine the target class. The following\n",
    "screenshot shows the resulting plot.\n",
    "\n",
    "Looking at the preceding screenshot, we can see that the separation between the red\n",
    "dots (corresponding to the Iris setosa) and green and blue dots (corresponding to the\n",
    "two other Iris species) is quite clear, while separating green from blue dots seems a\n",
    "very difficult task, given the two features available. This is a very common scenario:\n",
    "one of the first questions we want to answer in a machine learning task is if the\n",
    "feature set we are using is actually useful for the task we are solving, or if we need to\n",
    "add new attributes or change our method.\n",
    "\n",
    "Given the available data, let's, for a moment, redefine our learning task: suppose\n",
    "we aim, given an Iris flower instance, to predict if it is a setosa or not. We have\n",
    "converted our problem into a binary classification task (that is, we only have two\n",
    "possible target classes).\n",
    "\n",
    "If we look at the picture, it seems that we could draw a straight line that correctly\n",
    "separates both the sets (perhaps with the exception of one or two dots, which\n",
    "could lie in the incorrect side of the line). This is exactly what our first classification\n",
    "method, linear classification models, tries to do: build a line (or, more generally, a\n",
    "hyperplane in the feature space) that best separates both the target classes, and use\n",
    "it as a decision boundary (that is, the class membership depends on what side of the\n",
    "hyperplane the instance is).\n",
    "\n",
    "To implement linear classification, we will use the SGDClassifier from scikit-learn.\n",
    "SGD stands for Stochastic Gradient Descent, a very popular numerical procedure\n",
    "to find the local minimum of a function (in this case, the loss function, which\n",
    "measures how far every instance is from our boundary). The algorithm will learn the\n",
    "coefficients of the hyperplane by minimizing the loss function.\n",
    "\n",
    "To use any method in scikit-learn, we must first create the corresponding classifier\n",
    "object, initialize its parameters, and train the model that better fits the training data.\n",
    "You will see while you advance in this book that this procedure will be pretty much\n",
    "the same for what initially seemed very different tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
       "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
       "       learning_rate='optimal', loss='hinge', n_iter=5, n_jobs=1,\n",
       "       penalty='l2', power_t=0.5, random_state=None, shuffle=True,\n",
       "       verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create the linear model classifier\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "clf = SGDClassifier()\n",
    "# fit (train) the classifier\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SGDClassifier initialization function allows several parameters. For the\n",
    "moment, we will use the default values, but keep in mind that these parameters\n",
    "could be very important, especially when you face more real-world tasks, where the\n",
    "number of instances (or even the number of attributes) could be very large. The fit\n",
    "function is probably the most important one in scikit-learn. It receives the training\n",
    "data and the training classes, and builds the classifier. Every supervised learning\n",
    "method in scikit-learn implements this function.\n",
    "\n",
    "What does the classifier look like in our linear model method? As we have already\n",
    "said, every future classification decision depends just on a hyperplane. That\n",
    "hyperplane is, then, our model. The coef_ attribute of the clf object (consider, for\n",
    "the moment, only the first row of the matrices), now has the coefficients of the linear\n",
    "boundary and the intercept_ attribute, the point of intersection of the line with the\n",
    "y axis. Let's print them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-30.79833395  15.06870628]\n",
      " [  0.74533565  -4.07000427]\n",
      " [ 13.17646958   1.50202538]]\n",
      "[-18.46357161  -2.26800646 -16.31131755]\n"
     ]
    }
   ],
   "source": [
    "# print learned coeficients\n",
    "print clf.coef_\n",
    "print clf.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed in the real plane, with these three values, we can draw a line, represented by\n",
    "the following equation:\n",
    "    \n",
    ">_ -17.62477802 - 28.53692691 * x1 + 15.05517618 * x2 = 0 _\n",
    "\n",
    "Now, given x1 and x2 (our real-valued features), we just have to compute the value\n",
    "of the left-side of the equation: if its value is greater than zero, then the point is\n",
    "above the decision boundary (the red side), otherwise it will be beneath the line (the\n",
    "green or blue side). Our prediction algorithm will simply check this and predict the\n",
    "corresponding class for any new iris flower.\n",
    "\n",
    "But, why does our coefficient matrix have three rows? Because we did not tell the\n",
    "method that we have changed our problem definition (how could we have done\n",
    "this?), and it is facing a three-class problem, not a binary decision problem. What, in\n",
    "this case, the classifier does is the same we did—it converts the problem into three\n",
    "binary classification problems in a one-versus-all setting (it proposes three lines that\n",
    "separate a class from the rest).\n",
    "\n",
    "The following code draws the three decision boundaries and lets us know if they\n",
    "worked as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAD6CAYAAABwM0lXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYFNW5uN9vYAaabRQZURAZFZBFcYuIMdFxAzVu0VyN\n0ajRa0zigMZo1LihxDXrVWMM/ojXmAB6jYlbNGhkVDSKcQMFFJdBBZXBBRVZhpnv90fVzHT39N61\ndff3Pk89dHdVnXOq6Xf69KnznU9UFcMwDMMwDMN/qsJugGEYhmEYRqVgHS/DMAzDMIyAsI6XYRiG\nYRhGQFjHyzAMwzAMIyCs42UYhmEYhhEQ1vEyDMMwDMMIiIrueInI5SJyR9jtKFdEpF1Etg+7HUbu\nmBP+Yk6UHuaEv1SiE2Xf8RKR74jIcyLyuYisEJEHReSrcYcEtpCZiAwXkcdEZK2ILBaRA4Oq229E\nZJ6InJb0cuQWiRORU0TkybDbESYRc+JKEVkoIq0icllQ9QaBOVE6RMUJEakTkVluGz4RkSdFZEIQ\ndQeBOeFQ1h0vETkX+DXwc2BLYFvgd8CRITVpNvA8MBC4BLhbRLbws0IR6eFn+dmq96XQ4q5JiKDo\nQRFBJ5YB5wMPBFWhOdH9dMyJqDjRD1gA7IbzPfEn4EER6eNnpeZE99Px0wlVLcsNGAB8DhyT4ZjL\ngT/FPb8LeB/4BGgCxsbtOwx4FfgMeBc41319C+B+95yPgMfT1DUSWAf0jXvtceD7KY7dGvgS2Czu\ntd2AFqCH+/w0YLFb50PAtnHHtgM/Al4H3nRf+w3wIbAGeLnj2oB5wGlx554CPBn3POV5Se39ObDJ\nbfNnwA1x7TjTbcfHwE1J56W9hqTjhrtlnQYsB5rc1ycCT7nv/YvAfnHnnAq86bbnTeAEYLT7f9Dq\nfjY+DvtzWslOJNV7B3BZhv3mhDlRUU7ElbkG2M2cKB8nQv/g+yjUZGAjUJWHUKcCfYBqnF9AL8bt\nWwl81X1cC+zqPr4auBln9LAHsE+auo4GXk167Qbgf9Ic/yhwetzz64Gb3cdHuR/SUW69PwOeShLq\nn247ewGTgP8A/d39OwKDMwj1hPt4EvBcqvNStDehnLh23Af0B4YBq4BJuVxDGqH+F4i51zQEWA1M\ndo850H2+hft/uAYY4e4bDIxJvr5K26LmRFK9GTte5oQ5UWlOuOftitNR6W9OlI8T5XyrcQtgtaq2\n53qCqv6vqn6pqq3AlcAuItLf3b0RGCci/VV1jaq+5L7eivPLYztVbVPVp9IU3w/nPzmez3A+bKmY\nDXwn7vm3gb+4j88ErlHV193ruxbYVUSGxR1/tdvODW4b+wFjRURU9TVV/TDDW9FBq9u+fM+L5xpV\n/VxV38WRbtc8riEeBS5X1XXuNZ0EPKiq/wRQ1X/h/NE4zD2+DdhZRHqr6oequiTPdpcjUXMiX8yJ\nRMyJ4omsEyIyAOdW4zRV/TzNYeZEIiXhRDl3vD4CBolITtcoIlUicq2IvCEinwJv4/wnDnIPORb4\nBrDcnSA40X39epwhyrnuuRekqeILnGHteGpxhjJT8VdgoogMFpH9gHhZhwP/IyIfi8jH7rUqMDTu\n/Pc6HqjqPOAmnHkLH4rILSLSL8PbUdR5ScQL+CWO2LleQzLvxT0eDhzXcb6IfALsA2ytql8CxwM/\nBN4XkftFZMc8212ORM2JfDEnumNOFEcknRCR3jijQE+r6vUZDjUnuhN5J8q54/VvYAPOLb5cOBE4\nAjhAVTcD6nEm2AmAqj6vqkcDdcC9OPf5UdW1qnqequ6AMxnzXBHZP0X5rwLbi0jfuNd2cV/vhqp+\nCszF+QVzAjAnbvc7wJmqOtDdNlfVfqr6THwRSeXdpKpfAcbiDAWf7+5aizPk2sFWOZ7XrclpXk/H\nuzlcQ6Y63sUZ/o8/v3/HHylVfURVJ7nX8xowo8B2lhNRcyIvzImsdZgT+RM5J0SkBvg78I6q/iBT\nY8yJrHVE0omy7Xip6mc49+Z/JyJHiUhMRHqKyKEicm2KU/rhCPiJ2zm6BvfNF5FqccKNB6hqG84o\nVZu77xsisoNbxuc4kwe7DVur6jLgJeByEeklIscAO+H8YknHbOBknF9Rs+Je/wPwMxEZ67ahVkS+\nla4QEfmKiEwQkZ44kwbXx7XxJeAY9/0ZAZye43nJfAjksxbLLflcA90jX/4MHCEik9xfob1FZD8R\nGSIiW4rIkeJEArXijDZ2tPtDYBsRqc6jrWVB1Jxwj+3p/rqvAqpdNzL9XTIn4i4j6bk5kSdRc8L9\nXP0VZ9Tn1Bwvw5yIu4yk59F0QiMwwdHPDedXwHM4H/aVOJElEzVp0iTQF+dXxmc4w8cn4UizPc4k\nyodwhjk/BZ4F9nbPO8c9/nOcXxg/y9CWbXHuX38JLAH2z9L23m57FqbYdyKw0G3PcuD/xe1rA7aP\ne34ATqTJZzgTF+8A+rj7tsCZYLkGeBK4jK5Jk2nPS9GeiTi/GD4CfpumHX8ErszlGpLKHu6WVZX0\n+p44UUUf4YhyP7ANzq+XJpwolo+Bx4DR7jnV7nEfAavC/nyaE9yG88euLW472ZwwJyrRCWBft7wv\n3GM/d+tKOxnfnCg9J8StxDAMwzAMw/CZsr3VaBiGYRiGETWs42UYhmEYhhEQ1vEyDMMwDMMIiJ5h\nNyAXRMQmohmRQlV9yS+WK+aEETXMCcNIJJ0TJTPilS464PLLLw81GibM+sOse+NGpbb2cp5+uvLe\n+6gQtfclCvVX8rWHWX9UiNr7EoX6g6p70yalulpZvz461x5m/ZkomY6XES3uuAMGDoS99w67JYZh\nGEbY9OgBW28NK1aE3ZLoYx0vI282bYKrr4Z99w27JYZhGEZUGDYM3n037FZEn5LveDU0NFRs/WHV\nPWcODB0Kp54aTv0dhP1/H1XCfl8q0QmrP9qE/b5UihOpOl6V/N6noyQWUBURLYV2VgJtbTBuHNx0\nExx0UNitCQcRQSMwkdicMKKCOWEAXHABbLYZXHRR2C0Jn0xOlPyIlxEsd98Nm28OBx4YdksMwzCM\nKDFsGLzzTtitiD6hdrzchLjPisiLIrJIRC4Psz1GZtrb4ec/h0svBQn1t235Yk4YRiLmROlgc7xy\nI9R1vFR1g4jsr6pfikgP4CkReUhVF4TZLiM1994LvXrBoYeG3ZLyxZwwjETMidLBOl65EfqtRlX9\n0n3YC6cjaDfpI4gqTJ9uo11BYE4YRiLmRGlgHa/cCL3jJSJVIvIi8AHwiKo+F3abjO48+KAzsf7I\nI8NuSfljThhGIuZEaTBoEKxbB2vXht2SaBN6yiBVbQd2E5EBwN9FZKyqLk4+btq0aZ2PGxoaIhki\nWq50jHZdcklljnY1NTXR1NQUWH3mhBF1zAkjFSJdo16jR4fdmmDJx4lILSchIpcCa1X110mvW5hw\niMydC+ecA6+8AlWhj5GGT5Ch8+aEUQqYE0YHBx7oLCsxaVLYLQmXyC4nISKDRKTWfRwDDgaWhtmm\nKNPS0sJzzz1HS0tLYHWqwpVXOqNd1unyH3MiP8JwwggWcyI/wnbC5nllJ+yv0q2BeSLyEvAs8E9V\n/UfIbYokd86ezejhw/nBwQczevhw7pw9O5B6m5pg1So4/vhAqjPMiZwJywkjcMyJHImCE9bxyk6k\nbjWmo9KHkFtaWhg9fDjz1q1jPLAQ2D8WY+ny5dTV1fla9wEHwMknw6mn+lpNSWGrdIdPmE4Y3TEn\nwicqTsyYAc8+CzNnBlZlJInsrUYjN5qbm6mvqWG8+3w8MLy6mubmZl/rnT8fmpvhxBN9rcYw8iYs\nJwwjqkTFiW23tRGvbFjHqwSor6+neeNGFrrPFwLLW1upr6/3td7p0+HCC6G62tdqDCNvwnLCMKJK\nVJywtEHZsY5XCVBXV8fNM2eyfyzG7gMGsH8sxs0zZ/o6fLxgASxZYrcYjWgShhOGEWWi4kTHHK8K\nvuubFZvjVUK0tLQ4w8n19b7LdMQRcMghcNZZvlZTkth8lugQpBNGesyJ6BAFJ2pr4e23YeDAUKqP\nBJmcsI6X0Y0XX4TDD4c334TevcNuTfSwLxnDSMScMOLZaSf4y19gl13Cbkl42OR6Iy+mT4fzz7dO\nl2EYhpE/NsE+M6GnDDKixaJF8PTT8Oc/h90SwzAMoxSxtbwyYyNeRgJXXQXnngt9+oTdEsMwDKMU\nscjGzFjHy+hk6VJ47DH44Q/DbolhGIZRqtiIV2as42V0ctVVcPbZ0L9/2C0xDMMwShXreGXGOl4G\nAG+8AQ89BI2NYbfEMAzDKGWs45UZ63hFjLAyy19zjbNmV21toNUaRlbCcsIwokrUndhmG1ixAtrb\nw25JNLGOV4QIK7N8czP8/e/ObUbDiBJhOWEYUaUUnIjFnB/xq1aF3ZJoYguoRoQwM8v/4AfOCsNX\nX+1rNWWDLRYZDGE6YeSHOREMpeTEHnvA738PEyaE3ZJwsAVUS4CwMsu/9x7cdRf8+Me+VmMYeROW\nE4YRVUrJCZvnlR7reEWEsDLLX389nHYaROzHkmGE5oRhRJVScsI6XumxjldECCOz/PvvOyvUn3ee\nb1UYRsGE4YRhRJlScsLSBqXH5ngFQD7Z4oPMLP+Tn8CmTfA//+NrNWWHzWcpnqg6YRSGOVE85ebE\nnDlwzz3OVJZKxOZ4hUi+ESh1dXXsueeevsvU0gK33QY//amv1RhGN6LqhGGERTk6YWmD0mMjXj4S\n5QiUiy6CTz91ok6M/LBf94UTZSeMwjEnCqdcnXjnHdh7b2c9r0rERrxCIqoRKB9/DDNmwIUXhtoM\nowKJqhOGERbl6sSQIc6dldbWsFsSPazj5SNRjUD57W/hm9+E4cNDbYZRgUTVCcMIi3J1omdPGDwY\nVq4MuyXRwzpePhLFCJQ1a+Dmm51bjYYRNFF0wjDCpJydsCUlUhPqHC8R2Qb4EzAYaAduVdUbUhxX\nkvfuO4hSBMrPfw6vvw5/+lOozShp/JzPYk4YpYg5UTzl6MTxx8PRR8MJJ4TdkuDJ5ETYHa+tgK1U\n9SUR6Qc8DxylqkuTjitpofwiX1E//xx22AGeeAJGjw6ggWWKz18y5kQRlOOXVylgTkSXMJ047zxn\nce4LLgi02kgQ2cn1qvqBqr7kPv4CWAIMDbNNpUIhiVJvvhkOPNA6XVHGnCicUkgebOSPOVE4YTth\ntxpTE5nlJESkHmgCdnLlit9nv2TiKCT8eO1aZ7Tr0Udhp50CbW7ZEVTovDmRO+Uakl8qmBPRIwpO\n3HMP3H473HtvINVFikxO9Ay6Malwh4/vBs5OlqmDadOmdT5uaGigoaEhkLZFkc7w43XrgMTw43RC\nzZgB++xjna5CaGpqoqmpKdA6zYn8KMQJo3DMiegTBScqKW1QPk6EPuIlIj2BB4CHVDVl8hr7JZNI\nvr9k1q1zRrsefBB22y3w5pYdfv+6NyfyJwq/7isZcyJ6RMGJDz90fuy3tARSXaSI7Bwvlz8Ci9PJ\nZHQn3/DjmTPhK1+xTlcJYU7kSTmH5BuAOZE3UXCirs4J6vryy8CqLAnCjmrcB3gCWASou/1MVR9O\nOs5+yaQgl2iVDRtgxAjnXvueewbcwDLF5wguc6IILKoxHMyJ6BK2EzvsAA89BKNGBV51qER2OYlc\nKWehlixZwoIFC5gwYQJjxozxvPwZM5xO18MPZz/WyA3LS+cvfjtheI854S+l7ERDA1x6qRNRX0lE\n/VZjxXLOlCnsMXYsV596KnuMHcvZU6Z4Wn5rK1xzjfOhN4xSwG8nDKPUKHUnKmmCfa7YiFdILFmy\nhD3GjuUZ6Jz4OBF4fvFiz37R3HYb3HEHPPaYJ8UZLvbr3h+CcMLwB3PCH8rBiYsvht69K28AwEa8\nIsiCBQsYBgkZ6bdxX/eCTZvg6qvhsss8Kc4wfMdvJwyj1CgHJ2wR1e5YxyskJkyYwLuQkJH+Pfd1\nL7jzTthqK9hvP0+KMwzf8dsJwyg1ysGJYcPgnXfCbkW0sI5XSIwZM4YzGhuZCIzCGT4+o7HRk+Hj\ntja46ipnaFdCHfw3jNzx0wnDKEXKwQkb8eqOzfHygaeeeoq5c+cyadIk9tlnn4zH+hGtctdd8Otf\nw7//XZkdr0zh016EVtt8lvwJ24lKx5yIHpXixCefwPDh8NlnYbckkVCdUNXIb04zS4NDDz5YY6Aj\nQWOgh06aFGj9bW2qO++s+sADgVYbGWbNmqOx2ECtrd1dY7GBOmvWnJz25YP7eTQnciRsJyodcyJ6\nVJIT7e2q/fqpfvpp2C3pImwnQu9U5bKVilDz58/XGOjLoOr+GwOdP39+YG34299Ud9/d+bBXGqtW\nrdJYbKDCy+r8F7yssdhAXbVqVcZ9+WJfMrkTBScqGXMielSiE2PGqC5aFHYrHKLghM3x8pC5c+ey\nDYkRKEPd14NAFaZPr9y5Xc3NzdTU1BP/P1BdPZzm5uaM+wz/CNuJSseciB6V6ESU5nlFwQnreHnI\npEmTeI/ECJQV7utB8I9/OIumHnlkINVFjvr6ejZubCb+f6C1dTn19fUZ9xn+EbYTlY45ET0q0Yko\nRTZGwol0Q2FR2iiRIWRV1UMnTdIY6IiA7923t6vutZfqnXcGUl1k6bg/P2DAbmnv3afalw/YbZW8\nCMsJw8GciB6V5sS0aaoXXxx2K7oI2wmLavSBfKJVvOKRR2DqVHjlFejRI5AqI4tFcEWPMJwwujAn\nokclOXHbbdDUBLffHnZLugjTCet4+UC2/7T4/UDR/8GqsO++8IMfwIknFtNyIxfsSyZ/gnbCCBZz\nIn/y+eL3oiMQJo8+6mRSqaT0dbacRIDMmTVLB8ZiunttrQ6MxXTOrFlp9/evrtbampq0x+bKvHmq\nI0aotrZ6cAFGVrDbKnkRhhNGsJgT+ZHJieR9Zzc2ZvSnFFi61PmOqiQyOWEjXh7S0tLC6OHDmbdu\nXWdC0/1jMZYuX975q6Vj/9bAjkATpDw2Hw48EE46Cb73PY8vyEiJ/brPnbCcMILFnMidTE4ACfua\ngMMgIUl2KTrx5ZcwcCCsW1c5EfeWJDsgmpubqa+pSQgTHl5d3RmKGr+/GdgO0h6bK08/DW+95XS8\nDCNqhOGEYUSZTE4k7+sL3ZJkl6ITffpA377Q0hJ2S6KBdbw8pL6+nuaNGxPChJe3tnbOW4nfXw+8\nDWmPzZXp0+Gii6C6uvj2G4bXhOGEYUSZTE4k71sL3ZJkl6oTUVrLK2ys4+UhdXV13DxzJvvHYuw+\nYAD7x2LcPHNm55Bw/P7JAwbQWl3NvjU1KY/NhQUL4NVX4ZRT/LoiwyiOoJ0wjKiTyYnkfcfGYpzR\n2JjWn1Ji222t49WBzfHygaAiuI48EiZNgsZGL1pt5IrNZ8kfi2osb8yJ/KmkqEaAs86C0aNhypSw\nWxIMtpxEjhTz4Q5ajJdegsMOgzffhFjM9+p8p5T+sFTSl0yh/y+l9P8ZVUrpPTQn/DuvXLj2Wvj4\nY7j++sLLKKX30JaTyIFsIe9+nVsoxx6r+utf+15NIHiVDT4oqJDQ+UI/12H4UG6YE+ZEufHnP6se\nf3zh55eTE6F3qnLZ/BZq1apVOjAWS8gWPzAWyykjeTHnFsqiRaqDB6uuXetbFYHhZTb4oKiEL5lC\nP9dh+FBumBPmRDny+OOqX/1qYeeWmxM9cxgu6wUcixN01Hm8ql5Z0Phb9/JnAocDH6rq+GzH+0Fn\nCO+6dUBiyG624cxizi2Uq66CH//YCdEtdTqywa9b1z0bfJSHkkXkO5gTnp1ndGFOpCzbnChxiplc\nX6pOpCOXqMZ7gaOATTjRrR2bV9wGTPawvLzJFvLu17mF8Npr8K9/wY9+5EvxgRNYNnjvMSc8PM/o\nwpxIiTlR4gwdCh98AG1t+Z9bwk6kJt1QmHYN376S7ZhiN2A4sDDDfo8HAbvTcQ9+twEDCp7jVci5\n+XLyyarTp/tWfCh4lQ0+KAjgtkopOxGkD+WKOWFOlCNbb6367ruFnVtOTmSNahSRGcCNqrrI+25f\nZx3Dgfs1zRBy1KNVij03V958E/bay/m3ttaXKkKj1KJVgPHmhPfnGV2U0ntoTvh3Xjmx117wm9/A\nV79a2Pml9B5mimpMO8dLRBYB6h7zPRF5C9gACE5PLpT77H7SsYBdIaxevZrFixfTr18/oPB1iDJ9\nsK65xrnFWG6dLijuvQ+KnXfeueMLBuAFcyI98T4Uuw5RKf2x9RJzIpqYE4VT7Or1peBELmSaXH94\nYK3IgWnTpnU+bmhooKGhIbS2JHPOlCnMuOkmhuGkd5AePRjdrx/NGzdy88yZHH/CCTmVc+fs2fzo\n9NOpr6npdu7y5fC3v8GyZf5dh5GapqYmmpqaOPDAAwFYtGgRwIhQG0V0nUj2oWHSJJ598smUn+ts\nZHLCCA9zIj/MCYdyXr2+w4mcSHcPUrvum9+Ry2vFbDiRMIsy7C/+hqtPLF68WGNuiHBHqPDmoKs8\nXpbihz9UvfBCv6/GyAVS3Ls3JxxS+RADnVciS7UYhWFOpMec6OLXv1adOjXsVgRDKic6tlyiGsfF\nPxGRHsAeuXXrsiMis4CngVEi8o6IfM+rsoNgwYIF3bLH1wPN5JdJPlPG+hUrYM4cOPdcr1tveIE5\n0UUqH4YCfeOee+GEEW3MiS7MiS4sUbZD2o6XiFwkIp8D40XkM3f7HFiFs8SEJ6jqd1R1iKr2UtVt\nVfU2r8oOggkTJnTLHt+M0/nyalmK66+H730PyuDWdklzzTXX0L9/fwDMidSk8mEFXesKRHmpFiN/\nzInsmBNdWMfLJd1QmHYN316T7Ri/NyI6hNzB1MZGjYGOdIeQ+/To4dmyFO+/r7r55qorV/p4AUZe\nUAGrdBdDsg+HTppUEku1GIVjTmTGnHBYsUJ1yy3DbkUwZHIi7XISIrJ7lg7bC951/zIThTDhbFEk\nDzzwAH//+985+uij2WGHHViwYAETJkxgzJgxGc9dsmRJxmPPPx82bIAbbvDuWnLZb3TnhRecj/we\ne+wBKW6jmBNdxPtw+OGHZ/2cx5PPsV5dSy77je6YE+ZEPrS1QSwGn38OvXoVXVykKShJNjDP3f4N\ntAL/AZ53H/873Xl+bISc/DRbgtOz3V8zo9xfM72rqjqPPbuxMe25yedNbWxMKLelxRntynfBudlz\nZmntwL664+51Wjuwr86ek9jeUks2GhUaGhq0oaFBcZZZMSdy/Fx3/Lr3wolCMSf8wZwwJ/Klvl71\njTc8KSrSUEySbOAeYOe45zsBd2c7z8vNb6EyRYpkiyJJF7Gy2I1aSd7XcW7a8xYv7mzXRRepnnlm\n/tdSO7Cv/urlUfpX3UV/9fIorR3Yt7O9pZhsNGq4XzLmRB6f67+5z4t1otBrMSf8xZwwJ3Ll619X\nnTev6GIiT6aOVy5RjTtq3GrEqvoKMCaH80qGTJEi2aJIUkWsbAMswIlaSd7XcW7a8xYsAODjj+EP\nf4ALL8z/Wraq70P9+BgA9eNjDB7ep7O9HclG42vuSDZq5I45kXpfugiuNe7zYpwo5lrMCf8xJ8yJ\nXLAJ9rklyV4oIv9PRBrc7Va6AjTKgkyRItmiSFJFrLwHTMCJWkne13Fu2vMmTACcOV1HHw35BqvU\n19fzQfOXNC9cB0DzwnV8uPzLzvaWXbLRkDAnUu9LF8HVkWyhGCeKuRZzwn/MCXMiF6zjRU63GnsD\nPwb+5m4/BnpnO8/LjQDv3aeKFMkWRZIcsdK7qqrz2KnuvftU5yaf13Hv/tNPVbfYQnXZssKupePe\n/ajdMt+7L5Vko1ED57aKOZHj5zo5gqsQJ4rFnPAXc8KcyJWbbsp/Ck0pQoZbjVmTZEeBqEerQGLU\nyaBBgxKOzSdaBeCqq2DpUrjjDn+uJZf9RnoyRqsE14ZIO5EtCitfJ/y+llz2G+kxJ7LvMycc7r8f\nbrkFHnzQk+IiSyYnMi0ncZeqHheXLDsBDTD5aVBCFUP8BxQKT5L9xRew/fbwxBMwerQPDfUYv/4g\nRJHjjjuOu+66qyMp8KLk/eZEF/l8qZQb5kQX5kQX5oTjxIYNYzj5ZFhYVjeiu1PochJbu/8OT7Wl\nO8+PjQCGkIshPoy4f3W11tbUpF16IhvXX696/PE+NdRjGhvPVogpjFKIaWNjeSfhWumuYovzQ8Sc\nSENyWH2mUPlyw5wwJ1JhTnQ5cfrpF+pmm4XdKv+hyOUkTgdGZjvOzy3KQsWHEa/CSZBdaALTtWtV\nBw9WXbjQ50Z7wOLFi12ZusKNIVZ0mHMp4H7JmBMpSA6rzxQqX26YE+ZEKsyJ7k707t2mn30Wduv8\nJVPHK5eoxm2BP4jIWyLyfyIyRUR2zeG8iiA+jLgZ2I7UYcG5MGMGfPWrsPPOfrTUW5xw5u6BzsWG\nOZcQ5kQKksPqM4XKlxvmhDmRCnOiuxObbfZ5RUc2Zu14qerlqnoAMA54EjgfZ2Vig8QQ43rgbVKH\nBWdj/Xr4xS/g0kv9aqm3OOHM3QOdiw1zLhXMidQkh9VnCpUvN8wJcyIV5kR3J+rre1rHKxMicomI\nPATMBUYA5+Gs4WYAdXV13DxzJvvHYkweMIDW6mr2ralh9wED2D8W4+aZM3OaOPnHP8Luu8NuuwXQ\naA8YM2YMjY1nABOBUcBEGhvPKPvJxB2YE6mJ92H3AQM4NhbjjMbGzuf5OFFqmBPmRCrMiVRO9K3o\njlfW5SRE5AVgE/Ag8DhO/q0NAbQtvg2arZ1hU0xU48aNMGIE3H03lNqP40qK4OrAjeB6DnMiLRbB\nZU6YE4mYE11OXH658/oVV4TbLj/JFNXYM9vJqrq7iAwA9gEOBmaIyCpV/ZrH7fSdfNYsgcKXhKir\nq8vrnNtvhzFjiu90+ZWxPlO5Y8aMyfnLpcz+0ByEOZEz+TrhFeZEoJgTeVDJTgwbBk8/7W/7Ik26\nWffaFSmyE/BDYA7wBjAPuDLbeV5ueBCtkilzfPL+2poa7VddnXOob7ayM7Fxo+p226nOn1/QZXWS\nLdN8oeU0TpnqSYZ6r9oXBXAiuMyJAssNCnMiOMwJcyIfJx5+WPWgg/xrXxSgyOUkHgB+CnwVqM52\nvB9bsUIpw4PlAAAgAElEQVRlyhyfbv/m7vIQ2UJ9s5Wdjf/9X9X99y/q8rJmmi+0nCvmbd8tFLiQ\nDPVetS8quF8y5kSK/79iffAKcyJYzAlzIh8nXn1VddQof9oXFTJ1vHKJajxcVa9X1adVtbWwcbVw\nyZQ5Pt3+epzlIbKF+mYrOxNtbU56oGIjGZuzZJovtJzefXtQVTWUYjPUe9W+KGFOpP6MF+ODl5gT\nwWNOmBMO2Z3oSJStcVPyytGJdOSyjlfJkxzOmxy+m2p/M45U2UJ9s5WdiTvvhC23hIaG/K8puQ2Z\nMs0XWs76tW20t6+g2Az1XrXP8A6/nCjGBy8xJ4x8MScKK6cQJ/r3h1694OOPvW9fSZBuKCxKGx7e\nu0+V/T15f8e9+3TH5lt2KtraVMeOVf3nP4u6rE6yZZovtJzGKVM8yVDvVfuiABmGkIPaouxEIT74\ngTkRHOaEOZGvEzvvrPrii/60LwpkciLrchJRwKsw4WzREvEhr4MGDcoYsZEt03w27r7bWTD1mWdA\nUqfR9Pz6Ci2n0HKLfY+iSsbkp8G1IdJOJB8bFuZEMJgT5kS+5X7jG/CDH8CIERXoRLoeGXA/cF+6\nLd15fmwEkIMrU9RJ8r5DJ03SGOgoN+fW1MbGvOpqa1MdP171/vu9voro0DhlqkJMq6pGOMmCp0wJ\nu0lFc/jhh+sRRxzRMZHYnPDQiUrAnDAnjC7OPFN13/3uKjsnOqCQqEZgv0xbuvPy3YBDgKXA68AF\naY7x9Q3KFHWSal/MTXQa/zyfRLh//7vqbruptrf7eFEhUq7JgpuamrSpqanjS8ac8NCJcsecMCfM\niUSmTv1Q4Rdl50QHmTpeaRdQVdXH0+3zChGpAm4CDgRWAs+JyL2qutTvuuPpjDpZ50zqS446Sd43\nFCfRacfzbXCSgeayaKIqTJ/uRDJ6dYsxaixYsICqqqG0t3dFuVRVDcn5PYoq++23X+djv/yoRCcq\nAXOicMyJ8mTdutdw0giVlxO5kEuuxpEicreILHYzz78lIm95VP8EYJmqLlcnBHkOcJRHZedMpqiT\nVPtW4CQ67Xj+HuScCPfhh2HDBjgq8KsMjgkTJnSLcmlvX1lWyYLNCe+cqATMiaIwJ8qQffbZFhhI\nOTuRlnRDYdo1fDsf55fGQmA4MA2PViQGjgVmxD0/CbghxXFejwJ2I1PUSfK+jnv3I/O8d9/erjpx\nouqcwoKgSorGKVPce/c7lOW9e3PCGycqCXPCnDC6eOMN1f4DVpe1E5rmM51LkuznVXUPEVmkqjvH\nv1ZYVy+h7GOByar6fff5ScAEVZ2adJxe3pFVE2hoaKCh2MWvUpApmiJbVGMuPPooNDbCq69Cjx6e\nNz9ylEuy4KamJpqamjqfX3HFFaiqmBPFO1FplMt7ZE6YE8WyYQMMGADPP7+E558v/fconROpjs2l\n4/U08DXgbuAxnBHUa1V1x2IbKiITgWmqeoj7/EKcXuJ1ScdptnamI6jQ2Vzq2W8/OOMMOOmk3NuX\nj7j5/EEo9NqBot+HXMottH3FhPrnijiT83pgTqR938G7/89kzIn82leME62tsHQpvPyysw0cCBdd\n1P04cyJ1OUE4kW/nLkpOHHbYHrz8chVDhhRWbqHtC+p7Il3HK5dh3j2BfjhzA28D7gEmZjsvlw1H\n1DdwhqZrgJeAMSmOK2ioz6vEpMnlTG1sTHh+dtLzVPU0NanusINqa2vXa9kSgk49u1FrYlU6ZFQv\nrYlV6dSp6YeqM5U1a9acgpP6xp9bU1Orsb69OuuYenZjwYlSk8utru5XdPtisYHa2Hi2JwmMM4Fz\nW8WcSONE/+pqra2pKagOcyI8J1paVB99VPVXv1I95RTVXXdVjcVUd9xR9bjjVH/+c+fvWCrMiXCc\nyMeHbGWF4YTIC3rllXNzLrfUvic03Wc63Y5uB8IAoH+ux+dR7iHAa8Ay4MI0x+R90V4lJs0WJjzP\nfZ6tngMPVJ05M7HcTAlBFy9erDWxqoT9NbGqlKG2mcpatWqVxmIDtZCkvqnOre7dR/+4apxeMW/7\nhPblkyg1VbmwucKqIts3L+c2FEOHUOZEdydW4SQOLqQOcyIYJ3r3rtMnn/xIZ89WvfBC1UMPVR0y\nRLW2VvXrX1c96yzVGTNUn31Wde3arFWqqjkRhhP5+JCtrPCc+FRrak4J3Qk/vydSbblENX5FRBbh\nTJpcJCIvi0jR9+07UNWHVXVHVR2pqtd6VW5n6K/7vNDEpKnK2YauMOG+wDDIWM+//w1vvAHf/W5i\nuZkSgi5YsIBBw6oT9m+xTTULFixI2cZ0ZTU3N1NTU08hSX1TnduzZigtzRvp3bdHQvvySZSaqtz4\ndLOFt6/7/0YhCYxzwZxI7UQzsB2ZfchUrjnhtRMDgGMSyl2/fjknnNCH//s/iMXgzDPhqafgk0/g\niSfgppucKRETJkCfPlmr7MScCNaJfHzIVlZ4TtQisnPZfk+kI5ck2X8EfqSq9apaD5yFM5QcaTKF\n/hZbznt0hQmvBd6FjPVMn+7Mi6iuTiw3U0LQCRMmsPrd1oT9H73XmjLUNlNZ9fX1bNzYTD4JTOPL\nTT5308YV1NXXsH5tW0L78kmUmqrc+HSzhbev+/9GIQmMc8SciCunw4l64G0y+5CpXHOiMCfa2+H1\n12HhwlGsXXsC8Jm7ZzzQCHzkPn+D3r1H88ILn/PXv8JllzlL29TXe7KuoDkRV47fTuTjQ7aywnNi\nJZs2xcr5eyI16YbCtGv49sUUr72Q7TwvN4q8d19sYtLkcjru3ad7Hl/Pc8+pbrON6vr13cvNlhB0\n6lTn/v3WI3Ofz5KqrI7724Uk9Y0/t+PefUcdU6c2FpwoNbnc6up+RbfPuXc/1ZMExpkgxRCyOdHl\nQD93PkshdZgT2Z347DPV+fNVf/c71e9/X3WvvVT79lWtr1c96ijVY45ZpDU1p2i/fkdq795bmBNZ\nKGUn8vEhW1lhOFFdfZpOmPBOzuWW+vdEx5ZLVONvgRgwG2cC5fHAeuDP7if9BW+7ginboNnamY6w\noxqPOgoOOgimTCmsfRbBlV/7AoxqnIE5kfZ9B4tqLNYJVXj66fdZs2Y4zc21nZGFH3wA48bBLrt0\nbePHQ21t+mszJzJTyk6UclRjS8sIrrhic559trByC21f2FGNuXS85mXYrap6QDGNy4Wgss57zcsv\nw6GHwptvOnMpCqHQNmc7r9APc1DvYZQz1LtfMk1pdpsTPlOOTqxdC6+80rVsw8svw6JFzjpHHZ2r\n7bb7jM02W87XvrYVW20Vrf8rcyI8vOosheHEe+85cwlXrsyr2XnXEwZFLScRhQ0PViT2KmQ4H771\nLSc0u1CyhdYXel78/ljf3lpTU5tTWG2h7cmXYsKag4AMQ8hBbaXqRLGUuhPt7arLl6ved5/q9OnO\n34iRI51lG3bbTfXUU1V/8xvVxx5TXb266zxzwpxIRTF/k6PgxKZNqtXVqhs25NzsTkrZiVw+zIOB\nmcBD7vOxwOnZzvNyK1Yor0KG8+GVV1QHD1b94ovCzs8WWl/oefH7/7hqnFb37pNzuHsh7SnkugsN\naw4KnFsp5kTAlJoTX37pzPGcOVN16lTV/fZT3Xxz1a22Up08WfWnP1X9y19UFy1S3bgxc/vNCXMi\nmWL+Jkfpe2LYMNW3387/2kvBCU3zWc0lqvF/gX8CHWvLvg6ck8N5kaHZo5DhfLjqKjjnHOjbN/ux\nqWjOElpf6Hnx+1uaN1LdaxtyDXcvpD350lxEWHPAmBMBE1Unhu8c46MVW/LxioOo7nUBJ51Uxdix\nzirvp50G8+bBsGFw8cXOKvDvvw8PPwzXXQff+Q7stFNixHOqeswJ/yk1J4r5mxyl74lhw+Ddd3O8\n6Lh6SsSJlOTS8RqkqncB7QCquglo87VVHlPvUchwrrz+OjzyCJx1VuFl1GcJrS/0vPj9dfU1tG54\nj1zD3QtpT77UFxHWHCTmRPBEwYkNG+Cll+D222HmzHG8sfAuTh44j/N2ncNdV5zIZ5/UcsQR1cye\nDWvWwMKFcMcdcN55cPDBsOWWhV23OeE/peZEMX+To/Q9MWwYvPNOjhcdV08pOJGWdENh2jV82wRs\ngRsaDEwEHs92npcbHt67LzZkOBdOOUX1yiuLLydbaH2h58Xvj/XtpTU1tTmF1RbannwpJqw5CHBu\nq5gTIRCkE/36Hag1NcfoCSe8qCeeqLrzzqq9e6uOHat6wgmq116resEF87T/ZjvoyF3NCXMieIr5\nmxyV74nzz1e95pqcm91JKTihaT6ruUQ17g7cCOwEvALUAd9S1YUZT/SQUopWeestJ0rjjTdgs82K\nLy++zatXr04IG84nLDjT8+RyM4Un5xMF4+fyHfmc6zVuBNfTmBOh4LUTQ4fW89FHdTz55Gf8+99r\nee+9LVi0qIr169sYN24TEyf2pa5uJW1tz3PUUSPZddfRGcvN1F5zwj8q1Ynk9ib//Y7698ScOXUs\nXQq/+13x1x41J7SYqEagJzAOR6rqXM7xcsODXzJB8d//rXrJJd6Xm5wMdfLkyTlHdOSTGPXggw91\n81iNUohpY+PUnNvoV9RjPtErQUS64Py6NydCphAnVq9W/de/VL/73ee1uuYO7RVbqLBWt956jf7X\nfzmJoM8773Ht1WsnHTDAnMgVcyIalOL3xLk/eVyPOKKoy07Zxig4oek+q2l3wJ7AVnHPTwbuBW4A\nBqY7z4+tVIRavlx14MDEMHAvSE6Gmm/y3dwTo/6tW7kQS5t0Ndd6iiGf6BW/I10WLFig77//fqdQ\n5kR45OJEr14T9Q9/WJOQCHrAANW99tqoNb1u0f+67GK99tkT9ep/72JOFIg5ER1K9Xui34B9dKed\nMoT15kCUnOggU8cr0+T6PwAb3SGzfYFrgT8Ba3BWKDaSuO46J7nsFlt4W25yMtTefXsgVUMoNsqk\ne2TIGpzUrompXtMlXc21nmLIJ3rF70iXM888k5qaGsCcCJt4J9Z+2p+Vr00EuZD4//uNG+9izpxN\nxGLw/e/D/Pnw6adw440vsd24S/n2FQ8wcsIr7DgRc6JAzInoUKrfE1tu+3HeUY2pyo2KE7mQqePV\nQ1U/dh8fD8xQ1b+q6qXACP+bVlqsWAGzZ8O553pfdnIy1PVr29D2lRQbZdI9MqQWJ7VrYqrXdElX\nc62nGPKJXvE70qWtrY2BAwd2PDUnAqa9HZYtg7vvhmee+Qbvv/F//PeQB/n+sIf5x40/AR2C8/kF\neIXevXflzjvbuOwyOPpo2G47JxG0OWFOlCOl+j2xesU7rF/fky+/LOiyO8uNihM5kW4oDGeCZE/3\n8VJg3/h96c7zY6MEhpDPPlv1xz/2r/zkZKgd9+6LjTJJjgyZNKnj3v3Igu/dex31mE/0ip+RLuPG\njdPW1taO+SzmhI+kSwQ9fLjqkUeqXnqp6mGH3arVvUbqViN6mxPmRNk7kQul+j0xYoTq0qVFXXpk\nnOiAQqIaReRi4DBgNbAtsLuqqoiMAG5X1X087wWmwatoFb/48EMYMwZefRW23tq/evKJVkkmn8iW\nfJOu5lpPMUQhWuWqq67iH//4B08//TTAS5gTRaMKzc2JOQrTJYLeeefukcLmhDnRQbk4USyl6MQB\nB8DPfgYHHZT35ebc/mKOLYSCk2SLyERga2Cuqq51XxsF9NMAss3HtSPSQv30p7BuHdx4Y+r9fv0H\nP/XUU8ydO5dJkyaxzz6Jf9+K+QCWWjh1kDzzzDPsvffe4DhgTuRBRyLohQvhmWfW8fzzm3jrrb7U\n1laxyy4wfnxXJ2vkSOjRI/86zIngMSe8oZK/J045BRoa4HvfK6qYSGFJsn2kpcXJv/bOO6n3+xVO\nPnnyZIWYStUOCjGdPHlyQXUmh9U2Np4d6cSjUYAySQjsFx2JoO+/31me4b/+S3XUqK5E0Pvt96b2\n7nO+Dht5tPbfbJg5UQaYE8VR6d8TF1+sesUVRRUROTI5EXqnKpctykL97GfOHJRU+BVOPn/+/JTh\nvPPnz8+rzu5htfNyDj+uZOxLpot161T/85/cE0GbE+WJOVE45oTqLbeonn56MVcbPTI50dPXsbYy\n55NP4JZb4D//Sb0/U4huMUOzc+fORaqGoO1d4bBSNYS5c+dSU1OTc50dYbXr1nWU0xcYRqowW7u9\nUtmoOsmdO+ZgLVzo/PvWW85twY5bhIcf7vybLiehOWEYiZgTsO22cM89BZ1akuSSJNtIww03wFFH\nOWHqqfArnHzSpEndwoS1fSWTJk3Kq87uYbVrgXcp2cSjhids3NiVCPonP3EmvG65pTMH65e/hFWr\nYPJk+MtfnHWx8kkEbU4YRiLmhJMou9i1vEqKdENhUdqI4BDymjWqgwapLluW+Ti/wslzuXefS53J\nYbWNjVMjnXg0ClBGt1U+/FB17lzVX/xC9aSTuhJBjxmj+u1vO8lr//EP1RUrnLlbXmBOlB/l5EQY\nVLoTn36q2q+fd39jokAmJ7ImyY4CUYxWufpqWLwY/vzn7MdWcrRKOZIxWiW4NuTlRGsrvPZa4pIN\nCxfC+vWJSzbssguMHQuxmI+Nx5woN0rRiahR6U4MGADvvNN9yZhSpeDlJPxERL4FTAPGAHtqhrDj\nqAn1xRewww7Q1OSs3+UlxWSSNxxCzTpfXLmeOPHRR4nzsF5+GZYudYbzk5dtGDbMWc09ypgTxVPp\nTpQb5ejEuHFO9pfx47Mf6wWhOpFuKMzvDdgRGAk8hrPoXskMIf/iF6rHHed9uZnCe5P3TZ3a6Ev4\ncakTdtb5YrZ8ndi0SXXJEtU5c1Qvukj1sMNUhw5V7d9fdZ99VH/0I9U//EH1mWdUv/jC87chEMyJ\n4qkkJyqBcnVi8mTVBx4Ipq6wnQit49XZAJhXSkKtXeuEyi9c6G252bLDJ++riVXpFfO29zT8uNSJ\nQtZ5L7ZcnejTR3X77VW/+U3VadNU//Y31bfeKp95EuZE8VSaE+VOOTtxxhmqv/+9//VEwQlbTiJP\nbr0VJk500pd4SaaQYqDbvi22qaZ33x7djo3yULLfdA97Lu/Q//ffd+ZFlCvmRPFUmhPlTjk7EVRk\nYxSc8LXjJSKPAIPjX8JJpnqxqt6fT1nTpk3rfNzQ0EBDQ4MHLcyP9evh+uvhvvu8Lzs+vLd+fKxb\neG/yvo/ea2X92jbAu/DjUicx7Hk8XoX+NzU10dTUVHT7wFsnfv3raZ2Pw3LCT8yJ4qk0J6LwPeEn\n5ezEsGHw2GP+1xMJJ9INhQW1UUJDyL/7neo3vuFf+ZnCe5P3ddy79zr8uNQJO+u8F1spOeE35kTx\nmBPlRbk68eijqg0NwdQVthOhLychIvOA81T1+QzHaNjt3LjRWaH7rrtgr73yiyrJh0yZ5VevXl1w\n1vl86syHKEbMlGoEV1z5JeFEMuaEgznhPeZEIvGfz0GDBiWUk8mXKDvx+utw6KHw5psFN9HzNhVD\nVKMaj8ZZ/nYd8D7wUIZjveuGFsitt6oefLDzOFNERDHJTrtFpJzdFZES69tba2pqPY/CaJwyVSGm\nVVUjFGLaOGVKzucGERkSRfAvgquknIjHnHAwJ8yJDoJwok//XtqvtndnOY1Tpvry+QvCibVrVXv1\nUm1r86TJoZPJidA6XvlsYQu1caPqdtupPvFE5oiIYpKdJp97xbzttSZWpb96eZT+cdU4re7dx/Mo\njMWLF6dMorp48eKc2htEZEgU8etLJp8tbCfiMSeyvw/ljjmRSBBO/HHVOO27eY8EP/xI6B6kE4MG\nqX7wQVHNjQyZnLBcjTkwaxYMHw5f/3pXRESqBKHZIk4ykXxu7749GDSsmvrxMVqaN1Lda5uUdRbD\nggULqKoamlBuVdUQFixYkFN7070PRmVhTnS115wwIBgnWpo3Mni7mgQ/kj+7peZEpeRstI5XFtra\nnPRAl17qPE+MiID4iIj4iBPIL4ok+dz1a9tY/W4rzQvXUVdfQ+uG91LWWQwTJkygvX1FQrnt7SuZ\nMGFCTu1N9z4YlYU50dVec8KAYJyoq6/hw7c3JviR/NktNScqpeMV+m3EXDZCHEKeNUv1q19NXJQy\nU0REMclOM0WkxPr20pqaWs+jMBqnTHHv3e9Q8L37SksejN1W6YY54WBOmBMdBOFEn37OHK+Ochqn\nTPHl8xeUE2edpfrb33rR4vDJ5EToUY25EFa0Snu7s1Dqr34FhxySuM+vaJVMebYAX6Iwyi2Cy28s\nIXBqzInU7a0EzInUBOEEkNaPUnPiuuugpQV++ctiWxs+mZywleszcM890LcvTJ6c33l1dXUFf+BX\nr17N4sWL6devX2c58WX58Yd8zJgxeYvUQTHXalQO5oRhJOKVE2PGjOnmQ6k6se228ELaNOhlRLqh\nsChthDCE3N6uussuqvfd131fMaHAmZh6dqPWxKp0yKheWhOr0qlTGz0p1/AW7LZKN8yJysac6I45\nkT9PPqm6995ht8IbMjlhtxrTcN99cPnlTu9b4gYLW1paGDl6Oy6bN7QzLcOV+69g2dK3i16cbtc9\nduKaZ0Z0lnvRxDd46flXCv6VYfiD3VZJxJwwzIlEzInCWL4cvva18phgn8kJi2pMgSpMnw6XXJLY\n6YLsSUoLZcGCBZ2h8h3lbrFNdU4hu4YRJuaEYSRiThTGkCHw4YewaVPYLfEX63il4OGHYd06+OY3\nu+8rJhQ4ExMmTOgMle8o96P3WnMK2TWMMDEnDCMRc6Iwqquhrg5Wrgy7JT6T7h5klDYCvHff3u7c\nY549O/0xxYQCZ2LqVOfe/dYjy+/efTmBzWfphjlR2ZgT3TEnCmOvvVTnzw+7FcWTyQmb45XEv/4F\nZ50Fr74KPXqkPy6KIbtGMNh8ltSYE5WLOZEacyJ/jjsOjjkGvv3tsFtSHLacRBpSSTF9OvzsZ5k7\nXdA9PNYrwQYNGsTYsWMZNGhQwWXkSyWuO2SkppjPQrwTXn6mzAkjTMwJh6CcqIjV69MNhUVpw4ch\n5DmzZunAWEx3r63VgbGYzpk1Sx9/XHWHHVRbW/Mry6uwYb/Cj6NWZ6lDmd5WSeVEIXj5mTInSgNz\nIjPmRO785jeqeSyMH1kyORF6pyqXzWuhVq1apQNjMX3ZCWDUl0EHxmK6774b9P/9v/zLKjTTvB/l\nRL3OcqAcv2TSORHm59icKB3MiczlmBO5c/fdqkcd5UvRgZLJiYqMamxubqa+piYuXzoMkr1Ztkz5\n7nfzL8uLsGG/wo+jVqcRTVI5Mby6OtTPsTlhhIk5EU6dlXCrsSI7XvX19TRv3BiXLx3eXn8+55yz\nkZqa/MvyImzYr/DjqNVpRJNUTixvbQ31c2xOGGFiToRTZyV0vEK/jZjLho/37ncbMEAH9NpHBw5c\nq+vXF1aWV2HDfoUfR63OUocyvK2imuiEF/NZvPhMmROlgTmRGXMid9raVGtqVNet862KQMjkREUv\nJ9ERpXHZZeM59NBeTJ1afFnFRnyEEU1lEVz5Uc6h81H8HJsT0cecCK4cr8uKYp3bbw9z58KIEb5W\n4yuZnKjojhfAwoUweTK89RbEYr5UEQrZJLEvlsIp5y+Zcsac8A9zojSJqhP77efkSj7ggMCq9BzL\n1ZiBn/8cfvKT8up0zblzNiNHb8d3f/ANRo7ejjl3zs5rv2GUG+aEYSQSZSfKfZ5XRY94LV4M++8P\nb74J/fp5XnwotLS0MHL0dlw2b2hn9vor91/BsqVvU1dXl3W/kR37dV9amBP+Y06UFlF34qKLoG9f\nuOQS36vyDRvxSsNVV8E555RPpwuyh/5auLxRaZgThpFI1J3YdtvyHvEKreMlIteLyBIReUlE/ioi\nA4Ksf9kyZ/LeWWcFWav/ZAv9tXD56BK2E+WKOVG6mBP+EHUnyv1WY5ihvwcBVe7ja4FrMhxbaERn\nWk49VXXaNM+LjQTZQn8tXL448Cl0Pmwnyhlzwl/MidIjyk689JLqTjsFVp0vZHIiEnO8RORo4FhV\nTbluvNf37t96C/bcE954Azbf3LNiI0VUo1XKgSDmswTtRCVgTviHOVGaRNWJjz+G7baDNWsCq9Jz\nIr+chIjcB8xR1Vlp9nsq1Pe/D1tu6UQ0Gka+BPQlE6gThlEM5oThJarO3Ov334cBJXpzOZMTPX2u\n+BFgcPxLgAIXq+r97jEXA63pZPKad96Bu++G118PojbDSCSKThhGmJgTRjIiXfO8xo0LuzXe42vH\nS1UPzrRfRE4FDgOyLpM2bdq0zscNDQ00NDQU1Kbrr4f//m8YNKig040KpKmpiaamJk/KiqIThpEv\n5oThNx2RjaXS8crHidBuNYrIIcCvgH1V9aMsx3oyhLxyJey0EyxZAoMHZz/eMFLh122VMJwwDC8w\nJwyvOf10mDgRzjgj7JYURlTX8boR6Ac8IiIviMjNflf4i1/AKadYp8uILIE7YRgRx5yoUIYNc6YG\nlSO+3mrMhKqODLK+Vavg9tvhlVeCrNUwcidoJwwj6pgTlcuwYfDkk2G3wh8qZuX6X/0KTjgBhgzx\np/yWlhaee+45Wlpa/KnAMEoI88EwEjEn8qOcF1GtiI7X6tVw661wwQX+lG8Jdg2jC/PBMBIxJ/Kn\nnDtekVjHKxvFTpq85BLnVuOMGR42yiXsZKJG8FhC4PSYD5WJOZEec6IwvvgC6urgyy+d5SVKjahO\nrg+ETz+F3/8eLrzQn/LDTiZqGFHCfDCMRMyJwujXD2Ix545VuVH2Ha8bboAjjoDtt/en/LCTiRpG\nlDAfDCMRc6JwyvV2Y2hRjUHw2Wdw443w1FP+1VFXV8ctN9/KD/Y/g8HD+/Dh8i+55eZbbQjZqEjM\nB8NIxJwonI6O1+67h90SbynrOV7XXOMsH/GXv/jQqCQswW7lYPNZsmM+VBbmRHbMifz54Q+dlesb\nG8NuSf5EPkl2NgoRau1a5/bivHkwdqxPDTMqEvuSMYxEzAnDD665xpmnfd11Ybckfypycv0tt8C+\n+42IiVkAAAc7SURBVFqnyzAMwzBKEZvjVUKsWwe//CU8/HDYLTEMwzAMoxDKNW1QWY543Xor7LUX\n7LJL2C0xDMMwDKMQynXEq+zmeG3YADvsAPfeC3vs4XPDjIrE5rMYRiLmhOEHGzZA//7OXawePcJu\nTX5U1Byv226D8eOt02UYhmEYpUyvXjBwIHzwQdgt8ZaymuPV2grXXguzLQ2WYRiGYZQ8227r3G4c\nOjTslnhHWY143XEHjBwJe+8ddksMwzAMwyiWcpxgX/Idr6amJgA2bYKrr4ZLLw2n/jAIs26rP7qE\n/b6YE5Vbf1QJ+30xJwqn2An2YV9/Ksqm4zV7tjMUue++4dQfBmF/oCq9/qgS9vtiTlRu/VEl7PfF\nnCgc63hFlLY2uOqq4Ee7DMMwDMPwj3JcUqIsOl533w2bbw4HHhh2SwzDMAzD8IqOyfXlRMms4xV2\nGwwjniisWRRm/YaRjDlhGImUdJJswzAMwzCMcqAsbjUahmEYhmGUAtbxMgzDMAzDCAjreBmGYRiG\nYQREWXS8ROR6EVkiIi+JyF9FZECAdX9LRF4RkTYR2T3Aeg8RkaUi8rqIXBBUvW7dM0XkQxFZGGS9\nbt3biMhjIvKqiCwSkalBt6EUMCfMCSMRc8KciApl0fEC5gLjVHVXYBlwUYB1LwK+CTweVIUiUgXc\nBEwGxgEniMjooOoHbnPrDoNNwLmqOg7YGzgr4GsvFcwJc8JIxJwwJyJBWXS8VPVRVW13nz4DbBNg\n3a+p6jIgyFDqCcAyVV2uqq3AHOCooCpX1fnAJ0HVl1T3B6r6kvv4C2AJUEbpU73BnDAnwmhLlDEn\nzIkw2pKKsuh4JXEa8FDYjfCZoUD8knLvEaEPVVCISD2wK/BsuC2JPOZEhWBO5Iw5USFE0YmeYTcg\nV0TkEWBw/EuAAher6v3uMRcDrao6K+i6jWARkX7A3cDZ7i+aisOcMOIxJ8wJI5GoOlEyHS9VPTjT\nfhE5FTgMOCDoukNgBbBt3PNt3NcqAhHpiSPTHap6b9jtCQtzIgFzwpwwJxIxJyLqRFncahSRQ4Dz\ngSNVdUOYTQmonueAESIyXERqgG8D9wVUdwdCsPMV4vkjsFhV/yek+iOPOWFOGImYE+ZEVCiLjhdw\nI9APeEREXhCRm4OqWESOFpF3gYnAAyLi+7wBVW0DGnGidF4F5qjqEr/r7UBEZgFPA6NE5B0R+V6A\nde8DnAgcICIvuv/fhwRVfwlhTpgTRiLmhDkRCSxXo2EYhmEYRkCUy4iXYRiGYRhG5LGOl2EYhmEY\nRkBYx8swDMMwDCMgrONlGIZhGIYRENbxMgzDMAzDCAjreBmGYRiGYQSEdbxCQEQuFpFXRORld32R\nPT0ufz8R6ZaiIt3rHtR3VHzmdxGZJyK7e12PUb6YE4aRiDlRvpRMyqByQUQm4qSs2FVVN4nIQKDG\nh6rSLdDmx8JtRwMPAEt9KNsoc8wJw0jEnChvbMQreLYGVqvqJgBV/VhVPwAQkd1FpElEnhORh0Rk\nsPv6PBH5rbsC70IR+Yr7+p4i8rSIPC8i80VkZK6NEJE+IjJTRJ5xzz/Cff0UEfmrW/9rInJd3Dmn\nu689IyIzRORGEdkbOBK43v1Vtr17+HEi8qyILHVXETaMdJgThpGIOVHOqKptAW5AX+BFnF7/74B9\n3dd7Ak8BW7jPjwNmuo/nAX9wH38dWOQ+7gdUuY8PBO52H+8H3Jei7s7XgauA77iPa4HXgBhwCvCG\nW3YvoBkYivOH4G332B7AE8AN7vm3AcfE1TMP+IX7+FDgkbDfd9uiu5kTttmWuJkT5b3ZrcaAUdW1\n7n3trwMHAHNE5ELgeWAnnDxigjMauTLu1Nnu+U+KSH8RGQAMAP7k/oJR8rt1PAk4QkTOd5/X0JXJ\n/l+q+gWAiLwKDAfqgCZVXeO+/n9Apl9O97j/Pu+ebxgpMScMIxFzoryxjlcIqNPFfwJ4QkQWAScD\nLwCvqGq64dbke+4KTAceU9VjRGQ4zi+IXBHgWFVdlvCiM7dgQ9xL7XR9TvLJMt9RRhv2OTOyYE4Y\nRiLmRPlic7wCRkRGiciIuJd2BZbjDOHWuR9oRKSniIyNO+549/WvAWtU9XOc4dwV7v58M7//E5ga\n165dsxz/HLCviNSKSE/g2Lh9n+P8qkpHPiIaFYY5YRiJmBPljXW8gqcfcLs4YcIvAWOAaaraCnwL\nuM59/UVg77jz1ovIC8DNwGnua9cD14rI8+T/fzkdqHYnYb4CXJnmOAVQ1ZXA1cAC4Emc+/hr3GPm\nAOe7ky+3J/WvLsNIhzlhGImYE2WMuBPbjAgjIvOAn6jqCyG3o68796AH8DecSZ33htkmozIxJwwj\nEXOidLARr9IgKr3jaSLyIrAIeMtkMkLEnDCMRMyJEsFGvAzDMAzDMALCRrwMwzAMwzACwjpehmEY\nhmEYAWEdL8MwDMMwjICwjpdhGIZhGEZAWMfLMAzDMAwjIP4/92Fl0Sq/GFwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xb108240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_min, x_max = X_train[:, 0].min() - .5, X_train[:, 0].max() + .5\n",
    "y_min, y_max = X_train[:, 1].min() - .5, X_train[:, 1].max() + .5\n",
    "xs = np.arange(x_min,x_max,0.5)\n",
    "fig, axes = plt.subplots(1,3)\n",
    "fig.set_size_inches(10,6)\n",
    "for i in [0,1,2]:\n",
    "    axes[i].set_aspect('equal')\n",
    "    axes[i].set_title('Class ' + str(i) + ' versus the rest')\n",
    "    axes[i].set_xlabel('Sepal length')\n",
    "    axes[i].set_ylabel('Sepal width')\n",
    "    axes[i].set_xlim(x_min, x_max)\n",
    "    axes[i].set_ylim(y_min, y_max)\n",
    "    plt.sca(axes[i])\n",
    "    for j in xrange(len(colors)):\n",
    "        px = X_train[:, 0][y_train == j]\n",
    "        py = X_train[:, 1][y_train == j]\n",
    "        plt.scatter(px, py, c=colors[j])\n",
    "    ys = (-clf.intercept_[i]-xs*clf.coef_[i,0])/clf.coef_[i,1]\n",
    "    plt.plot(xs,ys,hold=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first plot shows the model built for our original binary problem. It looks like\n",
    "the line separates quite well the Iris setosa from the rest. For the other two tasks, as\n",
    "we expected, there are several points that lie on the wrong side of the hyperplane.\n",
    "Now, the end of the story: suppose that we have a new flower with a sepal width of\n",
    "4.7 and a sepal length of 3.1, and we want to predict its class. We just have to apply\n",
    "our brand new classifier to it (after normalizing!). The predict method takes\n",
    "an array of instances (in this case, with just one element) and returns a list of\n",
    "predicted classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\n"
     ]
    }
   ],
   "source": [
    "print clf.predict(scaler.transform([[4.7, 3.1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our classifier is right, this Iris flower is a setosa. Probably, you have noticed that\n",
    "we are predicting a class from the possible three classes but that linear models are\n",
    "essentially binary: something is missing. You are right. Our prediction procedure\n",
    "combines the result of the three binary classifiers and selects the class in which it is\n",
    "more confident. In this case, we will select the boundary line whose distance to the\n",
    "instance is longer. We can check that using the classifier decision_function method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 21.74978749  -3.59105443 -32.76527648]]\n"
     ]
    }
   ],
   "source": [
    "print clf.decision_function(scaler.transform([[4.7, 3.1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating our results\n",
    "We want to be a little more formal when we talk about a good classifier. What does\n",
    "that mean? The performance of a classifier is a measure of its effectiveness. The\n",
    "simplest performance measure is accuracy: given a classifier and an evaluation\n",
    "dataset, it measures the proportion of instances correctly classified by the classifier.\n",
    "First, let's test the accuracy on the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.830357142857\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "y_train_pred = clf.predict(X_train)\n",
    "print metrics.accuracy_score(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This figure tells us that 82 percent of the training set instances are correctly classified\n",
    "by our classifier.\n",
    "\n",
    "Probably, the most important thing you should learn from this chapter is that\n",
    "measuring accuracy on the training set is really a bad idea. You have built your\n",
    "model using this data, and it is possible that your model adjusts well to them but\n",
    "performs poorly in future (previously unseen data), which is its purpose. This\n",
    "phenomenon is called overfitting, and you will see it now and again while you\n",
    "read this book. If you measure based on your training data, you will never detect\n",
    "overfitting. So, never measure based on your training data.\n",
    "\n",
    "This is why we have reserved part of the original dataset (the testing partition)—we\n",
    "want to evaluate performance on previously unseen data. Let's check the accuracy\n",
    "again, now on the evaluation set (recall that it was already scaled):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.605263157895\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf.predict(X_test)\n",
    "print metrics.accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obtained an accuracy of 68 percent in our testing set. Usually, accuracy on the\n",
    "testing set is lower than the accuracy on the training set, since the model is actually\n",
    "modeling the training set, not the testing set. Our goal will always be to produce\n",
    "models that avoid overfitting when trained over a training set, so they have enough\n",
    "generalization power to also correctly model the unseen data.\n",
    "\n",
    "Accuracy on the test set is a good performance measure when the number of instances\n",
    "of each class is similar, that is, we have a uniform distribution of classes. But if you\n",
    "have a skewed distribution (say, 99 percent of the instances belong to one class), a\n",
    "classifier that always predicts the majority class could have an excellent performance in\n",
    "terms of accuracy despite the fact that it is an extremely naive method.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within scikit-learn, there are several evaluation functions; we will show three\n",
    "popular ones: precision, recall, and F1-score (or f-measure). They assume a binary\n",
    "classification problem and two classes—a positive one and a negative one. In our\n",
    "example, the positive class could be Iris setosa, while the other two will be combined\n",
    "into one negative class.\n",
    "\n",
    "* Precision: This computes the proportion of instances predicted as positives that were correctly evaluated (it measures how right our classifier is when it says that an instance is positive).\n",
    "* Recall: This counts the proportion of positive instances that were correctly evaluated (measuring how right our classifier is when faced with a positive instance).\n",
    "* F1-score: This is the harmonic mean of precision and recall, and tries to combine both in a single number.\n",
    "\n",
    "We can define these measures in terms of True and False, and Positives and Negatives:\n",
    "                    <pre>\n",
    "                                 | <b>Prediction: Positive           </b>|<b> Prediction: Negative </b>\n",
    "                    Target cass: | Positive True Positive  (TP)   | False Negative (FN)\n",
    "                    Target cass: | Negative False Positive (FP)   | True Negative (TN)\n",
    "                    </pre>\n",
    "\n",
    "With m being the sample size (that is, TP + TN + FP + FN), we have the\n",
    "following formulae:\n",
    "\n",
    "* Accuracy = (TP + TN) / m\n",
    "* Precision = TP / (TP + FP)\n",
    "* Recall = TP / (TP + FN)\n",
    "* F1-score = 2 * Precision * Recall / (Precision + Recall)\n",
    "\n",
    "Let's see it in practice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "     setosa       1.00      1.00      1.00         8\n",
      " versicolor       0.42      1.00      0.59        11\n",
      "  virginica       1.00      0.21      0.35        19\n",
      "\n",
      "avg / total       0.83      0.61      0.56        38\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print metrics.classification_report(y_test, y_pred, target_names=iris.target_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have computed precision, recall, and f1-score for each class and their average\n",
    "values. What we can see in this table is:\n",
    "* The classifier obtained 1.0 precision and recall in the setosa class. This means that for precision, 100 percent of the instances that are classified as setosa are really setosa instances, and for recall, that 100 percent of the setosa instances were classified as setosa.\n",
    "* On the other hand, in the versicolor class, the results are not as good: we have a precision of 0.43, that is, only 43 percent of the instances that are classified as versicolor are really versicolor instances. Also, for versicolor, we have a recall of 0.27, that is, only 27 percent of the versicolor instances are correctly classified.\n",
    "\n",
    "Now, we can see that our method (as we expected) is very good at predicting\n",
    "setosa, while it suffers when it has to separate the versicolor or virginica\n",
    "classes. The support value shows how many instances of each class we had in the\n",
    "testing set.\n",
    "\n",
    "Another useful metric (especially for multi-class problems) is the confusion matrix:\n",
    "in its (i, j) cell, it shows the number of class instances i that were predicted to\n",
    "be in class j. A good classifier will accumulate the values on the confusion matrix\n",
    "diagonal, where correctly classified instances belong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 8  0  0]\n",
      " [ 0 11  0]\n",
      " [ 0 15  4]]\n"
     ]
    }
   ],
   "source": [
    "print metrics.confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our classifier is never wrong in our evaluation set when it classifies class 0 (setosa)\n",
    "flowers. But, when it faces classes 1 and 2 flowers (versicolor and virginica), it\n",
    "confuses them. The confusion matrix gives us useful information to know what types\n",
    "of errors the classifier is making.\n",
    "\n",
    "To finish our evaluation process, we will introduce a very useful method known\n",
    "as cross-validation. As we explained before, we have to partition our dataset into\n",
    "a training set and a testing set. However, partitioning the data, results such that\n",
    "there are fewer instances to train on, and also, depending on the particular partition\n",
    "we make (usually made randomly), we can get either better or worse results.\n",
    "Cross-validation allows us to avoid this particular case, reducing result variance and\n",
    "producing a more realistic score for our models. The usual steps for k-fold\n",
    "cross-validation are the following:\n",
    "\n",
    "1. Partition the dataset into k different subsets.\n",
    "2. Create k different models by training on k-1 subsets and testing on theremaining subset.\n",
    "3. Measure the performance on each of the k models and take the average measure.\n",
    "\n",
    "Let's do that with our linear classifier. First, we will have to create a composite\n",
    "estimator made by a pipeline of the standardization and linear models. With this\n",
    "technique, we make sure that each iteration will standardize the data and then\n",
    "train/test on the transformed data. The Pipeline class is also useful to simplify\n",
    "the construction of more complex models that chain-multiply the transformations.\n",
    "\n",
    "We will chose to have k = 5 folds, so each time we will train on 80 percent of the\n",
    "data and test on the remaining 20 percent. Cross-validation, by default, uses accuracy\n",
    "as its performance measure, but we could select the measurement by passing any\n",
    "scorer function as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.76666667  0.63333333  0.76666667  0.76666667  0.9       ]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score, KFold\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# create a composite estimator made by a pipeline of the standarization and the linear model\n",
    "clf = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('linear_model', SGDClassifier())\n",
    "])\n",
    "# create a k-fold croos validation iterator of k=5 folds\n",
    "cv = KFold(X.shape[0], 5, shuffle=True, random_state=33)\n",
    "# by default the score used is the one returned by score method of the estimator (accuracy)\n",
    "scores = cross_val_score(clf, X, y, cv=cv)\n",
    "print scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obtained an array with the k scores. We can calculate the mean and the standard\n",
    "error to obtain a final figure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.767 (+/-0.042)\n"
     ]
    }
   ],
   "source": [
    "from scipy.stats import sem\n",
    "\n",
    "def mean_score(scores):\n",
    "    \"\"\"Print the empirical mean score and standard error of the mean.\"\"\"\n",
    "    return (\"Mean score: {0:.3f} (+/-{1:.3f})\").format(\n",
    "        np.mean(scores), sem(scores))\n",
    "\n",
    "print mean_score(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model has an average accuracy of 0.76."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Unsupervised Learning: Clustering\n",
    "\n",
    "Sometimes, is possible to take an unlabeled training set and try to find a hidden structure or patterns in the data: there is no given target class to predict or to evaluate the resulting model. We call these class of machine learning tasks _unsupervised learning_. For instance, _clustering_ methods try to group instances into subsets (called clusters): an instance should be similar to another in the same subset and different from those belonging to another subset.\n",
    "\n",
    "In this section, we will perform clustering of the Iris data set, to see if we could group instances using their petal and sepal width and length. The trainig set is the same we used we used for our last example on supervised classification.\n",
    "\n",
    "K-means is probably the most popular clustering algorithm, because it is very simple and easy to implement, and it has shown good performance on different tasks. It belongs to the class of partition algorithms that simultaneously partition data points into distinct groups, called clusters. We will apply k-means to the training data, using only sepal dimensions, building 3 clusters (note that we could have selected a different number of cluster to group data into)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train4, X_test4, y_train4, y_test4 = train_test_split(X_iris, y_iris, test_size=0.25, random_state=33)\n",
    "    \n",
    "# Standarize the features\n",
    "scaler = StandardScaler().fit(X_train4)\n",
    "X_train4 = scaler.transform(X_train4)\n",
    "X_test4 = scaler.transform(X_test4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeans(copy_x=True, init='k-means++', max_iter=300, n_clusters=3, n_init=10,\n",
       "    n_jobs=1, precompute_distances='auto', random_state=33, tol=0.0001,\n",
       "    verbose=0)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import cluster\n",
    "clf_sepal = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)\n",
    "clf_sepal.fit(X_train4[:,0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can show the label assigned for each instance (note that this label is a cluster name, it has nothing to do with our original target classes... actually, when you are doing clustering you have no target class!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 0 1 1 1 0 0 1 0 2 0 0 1 2 0 2 1 2 1 0 0 1 1 0 0 2 0 1 2 2 1 1 0 0 2 1 0\n",
      " 1 1 2 1 0 2 0 1 0 2 2 0 2 1 0 0 1 0 0 0 2 1 0 1 0 1 0 1 2 1 1 1 0 1 0 2 1\n",
      " 0 0 0 0 2 2 0 1 1 2 1 0 0 1 1 1 0 1 1 0 2 1 2 1 2 0 2 0 0 0 1 1 2 1 1 1 2\n",
      " 0]\n"
     ]
    }
   ],
   "source": [
    "print (clf_sepal.labels_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using NumPy's indexing capabilities, we can display the actual target classes for each cluster, just to compare the built clusters with our flower type classes..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf_sepal.labels_==0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 1 1 2 1 0 2 1 2 2 1 1 2 2 1 2 2 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1\n",
      " 2 1 2 1 1 2 1]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf_sepal.labels_==1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 1 2 2 2 2 1 1 2 2 1 2 2 1 1 2 2 2 2 2 2 1 2 2]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf_sepal.labels_==2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usually, is a good idea to display our instances and the clusters they belong to, to have a first approximation to how well our algorithm is behaving on our data: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]\n",
    "step = .01 \n",
    "margin = .1   \n",
    "sl_min, sl_max = X_train4[:, 0].min()-margin, X_train4[:, 0].max() + margin\n",
    "sw_min, sw_max = X_train4[:, 1].min()-margin, X_train4[:, 1].max() + margin\n",
    "sl, sw  = np.meshgrid(\n",
    "    np.arange(sl_min, sl_max, step),\n",
    "np.arange(sw_min, sw_max, step)\n",
    "    )\n",
    "Zs = clf_sepal.predict(np.c_[sl.ravel(), sw.ravel()]).reshape(sl.shape)\n",
    "centroids_s = clf_sepal.cluster_centers_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the data points and the calculated regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEoCAYAAACdERzXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYFNW1wH+HYUAQUWRTQQYVSVTEJYIaHUCjxrgkPoj7\ni0lMfCZicMviFuURt/hi4hI0xJggPg2iJDFBk6dJXEZRUVyjKIoBAZU9YARZxvP+uLeHmqa7p3q6\nuquq+/y+b77pqrp177m3llN3OeeIqmIYhmEYpdAhbgEMwzCM9GPKxDAMwygZUyaGYRhGyZgyMQzD\nMErGlIlhGIZRMqZMDMMwjJIxZRIxItIkImckQI6RIvJq3HIUi4h8TkT+WcHy3hCRQ0rMo05EPhGR\nAVHJlWRE5HIRuTVuOaKi2OsnIj8SkV/737uIyJrySlgcInK7iFxc6XLLokxE5J8icnhg+xQRWSki\njeUor9oQkW+IyKOl5KGqj6vq3lHJVA4KPMSRGD+FaUdV/bSqPhVBcaFkFpHdROSTCMqLrRxVvUpV\nzylH3iLyHyLykoj8S0SWisgjItK/HGVl0a57TlX/qardoxamFFT1LFW9rtLldix3ASLyVeAnwBdU\n9dlyl1clCCW8UEWkTlWbI5SnXJRUz5DkzL8MbSRFpKuEpXClyokMERkM/Br4oqo2icjWwOeBsitf\nwl8/Ix+qGvkf8E/gcOBsYCmwXxvpm4D/Bp4G/g38DtgeuAdY7ff3D6TfE3gEWAG8DowOHDseeNGf\nNx+4PHBsN9yN+RVgIbAE+EHg+IHAbH/u+8CPC8g8OlDOXOCIQF3O8L9/BPw6u/zA9jd8W60B3gZO\nAoYA64CNwIfAUp+2M/BT4F0v28+BTv7Y53w+l/hjd2T2BcpaCFwAvAKsAu4G6gPHM+cuBL7p22lA\nnrr3A/7k2/9N4OuBYz/y1+0uX69XgH3z5PMU0Oyv+RrgPwJ1+Z6/dxYBXwmck7cdcuT/DeDvWW3w\nXS/TusC+Ee24/hcH2utMX48BIe7BxT7th77OnwEGAX/37bkUmAJsEzjnUn/eatz9npFX/LG3/Xn3\nANvmKydHHe4CrghsZ98z+cptua9p+5nqAvwv7p77B/CDYBlZ8pwMzCrQ5oXqm5Hjm17mRcD5gXMP\nwr1HVvnjNwF1/lgdhe/3XYAnfDv8Gbg1u/5lepfd5eV8yF/Dp4CGQFvc7Nv7X8BLwKfzXNdvAW8B\ny7w8O2TV+7/88RXATYHzdgce9/kvBf634Hu80MH2/uFeBvfjHrYhIdI3AXOABmBb/3sOMAI3FHc3\nMMmn3drfKKf7Bt3PN9Lu/vgoYA//e2/fCMdk3XC3AvX+3I+B3fzxWcDJgXKG5ZH3s/6mHBV4ue4e\nqEshZdLsf2/jL9Kufrtv4GZo9RL0+24BpgPdgW7ADOC/Ay+Bjb68jrgX7ueAd7JepDOB3kAPnBI4\n0x87zrfpYNzDfw+Bl2OO+j8J3Bhow2VAY6DOHwFH+OtzPdCUJ5/Mzbxz1gttI3C5P3487qHs1lY7\n5Mg/lzJ5DtgR6BzYl3lJhr3+x+FeSJ/y7XUvrZXJKArfg81Z+e0OHObr28vfQ9cHXjbzgd5+uwEY\n6H9f5NPuAHQCfglMyVdOjnrkUibvhCg3lzLJ90z9BPgr7n7vB7xK4L7MkmcQ7kPqJ74Nu2Ydb6u+\nnwB34u7/ocDywLU9ABiGuycHAm8A52Tdh/nu91nAdb5+I3EK+te52pnS32XL2fwuucvfO/t5GacG\n6nsM8Aybn4tPA32yrytwFPAB7j7sBEwE/pZV79/jnqUGnEI53B+fBnzP/+4EHFzwfmrrRd+eP5wy\n+Rfw+5DpmzJC++0bgQcC2yfgv1iA0zKNETj+K+CSPHnfgv/CzFx4/APi983Gfw3gXpKXA9u3Ie+v\nyPPVSnHKZCXwJfyLLZAu+yUouIcs+NJtBOYGXgJrgY65Xgx+eyFwYmD7BuBm//tOAi9k3EsypzLB\nPYjrga0C+64Hfhmo80OBY3sDa/K01RYPsZd7DSCBfSuA/fO0w6GZdsiRfy5lcnpWmqAyCXv97wQm\nBLb3yNde+e7BNvIfAzzrfw/GfZQdjv+SDqSbi1fifntnNve4BoUop5AyKVRutjIp9EwtwH90+e2z\nyaNM/PGDcMp5qb+n78jca23UNyPHLln3+G15yrkIuDfffRhItwtOOQbv93uz65/1/EfyLvPX59bA\nseOBV/zvI3E9meEEnpXs6wpMBq4KHNsG2ATsFKj3sMDx6cCF/vfdOOWzU6H7KPNXztVc3wYGi8gd\nwZ1+pcGHIrJGRL4bOLQk8Htdju1u/ncDcKif0F8pIqtww0M7+vwPFpFH/eTdv3AvlF5BGVR1WWBz\nbSDvrwN7AW+KyDMi8oU8ddsZmFew9m2gqh8CpwLnAh+IyB9FZPc8yXfAfW29nKk3bpgpWK8lqrqp\njWKDbRqs9064l2qGheQfQ94JWK6qHwf2LcB9dWb4IKucrduQK5vl6u/mLFlztcMMsq5vGywqcCzs\n9c9urwUE2ivMPRhERPqKyL0issinn5xJr6pzcS++CcASEblbRPr4UwcAfwq0xSvAJ/64blFQEbRR\nbq70+Z6pHWnd5sF2y5XPM6p6sqr2wfUCDscNwULh+mYIlrUAd60QkU+JyAwReV9EVuOGosLcNzsB\nK3Lc74Uo9V22QyB99rPUDUBVHwF+AdyGe3/c6ueYcsnfIq9/76yi9fOa771wIa5H8ryIvNzWKtVy\nKpMluC+dxuAyQnUrDbZR1e6q+pN25LsQ+Kuqbu//evi8xvnjvwXuA/qp6na4L5tQk2uq+paqnqqq\nvXHj8tNFpFMeGXYLkeVHQNfA9o5Z5f2fqh6Ju3nm4W4O2PJFsATXG/hUoN7bqer2wexCyJOP94Hg\nipkBBfJ7D+glIl2y0i9uR7nFyhymHdpdZhHX/33cB0WGhqx8C92Ducr/Me7rdy+f/muB9KjqPap6\nKO4ruSNwrT+0EDgy61nYWlWXFqpngLbuz3zlFsMHbHlvhUJVnwP+gJtHhML1zRC8LgNw9yvAJNwQ\n266qui1wJeHeC+8DPUWkc3vq0Ab53mXnhTlZVW9W1c/g2mcv3Ms/m/dw9ycAIrINbpi70EdVJv8l\n/n29E+6j95ci0pAvfVntTFT1A5xC+byI/DSibP8I7CUip4pIRxGpF5Fhga/6bsAqVd0oIgcBp2Sd\nn/cGEpH/FJGefnMNrguYayXJHcA3vS2HiEi/PL2Kl4CRItJfRLbDTT5mytpBRI7zL+VNuAc7U9YS\noL+IdARQ1U9w3d+bRKSXP7+/iByZry5FMg34hogMFpGuuKGenKjqfOB54BoR6SQi++K+6O8qkH/O\nNvf1Wg7sGkbIcrdDEdd/GnCm/9rdGrgi63ihe3ApoCKyS2DfNrjr/6GI7IxbJJCR6dMiMsortfW4\nL9uMTJOAa/05iEgfETm+QDnZvAQcKyLbiciOwHdClptNoZfyNOBSEdnWL/HNu6RYREaIW87d22/v\ngRvaeTpEfTNy/FBEthKRvYGv4uYZwLXxalVd5/M9u4DMLajqO7ge0Hj/rhkBHJstepi8ctDWuywv\nPt0wEanDXZsNuGG+bH6Le7aHeIV4LfCEqr4foowTRWQnv7kad/3zroAslzJp+SpS1YU4hTJGRK5u\nK32bGauuwS0X/E/cV8N7wDW44Q9ww2vX+a7sxbjxzUJlBbePAeb4c68HTso1dKSqTwNn4cbCV+NW\n4mS+iIJ1/wtucutV3GTZA4Fs6nArlt7DTWAfDIz1xx7Bra5YIiKZL6vv4rqrs/xQyF9w4+JhKfRF\nPgPXXX4CNzH/pD+0Ps8pJ+PG1D/AvSwuVtWm9pSN+0L8re/mnxDi/FLaIZcc7bn+M3BjyY/jJnIf\nzkqS9x5U1X/jHuhnfZ33x7XBgbh5xj/gFq9k6OxlWYa7V7YDLvPHfopbXfQ3X9aTuInmfOVkM9nL\nvwC3Yui3IcvdokkKbF+JU2zzcdfqXvLfV6twK/peFWcIOAOnDDIfonnrG+BJ4B1f1tWq+rjffxHw\nNZ/vbWxWMvnqEOQU3NzcCtyQ25QC50b5LiuUV6bHuwpX3/eAn2Wfp6r/hxuq/ANu9KA/bsI/n7zB\n7QOB50TkQ9w9eY6q5u3RSOuhacMAERkCzFbVzm0mNowiEJFzgS/54d0o890NtxCjLsp8jfCYOxUD\nABE5wXezt8ctg/xD3DIZ6UdEdhK3IEH88NIFOFuHshRXpnyNEJgyMTKMxc1fzMWt6Dg3XnGMKqEz\ncDtuDur/cAsTflmmsmyYJUZsmMswDMMoGeuZGIZhGCVjysQoKyJyiYjkHdaQLA/TtYBfUl7QeK+I\nvH4jIhNKzKOta/RVESm0Ws8wTJlUAyJymog8J86zwGIReVBKjNHh871SRLKXQRaFql6rqv9VqixV\nSGLGl4PXSEQaxIUFyH43lCxvFPeTkVxMmaQcEbkQt/7+KqAPzjp3Is7YqxLlJ34FTZJk9EZmSSbj\nuj4xbZYhBW1X05gySTEi0h3nY+gcVX1AVdeparOqPqSqF/s0IiIXi8jbIrJMRKaKs8YPfoWeISIL\nxPmSutQf+zzO3ffJvsfzot//qIhcJSJPishHwC4isqOIPCAiK0Rkroh8MyDjlSJyV2D7KyIy38ty\naVZ9hvke1mpxPpRyutsRZ7H9Jy/vCv+7X+B4Lhm7i8gdIvKeiCwUFy0v5wvTyzxNRO4S50PuZRHZ\n3bfjEt9WRwTSf01EXvdp3xaR/wocG+nL+76IvI+L15Fd3jgR+Yd4a2NxnhFeFJFVvg57B9LuJyKz\nfRtNBbbKVQefdr6I7Od/n+6v9R5++0wR+V2gvpkeQ8bI71++Pgduzk7+R5wB5DwRObpAuT8Q52ds\njYjMEZHDCtxPRbWdiPT013uVv/aP5xHDqDRhvEHaXzL/cNazG4AOBdKch3M9vyPOhfZtwD3+WAPO\nRcIknEO3oTgfUZ/yx6/Eu7wO5Pcozpr507iPkY64F9AtPv99cBbPo7LzwLk1/xA4xKe9wcufcXk9\nE+/VF+czanieOm2Ps5TujHMieS8BD9V5ZPw9zk36VjgHf88AZ+XJ/0rc8ugj/Pl34qyML8F5Lvgm\nrT0yf4HN7tkbca5R9vXbI3Eu9a/xde7s973rj1+Bc0+zvd/eD+dO5wBc7+ArOC/c9f5vPjDOyzHG\nt9+EPPWYDFzgf0/CeVU422/fCZyX4xo14FxmBL02f9WXc6aX6VvA4jxlDsbFmunrtwfgPfmS+34q\ntu2u8dexg2+DQ+J+Du3P/VnPJN30xHnYLRSJ7mzgMlV9X1U34lwrfFk2j4krMF5VN6jqK8DLOIVQ\niMmq+oYvdwdcfJcfqOpGVX0Z5z8rl4fRMcCfVPUpL8sPaT0WvwEYJCI9VXWtqs7KVbiqrlTV36vq\nelX9COc2ZEQBGbfHvbQuUNWPVXU5zjX4qQXq2KSqf/Xn34dTQNepi844FWgQ1zNEVf+szmcZ6tzK\nPIx7MWZoBq707ZNxJdJBRG7AKaxRqrrS7z8L+IWqPq+Ou3DuRw7yfx3VOfhrVtXpuPgs+XgC90LG\ny3NtYHskm3shucjutc1X1V+rquIU0Q6S24twM+7DZIiIdFTVd1X1n/kKaUfbbcR9GO3i2yCKkMtG\nBJgySTcrcB58C13HBuD3stlt9+u4B7JvIE0+F9T5CK5E2glYqaprA/uyXdIH07ac689ZETj+DVws\nlTdE5FkRyXaoB4CIdBGRSX4Y51+4l+J2WcNWQRkbcF+278tmV9+/oLAL8my34UG3+OtwL9tuXp4v\niMjTfthlFU5xBfNe5pVnkO1wiuNadX60grJeJK3dkvfHtd1ObOmduZA79MdxXrt3wD3r03AuzxuA\n7qr6UoFzs2lxha6qreofRFXnAecD43G+5e7x5eekHW13Pc7D9sN+WOwHGInAlEm6eRr31ZrPQSK4\nIYcv6JZuu9v0Gkr+FTzB/e8B20vrWAr5XNK3ct0uzkNxxksvqjpPVU9T5wL+euB+ae3qPsNFuOiE\nw9S5bM/0SoLKJCjjQtzwXc9AG2ynqkPz1C804rzq3u/l7a2qPXDOCPPJkmElLmLjZBH5bJasV2dd\nr26qei+u/bKVdF536P7Fvg7nDfgJr7Q+wIVpfTLfafnyC4uqTlXVRja7Pv9xrrzb03aq+pGqfldV\ndwO+CFwoIoeVKrNROqZMUow6r6NXAhNF5Ev+i72jiBwtItf5ZJNw7uIHAIhIbxH5YiCbQqt2lgAD\n801UexkW4eY6rhWRziIyFNfDyOWS/n7gOBH5rIjU44bcgkGlThfvWh7njVnJ7fZ8G9xLco04X2Lj\nC9QBdaEQHgZ+JiLbiGNXce7ES6WT/1uuqp+IC6h1VJgTVfUJnAfX6SIyzO++HfiWiAwHEJGtReQY\nr6yfBjaJyHf8dR6Ni7RXiMdxrnEyQ1qPZW1nswzX5mHi9WyBuDAGh3lFsYHWruuz76ei205EjhXn\n1BHc/Nsm8rvGNyqIKZOUo6o/xQXFuRw38f0uzs9WxlHjTTjX9w+Lc9s9k9YvoEIuqO/DvexXiMjz\nedKDm3vYBddLmQ78UFUfzSHr61623/q0K2gdpOdo4DVxbsJ/hovHnstd+Y24Cfrlvj4PFahDhjNw\nL67Xcb2C+2gd0a5YFFpcvY8D7vPDiKfQOtRA4UxU/4pTvn8UkX1VdTZu+OvnPr+5uAlw/HDPaFz8\nmBXAibj2LsTjuOGoJ/JsZ8uzDrgaeMoPs+VTVvl6MJ1xjkIzrut7szlSYqv7ybfdeRTXdrsDfxXn\nFv0pYKJudjNvxIj55jIMwzBKxnomhmEYRsmYMjEMwzBKxpSJYRiGUTKmTAzDMIyS6Ri3AGEQEVsl\nYBiG0Q5UtSJOO1OhTAA2zihX2OhkM+HuqVxx+ilxi5EarL2Kp1CbrRtxRM79tcw111zDpZde2nbC\nBNC9e/eKlZUaZWIYfU48nVXr1rVs9+jShaX33R2jRIZhZDBlYqSGVevWtbKUk4BiMaLHeiVGMdgE\nfMIZufeQuEVIFdZexZOrzUyR5KexsbHtRDVIKizgRURrdc7E2Ez9caNb90yo3bm0cmKKpHro3r17\nxSbgrWdipIYeXbog0PLXo0suh8KGYcSBzZkYqcEm2w0juVjPxDCMFmyIy2gvpkwMwwBMkRilYcrE\nMAzDKBlTJoZhWK/EKBlTJoZR45giMaLAlIlhGIZRMqZMDMMwjJIxZWIYNYwNcRlRYcrEMGoUUyRG\nlJgyMYwaxBSJETWmTAzDMIySMWViGIZhlIw5ejTKjkVITBY2xGWUA1MmRtmxCInJwRSJUS5smMsw\nDMMoGVMmhlEjWK/EKCemTIyyYxESDaP6sTkTo+zYZHv8WK/EKDfWMzGMKscUiVEJTJkYhmEYJWPK\nxDCqGOuVGJXC5kxqFDMkNAwjSkyZ1ChmSFj9WK/EqCQ2zGUYVYgpEqPSmDIxDMMwSsaUSY1ihoTV\ni/VKjDiwOZMaxSbbqxNTJEZcxNozEZH+IvJ3EXlNRF4VkXFxymMYhmG0j7h7JpuAC1X1JRHpBswW\nkYdV9Y2Y5TIMwzCKINaeiap+oKov+d//BuYA/eKUyTDSig1xGXESd8+kBREZCOwLPBuvJEaaqVVj\nTFMkRtwkQpn4Ia77gfN8D8Uw2oUZYxpGPMSuTESkI06R3KWqD+RLN+HuqS2/R+49hJFDh1RAOsNI\nPtYrMTI0NTXR1NQUS9miqm2nKqcAIlOA5ap6YYE0unHG7yoolZFW6o8b3bpnAlT7vWPKxMhH9+7d\nUVWpRFlxLw0+BDgdOFxEXhSRF0Tk6DhlMtJNrRljmiIxkkLsPZMwWM/EMLbEFInRFjXTMzEMwzCq\nA1MmhpFCrFdiJA1TJoZhGEbJxL402Eg3XY8bzcbAdj2w1ua3DKPmMGVilMRG2GIprlFebIjLSCI2\nzGUYKcIUiZFUTJkYRkowRWIkGRvmMkqintZDW/VxCWIYRqyYMjFKwibbDcMAG+YyjFRgQ1xG0jFl\nYhgJxxSJkQZMmRhGgjFFUl00NxfernQ+UWLKpErpctxo6gN/XY4bHbdIhlHTNDfDUUd1pampDoCm\npjqOOqpr0Yogqnyixibgq5RNmDFh2rFeSXVRVwfjx6/njDO24qyzNnL77fVMmfIxdXXx5BM11jMx\nDMOoEI2NzZx11kauu64zZ521kcbG9nUnosonSkyZGEYCsV5JddLUVMftt9dz8cXruf32+pahqrjy\niRIb5qpSOtJ6aMsutGHES3MzjB/fmSlTPqaxsZnGxmbGj+/Mww+vLWqIKqp8osYiLRpGwrBeSfXS\n3EyrF372dtT5WKRFw6hRTJFUN9kv/Pb2JKLKJ0pMmRiGYRglY0PpVUqfE09n1bp1Lds9unRh6X13\nR55HFOUYhpF+TJlUKavWrWttZxJ44UeZRxTlGA4b4jLSjA1zGUYCMEVipB1TJoYRM6ZIjGrAlEmV\n0qNLFwRa/np06VKWPKIoxzCM9GN2JoYRM9YzMcqF2ZkYRo1gisSoFkyZGEZMmCIxqglTJoZhpJYk\nBomqVczOJARJM8xLmjxG8VivpHQyQaLGj19PY2MzTU11iXB4WKuYMglB0gzzkiaPURymSKIhqUGi\nahUb5jIMI7UkMUhUrWLKxDAqiPVKoiWJQaJqldiViYjcISJLROSVuGXJR9IM85Imj2HEQTBI1KWX\nbmDKlI8ZP76zTcLHROxGiyJyKPBvYIqqDs2TxowWjdRjvZLoiSrYVLVSU0aLqvoksCpuOQyjnJgi\nKQ9JDBJVq8SuTAzDMIz0Y8rEMAzDKJnU2JlMuHtqy++Rew9h5NAhMUoTL12PG83GwHY9sDZrTilJ\nho1JkiUObIjLqBRNTU00NTXFUnZSlElmYVJerjj9lAqJknw2QmujxRxpkmTYmCRZKo0pEqOSNDY2\n0tjY2LJ93XXXVazsNpWJiHQGxgADg+lVdUIUAojIPcAooKeIvAtcqaq/iSLvWqS5uZm33ns/bjEM\nTJEYtUWYnskDwGpgNrA+agFU9bSo86xlxlz1Yx587vm4xTAMo8YIo0z6q+rRZZfECE09rYe26oE/\nPjOLK6bczWvvLmzZH0wTp2Fjjy5dWg1tmZGlYVQfbRotisgvgVtU9dXKiJRTBjNaLMDaj9fT65Sv\nsHHTpi2O1XfsyPKpd9F1q84xSFa72BCXkQQqabSYt2ciIq/i5nk7Al8XkXdww1wCaD5rdaPy1Hes\nY5e+fZi7+L0tju3Stw/1Hc2Sq5KYIjFqkULDXMdVTAqjJOo7dmTSuLEc9oPLtjg2adxY6jsmZdGe\nYRjVSl6jRVVdoKoLgKsyv4P7KieiEYYOHXL3ZPPtNwqzbPVqnpv7FstWry7qvGCvxKIAGrVEmE/W\nvYIbIlIHfKY84iSTqIzuosonl9Hiv/94PxeNPoE3Fy1ixJC9uGzyXWz85BNGfu/SnGWVU5ZsA8q0\nce8TTYyddBt9G7ZiyYKPmXj2OZw84tCi8rAogEatUWjO5BLgUqCLiKzJ7AY2AL+sgGyJISqju6jy\nyWW02KFDB64784yWfd//9Z0FyyqnLGlm2erVjJ10G1c8MYCBQ7sw/5V1jB1xK4fvsze9t9224LnB\nXolFATRqjULDXNeq6jbA/6hqd/+3jar2VNVLKiijYVSM+UuW0rdhKwYOdcuXBw7tQt8BWzF/ydKi\n87IogEYtkVeZiMj+IrI/cF/md/CvgjIaRsUY2LcPSxZ8zPxXXE9t/ivrWPLuxwzs26fgeblWcFkU\nQKOWKDRncoP/vxVwAPAybhRjKPA8cHB5RUsOURndRZVPLqPFYsuqpCxpove22zLx7HMYO+JW+g7Y\niiXvujmTQkNcuRRJMApgY2MzjY3NNmdiVDVhjBZ/h/OX9arfHgKMV9UvV0C+jAxmtGhUlGWrVzN/\nyVIG9u1TtCLJYFEAjbhJhNFigE8Frd9V9R8iskcZZTKM2Om97bZtTri3hUUBNGqJMMrkFRH5FfC/\nfvt04JXyiWQYhmGkjTCRFr8OvAac5/9e9/sMo6apRrcpGzYU3jaMfLQ5Z5IEbM6kNWEMDms9umG5\nqVZF0tDQjcsvX8/YsRuZOLGeq67qzIIF/6ZTp7ilM9pDIuZMRGSaqp4UcPjYCnP0GB9hDA5rObph\nualGRQLQqRNcfvl6Lr20MzNmdGTmzDquuWa9KRIjFIXmTM7z/83ho2HUCGPHbmTGjI489VRHDjlk\nE2PHbmz7JMOgsAV8JvbrEUCnHM4eDcOoMiZOrGfmzDoOOWQTM2fWMXFi2i2HjEoRZgJ+ADBJRN4R\nkftE5Dsism+5BTPy06NLFwRa/nIZHIZJYxRPtQ5xgZszueqqzlxzzXr+/Od1XHPNeq66qrNNwhuh\nCD0BLyJdgLOA7wL9VLViq+ZtAt5ICtWsTMAplOAcSfa2kS4SMQGfQUQuBw4BugEv4pRJU5nlMozE\nUe2KBLZUHKZIjLCEMVocDWwCHgQeB55W1fVllcowEkYtKBLDKIU250xUdX/cJPws4EjgVRF5styC\nGUY5aW8kxbhYvnw5s2fPZvny5TmPpzGqY1t1MtJFmGGuIUAjMBLnPXghKRnmqmSExEoaCZpBYmkU\nG0kx7l7J/dPv44KLvkPfhq4sWbCWn93wc748ZrOf1TRGdWyrTkb6CDPMdR3wBHAz8JyqpmbheSUj\nJFbSSNAMEttPKZEU42D58uVccNF3uOLR/i3yXnDYuYwaOYpevXoB6YvqGKZORvoIM8x1nKper6oz\n06RIDCMXxUZSjLtXsmDBAvo2dG0lb58BXVmwoLWpV5qiOoatk5EuwtiZGEbVECaSYt3uH7f8dXt/\nRlyiAtDQ0MCSBWtbybv03bU0NDS0SpemqI5h62SkizDDXKmlkhESoyorKnmM3OSLpLjDAZ2Bj3Oe\n0+39Gfx7x3i8CvXq1Yuf3fBzLjjsXPoM6MrSd938QnA4KG1RHcPUyUgf5jXYqElW9lzK/EXLGNi/\nN7237x7qnLgUCrh5hgULFtDQ0JDzpZvGqI5t1ckonUoaLeZVJiLyJ3J4C86gql8sl1A5ZDFlYpRE\n3e65ex2b98pPAAAXe0lEQVTtIU6lUq00Nzczb948dtttN+qSrgVTRFIs4H9SCQEMoxxEqTyM8nPq\nqafyl7/8haOPPppp06bFLY7RDmyYy0g9cSkO66HkprkZVq3aPITVo0evvENuDz74IBMmTGDOnDkt\n+/bcc09++MMfcuyxx1ZI4uolEcNcLQlEdgeuBfYEtsrsV9VdyytaKxliVSaVNH7setxoguuv64G1\npki3ICk9D1MorWluhgMOWMcHS8+g36DZLH77M+zQZwrPP99lC4Wydu1adt55ZzZu3NLioL6+noUL\nF9K1a9cKSV6dVFKZhFka/BvgNpx/rsOAKcD/llOopJExEsz8rSrR+LFQPhsDx9VvG62X6yZFkQCx\nLx1OGqtWLeeDpWdAx3vY94s/go738MHSM1i1akuXKfX19XmXAzc0NFBfb7FU0kQYZdJFVf+G68Us\nUNXxQGT9TxE5WkTeEJG5IvKDqPI10k1SlUcuur0/w5SKZ8GCBfQbNJtjx01j2n9/i2PHTWOn3Wbn\nNEisr69n4sSJOfOZOHGiKZOUEcbOZL2IdADeEpFzgcU4d/Ql4/P9OfA54D3gORF5QFXfiCJ/I10k\nXWm0RZz2KEmhoaGBxW9/hsU3n8RJV/6CB28+CTb9IW8PpEOH3N+z+fYbySWMMjkP6AqMA34EHA58\nNaLyhwNvZcIAi8hU4EtAopRJJY0f63GREYPb1UralUcual2h9OjRix36TOGDpafx4gOzYdMf2KHP\nFHr0yP3MDBs2jPPOO4+5c+dy6KGH8uSTTzJ48GCGDx9eYcmNUikm0mJ3QFX1w8gKFxkDfF5V/8tv\n/ycwXFXHZaWz1VwppxoVR1vUqlIpZjWXUV6SYmcCgIgcgJuE38ZvrwbOVNXZZZatFRPuntrye+Te\nQxg5dEglizeKpBaVh+Goq3MuU8yqvfI0NTXR1BRPhJAwS4NfAcaqapPfPhS4VVWHlly4yEHAeFU9\n2m9fjOv9/DgrnfVMUoApkNzUag/FiJ+kLQ1uzigSAFV9ErdMOAqeAwaJSIOIdAJOAf4YUd5Gmcms\ntFrZcykvrHuNZSvXlJTfnHmLufN3jzNn3uKIJMzPspVreO6Vee2WublZCm4HqcRKr7CRFsNENyyU\nppIRHcOUlaQIk2mTN2rCKJPHRWSSiIwSkZEicivwmIjsLyL7l1K4qjYD5wIPA68BU1V1TuGzNtPn\nxNOpP250y1+fE08vRZyyE0beqNKUg1zLdac+NJPBx5zPGVffwOBjzmfqQzPblfd5V09mvzHf57Lf\nuP/nXT05MrmzKVXm5mah8bQjeOxZ57b+sWf70HjaEbEplEykxYzb+aamOo46qusWL6r7p9/HfgcM\n4ewLT2S/A4Zw//T7t8irUJqw5VSqTpWUp9rkLQdhhrkeLXBYVfXwaEXKKUPOYa7640a3jjgIJHk4\nLIy8UaWJikJDV8tWrmHwMedzxWOboxZOGPUucx+6MbQnXnA9kv3GfJ9rn9m9JZ9LDnqLF6dfzx67\n9YuiGpHL/NizfTjpvEM45/S3uPXu3Zl201OMOjB3gK1syjHs1dRUt0WkxWCArOXLl7PfAUNaRTec\ncNgiXnz+Hy1zG2HStFVOJetUaXnSKG+ihrlU9bACf2VXJEblyO55tDUHMn/RMnbYImphZ+YvWlZU\nubNefpteO3dqlU/P/p2Y9fLb7atIBWQedeBSzjn9LSb8fG/OOf2t0IoEytNLaSvSYpjohmHSVDKi\nY5iykhRhMm3yRk2bykRE+orIHSLyZ7+9p4h8o/yiGeWmVCvzgf1788EWUQvXM7B/76LyGb7PIJYv\n3NAqnxWLNjB8n0FFy1QpmR97tg+33r07V5z7KrfevXvLkFdYolYobUVaDBPdMEyaSkZ0DFNWkiJM\npk3eqAljtDgZtzT4Mr89F7gXuKNMMoUmbREHo4rY2N56R73aqvf23bntim/y7VG/ou+Azix5dz23\nXfHNooaLAPbYrR9nn3gklxz0CD37d2LFog2cfeKRkQ9xRSVzc7Nw6U/3aRnaGjV8KZf+dB+a7vkr\ndXXhvXBnFEqpw15hIi2GiW7YVppKRnQMU1aSIkymTd5yEGbO5DlVHSYiL6rqfn7fS6q6b0UkxJYG\nt5dKLdVdtnJN0VELczFn3mJmvfw2w/cZVBZFEqRUmZubpZXiyN4uligUSphIi2GiGxZKU8mIjmHK\nSlKEySTKmzQX9I8BY4BHVHV/bxvyY1UdWQH5MjKYMgmJ2XqkF7NHMaImURbwwIU424/dROQpoDfw\n5bJKZYTCFEd1Uet+vYx0E8o3l4h0BD6FW4X6pqpWNMyG9Uw2U80KJKrhsiTJ0lY+uYbL1vUvX4TB\nMMNc1UaShsIqTSKWBovIMBHZAUBVNwGfAa4GbhCR7SshXKmkzagxF2mK61EKURk/JkmWtvLJZ/zY\nZdGDZVk+HMZosdqodkPBJJG3ZyIiLwBHqOpKERkBTAW+A+wL7KGqFRvqam/PJG1GjdWsLAoRlSFh\nkmQJm09bxo9RDXuFMUisVpJk2FhpEtEzAepUdaX/fTLwS1Wdrqo/BKI3AKhRaqHX0RZRGRImSZaw\n+bRl/BhVDyWMQWK1Us2GgkmioDLxcyXgIiH+PXAszMS9kQdTIK2JypAwSbKEzSeM8WMUYYHDGCRW\nK9VsKJgkCimF3+KcPC4H1gEZF/SDgNUVkK1kkmDUaAqjbaIyfkySLGHyKdb4sZTVXmGMFquRajcU\nTBIFV3N5m5IdgYdV9SO/bzDQTVVfqIyI6VvNZQqkfdhqrnDGj6XMo9hqLlvNVS5Ch+2NkyQrE1Mc\nRhyYPYoRhqQZLRoBTHkYSSAqv16GERVhgmPVPDZhnhzaisZYTATEUik1WiOEk3fDhg55tysRxTGb\nMNEa00alIiDWeqTFmqLYmB5G5WgrGmN7IiC2lygMG8PIu2FDB3odOJobJw8G4MbJg+l14OjYFEo1\nGj5WyrCx2g0obc4EG7pKA2GjMZYSATEsURpZhpH3xsmDuei6/Wk8YClNz/fhhotf4Pyvzc2ZXzmH\nvarZ8LFSho01HWmxWrGeR7oIG42xlAiIYYnSyDKMvOd/bS6NByzlief60njA0ryKpNxUs+FjpQwb\nq9mAsuqVSa5hK1Mg6SNsNMZSIyCGIUojyzDy3jh5ME3P92HEsCU0Pd+nZcgrF1EYOOajmg0fK2XY\nWM0GlFW5msuURfURJhpjVBEQ2yIqw8Yw8m7Y0IErbhraMrR14+TBXHHTUM457W06dfokb97lcGdf\nrYaPlTJsrHYDyqqZMzEFUhu0FY0x6giIhYjCsDGMvBs2dGilOLK3C1GOOZRqNHyslGFjTUdaTALZ\nysQUh2EUh9mj1CY2AZ8Dm+8wjPYThz2KUVukRpkY0ROF0V3SaG6WVoaN7bUxqVTbhDFajMoQ0xSK\nUU5MmdQoSYpsGBXNzcLOI4ayzwkPctlvJrPPCQ+y84ihRb98K9U2YYwWozbELOdqL6O2qcrVXEZh\nlq1cw7cn/KqV0d23R/2Kzx00JHZvvaUwd/4ilq+ZROetp3PoqdN48OaTWL5mDHPnH5tzsj4XlWyb\nujrlmgtf3sJoMTgBHyaNYSQB65nUIEmKbBgls15+mz4Dn+bYcdOY9t/f4thx0+jd8PQWho2FqHTb\nhDFaLIchpvVQjKgxZVKDJCmyYZQM32cQS+cfzIM3n8RJV/6CB28+iWULDt7CsLEQlW6bMEaL5TTE\nNIViRIUNc9UgSYpsGBXNzZ/wySdCz21uZsWHY2i652nWf3QvvbpPZPDAV4Bww0KVbJswRouVMMQs\nh4GjUXukxs7kkzfviVuMqiNJkQ1L5Yvf+h9mPPoix47an+u/f2qLYePggf3b9dKtVNuEMVqspCGm\nKZXqwowWszBlYuTjgb8+z+U3TuO1txa17BsyeGd+dN6JfOmIA2KULL2YQqkeTJlkYcrEyMXadevp\nMeybbNy4pefV+vo6Vj33K7p26RyDZOnHFEp1UBMW8CLyZRH5h4g0i8j+cclhVIaojACD+dR3rGOX\nfrkno3fstR3/+nBtSWXVMrbaqzxYpMXy8CrwH8DjMcpgVICojACz85n+yCx+dfVZOdPq1uvZ64sX\nVYUxplEdVHukxdhWc6nqmwAiUpEumBEPURkB5svn7uu+kzP9t+7cga26dagKY8w4yfRObNirdOrq\nYPz49VtEWqwG9/NgdiZGmYnKCDBfPj2368Z3v3Ecxx+2P+POOJqtu3fkS9/rzacO7lo1xphJwIa8\noqGaIy2WtWciIo8AfYO7cAv+L1PVPxWT1/hb7m/5PWr4now6cM9IZDTKS9AIMNOjaI8RYL58dh3Q\nl+u/fxrgei93/ukxRvxnD0Skaowxk4LZo5ROdqTFTJCs6PJvoqmpKbL8iiH21Vwi8ihwkaq+UCCN\nreZKMVMfmsm3J7Q2AjzlmM+WJZ+oyjIKY0qleDJzJuPHr6exsZmmprqyR1qsqaXBXpl8V1VnF0hj\nyiTlRGUEGCafajLGTDKmUIrHIi2Wo2CRE4BbgF7Av4CXVPULedKaMjGMBGIKJdnUhDIpBlMmhpFs\nTKkkk5owWjRyE1VUPUhWJMUwsrSVJsq2SQrNzdKq3nHXqb1tbKu9DFMmCSLKqHpJiqQYRpa20kQd\ncTAJNDcLe3zhQHY9cjpnXH0Dux45nT2+cGBsdSq1jU2h1DY2zJUwHnu2zxZR9YoNhrRs5RoGH3N+\nKwO/CaPeZe5DN1Z8QjqMLGHljaJtksSylWvY9cjpUD+VY8e5yJBsPIV3HhkT28KBKNrYhrySgw1z\n1TBRRNVLUiTFMLKElbccEQfjZP6iZfQf9FyryJD9dpsVq5FlFG1sPZTaxJRJwogiql6SIimGkSWs\nvOWMOBgHA/v3ZtHbw1pFhlw8b3isRpZRtbE5iqw9LNJigogqql6SIimGkSVMmkpEHKw022+7LTv2\nnMT7K07hpT/Mgo2/Y8eek9h+22cJGxkySsrRxmY1XzvYnEnCiDKqXpKM96IwNqxkxMFK0dwsrFy9\nuqXe22+7bax1Klcbm0KJB7MzyaKWlIlhVCumUCqPTcAbhlF12DxKdWPKJIEkydgwKubMW8ydv3uc\nOfMWxy2KETOmUKoTm4BPGBmvtzs0bMUHCz6uCq+35109mUn3PUKvnTuxfOEGzj7xSG667Gtxi2XE\niE3MVx82Z5IgkmRsGBVz5i1mvzHf59pndm+p0yUHvcWL069nj936xS2ekQBMqZQPmzOpUZJkbBgV\ns15+m147d2pVp579OzHr5bdjlsxICjbsVR2YMkkQSTI2jIrh+wxi+cINreq0YtEGhu8zKGbJjCRh\nCiX92JxJgkiSsWFU7LFbP84+8UguOegRevbvxIpFbs7EhriMbDIKxYa90onNmSSQJBkbRsWceYuZ\n9fLbDN9nkCkSoyCmTKLDjBazqDVlYhiGKZUosAl4wzBqHptHSRemTFJK2gwb0yZvJbG2yY8plPRQ\n8xPwPfc/k1Uffdyy3WPrrVjxwq9jlKht0mbYmDZ5K4m1TdvYxHw6qPk5kw6fOq2Vs28Bkjw/kzbD\nxrTJW0msbYrHFEpx2JyJkZe0GTamTd5KYm1TPDbslVxMmaSMtBk2pk3eSmJt0z7M+3Ayqfk5kx5b\nb4VkzZkkmbQZNqZN3kpibWNUEzU/Z5JW0mbYmDZ5K4m1TWnYPEp+zGgxC1MmhmEUwhRKbmwC3jAM\nowhsDiV+TJkYRg6iMiQ0g8TKYRPz8WLKxDCymPrQTAYfcz5nXH0Dg485n6kPzYw1H6M4TKHEg82Z\nGEaAqAwJzSAxfmwexeZMDCM2ojIkNIPE+LFhr8piysQwAkRlSGgGiUatUfNGi4YRJCpDQjNITA7m\nKLIyxDZnIiLXA8cD64F5wNdVNeeSF5szMSpNVIaEZpCYLGpNodSE0aKIHAH8XVU/EZHrAFXVS/Kk\nNWViGEYk1JJCqYkJeFX9q6p+4jefAfrHJYthGLWDTcyXh6RMwJ8J/DluIQzDqB1MoURLWZWJiDwi\nIq8E/l71/48PpLkM2KiqNo5lGEZFMYUSHWVdzaWqRxY6LiJfA44BDm8rr/G33N/ye9TwPRl14J6l\nimcYhkG392dUzTxKU1MTTU1NsZQd5wT80cANwAhVXdFGWpuANwyj7FSLUslQExPwwC1AN+AREXlB\nRG6NURbDMAwb9iqB2IwWVXX3uMo2DMPIRzUNe1WSpKzmMgzDSAy2fLh4TJkYhmHkwRRKeEyZGIZh\nFMAUSjhMmaQUi+BnGJXDFErbmDJJIRbBzzAqjymUwlikxZRhEfwMI37SstqrVuxMjBA89uzrrbYt\ngl9hstvLaBtrs+J47NnXrZeSA1MmCeexWa0fdIvgV5js9jLaxtqsODLtZQqlNRZpMWVYBD/DSA4W\nxXEzpkxSyCnHfJbPHTTEIvgZRkIwq/kUTcDHLYNhGEYaqfqwvYZhGEb1YBPwhmEYRsmYMjEMwzBK\nxpRJChCRL4vIP0SkWUT2j1uepCIiR4vIGyIyV0R+ELc8SUdE7hCRJSLyStyypAER6S8ifxeR13wI\n8nFxy5QkTJmkg1eB/wAej1uQpCIiHYCfA58H9gJOFZFPxytV4vkNrr2McGwCLlTVvYCDgbF2j23G\nlEkKUNU3VfUtoCKrMlLKcOAtVV2gqhuBqcCXYpYp0ajqk8CquOVIC6r6gaq+5H//G5gD9ItXquRg\nysSoFvoBCwPbi7AH3SgTIjIQ2Bd4Nl5JkoMZLSYEEXkE6BvcBShwmar+KR6pDMPIRkS6AfcD5/ke\nioEpk8SgqkfGLUPKWQwMCGz39/sMIzJEpCNOkdylqg/ELU+SsGGu9GHzJrl5DhgkIg0i0gk4Bfhj\nzDKlAcHuqWL4NfC6qt4UtyBJw5RJChCRE0RkIXAQMENE/hy3TElDVZuBc4GHgdeAqao6J16pko2I\n3APMBAaLyLsi8vW4ZUoyInIIcDpwuIi8KCIviMjRccuVFMydimEYhlEy1jMxDMMwSsaUiWEYhlEy\npkwMwzCMkjFlYhiGYZSMKRPDMAyjZEyZGIZhGCVjysSoCkTkMu+m/2W//n9YxPmPFJEt3Nrk2x9B\neV8KeqQVkUct/ICRZMydipF6ROQg4BhgX1XdJCLbA53KUFQ+o6xyGGudAMwA3ihD3oYROdYzMaqB\nHYHlqroJQFVXquoHACKyv4g8JiLPicifRaSv3/+oiNzoLZlfEZED/P5hIjJTRGaLyJMisntYIUSk\nqw849Yw//3i//6siMt2X/6aI/Dhwzjf8vmdE5JcicouIHAx8Ebje97J29clPEpFnfQCwQ6JoOMOI\nClMmRjXwMDDAv2QnisgIaHHKdwswRlWH4YJBXRM4r4uq7geM9cfAxag4VFU/A1wJXFuEHJcBf1PV\ng4DDgZ+ISBd/bB/gRGAocLKI9BORHYHLcbFYDgE+DaiqPo3zK/Y9Vd1fVd/xedSp6oHABcD4IuQy\njLJjw1xG6lHVj/x8QiPuJT5VRC4GZgNDgEdERHAfT+8FTv2tP79JRLYRke5Ad2CK75EoxT0jRwHH\ni8j3/HYnNnsy/lvGXbmIvAY0AL2Bx1R1td9/H1CoJ/Q7/3+2P98wEoMpE6MqUOdk7gngCRF5FTgD\neAH4h6rmGxLKnutQ4EfA31V1tIg0AI8WIYbgekFvtdrp5nTWB3Z9wuZnrxiPvZk8mrFn10gYNsxl\npB4RGSwigwK79gUWAG8Cvf3LHBHpKCJ7BtKd7PcfCqxW1Q+BbdkcB6VYL7r/B4wLyLVvG+mfA0aI\nyLZ+SG5M4NiHuF5SPsxtvJEoTJkY1UA34E6/NPglYA9gvI8F/2Xgx37/i8DBgfM+FpEXgFuBM/2+\n64HrRGQ2xT8fPwLq/YT+P4AJedIpgKq+h5vDmQU0Af8EVvs0U4Hv+Yn8XcndizKMxGAu6I2aREQe\nBS5S1RdilmNrP+dTB/weuMMi+BlpxHomRq2SlK+o8SLyIvAq8I4pEiOtWM/EMAzDKBnrmRiGYRgl\nY8rEMAzDKBlTJoZhGEbJmDIxDMMwSsaUiWEYhlEypkwMwzCMkvl/nOM3hVHZPSQAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd909cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(1)\n",
    "plt.clf()\n",
    "plt.imshow(Zs, interpolation='nearest', \n",
    "           extent=(sl.min(), sl.max(), sw.min(), sw.max()),\n",
    "           cmap= plt.cm.Pastel1, aspect='auto', origin='lower')\n",
    "for j in [0,1,2]:\n",
    "    px = X_train4[:, 0][y_train == j]\n",
    "    py = X_train4[:, 1][y_train == j]\n",
    "    plt.scatter(px, py, c=colormarkers[j][0], marker= colormarkers[j][1])\n",
    "plt.scatter(centroids_s[:, 0], centroids_s[:, 1],marker='*',linewidths=3, color='black', zorder=10)\n",
    "plt.title('K-means clustering on the Iris dataset using Sepal dimensions\\nCentroids are marked with stars')\n",
    "plt.xlim(sl_min, sl_max)\n",
    "plt.ylim(sw_min, sw_max)\n",
    "plt.xlabel(\"Sepal length\")\n",
    "plt.ylabel(\"Sepal width\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat the experiment, using petal dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeans(copy_x=True, init='k-means++', max_iter=300, n_clusters=3, n_init=10,\n",
       "    n_jobs=1, precompute_distances='auto', random_state=33, tol=0.0001,\n",
       "    verbose=0)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_petal = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)\n",
    "clf_petal.fit(X_train4[:,2:4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print (y_train4[clf_petal.labels_==0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1\n",
      " 1]\n"
     ]
    }
   ],
   "source": [
    "print (y_train4[clf_petal.labels_==1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2]\n"
     ]
    }
   ],
   "source": [
    "print (y_train4[clf_petal.labels_==2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the clusters "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEoCAYAAACOxlwjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4HFW1t99fBkIQGUJIwAQOyIwBBCGEISaIKCCKV0Um\nRVEQERXBgUEuRFQGvSoOURG5V/ADgwwqo4LKcBBIuMjkJZAEISQhZCBhDCThsL4/9u6kTqerT/fp\nru7q7vU+z3lODbv2XrW7un69h7W2zAzHcRzHyZIBzTbAcRzHaX9cbBzHcZzMcbFxHMdxMsfFxnEc\nx8kcFxvHcRwnc1xsHMdxnMxxsWkikrolHZMDOyZIerTZdlSLpP0lPdXA8h6XtE+NeQyU9Kakzetl\nV56RdJaknzfbjlqo9jlLfq8lHSPppuysq556PMf9oeliI+kpSe9J7B8haYmk8c20q1WQ9FlJt9eS\nh5ndaWY71cumLCjzkq6Lo1gl9Whm25vZP+pQXEU2S9pK0pt1KK9p5ZjZd8zsC/XON/E8vCzpJUnP\nSPpehdd+W9J/V1lkv54zM7vczD7Qn2uzoo7PcVU0XWySSPoU8FPgIDPrbrY9LYKo4YUraWAdbcmS\nmu6zQkrmn0EdqYp0jfC6blQ59caAHc1sPeD9wKckfabJNjkp5EZsJJ0AfB94n5lNLZOuW9K3JN0r\n6RVJ10kaJulKSS/G46MT6XeUdJuk5yU9JukjiXMflPRgvO5pSWclzm0Vfzl9UtIcSQsknZY4v6ek\nB+K18yVdWMbmjyTKmSHpvSXS9Pq1VfxrM/7yfir+ipsl6eOSxhDEeXz8hbcwph0i6Yfx1958ST+T\ntFY8t3/M5wxJ84FfFXcTxPs9RdIjkpZKukLS4MT5M2K+cyQdV65bSNIoSTfE+n9C0rFF93ylpN/G\n+3pE0jtTqvHO+P+xmPY/Vmejr0taKGmupE8m8k+th76I9/Y1SY8ArySOvTtuV/P5n16oL+BTJF7s\n5Z7Bwj0nfr2/S9LWkv4e63OhpMslvTWR35mS5sX8HkvYq3huVrzuSknrp5VT4h5+K+nsxH7xM5NW\n7qrnWn1/p4ZK+n/xmfuXpNNUvvtK8Q8zmw78AxgT8xql8G5YKOlJSV+Ixz8AfAM4Ot7v/fH4Z6Pd\nL0maKemzZcotrpsDFbqmlkq6qGBTIt/b43ahNfb5+Dm8KOns+JneK+mF+F0bmLj+Q5IeinnfJekd\niXOp31NJG0u6KR5/XtIdRdcVPp8hkn4i6dl4/AeSBiU/Y6V/vw5J1Nkzkk4uW1Fm1tQ/4CngGmA+\nMKaC9N3AdKALWD9uTwfeTRDPK4CLY9q3AHOBo+MDsCuwCNgmnp8I7BC3dwIWAgfH/a2AN4GfA4Pj\nta8DW8Xz04DDE+XskWLv3sBSYGLcH5Uovxs4Jm5/G/jvxHVbAT1x+63AC8Db4/5IYPu4/Vng70Vl\n/hS4FlgPWBe4EfhWPLc/sDKWNwgYEo/9O3H9HOAeYGNgQ+AJ4DPx3CGxTrcFhgJXAj3A5in3fzdw\nUaIOFwHjE/f8KvDe+Pl8D+hOyWdg/Dw2Sxwr3MtZ8fwHCcKwbl/1UCL/XvUY6+B+YFNgSOLYu6v8\n/A8B5gHbxfq6Kllf9P0M9hTltw2wX7zf4fEZ+l48tyPwNLBx3O8CtojbX41pNwHWAn4FXJ5WTon7\n+C1wdlHd/7uCclc91/T9nfov4K+E530U8CiJ5zLleSjU4zuABcAn4rP0IHBaTPd24N/AfqW+a/HY\nB4CuxGeyjPg+ouj7UXTdxsDLwIdiWV8jPJOF7/Wq5yph8zXAOgRhXA78BdiM1e+zI2P6PQjvxd3i\nPX0amAUMquB7+j3gJ4R34iBg36Jnu/Acn0f4jg4jPE/3Af9Z4fdrIbBn3N4AeGfZZ6ivl3vWfwSx\neQH4Q4Xpu4GvJ/YvAv6U2P8wMC1uHwX8rej6XwNnpOT9U+DC5BeQ+AWKxx4APhK3744fwrA+7P11\nIc+Ue6lUbJYAhxJffIl0xS9JAa/R+6U8HpiReICWFR7YUl+m+DAeltj/AfCTuH0ZiRc24SVaUmyA\nLQhfprUTx74H/Cpxzzcnzu0EvJRSV71eLgm7XwKUOPY8q7+cxfWwb6EeSuRfSmyOLkqT/JJW+vlf\nBpyb2N8hrb7SnsE+8v8oMDVub0t4Ob0HGFiUbgZR5OP+ZsBrcXvrCsopJzblyi0Wm3LfqdnEH2Vx\n/wT6FpsXCN+NGQX7CD/wZhWlP4vVP0LXEJsS+d8AnFjq+1GU7ljgrqLv37OUF5vdE+kfAk5J7F/E\n6h8PvyK++BPnZwF7VfA9/S5B1N5ewubkc/w0sH/i3MH0fleU/H7F7bnAZ4ji09dfXrrRTgS2lXRp\n8qCkSxJN+68lTi1IbL9WYn/duN0F7Ksw4WCJpKXAxwm/VpG0l6TbYxPxBcKDMTxpg5ktSuwuS+R9\nLOHX1BOS7pN0UMq9bQY8Wfbu+8DMXgaOBL4IPCfpeknbpCTfhNBaebhw34QvTvK+FpjZG30Um6zT\n5H2/jfCwFphD+hjE24DFZvZ64thswq/WAs8VlfOWPuwqZrHFJ7/I1lL1cCNFn28fzC1zrtLPv7i+\nZtO7m6XPZzCJpJGSropdGi8AvymkN7MZhBbMucCC2K0yIl66OXBDoi4eAd6M522Ngqqgj3JLpU/7\nTm1K7zpP1lsaO5nZMDPb1szOjce6gK6i7/3XCT0CJYldQvfFLqelwAFU9qz0+nzjs1juuYHQIijQ\n1/vrtKL72ITe35+07+kFwDPA32K3YPL9WWz/M4n94u9n2vcL4D8IP4CfUejaHZtSBpCfMZsFBBUd\nr8Q0STM73szeambrmdl/9SPfOcBf48M4zMw2jHl9OZ7/HXA1MMrMNgAupcLBWzObaWZHmtnGwA+B\na1V6PGAO4RddX7xKaFoX2LSovL+Y2QGEh+1J4JeFU0X5LCC0JrZL3PcGZjYsmV0F9qQxHxid2N+8\nTH7PAsMlDS1KP68f5VZrcyX10O8yq/j85xN+cBToKsq33DNYqvwLCV1P74jpP51Ij5ldaWb7AlsS\nuk/Oj6fmAAcUfRfeYmYLy91ngr6ez7Ryq+E51ny2+qLU93UO4dd58l7XN7MPF8ztlYG0NuEz+C6h\n1bUhcFtK3sX0+nwlqegeamEOoRcheR/rmtk1fV1oZi+b2almtiWht+c0lZ7h+yzhmSzQRYXfTzO7\n38wOJXTj3QRMKZc+L2KDmT1HEJz3S/phnbK9HniHpCMlDZI0WNIeiVbBusBSM1spaRxwRNH1qQ+b\npE9I2ijuvkRoHpeaPnopcJyCL4sUBi5LtUoeAiZIGi1pA0J/c6GsTeIvr6HAG4QvfqGsBcDowqCe\nmb1J6Lr7saTh8frRkg5Iu5cq+T3wWUnbSlqH0D1REjN7Gvhf4DxJaykM/h9L6JJJo2Sdx/taTOh/\n75Os66GKz//3wGckbSfpLcDZRefLPYMLAZO0ZeLYWwmf/8uSNiOMERRs2l7SxCh6ywm/kgs2XQyc\nH69B0ghJHyxTTjEPAR+QtIGkTYEvVVhuMeVe4L8HzpS0vsIkn/5Omb4XWCHpVIUB8IGSxkjaLZ5f\nQOjiLTCEMIa0mFAPhxDeRZVwI/BOhYkeg4BTqa71XI5LgJMk7Q4gad3Ee6AsMV3hu/Iy4b3RUyLp\n74CzJW0kaWPC97nc97OQ/9rxvfpWM+shjOWUyn8VeRCbVb8yzGwO4UP+qKTv9pW+z4zNXiJMifwE\n4RfIs4QBsSExyYnABZJeBE4nDN6WKyu5fzAwPV77PeDjpbqmzOxe4HhCX/yLwN9Z/Usoee9/Bv5A\nGBS9D/hTIpuBhG6AZwkD7HsBJ8VztwEzCd0Xz8ZjXyM0h6fFrpY/E/rlK6XcL/obgV8AdxEGJO+O\np5anXHI4oU//OcLL5HQrP6293Od7DvC72KXw4ZQ0yetrqYdSdvTn878RmEyY8fU4cGtRktRn0Mxe\nIbQQpsZ73o1QB3sSxir+SOiXLzAk2rKI8KxsAHwznvshcAuhW+VFwue2e5lyivlNtH82cDPhJVVJ\nuWtUSZn9cwjC9zThs7qK9OeqVF7hYHj5HQyMjXktJPQEFGbtXQUMifd6n5m9SBCJPxLGJD5C6Hru\nk9gyPJwwuWERoVWTOpu2hM3lvmtTCc/HL2LX5+OEyU59XksYS/27pJcJY8MXmdk9Ja77FvAw8C/C\nD4p7CV1wldj/KeDp+N06tsi2NVDv7jjHqQ6F6dcPmNmQPhM7ThVI+iJwaOw+dlqcPLRsnBZD0odj\nl+Qwwq+gPzbbJqf1kfQ2hQkTkrQDcApwXbPtcuqDi43TH04i9G/PIMxO+WJzzXHahCGEcYqXCL4n\nVxOm/zptgHejOY7jOJnjLRvHcRwnc1xsnKajEGsttbtERZHBO4E4Vb4Sp8ZK8vofSef2nbJsHn19\nRp+S5MFznVRcbDoESUdJul8hIsM8hSB9Na9pIekcSZfXkoeZnW9mn6vVljYkN33cyc9IUpdCQMni\n90fN9tbjeXLyiYtNByDpVIKfxXeAEQTP7MmEwHqNKL/SkPpNI082Kv/LPhSWJMhNnRVogbrrWFxs\n2hxJ6xEct75gZn8ys9fMrMfMbjaz02MaKYTBnyVpkaQpMYpB8lfsMZJmK8TwOjOeez9wJnB4bDE9\nGI/fLuk7ku6W9CqwpaRNJf1JIfbUDEnHJWw8R9JvE/ufVAi3v6hQVuLcHrGFVgjtXzKMkYKn+w3R\n3ufj9qjE+VI2rifpUq0Ot/7tNBGKNv9eq5dHeFjSNrEeF8S6em8i/ae1Ohz7LEmfS5ybEMv7hsKy\nD2ss7CXpywph998W9w9RWJpgabyHnRJpd9Xq5Q+mAGuXuoeY9mlJu8bto+NnvUPc/4yk6xL3W2hx\nFJZ7eCHez56rs9P3FZwln5R0YJlyT1OI7/aSpOmS9ivzPFVVdwre8DdodXj9O1PMcBpJJdE6/a91\n/wgRFFYAA8qkOZkQqnxTQtiOXwBXxnNdhNAjFxNC0+9MiM21XTx/DjFUfSK/2wme29uzOsT5nYQo\nCoOBXQhe3ROL8yCEq38Z2Cem/UG0/z3x/D3EaMyEWF1jU+5pGCFQ4BBCcM+rSEQWT7HxD4Tw92uz\nOtz68Sn5n0OY9v3eeP1lhDD2ZxAiPhxH70jaB7E67P54QsiZd8b9CYRQ7ufFex4Sjz0Tz59NCPsz\nLO7vSgi5sjuhdfFJQvT0wfHvaeDL0Y6Pxvo7N+U+fkOMOhw/45nACXH/MuDkEp9RFyE0STIa8Kdi\nOZ+JNn0emJdS5raE4I8j4/7mwJZlnqdq6+68+DkOiHWwT7O/h/6Xn6jPTnZsRIjcWm7Z3xOAb5rZ\nfDNbSYje+zGt7pM3YJKZrTCzRwjhLXbpo9zfmNnjsdxNCGHfTzOzlWb2MCFu2TElrvsocIOZ/SPa\n8p/0HgtYAWwtaSMzW2Zm00oVbmZLzOwPZrbczF4lhGN5dxkbhxFeaqeY2etmtpgQ7v3IMvfYbWZ/\njddfTRCoCyyES5lCiDy8XrTnFgux4rAQrudWwouzQA9wTqyfQoiWAZJ+QBC0iWa2JB4/Hvilmf2v\nBX5LCOsyLv4NMrOfWGjBXktYlyeNuwgvbKI95yf2J7C6FVOK4lbf02b232ZmBKHaRKWjP/cQfriM\nkTTIzJ4xs6fSCulH3a0k/HDaMtZBw5dAdtbExab9eZ4QebncZ90F/EGrw88/RvjCJkOyp4UyTyM5\nk+ptwBIzW5Y4VhzKPJk2GbJ9WbyHAp8lxH16XNJUhZUX10Bh1ceLYzfRC4SX5gZF3WJJG7sIv4zn\na3U4919SPqhicWj4ZDj21wgv43WjPQcprMZYCGF/UFHei6K4JtmAICznW4hflrT1q+oden40oe7e\nxppRe2eXuYc7CdHWNyG8D35PWJajC1jPzB4qc20xq5aLMLNe95/EzJ4EvgJMIsT0uzKWX5J+1N33\nCJHRb43dbqfhNB0Xm/bnXsKv3rTAlRC6NA6yNcPPz68g/7QZSMnjzwLDFKIeF0hbaqA4ZPs6hNZZ\nyNTsSTM7ykJo/+8B16h0FNyvEla13MNCKP5CqyYpNkkb5xC6BzdK1MEGZrZzyv1VjEI05GuivYUQ\n9reUsaXAEsJKn7+RtHeRrd+1NUPPX0Wov2IRTw3VH1/8rxGiON8VRe054HOsDrK6xmVp+VWKmU0x\ns/GsDm9fWFa7OPx/1XVnZq+a2dfMbCvCCpqnStqvVpud2nCxaXMsRL4+B5gs6dD4i3+Qwrrpheiu\nFxOWAdgcVq1f/qFENuVmHS0AtkgbSI82zCWMtZyvEPJ9Z0ILpVQo82uAQyTtrbCe+rnJ8uMgduFX\n7YuEF02pLsK3El6iLynEcJtU5h6wsMTFrcCPJL1VgbcrrtVeI2vFv8Vm9qbCQmvvq+RCM7uLEE33\nWkl7xMOXAJ9XXKxK0lskHRzF/F7gDUlfip/zRwjRj8txJyHkUKHL7I6i/WIWEeq8knWa1kBheYr9\nopCsoPeSBMXPU9V1J+kDkgq2FcLrl+tGdhqAi00HYGY/JIRQP4swMP8MIb5ZIYDmjwlLGtyqEH7+\nHnq/oMqFRb+aIAbPS/rflPQQxj62JLRyriUsd3t7CVsfi7b9LqZ9nt4rHx4I/J+kl4AfAYcnxjiS\nXESYQLA43s/NZe6hwDGEF9tjhFbF1YTxpv5isCqE/5eBq2M35RH0XkKifCZmfyWI8/WS3mlmDxC6\n134W85tBGKAndid9hBDy/XngMEJ9l+NOQnfXXSn7xfa8Rlho7B+xGy9NzNJaQEMIAVwLSxJsTJhY\nAUXPU6y7k6mu7rYB/qoQXv8fwGQz8xlpTcZjozmO4ziZ4y0bx3EcJ3NcbBzHcZzMcbFxHMdxMsfF\nxnEcx8mcQc02oJ5I8tkOjuM4VWJmmQdVbSuxAXjziSsBmPTTa5j0pY812Zr+00z7X9n0kJrzOO+8\n8zjzzDP7TphT3P7m4vY3jvXWW68h5Xg3mrMG686/kXXn39hsMxzHaSNcbJxUXHCcVqWnp/x+XvLs\nJNpWbCaO3bHZJtREXuzvr+CMHz++70Q5xu1vLrXY39MD73vfOnR3h3XUursH8r73rVOTOFSbZ6vX\nfxa0VQQBSVYYs3GyoR7jOY6TNd3dAznmmLU5/viVXHLJYC6//HXGj6+tKZJFnnlgvfXWa8gEgbZt\n2TjZ4F1rTiswfnwPxx+/kgsuGMLxx6+siyhkkWcn4WLjVI1PIHDyTnf3QC65ZDCnn76cSy4ZvKr7\nK295dhLejebUhHerOXmjML4yadJyxo/vobt7IJMmDeHWW5cxsJ/6kEWeeaFR3WguNk7dcOFx8kJP\nD71EoHg/L3nmAR+zcVoO715z8kKxCNRDFLLIs5NwsXHqjguO4zjFtF24GicfJAXHu9c6i3btbnJq\nw1s2TuZ4S6dzyMKh0mkPvGXjNISC4Hgrp70ZOBAmTVq+hvOjt2wcb9k4DcVbOe2POz86pXCxcRqO\nz1prb9z50SmFi43TNFxw2o+eHpg0aQiXX/46Z565gssvf51Jk4b4mI3jTp1OPvCxnPbBZ6O1Fu7U\n6XQU3rXWPrjzo1MKFxsnV7joOE574mLj5BIXnM4mj6ti5tGmNPJoq4uNk1u8ldOZ5NExNI82pZFX\nW32CgNMy+CSCziGPq2Lm0aY0qrHVJwg4ThHe0ukc8ugYmkeb0sijrS42TsvhotP+5NExNI82pZFH\nW5vejSbpUuAQYIGZ7Vzi/ATgT8C/46HrzOw7KXl5N1oH4t1r7UUeV8XMo01pVGtrx6zUKWlf4BXg\n8jJi81Uz+1AFebnYdDAuOu1DHh1D82hTGtXY2jFjNmZ2N7C0j2SZV4TT+njXWvuQR8fQPNqURh5t\nbbrYVMhekh6SdJOkHZttjJNffDyns8ijP0lWtPq9tsJ6Ng8Am5vZMkkHAX8Etk1LPOmn16zanjh2\nRybu6drUiaw7/0bvVmtzWmkcpVbqea/d3d10d3dnY2gZmj5mAyCpC7ih1JhNibRPAe8ysyUlzvmY\njbMGLjrtSyv5vtRKVvfaMWM2EZEyLiNpZGJ7LEEg1xAax0nDu9Xalzz6k2RFq99r07vRJF0JTAQ2\nkvQMcA6wFmBm9ivgY5JOBFYCrwGHN8tWp3VJCo63dNqHYn+S8eN7Wu4lXCmtfq+56EarF96N5lSK\nC07r42M29bnXjvGzqScuNk61uOi0Nq3k+1IrWd1rp43ZOE5T8PGc1iaP/iRZ0er32vQxG8dpNj6e\n4zjZ4y0bx0ngLZ18UMqBsdJjaddXWk4eyKtdteBi4zhFeBSC5pK2+NcBB/Q+dsAB65RMt2JFZYuH\n5XWRsbzaVSs+QcBxKsC71xpLKQdGoKJjhdlalThA5tUptJF2+QQBx8kR3tJpLKUcGCs9lnZ9peXk\ngbzaVQsuNo5TId691jhKLf5V6bG06ystJw/k1a5a8NlojlMlHuQzW3p6YNKkIau6jsaP72HSpCGY\n0evYOecMQWKNdLfcsqzk9cUOkGnlNNspNK921YqP2ThODbjoZEMpB0ao7NjAgZU7QObVKbSRdvmY\njeO0AN6tlg2lHBgrOQY9zJgxA+hZI22l5eSBvNpVCy42jlMjPpaTH4488kh23313jjzyyGab4hTh\nYuM4daITRacaR8ssuemmm9hzzz3585//DMCf//xnxo0bx0033ZR94RVSa720uqOni43j1JlOEZ1S\nzodpjpZZvhiXLVvGMcccw/Tp03sdf+yxxzjmmGNYtmxZdoVXSK2Omu3g6Omz0RwnI9p91trAgTBp\n0vKKHC2zHHMYPHgwXV1dzJo1a41zXV1dDB48OLvCKyStriqtl1qvzwPesnGcDCm0ctq1pVONo2VW\nDB48mMmTJ5c8N3ny5FyIDdTuqNnqjp4uNo7TINpRcKpxtMySAQNKv8rSjjeDWuul1R09vRvNcRpI\nQXDaoXutlPNhmqNl1g6Je+yxByeffDIzZsxg33335e6772bbbbdl7Nix2RVaBbU6araDo6c7dTpO\nk2ll4anU+bJVXohZUqujpq/U6ThOTbRy91qlzpdO7fXS6vXqYuM4OaCdJxE4DrjYOE6uaDXBqdWp\ns5UcFVvJ1jziYuM4OaNVWjmVrqiZ5nzYSo6KrWRrXmn6BAFJlwKHAAvMbOeUND8BDgJeBT5tZg+l\npPMJAk5bkffJA5WuqJnmE5LXlTJL0Uq2VkMnTRD4H+D9aSclHQRsZWbbACcAv2yUYY7TbPLeyqnV\nqbOVHBVbydY80nSxMbO7gaVlkhwKXB7TTgXWlzSyEbY5Tl7Iq+DU6tTZSo6KrWRrHmkFp85RwJzE\n/rx4bEFzzHGc5pAUnDx0r1W6omaa82ErOSq2kq15peljNgCSuoAbSo3ZSLoBON/M7on7fwW+YWb/\nLJHWx2ycjiEvglOLU2deV8osRSvZWg2NGrNphZbNPGCzxP7oeKwkk356zartiWN3ZOKeO2ZnmeM0\nkTyEvqnE0bDcC7mVHBVbydZydHd3093d3fBy89Ky2YLQstmpxLmDgZPM7AOSxgEXmdm4lHy8ZeN0\nLPUUncWLFzN79my6uroYPnw4ACtWwFprrU5TvJ81rdSKaqVWUMfMRpN0JXAPsK2kZyQdK+kESZ8D\nMLObgackzQIuBr7QRHMdJ7fUa+baNddeza67j+GEUw9j193HcM2117BiBXR1rcvkySFc/+TJg+nq\nWpcVK2ouriJqXaitkX4y7pNTmly0bOqFt2wcJ9DfVs7ixYvZdfcxnH37aLbYeShPP/Ia5+43lwf/\n919cddWmnHnmEPbeu4d77hnIeect56STVtbZ8nRayaenlXxyOqZl4zhO/elvK2f27NmM7FqHLXYe\nCsAWOw9lxObrMHv2bE46aSV7793DP/4xiL337mmo0EBr+fS4T86auNg4ThtT7UqhXV1dLJi9jKcf\neQ2Apx95jYXPLKOrq4vJkwdzzz0D2WefN7jnnoGrutQaRSv59LhPzpp4N5rjdBCVdK9dc+01nPLV\nLzJi83VY+MwyfvSDn/GhD36Mrq51Oeus0HU2efJgvvOdIcye/UpDJgkUxkEmTVrO+PE9dHcPXLVQ\nW/JYOZ+e4uuz8pNpZFn1oFHdaC42jtOB9CU6PhutNnw22pq42DhOh5IHp1Cn+fgEAcdxMiXvQT6d\n9sLFxnHajJ4eld0vplbBWbx4MQ888ACLFy+uKR+nvXGxcZw2oqdHjD/qvdwxdQQAd0wdwfij3luR\n4PSnpVPKAdRxSuFjNo7TZtwxdQQfP3kfvnD0TH5+xTb8/sf/YOKeC6vKo5LxnHIOoIVJBU7+8TEb\nx3H6xcQ9F/KFo2dy7s924gtHz6xaaKCy8ZxyDqCOU4yLjeO0GXdMHcHPr9iGs7/4KD+/YptVXWr9\noZzglHMAdZxiWmGJAcdxKqSnR5z5w11WdZ1NHLuQM3+4C91X/pWBA/vXZZ62aNvw4cP50Q9+xin7\n9XYA9S40pxRtN2az8sbrGLjN6802xXGaRk+PeglL8X6tFI/nlHIAdVoHd+rsBwWxAVxwHCdj3Cm0\nPfAJAjXSM3PtZpvgOG2NO4Q61dDWYzYFwfFWjtNpLFryEk/PXcQWozdm42HrZVZO2nhOHrvWWile\nWTvSti2bJD0z1/aWjtMxTLn5HrY9+Csc890fsO3BX2HKzfc0pNzCdOk8Onr66pnNp23HbNLwVo7T\nzixa8hLbHvwVzr5j89WOlhOfYcbNF2Xawilbfk4cPVtp9cxG4mM2GeGtHKedeXruIjbpWruXo+XI\nzYfw9NxFTSs/L46evnpmc+lTbCTtI+k2STMk/VvSU5L+3QjjssQFx2lHthi9Mc/Nfr2Xo+WCZ5az\nxeiNm1Z+Xhw9ffXM5lLJBIFLgVOAB4C2+inQM3Nt71Zz2oqNh63HL84+jhMn/pqRmw9hwTPL+cXZ\nxzWkC61c+VusvI9XaN5U6Z4emDRpyKqus/Hje3K9emY70ueYjaSpZrZng+ypiUrGbIpxsXHakUbN\nRutP+c3yz/HZaKVpulOnpN3i5seBgcB1wPLCeTP7Z9bGVUt/xKaAi47jNBZ3Cs0HeRCb28tcZ2b2\nnmxM6j+75npvAAAYzElEQVS1iA244DhOo3HBaT6NEpvUMRsz2w9A0tvNrNeEAElvr5cBkg4ELiJM\nVrjUzC4sOj8B+BNQsOE6M/tOvcpP4k6gTl6otRvsihvu5uqb7uOwD4zj6A/uWze7pj85j2kPz2Ls\nLluzw1ajSqapJjZbwSnURaf9qWTM5p9mtlvRsQfM7F01Fy4NAGYA+wPPAvcDR5jZ44k0E4CvmtmH\nKsivppZNEhccp1lMufkeTjz312zStTbPzX6dX5x9HEccvHfF12/53i8xf/ESNho9mOfnrmSTYcN4\n+u8/rdmuk7/7Gy6++jaGb7YWi+es4ITDDuDH3/x0rzSFlULPO/VhJu65kDumjqgq6rSLTuNpestG\n0vbAO4D1JX0kaRtQr3nDY4GZZjY7ljkFOBR4vChd5hVRjLdynGawaMlLnHjur3s5RZ448dfsP25M\nRS2cK264m/mLl3D+fdusuv6McTO54oa7a2rhTH9yHhdffVtRvrfx+SMO6NXCGTjQOO/Uh9dYKbTS\nqNPe0mlfyvnZbAccAmwAfDDxtxtwfJ3KHwXMSezPjceK2UvSQ5JukrRjncquCPfHcRpJrU6ZV990\nHxuNHtzr+mGjBnP1TffVZNe0h2cxfLO1euW70ei1mPbwrDXS1mulUKe9KDdm8yfgT5L2MrN7G2hT\nMQ8Am5vZMkkHAX8Etk1LfO4VU1ZtT9hpDBN2HlOzAd7KcRpF0imy0IKoxinzsA+M48//+WCv65fM\nW8lhnx9Xk11jd9maxXNW9Mr3+bkrGLvL1mukLV4pdOLYhTULjrd06kd3dzfd3d0NL7fcbLSfAqlt\nXzP7cs2FS+OASWZ2YNw/PWTde5JA0TVPAe8ysyUlztVtzCYNFxwnawpjNkmnyGrGbLZ4z5d4bskS\nho0azJJ59R+z2Wj0Wjw/N5sxm3K44GRDHqY+fypu7gPsCFwV9w8DHjOzz9dcuDQQeIIwQWA+MA04\n0symJ9KMNLMFcXss8Hsz2yIlv8zFBlxwnOzplNlo/cFFp740XWxWJZDuA/Y1szfi/mCg28xqa5ev\nzv9A4Mesnvp8gaQTCC2cX0k6CTgRWAm8BpxiZlNT8mqI2BRw0XGc5uGiUx/yJDZPAHsVuq0kbQjc\nZ2bbZW1ctTRabMAFxylNI8PF1FpWqeubHe6mUlxwaqfpU58TXAA8GCMKCHg3MClLo1oJnzzgFFOr\nn0wjyyp1PdAw+2vFp0q3DhUtniZpE6AQjHOqmT2XqVX9pBktmyQuOE4jFy+rtaxS109692wGCM6+\ns6spi6/VggtO/2j64mnRqbMQkPNtBH+YOcDbEkE6nQS+MJvTyMXLai2r1PUbjhjE8M2GNG3xtVoo\nLEvtPjr5pJxT56nx/w9K/P1Xxna1NC44nUsjFy+rtaxS1y9d+AaL5yxv2uJr9cIFJ3+Uc+r8XPy/\nX+PMaR98YbbOpJGLl9VaVqnrL54UgoM0a/G1euLjOfmiktlodwN3At3AP8zs5UYY1h+aPWZTjItN\n5+Kz0fKFC046eZr6vCUwPv6NIyyg1m1mp2RtXLXkTWwKuOg4Tn5w4elN0ycIFDCzp4DbgL8BdwHr\nADtkbFdb4WM4jpMffDynOfQpNpKeJAS/HAlcCowpxDJzKsdnqrUWi5a8xP2PPMmiJS/VNd/pT87j\nsuvuZPqT88oe6+l5k8efnEdPz5t92lXpsSzvq9XwWWuNp5JutJOBfYHNCOvM3AncZWZPZm9edeS1\nG60Y71bLN1k5ZZZafAwouSDZhz7/fW68/UEO2W9Xrv/l11PtgjUdMEsdO+LgvRvqbNpqdHLXWm7G\nbFYllNYFjgW+Bow2s4FZGtYfWkVsCrjo5I+snDKnPzmPXT/6jV6Lj502dgYDBqj3sT1msMWmI5k1\ne8Gqa8dsuxlf++wHOPmC3/S2a8Js3jSYdFdXn06Z91zxbfY++j8b4mzaynSi6ORmzEbSDyRNBaYC\nOwNnA9tkbVgn4N1q+SMrp8xSi4+9Zf2BDB+9+timWw/hjRX0EhqAf82Yw3FnXsKIImfL4aOHsOHI\nwRU5ZU57eFbDnE1bGe9ay44+xQa4F/iQmb3DzI43s8vM7N9ZG9Yp+FhOvsjKKTO5+Fgh31df7GHx\n3NXH5k5/PXUB9K5Rw1nwTG+7Fs9dztIFKytyyhy7y9YNczZtdVxwsqHPQJxmdk0jDOl03Ak0H2Tl\nlLnDVqM44bADOGPc6sXHTjz8fQC9jn30fXtw7V/uX+P6/zn/BOYuXNLbrnPWdMBMc8rcYatRDXM2\nbQd8ldD6U/GYTSvQamM2pXDByQdZOTWWWnwseWzpi6+y75GT1rju7t9NYu/dtq3YATPN/nZz1mwU\n7Sw4uZsg0Aq0g9gUcNHpTN58801O/68pPPHvZ3n32B24a9p0tnv727jw60ciZf4+cPqgHUWn6WIj\naVi5CwuLqeWJdhIbcMFxnLzSTqKTh8XTHgCM0kOWBrw9E4ucVfjCbK1LI7uxsoiN5pTHg3xWT7mo\nz1s20hAnHZ880FqkOU9m4VSZxUqd7uhZOevOv9EFp0IqXalzQ4Jvzao5umZ2V4Z29Yt260YrhYtO\nvklzCs3CqTKLlTrd0bM2WlF48uTUeRwhAOdfgG/F/5OyNctJw31y8k2aU2gWTpVZrNTpjp614T46\n6VTi1HkysAcwOy6ktivwQqZWOWVxwckvaU6hWThVZrFSpzt61o4H+SxNn06dwOtm9rokJA0xs8cl\nbZe5ZU5ZfBwnn6Q5hWbhVJnFSp3u6Fk/fDynN5VEff4DIQDnV4D3AEuBwWZ2cPbmVUcnjNkU44KT\nT3w2mlNMXoWn6X42JRNLE4D1gVvMbGVdDJAOBC4idOldamYXlkjzE+Ag4FXg02b2UEpeHSc24ILj\nOK1CHgUnTxMEflvYNrM7zex64L/rUbikAcDPgPcD7wCOlLR9UZqDgK3MbBvgBOCX9Si7nfBgno7T\nGnTyWE4lEwTekdyRNBB4V53KHwvMNLPZsaU0BTi0KM2hwOUAZjYVWF/SyDqV31a0k+C00oqS1ayU\n6TidKjipEwQknQGcCQyV9BKrIwmsAH5Vp/JHAXMS+3MJAlQuzbx4bAHOGrRD1IFWcjSsdPXMvNrv\nNIdOjCpdLoLA+cD5ks43szMaaFNNnHvFlFXbE3Yaw4SdxzTRmubRqqKzaMlLnHjur3s5Gp448dfs\nP25M7gavS9o64ZI1Vs/Mq/1OPmh06Jvu7m66u7sbUlaSSqY+f1PSJ4AtzezbkjYDNjWzaXUofx6w\neWJ/dDxWnGazPtKs4uyjj6iDWU6zKOdomLeXdSlbh48ewsoVb7aE/U6+aNRU6fHjxzN+/PhV+xdc\ncEHmZUJlYzaTgb2Ao+L+K/FYPbgf2FpSl6S1gCOA64vSXA8cAyBpHPCCmXkXWoW02uSBVnI0LGVr\nqdUz82q/kz8KDqHtOK5TSctmTzPbTdKDAGa2NApDzZhZj6QvAreyeurzdEknhNP2KzO7WdLBkmYR\npj4fW4+yO41WcQJtJUfDkraWWD0zr/Y7+abdnEIrceqcCuwN3B9FZ2PgVjPbtREGVkOn+tlUSyuI\nTis5GlazUqbj9IcsRSc3Tp2SjgYOB3YDLgM+BpxlZldnbVy1uNhUTisIjuM4vclCdHIjNgDR0XJ/\nwvTnv5nZ9KwN6w8uNtXTrqIz/cl5THt4FmN32ZodthpVt7RZhIbxVpBTLfUUnaav1ClpbeDzwNbA\no8DFZvZG1gY5jaVVxnKq4eTv/oaLr76N4ZutxeI5KzjhsAP48Tc/XXPaLBYqA/fJcaqnFcdzUls2\nkq4CVgLdhLhkT5vZVxpoW9V4y6Y22kF0pj85j10/+g3Ov2+bVX4uZ4ybyYPXfm+NVks1abNYqGzS\nu2czQHD2nV2+eJnTb2oVnTzERtvRzD5hZhcTxmnenbUxTnNppSnSaUx7eBbDN1url5/LRqPXYtrD\ns2pKm8VCZRuOGMTwzYb44mVOTbTKVOlyYrMqqrN3n3UOrS44Y3fZmsVzVvTyc3l+7grG7rJ1TWmz\nWKhs6cI3WDxnufvkOHUh76JTzs9mlxgTDcLEgGSMNDMzb+e3Ka08jrPDVqM44bADOGPcbWw0ei2e\nnxvGYUoN/FeTNouFyi6e5D45TudQ1Xo2ecfHbOpLqwoO+Gw0x4HKxnNyNfW5VXCxqT+tLDiO4/Qt\nOHmYIOA4LRdbzXGc3uRlLKeS2GiO07JLFjiOE2j2GjresnGqwls5jtP6NKOl4y0bp2q8leM4rU+j\nBcdbNo7jOE7muNg4/cYnDziOUykuNk7NuOA4jtMXLjZOXfBWjuM45XCxceqKC47jOKVwsXHqjrdy\nHMcpxsXGyQwXHMdxCrjYOJnirRzHccDFxnEcx2kALjZOQ/AWjuN0Nk0TG0kbSrpV0hOS/iJp/ZR0\nT0t6WNKDkqY12k6nvrjgOE5n0syWzenAX81sO+DvwBkp6d4EJprZrmY2tmHWOZnhguM4nUczxeZQ\n4LK4fRnw4ZR0wrv72g7vVnOczqKZL/ERZrYAwMyeA0akpDPgNkn3Szq+YdY5DcEFx3E6g0yXGJB0\nGzAyeYggHmeVSJ62PvU+ZjZf0sYE0ZluZnfX2VSnifiSBY7T/mQqNmZ2QNo5SQskjTSzBZI2ARam\n5DE//l8k6Q/AWCBVbM69Ysqq7Qk7jWHCzmP6a77TYHpmru2C4zgZc8fUx7hj2mMNL1dmaQ2KjAuW\nLgSWmNmFkk4DNjSz04vSrAMMMLNXJL0FuBX4lpndmpKnrbzxusxtd7LFBcdxGseA7Y7CzJR5OVkX\nUIYLgQMkPQHsD1wAIGlTSYUl5EYCd0t6ELgPuCFNaJz2wcdxHKf9aFrLJgu8ZdN+eCvHcbKlE1o2\njtMn3spxnPbAxcbJPe6T4zitj4uN0zK44DhO6+Ji47QU3spxnNbExcZpSVxwHKe1yNSp01nNiMOO\nZulrr63a33DoUBZefUUTLWp9PPKA47QO3rJpEEtfew2DVX9J4XFqw1s5jpN/XGyctsAFx3HyjYuN\n0zb45AHHyS8uNg1iw6FDEaz623Do0CZb1L644DhO/vAJAg3CJwM0Fo8g7Tj5wls2TtviLRzHyQ8u\nNk5b4+M4jpMPXGycjsAFx3Gai4/ZNBl39mwcPo7jOM3DWzZNxp09G4u3cBynObjYOB2Hj+M4TuNx\nsXE6Fhccx2kcLjZNxp09m4u3chynMfgEgSbjkwHygU8ecJxs8ZaN40S8leM42eFi4zhFuOg4Tv1x\nsXEcx3Eyx8dsKqQa58uhh3yENxL7g4C3Dh1a8vpK83Xnz8bjK4E6Tv1oWstG0sck/UtSj6TdyqQ7\nUNLjkmZIOq2RNiapxvnyjUQ6i/tp11earzt/Ng/vUnOc2mlmN9qjwH8Ad6YlkDQA+BnwfuAdwJGS\ntm+MeY6zGh/HcZzaaJrYmNkTZjaT4F6SxlhgppnNNrOVwBTg0IYY6DglcMFxnP6R9wkCo4A5if25\n8VjDqcb5clAineJ+2vWV5uvOn/nBBcdxqifTCQKSbgNGJg8Rhhy+aWY3ZFHmuVdMWbU9YacxTNh5\nTF3yrWYw/rUbr6t7vj4ZIF+4E6jTqtwx9THumPZYw8uVmTW80F4GSLcDXzWzf5Y4Nw6YZGYHxv3T\nATOzC1PyspVVvOgdpx646DitzIDtjsLMyg1n1KecrAuokLQbvR/YWlKXpLWAI4DrG2eW4ziOUw+a\nOfX5w5LmAOOAGyXdEo9vKulGADPrAb4I3Ar8HzDFzKY3y2bHKYXPVHOcvml6N1o9ybIbbZ1DPsLK\nxP5gYNmN15U8vm6KA6fT/niXmtNqNKobzSMIVMhKwsyGAipzvOCAueqYO2B2DB51wHFKk5cxm7ag\np9kGOLnBu9UcpzcuNnXkw802wMkVPpbjOKvxbrQKGUzvKXODU44XSB5zB8zOxrvWHMcnCNTMsteX\nM/yIT7LyjTfWODd40CAWT/kt66w9pKE2OfnFBcfJG53mZ9OyDB40kC1Hjih5bsuRIxg8aGCDLXLy\njHerOZ2Ki02NDB40iIu/fFLJcxd/+SQGD/KeSqc3LjhOJ+JiUwcGDCjdAk077jg+ecDpNDpmzKaa\nlS5LOWoWFkErMIgQcLN4Vc4BrF7gLJmWlOt9BU7Hx3GcZuJOnXWmGkfLNAfOUsfeqCJtqWPuAOr4\nbDWnE/BuNMfJCd6t5rQzLjaOkyN8LMdpVzpGbKpZ6XJwIp3ifqnVN0k5Xumxau1yHMdpVTpmgoDj\ntCI+juNkjTt1Oo7jOG2Di43j5Bgfw3HahY6Z+pxGFn4uxb43BZ8ax+kvPTPX9i41p6XpeLHJws8l\nzffGcWrB/XGcVsa70RynxfCuNacVcbFxnBbFBcdpJTpebLLwc0nzqXGceuOtHKdVaFs/mzsf+RcT\ndh7TZIv6j9vfXFrR/uRYzh1TH2Pinjs20ZracPsbh/vZ1Midj/6r2SbUhNvfXFrR/mQL545pjzXR\nktpx+9uPthUbx+lEvEvNySsuNo7TZvTMXBt73kcKnXzRdmM2zbbBcRyn1WjEmE1biY3jOI6TT7wb\nzXEcx8kcFxvHcRwnc9pGbCR9TNK/JPVI2q1MuqclPSzpQUnTGmljOaqw/0BJj0uaIem0RtpYDkkb\nSrpV0hOS/iJp/ZR0uar/SupT0k8kzZT0kKR3NtrGcvRlv6QJkl6Q9M/4d1Yz7CyFpEslLZD0SJk0\nea77svbnue6bgpm1xR+wHbAN8HdgtzLp/g1s2Gx7+2M/4cfBLKCLsIDoQ8D2zbY92nYh8I24fRpw\nQd7rv5L6BA4CborbewL3NdvuKu2fAFzfbFtT7N8XeCfwSMr53NZ9hfbntu6b8dc2LRsze8LMZtJ3\nkGWRwxZdhfaPBWaa2WwzWwlMAQ5tiIF9cyhwWdy+DPhwSro81X8l9XkocDmAmU0F1pc0srFmplLp\n85DLwONmdjewtEySPNd9JfZDTuu+GeTlS99IDLhN0v2Sjm+2MVUyCpiT2J8bj+WBEWa2AMDMngNG\npKTLU/1XUp/FaeaVSNMsKn0e9ordUDdJao0YKoE8132ltGrd152W8vySdBuQ/GUjwsvrm2Z2Q4XZ\n7GNm8yVtTHjpTY+/UDKnTvY3jTL2l+qLTptT37T671AeADY3s2WSDgL+CGzbZJs6Ba/7BC0lNmZ2\nQB3ymB//L5L0B0JXRENednWwfx6weWJ/dDzWEMrZHwdKR5rZAkmbAAtT8mha/ZegkvqcB2zWR5pm\n0af9ZvZKYvsWST+XNMzMljTIxlrIc933SYvXfd1p1260kv2kktaRtG7cfgvwPiCPERfT+nnvB7aW\n1CVpLeAI4PrGmVWW64FPx+1PAX8qTpDD+q+kPq8HjgGQNA54odBdmAP6tD85xiFpLMGRO08vu8JK\nHKXIc90XSLW/Beq+sTR7hkK9/ggD0nOA14D5wC3x+KbAjXF7S8KMnQeBR4HTm213NfbH/QOBJ4CZ\nObN/GPDXaNutwAatUP+l6hM4AfhcIs3PCLO+HqbMTMc82g+cRBD0B4F7gD2bbXPC9iuBZ4HlwDPA\nsS1W92Xtz3PdN+PPw9U4juM4mdOu3WiO4zhOjnCxcRzHcTLHxcZxHMfJHBcbx3EcJ3NcbBzHcZzM\ncbFxHMdxMsfFxuk44jIO/5T0qKSrJK3dR/ozKsz3KUnDKj1eC9GR88jE/qck/bSeZThOPXGxcTqR\nV81sNzPbCVgJfL6P9GdWmG+a01oWzmxbAkc1oBzHqQsuNk6n0w1sDSDpaElTY6vnF5IGSDofGBqP\n/Tam+0OMWv2opOMSeaWFXVl1vEQZisdflvSdGCH4nhioFElvl3RvXHDu25JejlmdD+wb8zk5Hhsl\n6RaFBewurFsNOU4dcLFxOpHCC34QYYGuRyVtDxwO7G1muwFvAkeZ2RnAstgS+mS8/lgz2wPYAzhZ\n0oYVFVq6jKPj6bcA95jZOwkCWFh+4cfAj8xsF8ISAoXWy+lAd7Trx/HYLsBhwM7A4ZJaLRy/08a0\nVNRnx6kTQyX9M27fBVxKiGm1G3B/bG2sDTwX0xS3WL4iqbA43GjCCqvllrguCMT+ZcpYYWY3x+0H\ngPfG7b1YvSDalcD3y5TzN4uRhiU9RljBs2WiJDvtjYuN04ksiy2LVcSX/2Vm9s1yF0qaALyHEFRx\nuaTbCaJRCeXKWJHY7mH1dzM5DtPXqo/LU/JwnKbj3WhOJ1Lqpf034GOJsZINJRXWUlkhaWDcXh9Y\nGoVme2BcFeWVKyNNSO4DPha3j0gcfxl4awVlO04ucLFxOpE1Zm2Z2XTCiqO3SnqYsEzCpvH0rwjj\nOr8FbgEGS/o/4Dzg3nL5Jo/3UUbatacAp0p6CNgKeDEefwR4U9KDcYJA8fU+M83JFb7EgOPkGElD\nzey1uH04cISZ/UeTzXKcqvE+XcfJN++S9DNCN9tS4DNNtsdx+oW3bBzHcZzM8TEbx3EcJ3NcbBzH\ncZzMcbFxHMdxMsfFxnEcx8kcFxvHcRwnc1xsHMdxnMz5/z2Cr/BcYLO3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xdc45b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]\n",
    "step = .01 \n",
    "margin = .1\n",
    "sl_min, sl_max = X_train4[:, 2].min()-margin, X_train4[:, 2].max() + margin\n",
    "sw_min, sw_max = X_train4[:, 3].min()-margin, X_train4[:, 3].max() + margin\n",
    "sl, sw  = np.meshgrid(\n",
    "    np.arange(sl_min, sl_max, step),\n",
    "    np.arange(sw_min, sw_max, step), \n",
    "    )\n",
    "Zs = clf_petal.predict(np.c_[sl.ravel(), sw.ravel()]).reshape(sl.shape)\n",
    "centroids_s = clf_petal.cluster_centers_\n",
    "plt.figure(1)\n",
    "plt.clf()\n",
    "plt.imshow(Zs, interpolation='nearest', extent=(sl.min(), sl.max(),\n",
    "                                                sw.min(), sw.max()), cmap= plt.cm.Pastel1, aspect='auto', origin='lower')\n",
    "for j in [0,1,2]:\n",
    "    px = X_train4[:, 2][y_train4 == j]\n",
    "    py = X_train4[:, 3][y_train4 == j]\n",
    "    plt.scatter(px, py, c=colormarkers[j][0], marker= colormarkers[j][1])\n",
    "plt.scatter(centroids_s[:, 0], centroids_s[:, 1],marker='*',linewidths=3, color='black', zorder=10)\n",
    "plt.title('K-means clustering on the Iris dataset using Petal dimensions\\nCentroids are marked with stars')\n",
    "plt.xlim(sl_min, sl_max)\n",
    "plt.ylim(sw_min, sw_max)\n",
    "plt.xlabel(\"Petal length\")\n",
    "plt.ylabel(\"Petal width\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, calculate the clusters, using the four attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeans(copy_x=True, init='k-means++', max_iter=300, n_clusters=3, n_init=10,\n",
       "    n_jobs=1, precompute_distances='auto', random_state=33, tol=0.0001,\n",
       "    verbose=0)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)\n",
    "clf.fit(X_train4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf.labels_==0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 2 1]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf.labels_==1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 1 2 2 1 2 2 1 2 2 2 1 2 1 2 2 2 1 2 2 2 2 2 1 1 2 2 2 2 2 2 2 1 2 2]\n"
     ]
    }
   ],
   "source": [
    "print (y_train[clf.labels_==2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure precision & recall in the testing set, using all attributes, and using only petal measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "     setosa       1.00      1.00      1.00         8\n",
      " versicolor       0.64      0.64      0.64        11\n",
      "  virginica       0.79      0.79      0.79        19\n",
      "\n",
      "avg / total       0.79      0.79      0.79        38\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred=clf.predict(X_test4)\n",
    "print (metrics.classification_report(y_test, y_pred, target_names=['setosa','versicolor','virginica']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "     setosa       1.00      1.00      1.00         8\n",
      " versicolor       0.85      1.00      0.92        11\n",
      "  virginica       1.00      0.89      0.94        19\n",
      "\n",
      "avg / total       0.96      0.95      0.95        38\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_petal=clf_petal.predict(X_test4[:,2:4])\n",
    "print (metrics.classification_report(y_test, y_pred_petal, target_names=['setosa','versicolor','virginica']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait, every performance measure is better using just two attributes. It is possible that less features give better results? Although at a first glance this seems contradictory, we will see in future notebooks that selecting the right subset of features, a process called feature selection, could actually improve the performance of our algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised Learning: Regression\n",
    "In every example we have seen so far the output we aimed at predicting belonged to a discrete set. For classification, the set was the target class, while for the clustering algorithm the set included different calculated clusters. What if we want to predict a value extracted from the real line?. In this case, we are trying to solve a regression problem. \n",
    "\n",
    "To show how regression works in scikit-learn, we will apply to a (very) simple and well-know problem: trying to predict the price of a house given some of its. As the dataset, we will use the Boston house-prices dataset (find the dataset description and attributes [here](http://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boston dataset shape:(506L, 13L)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "print ('Boston dataset shape:{}'.format(boston.data.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'\n",
      " 'B' 'LSTAT']\n"
     ]
    }
   ],
   "source": [
    "print (boston.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create training and testing sets, and scale values, as usual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Miniconda2\\lib\\site-packages\\sklearn\\preprocessing\\data.py:583: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.\n",
      "  warnings.warn(DEPRECATION_MSG_1D, DeprecationWarning)\n",
      "C:\\Miniconda2\\lib\\site-packages\\sklearn\\preprocessing\\data.py:646: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.\n",
      "  warnings.warn(DEPRECATION_MSG_1D, DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "X_train_boston=boston.data\n",
    "y_train_boston=boston.target\n",
    "X_train_boston = StandardScaler().fit_transform(X_train_boston)\n",
    "y_train_boston = StandardScaler().fit_transform(y_train_boston)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a method for training and evaluating a model. This time, to evaluate our model we will a different approach: instead of separating the training set, we will use _cross-validation_. \n",
    "\n",
    "Cross-validation usually involves the following steps:\n",
    "1. Partition the dataset into k different subsets.\n",
    "2. Create k different models by training on k-1 subsets and testing on the remaining one.\n",
    "3. Measure the performance of each of the k models and use the average value as you performance value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_and_evaluate(clf, X_train, y_train, folds):\n",
    "    clf.fit(X_train, y_train)\n",
    "    print ('Score on training set: {:.2f}'.format(clf.score(X_train, y_train)))\n",
    "    #create a k-fold cross validation iterator of k=5 folds\n",
    "    cv = sklearn.cross_validation.KFold(X_train.shape[0], folds, shuffle=True, random_state=33)\n",
    "    scores = sklearn.cross_validation.cross_val_score(clf, X_train, y_train, cv=cv)\n",
    "    print ('Average score using {}-fold crossvalidation:{:.2f}'.format(folds,np.mean(scores)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the main advantages of cross-validation is reducing the variance of the evaluation measures. Note that when you have a training and testing set, you result will depend on how you created them. Evaluation within machine learning generally assumes that the distribution of classes on your training and testing sets are similar. If this is not the case, you may get results that are not an accurate measure of the actual classifier’s performance. Cross-validation allows us to mitigate this risk: since we are averaging on k different models built on k different datasets, we are reducing variance and probably producing more realistic performance scores for our models.\n",
    "Scikit-learn includes a linear model method called linear_model.SGDRegressor , which tries to minimize least squares loss using stochastic gradient descent. We will use it to find the model for Boston house values, and predict a new house value, given its 13 learning features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score on training set: 0.73\n",
      "Average score using 5-fold crossvalidation:0.70\n"
     ]
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn import linear_model\n",
    "clf_sgd = linear_model.SGDRegressor(loss='squared_loss', penalty=None, random_state=33)\n",
    "train_and_evaluate(clf_sgd, X_train_boston, y_train_boston,5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you remember the classification, we used accuracy, the proportion of correctly classified instances in the evaluation set, to summarize in just one number our method’s performance. For regression, accuracy is not a good idea, since we are predicting real values, and it is almost impossible that we may exactly predict the final real value. For regression, the default score function in scikit-learn is the coefficient of determination (or $R^2$ score), which measures the proportion of outcome variation explained by the model.; $R^2$ score ranges from 0 to 1, and reaches its maximum value of 1 when the model perfectly predicts all the target values. \n",
    "\n",
    "Let us show the the model  hyperplane coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.06777406  0.06767528 -0.04290825  0.08828856 -0.11797833  0.3394894\n",
      " -0.01969258 -0.23195707  0.09594823 -0.05271866 -0.19913907  0.10355794\n",
      " -0.36852382]\n"
     ]
    }
   ],
   "source": [
    "print(clf_sgd.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You probably noted the `penalty=None` parameter when we called the method. The penalization parameter for linear regression methods is introduced to avoid overfitting. It does this by penalizing those hyperplanes having some of their coefficients too large, seeking hyperplanes where each feature contributes more or less the same to the predicted value. This parameter is generally the L2 norm (the squared sums of the coefficients) or the L1 norm (the sum of the absolute value of the coefficients). Let us see how our model works if we introduce a L2 penalty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score on training set: 0.73\n",
      "Average score using 5-fold crossvalidation:0.70\n"
     ]
    }
   ],
   "source": [
    "clf_sgd1 = linear_model.SGDRegressor(loss='squared_loss', penalty='l2', random_state=33)\n",
    "train_and_evaluate(clf_sgd1, X_train_boston, y_train_boston,folds=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of the implementation here:\n",
    "To end with this chapter, we will summarize the main common steps to apply a supervised learning method in scikit-learn:\n",
    "-\tGet your dataset. Select your learning features, and create a 2D data array, with one column for each feature, and one row for each learning instance. Every feature is represented by a real number. Your data probably does not look like that in its original format. In the real world, this the preprocessing stage usually takes a lot of time. In the next chapters we will show several examples of preprocessing different types of data. \n",
    "-\tCreate an instance of an estimator (a classifier or a regressor). In scikit-learn, this corresponds to an object that implements the methods `fit(X,y)` and `predict(T)`. This estimator takes as arguments the model’s parameters. You can set these parameters by hand, or using tools for model selection (we will address this later in this book).\n",
    "-\tSeparate you training and testing sets (or, alternatively, use cross-validation)\n",
    "-\tBuild the model using the `fit(X,y)` method, being X your training data, and y the corresponding target class. \n",
    "-\tEvaluate you estimator on the testing data, using the `predict(T)` method, being T your testing dataset.\n",
    "-\tCompare your results with the original target classes, using one of the several measures in the metrics module.\n",
    "\n",
    "The previous steps are only an overview: scikit-learn offers several additional methods implementing different machine learning operations (such as dimensionality reduction, clustering, or semi-supervised learning), and many dataset transformation methods. \n",
    "We would look at most of them in the subsequent sections.\n",
    "___\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to do when you are stuck\n",
    "\n",
    "We try to convey every idea necessary to reproduce the steps throughout this\n",
    "website. Nevertheless, there will be situations where you are stuck. The reasons\n",
    "might range from simple typos over odd combinations of package versions to\n",
    "problems in understanding.\n",
    "\n",
    "In this situation, there are many different ways to get help. Most likely, your problem\n",
    "will already be raised and solved in the following excellent Q&A sites:\n",
    "\n",
    "http://metaoptimize.com/qa: This Q&A site is laser-focused on machine learning\n",
    "topics. For almost every question, it contains above average answers from machine\n",
    "learning experts. Even if you don't have any questions, it is a good habit to check it\n",
    "out every now and then and read through some of the answers.\n",
    "\n",
    "http://stats.stackexchange.com: This Q&A site is named Cross Validated,\n",
    "similar to MetaOptimize, but is focused more on statistical problems.\n",
    "\n",
    "http://stackoverflow.com: This Q&A site is much like the previous ones,\n",
    "but with broader focus on general programming topics. It contains, for example,\n",
    "more questions on some of the packages that we will use in this book, such as\n",
    "SciPy or matplotlib.\n",
    "\n",
    "http://www.TwoToReal.com: This is the instant Q&A site written by the authors to\n",
    "support you in topics that don't fit in any of the preceding buckets. If you post your\n",
    "question, one of the authors will get an instant message if he is online and be drawn\n",
    "in a chat with you.\n",
    "\n",
    "As stated in the beginning, this book tries to help you get started quickly on your\n",
    "machine learning journey. Therefore, we highly encourage you to build up your own\n",
    "list of machine learning related blogs and check them out regularly. This is the best\n",
    "way to get to know what works and what doesn't.\n",
    "\n",
    "The only blog we want to highlight right here is http://blog.kaggle.com, the blog of the Kaggle company, which is carrying out machine learning competitions. Typically, they encourage the winners of the competitions to\n",
    "write down how they approached the competition, what strategies did not work, and\n",
    "how they arrived at the winning strategy. Even if you don't read anything else, this is a must."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}