Understand Your Data With Descriptive Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook you will discover 7 recipes that you can use in Python to better understand your machine learning data. After\n",
    "reading this lesson you will know how to: <br>\n",
    "  1. Take a peek at your raw data.\n",
    "  2. Review the dimensions of your dataset.\n",
    "  3. Review the data types of attributes in your data.\n",
    "  4. Summarize the distribution of instances across classes in your dataset.\n",
    "  5. Summarize your data using descriptive statistics.\n",
    "  6. Understand the relationships in your data using correlations.\n",
    "  7. Review the skew of the distributions of each attribute.\n",
    "\\end{enumerate}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Peek at Your Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no substitute for looking at the raw data. Looking at the raw data can reveal insights that you cannot get any other way. It can also plant seeds that may later grow into ideas on how to better pre-process and handle the data for machine learning tasks. You can review the first 20 rows of your data using the head() function on the Pandas DataFrame.<br>\n",
    "Details of each feature in the dataset:\n",
    "1. Number of times pregnant\n",
    "2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test\n",
    "3. Diastolic blood pressure (mm Hg)\n",
    "4. Triceps skin fold thickness (mm)\n",
    "5. 2-Hour serum insulin (mu U/ml)\n",
    "6. Body mass index (weight in kg/(height in m)^2)\n",
    "7. Diabetes pedigree function\n",
    "8. Age (years)\n",
    "9. Class variable (0 or 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    preg  plas  pres  skin  test  mass   pedi  age  class\n",
      "0      6   148    72    35     0  33.6  0.627   50      1\n",
      "1      1    85    66    29     0  26.6  0.351   31      0\n",
      "2      8   183    64     0     0  23.3  0.672   32      1\n",
      "3      1    89    66    23    94  28.1  0.167   21      0\n",
      "4      0   137    40    35   168  43.1  2.288   33      1\n",
      "5      5   116    74     0     0  25.6  0.201   30      0\n",
      "6      3    78    50    32    88  31.0  0.248   26      1\n",
      "7     10   115     0     0     0  35.3  0.134   29      0\n",
      "8      2   197    70    45   543  30.5  0.158   53      1\n",
      "9      8   125    96     0     0   0.0  0.232   54      1\n",
      "10     4   110    92     0     0  37.6  0.191   30      0\n",
      "11    10   168    74     0     0  38.0  0.537   34      1\n",
      "12    10   139    80     0     0  27.1  1.441   57      0\n",
      "13     1   189    60    23   846  30.1  0.398   59      1\n",
      "14     5   166    72    19   175  25.8  0.587   51      1\n",
      "15     7   100     0     0     0  30.0  0.484   32      1\n",
      "16     0   118    84    47   230  45.8  0.551   31      1\n",
      "17     7   107    74     0     0  29.6  0.254   31      1\n",
      "18     1   103    30    38    83  43.3  0.183   33      0\n",
      "19     1   115    70    30    96  34.6  0.529   32      1\n"
     ]
    }
   ],
   "source": [
    "# View first 20 rows\n",
    "from pandas import read_csv\n",
    "filename = \"pima-indians-diabetes.csv\"\n",
    "names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']\n",
    "data = read_csv(filename, names=names)\n",
    "peek = data.head(20)\n",
    "print(peek)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Dimensions of Your Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You must have a very good handle on how much data you have, both in terms of rows and columns.<br>\n",
    "* Too many rows and algorithms may take too long to train. Too few and perhaps you do not have enough data to train the algorithms.\n",
    "* Too many features and some algorithms can be distracted or suffer poor performance due to the curse of dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(768, 9)\n",
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "# Dimensions of your data\n",
    "shape = data.shape\n",
    "print(shape)\n",
    "print(type(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data Type For Each Attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The type of each attribute is important. Strings may need to be converted to floating point values or integers to represent categorical or ordinal values. You can get an idea of the types of attributes by peeking at the raw data, as above. You can also list the data types used by the DataFrame to characterize each attribute using the dtypes property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preg       int64\n",
      "plas       int64\n",
      "pres       int64\n",
      "skin       int64\n",
      "test       int64\n",
      "mass     float64\n",
      "pedi     float64\n",
      "age        int64\n",
      "class      int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "types = data.dtypes\n",
    "print(types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Descriptive Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descriptive statistics can give you great insight into the shape of each attribute. Often you can create more summaries than you have time to review. The describe() function on the Pandas DataFrame lists 8 statistical properties of each attribute. They are:<br>\n",
    "* Count.\n",
    "* Mean.\n",
    "* Standard Deviation.\n",
    "* Minimum Value.\n",
    "* 25th Percentile.\n",
    "* 50th Percentile (Median).\n",
    "* 75th Percentile.\n",
    "* Maximum Value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          preg     plas     pres     skin     test     mass     pedi      age    class\n",
      "count  768.000  768.000  768.000  768.000  768.000  768.000  768.000  768.000  768.000\n",
      "mean     3.845  120.895   69.105   20.536   79.799   31.993    0.472   33.241    0.349\n",
      "std      3.370   31.973   19.356   15.952  115.244    7.884    0.331   11.760    0.477\n",
      "min      0.000    0.000    0.000    0.000    0.000    0.000    0.078   21.000    0.000\n",
      "25%      1.000   99.000   62.000    0.000    0.000   27.300    0.244   24.000    0.000\n",
      "50%      3.000  117.000   72.000   23.000   30.500   32.000    0.372   29.000    0.000\n",
      "75%      6.000  140.250   80.000   32.000  127.250   36.600    0.626   41.000    1.000\n",
      "max     17.000  199.000  122.000   99.000  846.000   67.100    2.420   81.000    1.000\n"
     ]
    }
   ],
   "source": [
    "from pandas import set_option\n",
    "set_option('display.width', 100)\n",
    "set_option('precision', 3)\n",
    "description = data.describe()\n",
    "print(description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Class Distribution (Classification Only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On classification problems you need to know how balanced the class values are. Highly imbalanced problems (a lot more observations for one class than another) are common and may need special handling in the data preparation stage of your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class\n",
      "0    500\n",
      "1    268\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class_counts = data.groupby('class').size()\n",
    "print(class_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Correlations Between Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Correlation refers to the relationship between two variables and how they may or may not change together. \n",
    "* The most common method for calculating correlation is Pearson's Correlation Coefficient, that assumes a normal distribution of the attributes involved. A correlation of -1 or 1 shows a full negative or positive correlation respectively. Whereas a value of 0 shows no correlation at all. \n",
    "* Some machine learning algorithms like linear and logistic regression can suffer poor performance if there are highly correlated attributes in your dataset. \n",
    "* As such, it is a good idea to review all of the pairwise correlations of the attributes in your dataset. You can use the     corr() function on the Pandas DataFrame to calculate a correlation matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        preg   plas   pres   skin   test   mass   pedi    age  class\n",
      "preg   1.000  0.129  0.141 -0.082 -0.074  0.018 -0.034  0.544  0.222\n",
      "plas   0.129  1.000  0.153  0.057  0.331  0.221  0.137  0.264  0.467\n",
      "pres   0.141  0.153  1.000  0.207  0.089  0.282  0.041  0.240  0.065\n",
      "skin  -0.082  0.057  0.207  1.000  0.437  0.393  0.184 -0.114  0.075\n",
      "test  -0.074  0.331  0.089  0.437  1.000  0.198  0.185 -0.042  0.131\n",
      "mass   0.018  0.221  0.282  0.393  0.198  1.000  0.141  0.036  0.293\n",
      "pedi  -0.034  0.137  0.041  0.184  0.185  0.141  1.000  0.034  0.174\n",
      "age    0.544  0.264  0.240 -0.114 -0.042  0.036  0.034  1.000  0.238\n",
      "class  0.222  0.467  0.065  0.075  0.131  0.293  0.174  0.238  1.000\n"
     ]
    }
   ],
   "source": [
    "correlations = data.corr(method='pearson')\n",
    "print(correlations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Skew of Univariate Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Skew refers to a distribution that is assumed Gaussian (normal or bell curve) that is shifted or squashed in one direction or another. \n",
    "*  Many machine learning algorithms assume a Gaussian distribution. Knowing that an attribute has a skew may allow you to perform data preparation to correct the skew and later improve the accuracy of your models. \n",
    "* You can calculate the skew of each attribute using the skew() function on the Pandas DataFrame.\n",
    "* The skew result show a positive (right) or negative (left) skew. Values closer to zero show less skew."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preg     0.902\n",
      "plas     0.174\n",
      "pres    -1.844\n",
      "skin     0.109\n",
      "test     2.272\n",
      "mass    -0.429\n",
      "pedi     1.920\n",
      "age      1.130\n",
      "class    0.635\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "skew = data.skew()\n",
    "print(skew)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.  Tips To Remember"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section gives you some tips to remember when reviewing your data using summary statistics.<br>\n",
    "* <b>Review the numbers:</b> Generating the summary statistics is not enough. Take a moment to pause, read and really think about the numbers you are seeing.\n",
    "* <b>Ask why:</b> Review your numbers and ask a lot of questions. How and why are you seeing specific numbers. Think about how the numbers relate to the problem domain in general and specific entities that observations relate to.\n",
    "* <b>Write down ideas:</b> Write down your observations and ideas. Keep a small text file or note pad and jot down all of the ideas for how variables may relate, for what numbers mean, and ideas for techniques to try later. The things you write down now while the data is fresh will be very valuable later when you are trying to think up new things to try."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}