mnist_predict_digit_cnn_v2.3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Convolutional Neural Network (CNN) for MNIST Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "Convolutional Neural Networks are very similar to ordinary Neural Networks, they are made up of neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity. The whole network still expresses a single differentiable score function: from the raw image pixels on one end to class scores at the other. And they still have a loss function (e.g. SVM/Softmax) on the last (fully-connected) layer and all the tips/tricks we developed for learning regular Neural Networks still apply. ConvNet architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties into the architecture. These then make the forward function more efficient to implement and vastly reduce the amount of parameters in the network.\n",
    "\n",
    "\n",
    "MNIST is a simple computer vision dataset. It consists of images of handwritten digits. It also includes labels for each image, telling us which digit it is. \n",
    "\n",
    "\n",
    "In this project we will develop a deep learning model on the MNIST handwritten number recognition in Python using the Keras deep learning library running Theano in the backend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Load Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Keras deep learning library provides a method to load the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT8AAAD8CAYAAAABraMFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF1tJREFUeJzt3XtsVVX2B/DvEsUXASmaWgFBTcHUCb4RHcQ6gEHUgG8J\nSInEmggGDRrQQaPxVUVJfOCDKG8CDkEENUaZWiBEbAAfMzwsRRMQrCCi8lIZdP3+6GF79vn1trf3\nnnvOuXd/P0nTte++9541dLnmvI+oKoiIXHNU3AkQEcWBzY+InMTmR0ROYvMjIiex+RGRk9j8iMhJ\nbH5E5KSsmp+IDBKROhHZIiITw0qKKG6s7cInmZ7kLCJtAGwGMBDAdgBrAAxT1Y3hpUcUPda2G47O\n4rO9AWxR1W8AQEQWABgCIGWBiAgvJ0mO3ap6StxJJBRrO4+pqqTzvmw2ezsD+NY33u69Rvlha9wJ\nJBhr2wHZrPmlRUQqAVTmejlEUWNt57dsmt8OAF194y7eaxZVnQZgGsBNA8obrG0HZLPZuwZAqYic\nISJtAdwGYGk4aRHFirXtgIzX/FT1sIiMBfAhgDYApqvqhtAyI4oJa9sNGZ/qktHCuGmQJOtU9aK4\nkygUrO3kiOJoLxFR3mLzIyInsfkRkZPY/IjISWx+ROQkNj8ichKbHxE5KefX9hJRfrrwwgut8dix\nY008cuRIa2727Nkmfumll6y5zz77LAfZZY9rfkTkJDY/InISmx8ROYnX9jahTZs21rhDhw5pf9a/\nX+SEE06w5nr27GniMWPGWHPPPfeciYcNG2bN/fbbbyauqqqy5h577LG0cwvgtb0hypfabs55551n\njT/++GNr3L59+7S+55dffrHGnTp1yi6xVuK1vUREzWDzIyInFfSpLqeffro1btu2rYkvu+wya65v\n374mPumkk6y5G2+8MZR8tm/fbuIXX3zRmrv++utNvG/fPmvuyy+/NPGKFStCyYUIAHr37m3iRYsW\nWXPB3T3+XWTBGj106JCJg5u5ffr0MXHwtBf/56LGNT8ichKbHxE5ic2PiJxUcKe6+A/XBw/Vt+aU\nlTD8+eef1viOO+4w8f79+1N+rqGhwRr/9NNPJq6rqwspO57qEqYkn+riP+XqggsusObmzp1r4i5d\nulhzIvYZI/5eEdx39+yzz5p4wYIFKb9n0qRJ1tzTTz/dbO6Z4KkuRETNYPMjIicV3Kku27ZtM/GP\nP/5ozYWx2VtbW2uNf/75Z2t85ZVXmjh4GH/OnDlZL5+otV5//XUTB68eylRw87ldu3YmDp6OVV5e\nbuJevXqFsvwwcM2PiJzE5kdETmLzIyInFdw+vz179pj4gQcesOauvfZaE3/++efWXPByM78vvvjC\nxAMHDrTmDhw4YI3POeccE48bNy6NjInCFbwD8zXXXGPi4OkrfsF9de+++6419t956LvvvrPm/P89\n+U/NAoB//OMfaS0/alzzIyIntdj8RGS6iOwSkfW+14pEZJmI1Hu/O+Y2TaLwsbbd1uIVHiLSD8B+\nALNV9W/ea88C2KOqVSIyEUBHVZ3Q4sJiPgvefzPG4F0p/KcDjB492pobMWKEiefPn5+j7CLn/BUe\nhVTbzV3Z1NxNSD/44AMTB0+DueKKK6yx/zSVN954w5r74YcfUi7jjz/+MPHBgwdTLiOsBx2FdoWH\nqq4EsCfw8hAAs7x4FoChrcqOKAFY227L9IBHsaoeuQD1ewDFqd4oIpUAKjNcDlHUWNuOyPpor6pq\nc6v8qjoNwDQg/k0DotZgbRe2TJvfThEpUdUGESkBsCvMpHJl7969KeeCD13xu/POO0381ltvWXPB\nO7dQ3suL2u7Ro4c19p/WFbyMc/fu3SYO3jFo1qxZJg7eaej9999vdpyJ448/3hqPHz/exMOHD8/6\n+1sj01NdlgKo8OIKAEvCSYcodqxtR6Rzqst8AKsB9BSR7SIyGkAVgIEiUg9ggDcmyiusbbcV3M1M\nM3XiiSeaOHhmu/9w/NVXX23NffTRR7lNLHecP9UlTFHU9rHHHmvihQsXWnODBw82cXDz9dZbbzXx\n2rVrrTn/Zqj/AVth8p/qEuw3q1evNvHll18eyvJ4M1Miomaw+RGRk9j8iMhJ3OfXhLPOOssa+y+7\nCd65uaamxhr796lMnTrVmovy3zoN3OcXoihq2//w71WrVqV8X//+/a1x3A+65z4/IqIEYfMjIicV\n3M1Mw/D1119b41GjRpl4xowZ1tztt9+ecuw/fQYAZs+ebeLgmfZELZkyZYqJgzcF9W/axr2ZG3TU\nUX+tYyXpiiiu+RGRk9j8iMhJbH5E5CTu80vD4sWLTVxfX2/N+ffDAPZpBk899ZQ1161bNxM/+eST\n1tyOHTuyzpMKi/+BW4B9t+bgKSNLly6NJKdM+PfzBfP2PxwsalzzIyInsfkRkZPY/IjISdzn10rr\n16+3xrfccos1vu6660wcPCfwrrvuMnFpaak1F3wYOlHwrsdt27Y18a5d9g2mg3cYj5r/dluPPvpo\nyvcFnyz34IMP5iqlFnHNj4icxOZHRE7iZm+Wgnd5mTNnjomDD3Y++ui//rn79etnzZWXl5t4+fLl\n4SVIBen333+3xlFfLunfzAWASZMmmdj/MCXAvkP0888/b80F7zodJa75EZGT2PyIyElsfkTkJO7z\na6VevXpZ45tuuskaX3zxxSb27+ML2rhxozVeuXJlCNmRK+K4nM1/eV1wv57/CXFLltiPOr7xxhtz\nm1iGuOZHRE5i8yMiJ3Gztwk9e/a0xmPHjjXxDTfcYM2deuqpaX+v/0EuwVMTknSHW0qG4N2a/eOh\nQ4dac+PGjQt9+ffdd581fvjhh03coUMHa27evHkmHjlyZOi55ALX/IjISS02PxHpKiI1IrJRRDaI\nyDjv9SIRWSYi9d7vjrlPlyg8rG23pbPmdxjAeFUtA9AHwBgRKQMwEUC1qpYCqPbGRPmEte2wFvf5\nqWoDgAYv3icimwB0BjAEQLn3tlkAlgOYkJMscyC4r27YsGEm9u/jA4Du3btntAz/A8wB++7NSb7z\nriuSXtvBux77x8H6ffHFF008ffp0a+7HH380sf/B54D9tMFzzz3XmuvSpYs13rZtm4k//PBDa+6V\nV175//8DEq5V+/xEpDuA8wHUAij2igcAvgdQHGpmRBFibbsn7aO9ItIOwCIA96rqXv+RJ1VVEdEU\nn6sEUJltokS5wtp2kwRXrZt8k8gxAN4D8KGqTvFeqwNQrqoNIlICYLmq9mzhe1peWIiKi+3/wy4r\nKzPxyy+/bM2dffbZGS2jtrbWGk+ePNnEwTPdE3Y6yzpVvSjuJOKW5Nq++eabrfH8+fPT+tzOnTut\n8d69e00cvIluc1avXm2Na2pqTPzII4+k/T1RU1Vp+V3pHe0VAG8C2HSkODxLAVR4cQWAJcHPEiUZ\na9tt6Wz2/h3A7QD+KyJHnjP3EIAqAP8SkdEAtgK4JcXniZKKte2wdI72rgKQajWyf4rXiRKPte22\ntPb5hbawHOwXKSoqssavv/66if13oQCAM888M6NlfPLJJyYO3ok2eMj/119/zWgZMeA+vxDloraD\np5osXLjQxP67BzWRizVu7r9x/2kwCxYssOZycclcFELb50dEVIjY/IjISXmx2XvJJZdYY/+NFHv3\n7m3Nde7cOZNF4ODBgyb2ny0PAE899ZSJDxw4kNH3JxA3e0MUxWlcJSUlJvY/AxqwHyDU3GbvCy+8\nYM29+uqrJt6yZUsoecaNm71ERM1g8yMiJ7H5EZGT8mKfX1VVlTUOPjwlleBDgt577z0THz582Jrz\nn8ISfBB5geI+vxBFfekmpcZ9fkREzWDzIyIn5cVmL+UEN3tDxNpODm72EhE1g82PiJzE5kdETmLz\nIyInsfkRkZPY/IjISWx+ROQkNj8ichKbHxE5ic2PiJyUzqMrw7QbjY8CPNmLk8DVXLpFtBxX7AZw\nAMmpJcDN2k67riO9ttcsVGRtUq4rZS4UlqT9/ZKUT5JyOYKbvUTkJDY/InJSXM1vWkzLbQpzobAk\n7e+XpHySlAuAmPb5ERHFjZu9ROQkNj8iclKkzU9EBolInYhsEZGJUS7bW/50EdklIut9rxWJyDIR\nqfd+d4wol64iUiMiG0Vkg4iMizMfyk6ctc26zkxkzU9E2gCYCuBqAGUAholIWVTL98wEMCjw2kQA\n1apaCqDaG0fhMIDxqloGoA+AMd6/R1z5UIYSUNszwbputSjX/HoD2KKq36jqIQALAAyJcPlQ1ZUA\n9gReHgJglhfPAjA0olwaVPUzL94HYBOAznHlQ1mJtbZZ15mJsvl1BvCtb7zdey1uxara4MXfAyiO\nOgER6Q7gfAC1SciHWi2JtR17HSW9rnnAw0cbz/uJ9NwfEWkHYBGAe1V1b9z5UOFhXTctyua3A0BX\n37iL91rcdopICQB4v3dFtWAROQaNBTJPVd+OOx/KWBJrm3Xdgiib3xoApSJyhoi0BXAbgKURLj+V\npQAqvLgCwJIoFioiAuBNAJtUdUrc+VBWkljbrOuWqGpkPwAGA9gM4GsA/4xy2d7y5wNoAPA/NO6X\nGQ2gExqPPtUD+DeAoohy6YvGVf//APjC+xkcVz78yfrvGVtts64z++HlbUTkJB7wICInZdX84r5i\ngyhXWNuFL+PNXu+s9s0ABqJxP8MaAMNUdWN46RFFj7Xthmye4WHOagcAETlyVnvKAhER7mBMjt2q\nekrcSSQUazuPqaqk875sNnuTeFY7pW9r3AkkGGvbATl/epuIVAKozPVyiKLG2s5v2TS/tM5qV9Vp\n8G5hzU0DyhOsbQdks9mbxLPaicLA2nZAxmt+qnpYRMYC+BBAGwDTVXVDaJkRxYS17YZIr/DgpkGi\nrNOEPUQ6n7G2kyOKo71ERHmLzY+InMTmR0ROYvMjIiex+RGRk9j8iMhJbH5E5CQ2PyJyEpsfETmJ\nzY+InMTmR0ROyvn9/Cg9/fv3N/G8efOsuSuuuMLEdXV1keVElK5JkyaZ+LHHHrPmjjrqr3Ws8vJy\na27FihU5zas5XPMjIiex+RGRk/Jis7dfv37WuFOnTiZevHhx1OnkxMUXX2ziNWvWxJgJUctGjRpl\njSdMmGDiP//8M+XnoryFXku45kdETmLzIyInsfkRkZPyYp9f8PB4aWmpifN1n5//8D8AnHHGGSbu\n1q2bNSeS1l25iSITrNHjjjsupkwyxzU/InISmx8ROSkvNntHjhxpjVevXh1TJuEpKSmxxnfeeaeJ\n586da8199dVXkeRE1JwBAwaY+J577kn5vmC9XnvttSbeuXNn+IlliGt+ROQkNj8ichKbHxE5KS/2\n+QVPCykEb7zxRsq5+vr6CDMhalrfvn2t8YwZM0zcoUOHlJ+bPHmyNd66dWu4iYWkxa4iItNFZJeI\nrPe9ViQiy0Sk3vvdMbdpEoWPte22dFapZgIYFHhtIoBqVS0FUO2NifLNTLC2ndXiZq+qrhSR7oGX\nhwAo9+JZAJYDmIAQ9erVy8TFxcVhfnUiNLfZsGzZsggzcVdctZ0vKioqrPFpp52W8r3Lly838ezZ\ns3OVUqgy3ZlWrKoNXvw9gMLrTuQq1rYjsj7goaoqIilv0iUilQAqs10OUdRY24Ut0zW/nSJSAgDe\n712p3qiq01T1IlW9KMNlEUWJte2ITNf8lgKoAFDl/V4SWkaewYMHm/j4448P++tj4d936b+LS9CO\nHTuiSIealvPaTqqTTz7ZGt9xxx3W2H+H5p9//tmae+KJJ3KXWI6kc6rLfACrAfQUke0iMhqNhTFQ\nROoBDPDGRHmFte22dI72Dksx1T/F60R5gbXttsRe4dGzZ8+Ucxs2bIgwk/A899xzJg6evrN582YT\n79u3L7KcyG3du3c38aJFi9L+3EsvvWSNa2pqwkopMoV33RgRURrY/IjISWx+ROSkxO7za06SHurd\nvn17azxo0F+Xio4YMcKau+qqq1J+z+OPP27i4GkERLnir1f/JaVNqa6uNvELL7yQs5yiwjU/InIS\nmx8ROSkvN3uLiooy+ty5555r4uCzcP0PZ+nSpYs117ZtWxMPHz7cmgveaPXXX381cW1trTX3+++/\nm/joo+1/+nXr1jWbO1EYhg4dao2rqlKfw71q1Spr7L/Lyy+//BJuYjHgmh8ROYnNj4icxOZHRE5K\n7D4//74zVfuWaq+99pqJH3roobS/038oP7jP7/DhwyY+ePCgNbdx40YTT58+3Zpbu3atNV6xYoWJ\ngw9o3r59u4mDd6rhg8kpVzK9hO2bb76xxkl64HgYuOZHRE5i8yMiJ7H5EZGTErvP7+677zZx8KHH\nl112WUbfuW3bNhO/88471tymTZtM/Omnn2b0/UGVlfbjHU455RQTB/enEOXKhAl/PXzOfzfmljR3\nDmAh4JofETmJzY+InJTYzV6/Z555Ju4UMtK/f+q7obfmlAOi1jjvvPOscXN3E/JbssR+VlNdXV1o\nOSUR1/yIyElsfkTkJDY/InJSXuzzK0SLFy+OOwUqUB999JE17tixY8r3+k/rGjVqVK5SSiSu+RGR\nk9j8iMhJ3OwlKjCdOnWyxs1d1fHKK6+YeP/+/TnLKYm45kdETmqx+YlIVxGpEZGNIrJBRMZ5rxeJ\nyDIRqfd+p96rSpRArG23pbPmdxjAeFUtA9AHwBgRKQMwEUC1qpYCqPbGRPmEte2wFvf5qWoDgAYv\n3icimwB0BjAEQLn3tlkAlgOY0MRXkMd/9+gePXpYc2HdSYbSV0i1PWPGDBMHnyjYnE8++SQX6eSF\nVh3wEJHuAM4HUAug2CseAPgeQHGKz1QCqGxqjigpWNvuSfv/IkSkHYBFAO5V1b3+OW18yIY29TlV\nnaaqF6nqRVllSpQjrG03pbXmJyLHoLE45qnq297LO0WkRFUbRKQEwK5cJVko/A9ias2mCeVOvtZ2\n8M4tAwYMMHHw1JZDhw6ZeOrUqdZcoT2UqDXSOdorAN4EsElVp/imlgI48gj3CgBLgp8lSjLWttvS\nWfP7O4DbAfxXRL7wXnsIQBWAf4nIaABbAdySmxSJcoa17bB0jvauAiApplPfrZMo4VjbbuPlbTG5\n9NJLrfHMmTPjSYTy0kknnWSNTz311JTv3bFjh4nvv//+nOWUb7jXnYicxOZHRE7iZm+E/Fd4EFG8\nuOZHRE5i8yMiJ7H5EZGTuM8vhz744ANrfPPNN8eUCRWar776yhr7787St2/fqNPJS1zzIyInsfkR\nkZPEf6eRnC9MJLqFUUvW8VZM4WFtJ4eqpnVOGdf8iMhJbH5E5CQ2PyJyEpsfETmJzY+InMTmR0RO\nYvMjIiex+RGRk9j8iMhJbH5E5KSo7+qyG42PAjzZi5PA1Vy6RbQcV+wGcADJqSXAzdpOu64jvbbX\nLFRkbVKuK2UuFJak/f2SlE+ScjmCm71E5CQ2PyJyUlzNb1pMy20Kc6GwJO3vl6R8kpQLgJj2+RER\nxY2bvUTkpEibn4gMEpE6EdkiIhOjXLa3/OkisktE1vteKxKRZSJS7/3uGFEuXUWkRkQ2isgGERkX\nZz6UnThrm3Wdmcian4i0ATAVwNUAygAME5GyqJbvmQlgUOC1iQCqVbUUQLU3jsJhAONVtQxAHwBj\nvH+PuPKhDCWgtmeCdd1qUa759QawRVW/UdVDABYAGBLh8qGqKwHsCbw8BMAsL54FYGhEuTSo6mde\nvA/AJgCd48qHshJrbbOuMxNl8+sM4FvfeLv3WtyKVbXBi78HUBx1AiLSHcD5AGqTkA+1WhJrO/Y6\nSnpd84CHjzYe+o708LeItAOwCMC9qro37nyo8LCumxZl89sBoKtv3MV7LW47RaQEALzfu6JasIgc\ng8YCmaeqb8edD2UsibXNum5BlM1vDYBSETlDRNoCuA3A0giXn8pSABVeXAFgSRQLFREB8CaATao6\nJe58KCtJrG3WdUtUNbIfAIMBbAbwNYB/Rrlsb/nzATQA+B8a98uMBtAJjUef6gH8G0BRRLn0ReOq\n/38AfOH9DI4rH/5k/feMrbZZ15n98AoPInISD3gQkZPY/IjISWx+ROQkNj8ichKbHxE5ic2PiJzE\n5kdETmLzIyIn/R+KxB3xQTzcVwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xcbe0320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot ad hoc mnist instances\n",
    "from keras.datasets import mnist\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# load (downloaded if needed) the data, shuffled and split between train and test sets\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "\n",
    "# plot 4 images as gray scale\n",
    "plt.subplot(221)\n",
    "plt.imshow(X_train[0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(222)\n",
    "plt.imshow(X_train[1], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(223)\n",
    "plt.imshow(X_train[2], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(224)\n",
    "plt.imshow(X_train[3], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "# show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Import Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Importing the Keras classes and other functions we will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers.convolutional import Conv2D, MaxPooling2D\n",
    "from keras.utils import np_utils\n",
    "from keras import backend as K\n",
    "K.set_image_dim_ordering('th')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Initialize Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Intializing variables that we need in our program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variables intialized\n"
     ]
    }
   ],
   "source": [
    "# set batch and epoch sizes\n",
    "batch_size = 200\n",
    "epochs = 10\n",
    "\n",
    "# fix random seed for reproducibility\n",
    "seed = 93\n",
    "numpy.random.seed(seed)\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "print(\"Variables intialized\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Reshape data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Reduce the images down into a vector of pixels in the size of 28×28 and will be 784 pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# reshape to be [samples][pixels][width][height]\n",
    "X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols).astype('float32')\n",
    "X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols).astype('float32')\n",
    "input_shape = (1, img_rows, img_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The pixel values are scale between 0 and 255. We will normalize the pixel values to the range 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('x_train shape:', (60000L, 1L, 28L, 28L))\n",
      "(60000L, 'train samples')\n",
      "(10000L, 'test samples')\n"
     ]
    }
   ],
   "source": [
    "# normalize inputs from 0-255 to 0-1\n",
    "X_train /= 255\n",
    "X_test  /= 255\n",
    "\n",
    "print('x_train shape:', X_train.shape)\n",
    "print(X_train.shape[0], 'train samples')\n",
    "print(X_test.shape[0], 'test samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The output is an integer from 0 to 9. Since it is a multi-class classification problem. As such, it is good practice to use a one hot encoding of the class values, transforming the vector of class integers into a binary matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# one hot encode outputs\n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)\n",
    "num_classes = y_test.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Define Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will create a Sequential model and add layers to complete our network topology."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# create model\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, (5, 5), input_shape=input_shape, activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(16, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(50, activation='relu'))\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Compile Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will use logarithmic loss as \"binary_crossentropy\" and efficient gradient descent algorithm “Adam” for compiling our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Compile model\n",
    "model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Fit Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The training process will run for a fixed number of iterations through the dataset called epochs. We can also set the number of instances that are evaluated before a weight update in the network is performed, called the batch size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "50s - loss: 0.3879 - acc: 0.8774 - val_loss: 0.0729 - val_acc: 0.9763\n",
      "Epoch 2/10\n",
      "49s - loss: 0.0957 - acc: 0.9710 - val_loss: 0.0525 - val_acc: 0.9828\n",
      "Epoch 3/10\n",
      "49s - loss: 0.0688 - acc: 0.9786 - val_loss: 0.0348 - val_acc: 0.9892\n",
      "Epoch 4/10\n",
      "49s - loss: 0.0559 - acc: 0.9822 - val_loss: 0.0345 - val_acc: 0.9886\n",
      "Epoch 5/10\n",
      "52s - loss: 0.0464 - acc: 0.9852 - val_loss: 0.0298 - val_acc: 0.9900\n",
      "Epoch 6/10\n",
      "149s - loss: 0.0399 - acc: 0.9874 - val_loss: 0.0252 - val_acc: 0.9914\n",
      "Epoch 7/10\n",
      "195s - loss: 0.0362 - acc: 0.9887 - val_loss: 0.0268 - val_acc: 0.9911\n",
      "Epoch 8/10\n",
      "88s - loss: 0.0323 - acc: 0.9897 - val_loss: 0.0236 - val_acc: 0.9927\n",
      "Epoch 9/10\n",
      "51s - loss: 0.0298 - acc: 0.9904 - val_loss: 0.0265 - val_acc: 0.9916\n",
      "Epoch 10/10\n",
      "64s - loss: 0.0280 - acc: 0.9906 - val_loss: 0.0220 - val_acc: 0.9931\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x10474048>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the model\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), batch_size=batch_size, epochs=epochs, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Evaluate Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can evaluate the model on training dataset. This will generate a prediction for each input and output pair and collect scores, including the average loss and any metrics you have configured, such as accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.022037403795970023, 0.99309999999999998]\n"
     ]
    }
   ],
   "source": [
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss:     0.02%\n",
      "Test Accuracy: 99.31%\n",
      "Test Error:    0.69%\n"
     ]
    }
   ],
   "source": [
    "print('Test Loss:     %.2f%%' % (scores[0]))\n",
    "print('Test Accuracy: {:.2%}'.format(scores[1]))\n",
    "print('Test Error:    {:.2%}'.format(1-scores[1]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}