Tennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaboration and Competition\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the third project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from p3_mad4pg import MAD4PG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\".\\Tennis_Windows_x86_64\\Tennis.exe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of +0.1.  If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01.  Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of 8 variables corresponding to the position and velocity of the ball and racket. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.          0.\n",
      "  0.          0.         -6.65278625 -1.5        -0.          0.\n",
      "  6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agents and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agents' performance, if they select actions at random with each time step.  A window should pop up that allows you to observe the agents.\n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agents are able to use their experiences to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (max over agents) from episode 1: 0.0\n",
      "Score (max over agents) from episode 2: 0.0\n",
      "Score (max over agents) from episode 3: 0.0\n",
      "Score (max over agents) from episode 4: 0.10000000149011612\n",
      "Score (max over agents) from episode 5: 0.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(1, 6):                                      # play game for 5 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Score (max over agents) from episode {}: {}'.format(i, np.max(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(check_pt_path,n_episodes=10000,isSolve=True):\n",
    "    \"\"\"Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    env=None\n",
    "    env = UnityEnvironment(file_name='.\\Tennis_Windows_x86_64\\Tennis.exe')\n",
    "    brain_name = env.brain_names[0]\n",
    "    brain = env.brains[brain_name]\n",
    "    \n",
    "    avgscores=[]\n",
    "    scores=[]\n",
    "    scores1 = []\n",
    "    scores2 = []# list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states= env_info.vector_observations\n",
    "        score1 = 0\n",
    "        score2 = 0\n",
    "        while True:\n",
    "            actions = multiagent.acts(states)\n",
    "            env_info=env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = env_info.local_done\n",
    "            multiagent.step(states,actions, rewards, next_states, dones)\n",
    "            states = next_states\n",
    "            score1 += rewards[0]\n",
    "            score2 += rewards[1]\n",
    "            if dones[0]:\n",
    "                break\n",
    "\n",
    "        score=max(score1,score2)        \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        scores1.append(score1)\n",
    "        scores2.append(score2)\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "            avgscores.append(np.mean(scores_window))\n",
    "        if np.mean(scores_window)>0.5 and isSolve:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(multiagent.mad4pg_agent[0].actor_local.state_dict(),check_pt_path+'_actor1_solved.pth')\n",
    "            torch.save(multiagent.mad4pg_agent[1].actor_local.state_dict(),check_pt_path+'_actor2_solved.pth')\n",
    "            isSolve=False\n",
    "    env.close()\n",
    "    torch.save(multiagent.mad4pg_agent[0].actor_local.state_dict(),check_pt_path+'_actor1_last.pth')\n",
    "    torch.save(multiagent.mad4pg_agent[1].actor_local.state_dict(),check_pt_path+'_actor2_last.pth')\n",
    "    return scores,scores1,scores2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: 0.01\n",
      "Episode 200\tAverage Score: 0.01\n",
      "Episode 300\tAverage Score: 0.02\n",
      "Episode 400\tAverage Score: 0.08\n",
      "Episode 500\tAverage Score: 0.00\n",
      "Episode 600\tAverage Score: 0.00\n",
      "Episode 700\tAverage Score: 0.00\n",
      "Episode 800\tAverage Score: 0.00\n",
      "Episode 900\tAverage Score: 0.00\n",
      "Episode 1000\tAverage Score: 0.02\n",
      "Episode 1100\tAverage Score: 0.04\n",
      "Episode 1200\tAverage Score: 0.01\n",
      "Episode 1300\tAverage Score: 0.00\n",
      "Episode 1400\tAverage Score: 0.01\n",
      "Episode 1500\tAverage Score: 0.01\n",
      "Episode 1600\tAverage Score: 0.01\n",
      "Episode 1700\tAverage Score: 0.02\n",
      "Episode 1800\tAverage Score: 0.04\n",
      "Episode 1900\tAverage Score: 0.04\n",
      "Episode 2000\tAverage Score: 0.05\n",
      "Episode 2100\tAverage Score: 0.06\n",
      "Episode 2200\tAverage Score: 0.10\n",
      "Episode 2300\tAverage Score: 0.09\n",
      "Episode 2400\tAverage Score: 0.09\n",
      "Episode 2500\tAverage Score: 0.09\n",
      "Episode 2600\tAverage Score: 0.09\n",
      "Episode 2700\tAverage Score: 0.12\n",
      "Episode 2800\tAverage Score: 0.10\n",
      "Episode 2900\tAverage Score: 0.07\n",
      "Episode 3000\tAverage Score: 0.04\n",
      "Episode 3100\tAverage Score: 0.03\n",
      "Episode 3200\tAverage Score: 0.06\n",
      "Episode 3300\tAverage Score: 0.08\n",
      "Episode 3400\tAverage Score: 0.06\n",
      "Episode 3500\tAverage Score: 0.06\n",
      "Episode 3600\tAverage Score: 0.06\n",
      "Episode 3700\tAverage Score: 0.07\n",
      "Episode 3800\tAverage Score: 0.09\n",
      "Episode 3900\tAverage Score: 0.09\n",
      "Episode 4000\tAverage Score: 0.11\n",
      "Episode 4100\tAverage Score: 0.11\n",
      "Episode 4200\tAverage Score: 0.11\n",
      "Episode 4300\tAverage Score: 0.11\n",
      "Episode 4400\tAverage Score: 0.14\n",
      "Episode 4500\tAverage Score: 0.13\n",
      "Episode 4600\tAverage Score: 0.14\n",
      "Episode 4700\tAverage Score: 0.13\n",
      "Episode 4800\tAverage Score: 0.18\n",
      "Episode 4900\tAverage Score: 0.16\n",
      "Episode 5000\tAverage Score: 0.18\n",
      "Episode 5100\tAverage Score: 0.15\n",
      "Episode 5200\tAverage Score: 0.14\n",
      "Episode 5300\tAverage Score: 0.16\n",
      "Episode 5400\tAverage Score: 0.17\n",
      "Episode 5500\tAverage Score: 0.17\n",
      "Episode 5600\tAverage Score: 0.19\n",
      "Episode 5700\tAverage Score: 0.17\n",
      "Episode 5800\tAverage Score: 0.16\n",
      "Episode 5900\tAverage Score: 0.16\n",
      "Episode 6000\tAverage Score: 0.19\n",
      "Episode 6100\tAverage Score: 0.14\n",
      "Episode 6200\tAverage Score: 0.18\n",
      "Episode 6300\tAverage Score: 0.27\n",
      "Episode 6400\tAverage Score: 0.20\n",
      "Episode 6500\tAverage Score: 0.18\n",
      "Episode 6600\tAverage Score: 0.17\n",
      "Episode 6700\tAverage Score: 0.19\n",
      "Episode 6800\tAverage Score: 0.16\n",
      "Episode 6900\tAverage Score: 0.12\n",
      "Episode 7000\tAverage Score: 0.18\n",
      "Episode 7100\tAverage Score: 0.23\n",
      "Episode 7200\tAverage Score: 0.20\n",
      "Episode 7300\tAverage Score: 0.23\n",
      "Episode 7400\tAverage Score: 0.22\n",
      "Episode 7500\tAverage Score: 0.17\n",
      "Episode 7600\tAverage Score: 0.26\n",
      "Episode 7700\tAverage Score: 0.30\n",
      "Episode 7800\tAverage Score: 0.33\n",
      "Episode 7900\tAverage Score: 0.26\n",
      "Episode 8000\tAverage Score: 0.30\n",
      "Episode 8100\tAverage Score: 0.29\n",
      "Episode 8200\tAverage Score: 0.40\n",
      "Episode 8235\tAverage Score: 0.52\n",
      "Environment solved in 8135 episodes!\tAverage Score: 0.52\n",
      "Episode 8300\tAverage Score: 0.58\n",
      "Episode 8400\tAverage Score: 0.44\n",
      "Episode 8500\tAverage Score: 0.73\n",
      "Episode 8600\tAverage Score: 0.56\n",
      "Episode 8700\tAverage Score: 0.74\n",
      "Episode 8800\tAverage Score: 1.00\n",
      "Episode 8900\tAverage Score: 1.51\n",
      "Episode 9000\tAverage Score: 1.73\n",
      "Episode 9100\tAverage Score: 1.44\n",
      "Episode 9200\tAverage Score: 1.53\n",
      "Episode 9300\tAverage Score: 1.31\n",
      "Episode 9400\tAverage Score: 1.17\n",
      "Episode 9500\tAverage Score: 0.70\n",
      "Episode 9600\tAverage Score: 0.64\n",
      "Episode 9700\tAverage Score: 0.46\n",
      "Episode 9800\tAverage Score: 0.22\n",
      "Episode 9900\tAverage Score: 0.21\n",
      "Episode 10000\tAverage Score: 0.49\n",
      "Episode 10100\tAverage Score: 0.43\n",
      "Episode 10200\tAverage Score: 0.36\n",
      "Episode 10300\tAverage Score: 0.50\n",
      "Episode 10400\tAverage Score: 0.81\n",
      "Episode 10500\tAverage Score: 1.20\n",
      "Episode 10600\tAverage Score: 1.20\n",
      "Episode 10700\tAverage Score: 1.19\n",
      "Episode 10800\tAverage Score: 1.22\n",
      "Episode 10900\tAverage Score: 1.16\n",
      "Episode 11000\tAverage Score: 1.25\n",
      "Episode 11100\tAverage Score: 1.55\n",
      "Episode 11200\tAverage Score: 1.33\n",
      "Episode 11300\tAverage Score: 0.98\n",
      "Episode 11400\tAverage Score: 1.29\n",
      "Episode 11500\tAverage Score: 0.96\n",
      "Episode 11600\tAverage Score: 0.75\n",
      "Episode 11700\tAverage Score: 0.86\n",
      "Episode 11800\tAverage Score: 0.42\n",
      "Episode 11900\tAverage Score: 0.51\n",
      "Episode 12000\tAverage Score: 0.52\n",
      "Episode 12100\tAverage Score: 0.78\n",
      "Episode 12200\tAverage Score: 0.98\n",
      "Episode 12300\tAverage Score: 0.87\n",
      "Episode 12400\tAverage Score: 0.57\n",
      "Episode 12500\tAverage Score: 0.90\n",
      "Episode 12600\tAverage Score: 1.07\n",
      "Episode 12700\tAverage Score: 0.95\n",
      "Episode 12800\tAverage Score: 1.06\n",
      "Episode 12900\tAverage Score: 1.44\n",
      "Episode 13000\tAverage Score: 1.57\n",
      "Episode 13100\tAverage Score: 1.48\n",
      "Episode 13200\tAverage Score: 1.25\n",
      "Episode 13300\tAverage Score: 1.18\n",
      "Episode 13400\tAverage Score: 1.20\n",
      "Episode 13500\tAverage Score: 1.00\n",
      "Episode 13600\tAverage Score: 0.84\n",
      "Episode 13700\tAverage Score: 1.22\n",
      "Episode 13800\tAverage Score: 0.92\n",
      "Episode 13900\tAverage Score: 0.95\n",
      "Episode 14000\tAverage Score: 1.13\n",
      "Episode 14100\tAverage Score: 1.21\n",
      "Episode 14200\tAverage Score: 1.20\n",
      "Episode 14300\tAverage Score: 1.43\n",
      "Episode 14400\tAverage Score: 1.44\n",
      "Episode 14500\tAverage Score: 1.15\n",
      "Episode 14600\tAverage Score: 1.38\n",
      "Episode 14700\tAverage Score: 1.49\n",
      "Episode 14800\tAverage Score: 1.28\n",
      "Episode 14900\tAverage Score: 1.03\n",
      "Episode 15000\tAverage Score: 1.07\n",
      "Episode 15100\tAverage Score: 1.11\n",
      "Episode 15200\tAverage Score: 1.10\n",
      "Episode 15300\tAverage Score: 1.00\n",
      "Episode 15400\tAverage Score: 1.42\n",
      "Episode 15500\tAverage Score: 0.97\n",
      "Episode 15600\tAverage Score: 1.27\n",
      "Episode 15700\tAverage Score: 1.28\n",
      "Episode 15800\tAverage Score: 1.41\n",
      "Episode 15900\tAverage Score: 1.54\n",
      "Episode 16000\tAverage Score: 1.37\n",
      "Episode 16100\tAverage Score: 1.15\n",
      "Episode 16200\tAverage Score: 1.20\n",
      "Episode 16300\tAverage Score: 1.07\n",
      "Episode 16400\tAverage Score: 1.17\n",
      "Episode 16500\tAverage Score: 1.37\n",
      "Episode 16600\tAverage Score: 1.06\n",
      "Episode 16700\tAverage Score: 1.03\n",
      "Episode 16800\tAverage Score: 0.94\n",
      "Episode 16900\tAverage Score: 1.10\n",
      "Episode 17000\tAverage Score: 0.98\n",
      "Episode 17100\tAverage Score: 1.10\n",
      "Episode 17200\tAverage Score: 1.00\n",
      "Episode 17300\tAverage Score: 1.65\n",
      "Episode 17400\tAverage Score: 1.60\n",
      "Episode 17500\tAverage Score: 1.42\n",
      "Episode 17600\tAverage Score: 1.30\n",
      "Episode 17700\tAverage Score: 1.21\n",
      "Episode 17800\tAverage Score: 1.43\n",
      "Episode 17900\tAverage Score: 1.12\n",
      "Episode 18000\tAverage Score: 0.92\n",
      "Episode 18100\tAverage Score: 1.05\n",
      "Episode 18200\tAverage Score: 1.02\n",
      "Episode 18300\tAverage Score: 1.36\n",
      "Episode 18400\tAverage Score: 1.09\n",
      "Episode 18500\tAverage Score: 1.15\n",
      "Episode 18600\tAverage Score: 1.18\n",
      "Episode 18700\tAverage Score: 1.44\n",
      "Episode 18800\tAverage Score: 1.50\n",
      "Episode 18900\tAverage Score: 1.27\n",
      "Episode 19000\tAverage Score: 1.41\n",
      "Episode 19100\tAverage Score: 1.08\n",
      "Episode 19200\tAverage Score: 1.18\n",
      "Episode 19300\tAverage Score: 1.16\n",
      "Episode 19400\tAverage Score: 1.19\n",
      "Episode 19500\tAverage Score: 1.23\n",
      "Episode 19600\tAverage Score: 1.11\n",
      "Episode 19700\tAverage Score: 1.05\n",
      "Episode 19800\tAverage Score: 0.76\n",
      "Episode 19900\tAverage Score: 0.96\n",
      "Episode 20000\tAverage Score: 1.32\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFPWd//HXh5kBhkOQMCoiCCieieeId9bEG+ORxI3k\n53olWaJrEs2azeKxRjfJxpiNUWMUjffGoCaeu4BREQNeyIDIfcs1XMM1XANzfX5/dE3Zc/f0dHX3\nTL+fj8c8qK76VtVnqof61Pdb3/qWuTsiIiIAXTIdgIiIZA8lBRERCSkpiIhISElBRERCSgoiIhJS\nUhARkZCSgoiIhJQUREQkpKQgIiKh/EwH0Fb9+/f3IUOGZDoMEZEOZcaMGZvcvai1ch0uKQwZMoSS\nkpJMhyEi0qGY2cpEyqn5SEREQkoKIiISUlIQEZGQkoKIiISUFEREJKSkICIiISUFEREJdbjnFERS\nbcbKrXzzkQ+aXX7B0Qfw1SP241snDWrzthdv2MF5v5sCwAmD+3LxsQdSXlHF/W8v4fnRp7By8y4m\nzl1PQV4X3pq/IVxv4k1ncsuLnzJ/3XZOHtqPey8/hlc/Wcsxg/rwlcP3a7SfPVU1jJ+9jncWbWRg\n30JuG3lkuOzapz6md/cCdu6pYvKislZjHnXSID5avpnTD+3PDWcdwtQlm5ixcit/nbGGk4bsy83n\nHEbPbvks3biTeyYuYNPOyia388OvHsrv31lab94dFx3JsrJdfLJqKwP7FtKrez4H7NOdR6cs5+Jj\nD+R/P10LwEH7FrJmawV9Cgsor6hqcvtnH7EffXt0ZfOuveypqmHe2u0MK+rFoH0LmbK4jH865WBG\nfmkAj09dzquzYtv90VcPZcH6Hbw1fwP9enZly67GsRf17kbZjr0AdM3vQmV1Lb265fPf/3gsxxzU\nh7ml5bw6q5Q5peWs3lJBj655PPe9kyndVsGrn5RyxAH78NDkz3/vQf0Kqa2Fsp17+crhRfTv1Y1j\nD+rLr99YyOZdlRy+f29GDO3HzFVbOfeo/Xl5Zin3fPNLnHZIf6YsLmO/fbpxwf1TOf/o/Xn0quJW\nv7/2so72jubi4mLXw2uSSkPGjE+o3Ip7Lops223RVBx3vjaXZz9c2WSZKGLIVX17FLBtd9NJKtVW\n3HNRo+/u/TFfZWDfwqS2Z2Yz3L3VrKLmI5FOYOP2vZkOISekKyE0p7K6NvJ9KCmIiEhISUFEREJK\nCiIiElJSEBGRkLqkStJqa50de6rp06MgZdvctruSvj261pu3Y08V3QvyKMirfw2zt7qGnXuq6dkt\nn71VtfTunk+XLgZAeUUV2yuq2H+f7uzaW03Zzr24Q0VVDfv2KMAw9lTXUFFZk3BsM1dtZd8eXSks\nyAPggD7dw5gra2rZsquS/Xt3p09hQRhHFN5dtJFD9+tFz675LN+0i97d83lj3vp6ZUq3VdCjII/V\nW3dHFodE668z1jSaV5uG3qLqkipJ+91bi3lg0hJK7jiH/r26tXt7s1Zv47I/vM/vv308Fx97YDh/\nyJjxnH3Efjxx7Un1yn/53sms2rIbM3CP9Yu/5bzDw3WituKei5i8aCPXPTW93vwfn3MYN50znF+/\nsZBH3l0WeRySO4YV9eSdW85Kal11SZXIvTE3dnW6aWdqukPOW1sOwAfLNjdaNmnhxkbzVm2JXQXX\nXddMmLMuJXG0RcmKLY3m/S24an9i6mfpDkc6ueVluyLfh5KCiIiElBSk08i6htDobiuIRCaypGBm\ng8xsspnNN7N5ZnZTE2XOMrNyM5sV/NwZVTwiItK6KHsfVQO3uPtMM+sNzDCzt9x9foNyU939axHG\nIZIRqihIRxRZTcHd17n7zGB6B7AAGBjV/qQzSbIhKOvaj0Q6nrTcUzCzIcDxwLQmFp9mZrPNbKKZ\nHZ2OeCQ7WSe5tq5159KH3mNvGgYvE0m1yB9eM7NewEvAze6+vcHimcBgd99pZiOBV4HhTWxjNDAa\nYPDgwRFHLG2VLY+6ZCKMpn73iqoaFq7fkf5gRFIg0pqCmRUQSwjPufvLDZe7+3Z33xlMTwAKzKx/\nE+Uec/didy8uKiqKMmRpA+scF/YiEifK3kcGPAEscPf7milzQFAOMxsRxNP4ySUREUmLKJuPTgeu\nAuaY2axg3m3AYAB3HwtcDtxgZtVABTDKO9q4G5I1suVPRxUo6cgiSwru/h6t/P9w94eAh6KKQTqm\nLDm3i+QkPdEsWUP3KEQyT0lBOg1VMETaT0lBRERCSgrSbrl8DyCHf3XppJQUpNPIluSUJWGIJEVJ\nQbJOtpzcRXKRkoJkjfZ2PnJdo4u0m5KCSIqpZ610ZEoKErkN2/dkZN1MqdToqNKBKSlIpMbPXsfJ\n/zWJD5ZtavO6E+bE1k1UttyLWFve8RKZSB0lBYnUjJVbAZi/tuGo6a2bGawrIumjpCDtluobvB3p\nhnG21E5EUkVJQZJmKR6sqL2b0wlapP2UFEREJKSkICIiISUFEREJRfnmNZHQL8YvwMz47hlD272t\nF6av4ukPVqYgKhFpSDUFSZuf/9/8hMrV3TBu7sbzv780hwXrGndxzZbXcYp0ZEoKkjVMA0SIZJyS\ngrRbLl+gd6RnKkQSoaQgScu263qdnkXaT0lBIpXi59tEJGJKChKpZJqWdMUvkjlKCpI9GtQq2jqM\nRrrvbai3k3RGSgoiSZq0YGOmQxBJOSUF6TTS3ROooqpG3Wil01FSEEmSGo+kM1JSEEmSu+s5Bel0\nIksKZjbIzCab2Xwzm2dmNzVRxszsQTNbamazzeyEqOKR1Iuqu2my929131ek/aIcEK8auMXdZ5pZ\nb2CGmb3l7vED4FwIDA9+TgYeCf6VHNTRWueVhKQziqym4O7r3H1mML0DWAAMbFDsUuBZj/kI6Gtm\nA6KKSTqW1Vt2ZzqEFlVW12Y6BJGUS8s9BTMbAhwPTGuwaCCwOu7zGhonDsxstJmVmFlJWVlZVGFK\nhsVfeM9es42Jc9cnvX463PrKnDTvUSR6kScFM+sFvATc7O6NxztOgLs/5u7F7l5cVFSU2gAlUsnc\ndzCDzzbtSn0wKVZTq/Yj6XwiTQpmVkAsITzn7i83UaQUGBT3+aBgnnQAalMX6Xyi7H1kwBPAAne/\nr5lirwNXB72QTgHK3X1dVDFJx6DeRyKZE2Xvo9OBq4A5ZjYrmHcbMBjA3ccCE4CRwFJgN3BdhPFI\nlutovY8APcEmnU5kScHd36OV/+ceG1HsxqhikGglcr8gvVfvOkOLtJeeaBYRkZCSgnQam3ZWctPz\nn6R1n49OWZ7W/YlETUlBOpXXZq3NdAgiHZqSgmQdDTInkjlKCpI12vqmNRFJPSUFiZTO8yIdi5KC\ntJseGhPpPJQUJGmqBYh0PlE+0SySUo9NWcaxB/VttdzSjTvTEI1I56SkINmnmeao/5qwMKHVz7nv\n7ykMRiS3qPlIsoZao0QyT0lBRERCSgoSKfVMEulYlBRERCSkpCDtpmEpRDoP9T6SpOyurGb1lopI\ntj1z1Vb69CiIZNsi0jLVFCQp33l6OuUVVa2Wa8sDbnVlV2zezVPvr0guMBFpFyUFScpHy7dkOgQR\niYCSgoiIhJQUREQkpKQgIiIhJQUREQkpKUi7peqpZQ3FLZJ5SgqSMVOXlDFkzHg279yb6VBEJKCk\nIBnz+NTPAJhdWp7hSESkjpKCiIiElBRERCQUWVIwsyfNbKOZzW1m+VlmVm5ms4KfO6OKRUREEhPl\ngHhPAw8Bz7ZQZqq7fy3CGKQDMb17TSTjIqspuPsUQAPkiIh0IAknBTM7w8yuC6aLzGxoCvZ/mpnN\nNrOJZnZ0CrYnGdCWxxSqamq5942F7NjT+girIpJ+CTUfmdnPgGLgcOApoAD4E3B6O/Y9Exjs7jvN\nbCTwKjC8mf2PBkYDDB48uB27lEx75ZNSHn53GbsrazIdiog0IdGawteBS4BdAO6+Fujdnh27+3Z3\n3xlMTwAKzKx/M2Ufc/didy8uKipqz24lw6pqagHYW12b4UhEpCmJJoVKd3eClgIz69neHZvZAWax\ngQ3MbEQQy+b2bldERJKXaO+jF83sUaCvmf0z8B3gjy2tYGbjgLOA/ma2BvgZsWYn3H0scDlwg5lV\nAxXAqCDxSI7S2EcimZdQUnD3/zazc4HtxO4r3Onub7WyzrdbWf4QsS6rIiKSJVpNCmaWB7zt7l8B\nWkwEIklR/VAka7R6T8Hda4BaM+uThngky81ZU05tbeJn8ZZahNRcJJJ9Er2nsBOYY2ZvEfRAAnD3\nH0USlWSlGSu38s1HPuDfzj+83vxU3QrSHSWRzEs0Kbwc/EgOW7utAoD567YnvE7D87xO/CLZLdEb\nzc+YWVfgsGDWInfXI6mStKaajtScJJJ5iT7RfBbwDLCCWDPxIDO7JhjfSEREOolEm49+C5zn7osA\nzOwwYBxwYlSBiYhI+iX6RHNBXUIAcPfFBA+iSQ5qcF/A1O4j0mkkWlMoMbPHiQ2CB3AlUBJNSJKt\ndO4X6fwSTQo3ADcCdV1QpwIPRxKR5IQ/T1sVTlfXOkPGjOe4QX0zGJGIQOJJIR94wN3vg/Ap526R\nRSUdSnufU9hdWQ3ArNXbUhGOiLRDovcUJgGFcZ8LgbdTH450BK5xKUQ6rUSTQve6dx8ABNM9oglJ\nslUy71BuuIbSiUh2SzQp7DKzE+o+mFkxseGuRUSkE0n0nsLNwF/MbG3weQBwRTQhSbZSs5FI59di\nTcHMTjKzA9x9OnAE8AJQBbwBfJaG+CQLJdOMJCIdQ2vNR48ClcH0qcBtwB+ArcBjEcYlKfbG3HVM\nnLMuJdtqT43h5/83v9G8Ret3tCccEUmh1pJCnrtvCaavAB5z95fc/T+AQ6MNTVLp+j/N5IbnZrZr\nG63VEG59eTajHvuw3ryG6aOyurbReg+/u6xdcYlI6rR2TyHPzPLdvRo4GxjdhnUlR9Sd+Md9vDqj\ncYhI+7V2Yh8H/N3MNhHrbTQVwMwOBcojjk1ERNKsxaTg7r80s0nEehu96Z8/utoF+GHUwUl20oty\nRDqvVpuA3P2jJuYtjiYcyWYaEE+k80v04TUREckBSgrSZsvKdtb7XFFZw+w1TQ9mp8qFSMeiHkTS\nZos31E8KP35hFht37A0//2HyUv7xxIPYb5/u6Q5NRNpJNQVpt/iEAPCbvy3iB+M+yVA0ItIeSgoS\nibp3JIhIx6KkICIiociSgpk9aWYbzWxuM8vNzB40s6VmNjt+aG4REcmMKGsKTwMXtLD8QmB48DMa\neCTCWEREJAGRJQV3nwJsaaHIpcCzHvMR0NfMBkQVj2SGHn4W6VgyeU9hIBA/gtqaYJ5kKT1zINL5\ndYgbzWY22sxKzKykrKws0+FIAvQiHpGOKZNJoRQYFPf5oGBeI+7+mLsXu3txUVFRWoITEclFmUwK\nrwNXB72QTgHK3T01rwaTrKH6gkjHEtkwF2Y2DjgL6G9ma4CfAQUA7j4WmACMBJYCu4HroopFUkOj\npIp0fpElBXf/divLHbgxqv2LiEjbdYgbzSIikh5KCjlm0oINSa3n7vxq4sKEy88pjb2t9fH3Pktq\nfyKSGUoKOea7z5Qktd6G7XtZuXl3iqMRkWyjpCAiIiElBRERCSkpSELUHVUkNygpiIhISElBEqKK\ngkhuUFLIYQvXb2ddeUWTy2au2kr57qrwc+m2psu15P2lm5KOTUQyQ0khh11w/1RO/dU7jea7O994\n+AOuenJaOO/rD3/Q5u1f+fi01guJSFZRUpBmzV5TnukQRCTNlBRERCSkpCAiIiElBRERCSkpiIhI\nSElBRERCSgoiIhJSUhARkZCSgoiIhJQUREQkpKQgIiIhJQUREQkpKYiISCink8KvJiygZMWWTIeR\nEX+etiqcLt1WQXVNLbe8+ClXPPohKzbvDpcNGTOet+dvyESIIpIBOZ0UHp2ynMvHfpjpMDLitlfm\nhNN3vjqXmau28dLMNUz7bAu3vDirXtnvPVuS7vBEJENyOimIiEh9SgoiIhJSUpBGPNMBiEjGRJoU\nzOwCM1tkZkvNbEwTy88ys3IzmxX83BllPNI8s7jpzIUhIhmWH9WGzSwP+ANwLrAGmG5mr7v7/AZF\np7r716KKQ1pnBq7qgYgQbU1hBLDU3Ze7eyXwPHBphPtL2tQlZRnZ78bte5ixcgvryiuYuWore6pq\neGdh9N0/Jy2ov48pizexrrwi/Dxz1bbIYxCR7BRZTQEYCKyO+7wGOLmJcqeZ2WygFPiJu89rWMDM\nRgOjAQYPHpzyQK964mM+vPWrDOhTmPJtt+S8+6ewbXcV3fK7sLe6litPHsxz01bx2o2nc+ygvpHt\n97vP1O9iWllTy03Pz2qmtIjkkkzfaJ4JDHb3Y4DfA682VcjdH3P3YncvLioqiiSQisqaSLbbkm27\nqwDYW10LwMrgobHte6rSHouICESbFEqBQXGfDwrmhdx9u7vvDKYnAAVm1j/CmDoEte+LSKZEmRSm\nA8PNbKiZdQVGAa/HFzCzA8xi/V7MbEQQz+YIY2pWNpyHTd1+RCTDIrun4O7VZvYD4G9AHvCku88z\ns+uD5WOBy4EbzKwaqABGuWfmOllX5yIi0d5ormsSmtBg3ti46YeAh6KMIXHZkxWyJxIRyTWZvtGc\ns1qqELk7ldW1LZaJV1Pr1NY2X7aqprbN8YlIblJSyJDnp69udtm1T03nsDsmcv/bSxLa1iG3TWDk\ng1PZuGMPi9bvaLR8+O0Tk45TRHKLkkIg3fcUXptV2mqZcR+varVMnYXrd/Dleydz/v1T2hOWiOQ4\nJYVANrbjtzWmPVVqJhKR9lFSyGLqESUi6aakEEj3CdgSGotUWUFE0ktJIeBZcAK2Bk+vqaYgIumm\npBDIxhNwFoYkIp2ckkKarN6ym+ue+piyHXu56olpfLi88WgeUxbXH8J7y65KTr/nHdaX7wGgdFsF\n1z71Mb+ftITHpy4H4Kd//bTRdhas2873/6eEX46fz5Ax4yP4bUSks4r0iWb53D1vLGTyojL+/aXZ\nTF2yKeH1SrdV8NDkJfzisi/x2zcX8e6iMt5dFEse3ztzGC+WrGm0zoUPTE1Z3CKSW1RTCKSr+SiZ\nMe+aiy1Dw0SJSCempBBI143mhjeTE1E3gkXDHkvKCSKSakoKHULs7N8wnygniEiqKSkE0tZ8lET7\nUW0zDyrXqqogIimmpJBmXZJICnVNWw1XVVIQkVRT76MElG6r4IG3F/Prbx6T1D0BgClBj6G/zdvQ\n5nVfLFmDYfxlRv2eRoff8UZSsYiINCdnagruzt3/O48lGxoPLd2a0+95hxdL1vD+0ra/KfSBt5fw\n8Wdb2LG3us3rxnuhpPmhtkVEUiVnksLa8j089f4Krnny4yaXJ9ISs3V3ZZv3+7u3F/OtRz9s83oi\nIpmQM0mhrtGnuXN/Il1Sa1p4u5mISGeQM0mhS3AvoD03Z5UURKSzy5mkUHd/uPmng1vfhpKCiHR2\nuZMUgn+bbz5qXbWSgoh0cjnTJbWuK2nZjr1s3LGHvQ1eXbm3qqbJ9aprPi837bPNHDeoL4P6FVJe\nUUV+ly7kdTGqamrpU1jAnqoatu+ppk9hAYUFeSzduDO6X0hEJAI5lBQ+nx7xy0mNll/x2EesuOei\n8PPqLbvZVVnN7a/MDee9Nmstr81aG2mcIiKZlDNJ4f2liQ9XDXDmvZMjikREJHvlTFLYuqvtzxh0\nVmcfsR+TFm5sNL9fz67MuOMc4PPmNnev9xS3u7N9TzXH3v1mOO+IA3qzcH3LDwX26pbPnLvOY+it\nE1LxKzTy88u+SMmKLZHU5Ab2LaR0W0XKtyvSnJvPGc79by/JyL5zJink5eXMPfVWdWlmAKYu1nho\n76Y+JzPShzWxrVQyohvUUL3OJN0aDpOfTpGeKc3sAjNbZGZLzWxME8vNzB4Mls82sxOiiiWZgeg6\nq7xmTs5dEjxpezOjtrYo4uOfaOzJUK8zSbdMnq8iSwpmlgf8AbgQOAr4tpkd1aDYhcDw4Gc08EhU\n8TR3IsxFXZr51hM9sdZk4eisZtG9X6KmubHLRSKSydNVlDWFEcBSd1/u7pXA88ClDcpcCjzrMR8B\nfc1sQBTBNNdkkovymskKiR6i6iROklEf/Si/XtUUJN2ibGptTZT3FAYC8UN7rgFOTqDMQGBdqoP5\nZNW2Vsuce9/fU73brFRY0HRSKOyal9D6DSsK3QtaX69H12hvX+V36UK3/GiucboX5LFjT/tGuRVp\ni64ZvAfaIe6+mtloMysxs5KysrKktnH5iQNbLTN8/17hT1HvbgAM6leY1P6y1bCintw+8ije/PGX\n6dUtn6evO4k7LjqSW849jKevG5HQNvbr3Y2fnHcYf/+3s7j5nOE8fOUJ/PHqYn509vCwzFmHF9Gn\nsIBrTj2YG79yCONGnwLAHRcdCcDPLj6K6//hEK49bQgA/3np0RxzUB96dcvnF5d9kbsvORqAot7d\nwul4fQoLwulrTj2YS447kP+46ChOHfYFbr3wCE48eF8G9OkOwAOjjuOqUw6mb48CRgztB8B3Th/K\nP585tMXf88SD9+WSYw/kzZu/nNBxAbjomAHcPvJIhhX15MIvHsC/nnsYxw7qC8AXB+6T8HaS8fXj\nY3/jz35nBN89I/a7/eelRzNiSL9m1+nZwoXAKcOaXw9ivyvAxcce2GJC/v4/DOMbJwzk1guPaLZM\n3x6ff5+H7d+r3rKCPOOXX/9is+vW/V9tzXWnD6GodzeeD/4Wf3rB4fWWnzRkX0475As8MOo4INaj\nDSCvlWroZccdGE7/5vJjOO2QLzRZ7qa4/x8AY+KOR4+ueUy//RzOOXI/fnrB4Vx16sF8/8vDGm2j\n7v9PlMwjah82s1OBu9z9/ODzrQDu/qu4Mo8C77r7uODzIuAsd2+2plBcXOwlJSWRxCwi0lmZ2Qx3\nL26tXJQ1henAcDMbamZdgVHA6w3KvA5cHfRCOgUobykhiIhItCJr6HX3ajP7AfA3IA940t3nmdn1\nwfKxwARgJLAU2A1cF1U8IiLSukjv/rn7BGIn/vh5Y+OmHbgxyhhERCRxHeJGs4iIpIeSgoiIhJQU\nREQkpKQgIiIhJQUREQlF9vBaVMysDFiZ5Or9gba9bSc9sjUuyN7YFFfbKK626YxxHezuRa0V6nBJ\noT3MrCSRJ/rSLVvjguyNTXG1jeJqm1yOS81HIiISUlIQEZFQriWFxzIdQDOyNS7I3tgUV9sorrbJ\n2bhy6p6CiIi0LNdqCiIi0oKcSQpmdoGZLTKzpWY2JuJ9DTKzyWY238zmmdlNwfy7zKzUzGYFPyPj\n1rk1iG2RmZ0fN/9EM5sTLHvQUvCePjNbEWxzlpmVBPP6mdlbZrYk+HffdMZmZofHHZdZZrbdzG7O\nxDEzsyfNbKOZzY2bl7LjY2bdzOyFYP40MxvSjrh+Y2YLzWy2mb1iZn2D+UPMrCLuuI2NWycdcaXs\ne0txXC/ExbTCzGZl4Hg1d37I+N8YAO7e6X+IDd29DBgGdAU+BY6KcH8DgBOC6d7AYuAo4C7gJ02U\nPyqIqRswNIg1L1j2MXAKsdccTwQuTEF8K4D+DebdC4wJpscAv85EbHHf13rg4EwcM+DLwAnA3CiO\nD/AvwNhgehTwQjviOg/ID6Z/HRfXkPhyDbaTjrhS9r2lMq4Gy38L3JmB49Xc+SHjf2PunjM1hRHA\nUndf7u6VwPPApVHtzN3XufvMYHoHsIDYu6ebcynwvLvvdffPiL1fYoSZDQD2cfePPPbtPgtcFlHY\nlwLPBNPPxO0nE7GdDSxz95YeUowsLnefAmxpYn+pOj7x2/orcHYitZmm4nL3N9297gXSHwEHtbSN\ndMXVgowerzrB+t8CxrW0jYjiau78kPG/Mcid5qOBwOq4z2to+SSdMkG17XhgWjDrh0FV/8m46mFz\n8Q0MphvOby8H3jazGWY2Opi3v3/+1rv1wP4Zig1iVzbx/1mz4Zil8viE6wQn9HKg6Rf7ts13iF0t\n1hkaNIX83czOjNt3uuJK1fcWxfE6E9jg7kvi5qX9eDU4P2TF31iuJIWMMLNewEvAze6+HXiEWBPW\nccA6YtXXTDjD3Y8DLgRuNLN6b6YPrjoy0i3NYq9uvQT4SzArW45ZKJPHpzlmdjtQDTwXzFoHDA6+\n538F/mxm+6QxpKz73hr4NvUvPNJ+vJo4P4Qy+TeWK0mhFBgU9/mgYF5kzKyA2Bf+nLu/DODuG9y9\nxt1rgT8Sa9ZqKb5S6jcHpCRudy8N/t0IvBLEsSGojtZVmTdmIjZiiWqmu28IYsyKY0Zqj0+4jpnl\nA32AzckGZmbXAl8DrgxOJgRNDZuD6RnE2qEPS1dcKf7eUn288oFvAC/ExZvW49XU+YEs+RvLlaQw\nHRhuZkODK9FRwOtR7Sxou3sCWODu98XNHxBX7OtAXa+I14FRQY+BocBw4OOgKrndzE4Jtnk18Fo7\nY+tpZr3rpondqJwbxHBNUOyauP2kLbZAvSu4bDhmcftL1fGJ39blwDt1J/O2MrMLgJ8Cl7j77rj5\nRWaWF0wPC+Jansa4Uvm9pSyuwDnAQncPm17SebyaOz+QLX9jid6R7ug/wEhid/mXAbdHvK8ziFX9\nZgOzgp+RwP8Ac4L5rwMD4ta5PYhtEXG9ZYBiYv+hlgEPETxw2I7YhhHryfApMK/uWBBrb5wELAHe\nBvplILaexK5m+sTNS/sxI5aU1gFVxNppv5vK4wN0J9Y8tpRY75Fh7YhrKbG247q/s7oeJ98Mvt9Z\nwEzg4jTHlbLvLZVxBfOfBq5vUDadx6u580PG/8bcXU80i4jI53Kl+UhERBKgpCAiIiElBRERCSkp\niIhISElBRERCSgqSM8ysxuqPxNriaLlmdr2ZXZ2C/a4ws/5JrHe+md1tsdEzJ7a+hkj75Wc6AJE0\nqvDYMAYJcfexrZeK1JnA5ODf9zIci+QI1RQk5wVX8vdabFz6j83s0GD+XWb2k2D6RxYb/362mT0f\nzOtnZq8G8z4ys2OC+V8wszctNlb+48SGNa7b1z8F+5hlZo/WPUXbIJ4rLDbO/4+A+4kNE3GdmUX2\nFL5IHSV/rZmHAAABw0lEQVQFySWFDZqProhbVu7uXyL2VOj9Taw7Bjje3Y8Brg/m3Q18Esy7jdjQ\nxQA/A95z96OJjS01GMDMjgSuAE4Paiw1wJUNd+TuLxAbOXNuENOcYN+XtOeXF0mEmo8kl7TUfDQu\n7t/fNbF8NvCcmb0KvBrMO4PY8Ai4+ztBDWEfYi93+UYwf7yZbQ3Knw2cCEyPDVVDIZ8PetbQYcDy\nYLqnx8bdF4mckoJIjDczXeciYif7i4HbzexLSezDgGfc/dYWC8VekdofyDez+cCAoDnph+4+NYn9\niiRMzUciMVfE/fth/AIz6wIMcvfJwL8TG4a4FzCVoPnHzM4CNnlsXPwpwP8L5l8I1L1gZhJwuZnt\nFyzrZ2YHNwzE3YuB8cTennUvsUELj1NCkHRQTUFySWFwxV3nDXev65a6r5nNBvYSG747Xh7wJzPr\nQ+xq/0F332ZmdwFPBuvt5vOhiu8GxpnZPOADYBWAu883szuAN4NEUwXcCDT12tETiN1o/hfgviaW\ni0RCo6RKzjOzFUCxu2/KdCwimabmIxERCammICIiIdUUREQkpKQgIiIhJQUREQkpKYiISEhJQURE\nQkoKIiIS+v/o8Y6zk3gkWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x179ba370898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOW9x/HPLyEhCAgCUdYIIlqwWlRU6kq17raopa7V\nVm25WqvVLreoreL12moXbV1uUVuLK9q6UCquuFSsiiwisogiIltYZEnCGpL87h9zMkxClsnMnJlJ\n5vt+vfLKmedsv5wk53ee5zznOebuiIiIAORlOgAREckeSgoiIhKlpCAiIlFKCiIiEqWkICIiUUoK\nIiISpaQgIiJRSgoiIhKlpCAiIlHtMh1AS/Xo0cP79++f6TBERFqVmTNnfuHuxc0t1+qSQv/+/Zkx\nY0amwxARaVXM7PN4llPzkYiIRCkpiIhIlJKCiIhEKSmIiEiUkoKIiEQpKYiISJSSgoiIRLW65xRE\nUm7Ze/DXE6MfN3fam46bIl26P6vZi9/v+zDHDe7DOYf1a/m21yyg4m+jeK7odI7r7XwxYCQ9V7zM\nnu//iU9OfYI5eYPZb8mjzFxfRL/lz3FC/vsArDx/Cp2m383ui/7JlAN/x6ARFzJ33lxO61mG7XfS\nLrvZtqOayXNKOXz1BPJ260GfEZdE573+z/HMqe7Pii15/HbRN3ilwyk8VnYg4wt/V2cbL/b9Macs\n/xOrS05n8spOnDBod0pOvponPoaZn2/g26vu4PCOq7ERY6CwM6sWzaL97PFctnoU3yyYzqn2H9q1\n70j3yhVs8E7MOXYcx039Dq/VHMLxebN4o/u5lB16FdXlqygpn8mwqveZ1e1UKtd8yrT1u7G054nc\nsuA0drPtbC4ZQcelb/DTysv53w6P0aG6gmd3/w5nlT8KwJ+qzuKi/Z1unXZjVV4xy2qKeX7mJ+zV\nozudCvM4ec1feGf/69jn2POZ8J+PuW7ZaDpVLGbFieP49eJ9uGHz7cwo78LBPp+PNnXkxPyZAHzS\n6xv0XP0mnWvKqCjoQaeaCqx6Oyt6nUSfYy+G3gfjK2ez4s2HmNTuJE5Z/QAdeg6i10nXUr1xBQ+/\n8zkje61j+ayXOahmfvTYPrn796go3JP8POPLHTYw9MCD8Pfu57ntQ9m3dzE9fTW7fzGb7UMv4bO5\n79DlsHMZcOhJvPnxWgZ0hl5/OZBnDrqPc0aObPnfYAtZa3tH87Bhw1wPr0lKje3S5Ow3qw/k4h3X\nseS201O+7Vt2XMivCh5rdjP9tz3O3PaX0sm2wdiyXebf+M+5PPzO5ywpuiDYb8wyY7uwwrvTx9a1\nKHSAHQWdGVRxH4Xs4OOi77Z4/freqR7CV/PnNzhvfs3eDMmL6/mquB207X5+3O5ZLmv3QrTs9h3n\n8YuCJxLbYIc9YOuGFEXXjLFl9B8zmX8XXsPeeWsAWHHNKvp07ZDQ5sxsprsPa245NR+JNKPE1oS2\n7e5WEfeynWxbo/PWlG9vct1EEgJAwY5IfEZqLh572xeNzhtoK1Kyj1jtqKHYNtYp6267JtW4pSsh\nxKhNCACVVTWh709JQUSyQqoST/P7kaYoKYhITklX8glDOhKakoKISCthacgKSgoibUbrvQKG8K6C\n69cMWnNNIR3UJVUSVl1VxbrVy9izz4CUbXPV0k/Yq+9ALG/n9cqGz2bRoXgfijp1rbPs5oqNbFxb\nSpcePdlSUUa3PXvTrqAQgLUrl7B2YzkDBg5mQ+liKlctZGvlDjaVb6BowBH0yi+jdPUatm9aR3Pd\nMfrnrebadv9gxqSl7Fbcnw67dwPLY8CQw6Ixb9+6mY0rPqF4n4PYq+9A8tu1w92bPdH9sN2kuI7L\n3QV3RadnTBpHl5ID6NitN6Wznid/jxL6ffQi/91uU3SZT+e8RYfOPShd8B8OjWsPjbssf3KSW9gp\n9qZpfQVWnbL91Lq5YDzfyH+3TtnF+S+nfD9hmH3XuZyd179OWU116o9RfeqSKgl7d9wPGb7qMb4Y\nPYcevfdOensLpr3E4BfOYfrQWznszB9FCmuq4X+6RabrdcXcMLYfe1C+M569zmP4FfcFy0a6gv52\nx7n8d8GTScfWoLFlvP/yoxz89pV1it/pdTFf/a+7ufeJiVz5UfLdOEVqfUZfBoydl9C66pIqodtr\nzVsAVGxYlZLtlS/9EICapdN2FnrjXfBiEwJAr7Vv7bLMkXlzUxJbY7Z9Pn2Xsj2D4/Lx3Fmh7lty\nzwCWh74PJQUREYlSUhAJiaejq4hIioWWFMysn5m9bmbzzWyemf24gWVGmFmZmc0Ovm4MKx4REWle\nmL2PqoCfuvssM+sMzDSzV9y9/sAnU939jBDjEBGROIVWU3D3UnefFUxXAAuAPmHtT9qolvSOy7KO\ndM13SBXJPmm5p2Bm/YGDgWkNzD7SzOaY2QtmdkA64pHsZNHvqTu7H52fWPe9ZDjQf8xkRvJG2vct\nkqzQH14zs07A08A17l5eb/YsoMTdN5nZacBEYFAD2xgNjAYoKSkJOWJpqVQ96tIar6wbSmDuzu5s\n4uvBuxFEWpNQawpmVkAkITzm7s/Un+/u5e6+KZh+Higwsx4NLHe/uw9z92HFxcVhhizSQg0nst1t\nS5rjEEmNMHsfGfBXYIG739HIMj2D5TCzw4N4Ehv4XUREkhZm89FRwEXAh2Y2Oyi7HigBcPdxwCjg\nCjOrArYC53lrG3dDpJ7W2AwmUiu0pODub9HMwIfufg9wT1gxSOtU97KgFV4jtMKQRWrpiWbJGnoA\nWCTzlBQk67SF8e6/YoszHYJIQpQUJIu0japCHjXcW3hX8wuKZCElBZEkWAP9IorYnoFIRFJDSUGS\n1vobe5LQNio3IlFKCpLd1ENZJK2UFEREJEpJQUREopQUJFw11fDR5BY2AwXL1tTAwsktWCs7mpqq\na7IjDpFEKClIuKaNgycugLlPN79svafXtv3nXnjq0pACC0+N7oNIK6akIOEqWx75vml1i1f9YF76\n34UgkuuUFCRrtYYL7oaevlYvVWnNlBQkaak6d+tkKpJ5SgoiIhKlpCAiIlFKCpIWldU17KiuiW/h\nZm4mVFXXsKWyKgVRiUh9SgqSvCZO4rXdM19+6V9c+0DTzxzE+8ayXz76GpeP/W388aVZsW3MdAgi\nCQvzdZzS1sXxVpx5K8s5EDgjfxonls4Cvkh6t1ctHk2fwux9lXdH0yip0nqppiCJi6PP6Obt1dHp\n9rYjJbvtY9mbEERaOyUFSV6K3qPZKruktoaHKURaQElBkpYt58VWmVREsoySgiQs9JNwimogYcr+\nCEVaRklBkpctVQURSZqSgiTMU30lv8v2svw6fLNueEvbo6QgCWvopfX1xfvsQWu08oOXU58YRTJM\nSUGSl6MnxtKNWzIdgkjKKSmIJEi3UqQtCi0pmFk/M3vdzOab2Twz+3EDy5iZ3WVmi8xsjpkdElY8\n0val+yTt7nE1oYm0JmEOc1EF/NTdZ5lZZ2Cmmb3i7vNjljkVGBR8HQH8OfgurUlTJ8bcbFkSabVC\nqym4e6m7zwqmK4AFQJ96i40EHvaId4GuZtYrrJgktcK6ydrQ28yyknvO3k+Rtist9xTMrD9wMDCt\n3qw+wLKYz8vZNXFIlorr5N2C83ttknEHaqoZvuqxFkaU3mSybpMGvpO2J/SkYGadgKeBa9y9PMFt\njDazGWY2Y+3atakNUJLX1NVyghfSFbOfTWzFNKqaPynTIYikXKhJwcwKiCSEx9z9mQYWWQH0i/nc\nNyirw93vd/dh7j6suLg4nGAla5jBwpXrMx1Gs87Ir1/xFWn9wux9ZMBfgQXufkcji00CLg56IQ0H\nyty9NKyYJLupJ49I5oXZ++go4CLgQzObHZRdD5QAuPs44HngNGARsAW4JMR4JNVqz+EpPpk7pk5L\nIhkSWlJw97dopkXZ3R24MqwYJFwa4gHSfXNbJGx6olkSFlpzj5qRRDJGSUESV1tRSFWNIcntVNc4\nyzekdzyibluWpHV/ImFTUpBQpXOU1IF5pfzyd431aQhpnxumpnV/ImFTUpA2ZXzh7zIdgkirpqQg\nydM9AJE2Q0lBEhZu05B6NolkgpKCJCy8getU8xDJFCUFSV7KnldQ7UAk05QUJHsoJ4hknJKCJK3p\n+8wpPNOP7cK5d/yr2cWuHf/v1O1TJMcoKUjS0jmQ3ea1S5pd5tefjQo/EJE2SklBEpbOB9NaooNV\nZjoEkVZLSUES1mpemykicVNSkKSlarTU7Kx3iOQWJQXJOoZnbdOUSFunpCBZI55EoGQhEq4w37wm\nbZh/9Dz9qz8HUt/76LCyl2DGSw3O030MkXCppiAJsSfOj057Gk/Uz7X/Zdr2JZKLlBRERCRKSUGS\np95HIm2GkoKIiEQpKYiISJSSgiQvVb2PUjYEt4gkSklBwtXEib5i2w4eensJrtd5imQNPacgGfOr\niXOZOHslg/bqxJEDe2Q6HBFBNQVJgUSfMt6wZQcA26tqUhmOiCRBSUFERKKUFEREJCq0pGBmD5rZ\nGjOb28j8EWZWZmazg68bw4pFwqYbxSJtRZg3mscD9wAPN7HMVHc/I8QYpDWozSnqkiqScaHVFNz9\nTWB9WNuX1qGhm9ALV1Xg7soBIlko7qRgZkeb2SXBdLGZDUjB/o80szlm9oKZHdDEvkeb2Qwzm7F2\n7doU7FYy5e1FX3DyH9/ksWlL6VCzme/lv5i6h99EJGlxJQUzuwn4BXBdUFQAPJrkvmcBJe5+EHA3\nMLGxBd39fncf5u7DiouLk9ytpF78l/yfrdsMwLyV5Vy0/m7GFjxM1zXTgq2o6iCSafHWFM4Cvgls\nBnD3lUDnZHbs7uXuvimYfh4oMDM9wdQKJfo+hU7V5QDkVW9PZTgikoR4k0KlR8YicAAz65jsjs2s\np1mkVdnMDg9iWZfsdkVEJHHx9j76u5ndB3Q1sx8AlwIPNLWCmU0ARgA9zGw5cBORZifcfRwwCrjC\nzKqArcB5rkFwcojHTOnXLpIt4koK7v57MzsRKAf2B25091eaWef8ZubfQ6TLquSQpu4bKDWIZF6z\nScHM8oEp7v41oMlEICIirVuz9xTcvRqoMbMuaYhHstjyDVvoP2Yyry9cU29Oy3oN7W9L63RDNdUR\nRLJGvPcUNgEfmtkrBD2QANz96lCikqz0/tKNADw1czlfqzMn/pP6Xuum8VL7MTy77icpjU1EUiPe\npPBM8CWSlM6bPweg97ZF0bJEh94WkdSL90bzQ2ZWCOwXFC109x3hhSUiIpkQV1IwsxHAQ8ASIg3I\n/czsu8H4RpLj1JNYpO2It/noD8BJ7r4QwMz2AyYAh4YVmGQx5QCRNiveJ5oLahMCgLt/TPAgmuSO\nxkc1beqegDX5Mb7ti0i6xJsUZpjZX4IX44wwsweAGWEGJq1Jy6sOn67ZxMYtlcHqzthJ81i2fmuK\n4xKRloo3KVwBzAeuDr7mB2WSQzpsWcGSogsYsnlaSrZXE+SSrTuqGf/2Et5YqGHRRTIt3nsK7YA/\nufsdEH3KuX1oUUlW6rr+AwCGl79Yp9yaaPepX4ewBioV6pIqkj3irSm8CnSI+dwBmJL6cCRXKBGI\nZKd4k0JR7bsPAILp3cIJSbKVXoIj0vbFmxQ2m9khtR/MbBiR4a5FUvY2TfV0Fcm8eO8pXAP8w8xW\nBp97AeeGE5Jkq8bfe9DU6Vy1C5HWpMmagpkdZmY93X068CXgSWAH8CLwWRrikxSZu6KMuSvKMh0G\nJate3qWsaJt6HYlki+aaj+4Dgs7kfBW4HrgX2ADcH2JckmJn3P0WZ9z9Voq21kjN4MOn4J17m1xz\nr3WR7qwXtnuVY/LmAHDI7F+lKC4RSVZzzUf57r4+mD4XuN/dnwaeNrPZ4YYm2caau4Z4+rLI969e\nGdf28hvqnyoiGdVcTSHfzGoTxwnAazHz4r0fISIirURzJ/YJwL/N7AsivY2mApjZvkDmG6glS+hm\nskhb0WRScPdbzexVIr2NXvadYyTnAVeFHZxkm0R6H7Vk60ouIpnWbBOQu7/bQNnH4YQjrYFO3iJt\nV7wPr4lEU0Gn6vI65dU1zpqKbekPSERSTjeLc8SXbXHS2/Bg4LsvbZ1Vp/zxN95n3VMvc19hUDD3\nadj361DUJe5tn5Q3nR5W3vyCIhIqJYUc8Vz7XwZTqb8VNKbsf9m9MGbUk6cuhf1Ph/Mfj3sb9xfe\nmfK4RKTl1HwkSdvdGhgGq3x5+gMRkaQpKYiISFRoScHMHjSzNWY2t5H5ZmZ3mdkiM5sTOwqrtAGp\nGjpVRNIqzJrCeOCUJuafCgwKvkYDfw4xFsmUJt7KJiLZJ7Sk4O5vAuubWGQk8LBHvAt0NbNeYcUj\nqaATvEhbl8l7Cn2AZTGflwdl0gaUb9uR6RBEJAGt4kazmY02sxlmNmPtWo29nznx3ydYW7E9xDhE\nJCyZTAorgH4xn/sGZbtw9/vdfZi7DysuLk5LcLIrNR6JtH2ZTAqTgIuDXkjDgTJ3L81gPJJCpjcu\ni7RKoT3RbGYTgBFADzNbDtwEFAC4+zjgeeA0YBGwBbgkrFhERCQ+oSUFdz+/mfkOxPeKLskSLW9A\nUn1BpHVpFTeapfWxet9FpHVQUsg1i6YktJq7M/nD+G/5bNtRDcARq+IfFE9EMk9JIdc8+q2EVltT\nsZ2VGxsY+K4Rg/OWAlDglQntT0QyQ0lB4qKhjERyg5KCiIhEKSmIiEiUkoKIiEQpKYiISJSSQg6b\nu6KMFY30KJq+ZD0bNu/sOZRf9jlfCnoUxev1hWuSik9E0k9JIYedcfdbHHXba7uUuzvfHvcOF/5l\nWrSs+MHDubrdxBZt/9NHrk46RhFJLyUFadT80vKk1v9+uxdSFImIpIuSgoiIRCkpiIhIlJKCiIhE\nKSmIiEiUkoKIiEQpKeSwA+wz+rC2wXmH2wK6UpHmiEQk00J785pkv8ntbwimvld3hjt/b38L82v2\nBs5Lc1QikkmqKUijhuR9nukQRCTNlBRERCRKSUFERKKUFEREJEpJQUREonI6KYy6ewrPzMzNm6m/\nfGpGdPqDZRvZvL2KQ255hf5jJjPts/XReVNvPp7qN36biRBFJANyOik8te5bMPGHmQ4jI/537gnR\n6bte/YR5K8tZH7w/4fYXP4rOO8Znkv/GrWmPT0QyI6eTAsDZ+W9lOoSs457pCEQkU3I+KciulBNE\ncleoScHMTjGzhWa2yMzGNDB/hJmVmdns4OvGMOOR+LiqCiI5K7RhLswsH7gXOBFYDkw3s0nuPr/e\nolPd/Yyw4pD4mO2cVk4QyV1h1hQOBxa5+2J3rwSeAEaGuD9JUGxCEJHcFmZS6AMsi/m8PCir70gz\nm2NmL5jZASHG06i/TF2cid3Cqrnw/mNQ+gF88AQbNldy7+uLQm++ufvVT+p87vXxoyz6aA4AedRw\n3tbHQ92/iGSvTI+SOgsocfdNZnYaMBEYVH8hMxsNjAYoKSlJeRAnvXISS/efTsmee6R8200ad1Sd\nj2MG7stL81ZzSMkefHVg99B2+8Ar73NV0c7PtxSMh3fHc1r73ehiW2BraLsWkSwXZk1hBdAv5nPf\noCzK3cvdfVMw/TxQYGY96m/I3e9392HuPqy4uDjlgZbkrcXKl6d8uy21eXs1AFU1NaHu55C8Txos\n72JbQt2viGS/MJPCdGCQmQ0ws0IiA/NPil3AzHqaRVq0zezwIJ51IcYkgDqdikhjQms+cvcqM/sR\n8BKQDzzo7vPM7PJg/jhgFHCFmVURabQ4z9UfUkQkY0K9pxA0CT1fr2xczPQ9wD1hxhCvXOqAk0s/\nq4i0jJ5oDuRSBcXUfCQijVBSyJTqHY3O+mLTdp6bs5KtldXxbWv9YigvbXT2ax+trvN5sC2Nb7si\nknMy3SU1d71y0y5FnarLuDJ/Io/+fSG9bD3PDT6LcRcd2vy27jo48v0Hr8G6T+Ggc+rM/tvDD3J8\n4c7PPy/4ezKRi0gbpqQQSHs7+6o5uxR9f8OdDCt4O/r5iGUjWrbNB46PfK+XFB4pvK2l0YlIjlLz\nURZp76l/aiyX7pWISPJUU8gq6hckEq8dhV1Zfsgv2NZlH3Lqf2fBgiZnFxUV0bdvXwoKChLavJJC\nFvEQ/rDdc+rfRXLI8kN+Qed9htG/Yzssl0Z17D240Vnuzrp161i+fDkDBgxIaPNqPsoqdf+w1fIj\n0rhtXfahe64lhGaYGd27d2fbtm0Jb0NJIVMa+EMOIwcor0jbZUoIDUj2mCgpRIV7+nxx7ir6j5nM\nax+t5lvX3UHV4l3fDf2V7TPqfB60eQbnXPd73ly4BoAp81fzpTHP8F/Xj+WsX08A4Ozr7txlO3Pv\nuxTGdmHbbwaS/z9dQ/hpRCRsG8sq+L/xdbuPn3LhlXTt2pUzzgjvvWRKCoGwLzj+NWclANNfn8TT\n7W+mnTU/Eupjhb/h7+1vofzNyEggz39Yyh8L/o/7Cu/k2crLAXim/dhd1vty6dMAFG3/IkXRi0i6\nbSyv4P8e/kedsp9ffjGPPPJIqPvVjeao9FRDu1W3/ETdY9vn0emheYtSGY6IJOHMS3/CspWr2La9\nkh9fdj6jv/Mt/jphIrffO56uXTrzlSH70b6wgHtuHcPadRu4fMytLF2xCoA/3vwzjjpsKGP/MI6l\nK1axeOkKlq5YxTXfv4CrLzufMb++i08/X87QE8/jxGOP4He/upYTjjmCNz4uC/VnUlIIpO2mbiJV\nkpjYNG6RyK5ufrOM+WsbHzomEUOKC7jp2C5NLvPgH26i2x5d2Lp1G4edfhGnn3AMt/zxAWa9+Did\nO+3G8ef8F18Zsh8AP77xd1z7gws5+vCDWbqilJMvuJIF/34GgI8WLeH1f9xPxebN7H/M2Vxx8Shu\nu/5q5i78lNmvPJHSn6s5Sgpp5gm12CkRiGSjux6cwLMvvA7AspWreeTpyRw3/FC67RFJJt8+40Q+\nXhyp6U+ZOo35H+989W/5ps1s2hx5sdXpJxxN+/aFtG9fyJ499mD12vVp/kl2UlIImKXnxJtIRcHU\nN1WkSc1d0YfhjbdnMGXqe7zzr/Hs1qEDI0b9gC/t258Fn3zW4PI1Nc67/3qIoqL2u8xr337n4GT5\n+flUVcc5GGYIdKO5FXDVFESyTlnFJvbo0pndOnTgo0Wf8e6sD9m8ZSv/fncmGzaWU1VVxdPPvxpd\n/qTjhnP333Y2Bc2eu7DJ7XfuuBsVmzaHFn9jlBTiMH3GND6+7Rh8e0XC25g8p5QlRRfwgzW3tnjd\n4Rv+BWO7cMeC49jLNu6cMTb9V0ciEnHKiCOpqq5m8HFnM+bXdzP8kAPp03NPrr/qUg4//SKOOvNS\n+vftTZfOnQC465afM+OD+Rz09XMYMuJbjHvkqSa3371bV446bChfPv7b/PyWSNfzY866lG9/+9u8\n+uqr9O3bl5deeinlP1fuNB9t3wS/6QMn3AjH/LRFq67/5/Uclj+H6kWvkX/AyJbtd2wXGHohZ+bp\nBC7SlrRvX8gLj+764shhXxnC6O98i6qqKs667KececoIAHp024Mnx92+y/Jjf3p5nc9zX9vZDfXx\ne39dZ97UZx+E3genIPrG5U5NYeuGyPfpDzY4u6lm+5qgu2rCPX9mP8Z32k1JbF0RaVXG/uE+hp54\nHl8+/hwGlPThzFO+lumQWiR3agq1d3i9+YfG6osOVJfEDd88Wr5fEWl9fn/jtZkOISm5U1Ow2h+1\n5Sf26BpJJQXdLBaR7Jc7SYGmawpNvYxm55DWiZ/Y9dCZiLQGuZMUrOkmoKZO2Wo+EpFckUP3FIL8\nt3kN373pTpZXduTVmGdI3p83n5L9hu6y2sJVFdGkkPfMZfDMZfxmn0e5b35ke0fnfUiXvG2MPWN/\nzn5zL5at38rA4o7cd/Rm7v3nVO4Mnkk5MG9JmD+diEhK5ExSWLJuC/2D6YdsLNR7qHDkB1fAWRfs\nLHjvAdj4OSe/9lXmtP+gzrLXLf4OXysczHcrf8Gjhb+JFL4IUwGKgArgBaIJQUSkpTaWVfD4sy/w\nw++dA0Qedrviul9Tvq2a/Px8brjhBs4999yU7zdnmo9e/Whty1Z4/mfw9t0A7G5bdpk9PG8BB9ri\nXcpFRFKh/tDZu3Uo4uE/3cK8efN48cUXueaaa9i4cWMTW0hMztQU8vNTn//Uo0gkt6V76OxavXv3\nZs8992Tt2rV07ZraF2nlTlLIy5lKkUjueftuWJfid4103xeOvKrJRTI1dPZ7771HZWUlAwcOTO3P\nTMhJwcxOAf4E5AN/cffb6s23YP5pwBbge+4+K4xY8kN4h47eDiuS2zIxdHZpaSkXXXQRDz30EHkh\nXOyGlhTMLB+4FzgRWA5MN7NJ7j4/ZrFTgUHB1xHAn4PvKZefl/pTeF4cr9QUkTRo5oo+DJkYOru8\nYhOnn386t956K8OHD0/ND1JPmG0qhwOL3H2xu1cCTwD1R5MbCTzsEe8CXc2sVxjBbNhS2ewym8f2\n5JP/GVpn9NGJhb9qdPkJhS0f8VRE2oZ0D51dWbmDsy77KRdffDGjRo1K/Q8UCLP5qA+wLObzcnat\nBTS0TB+gNHYhMxsNjAYoKSlJKJgeCyc0u0xHtjKopm6WH5r3aUL7E5G27ZQRRzLukacYfNzZ7D+w\n/y5DZ3fbowtfGti/ztDZV15/Gwd9/Ryqqqo59ohDGHf7DY1uP3bo7FO/diRfGbIfb057n3WbKhk/\nfjwA48ePZ+jQXZ+vSoY1NbxDUhs2GwWc4u7fDz5fBBzh7j+KWeY54DZ3fyv4/CrwC3ef0dh2hw0b\n5jNmNDq7Ucve+Bv93rimxeuJvFJ9KCfmz8x0GFLPgpP/zuC998x0GLvYtHkLnTruFh06+9LzRnLW\nqcenZNvVee3J7zmk2eUWLFjA4MGD65SZ2Ux3H9bcumHWFFYA/WI+9w3KWrpMSvQbcQmMuCTh9UvL\ntjLhvWVc+/VBWCLv1EyBrZXV3DnlY679+n50KMxPzUZ3bIXXb4UR10Fhx9Rss4U+WV3BLyfO5dKj\nB3DyAT1Tss0FpeW88+k6Lj16QNLbOjEF8cRl9uPQtQT6H52uPbZuCxZA7+DE5w6bVkOHPaDdrm32\n6TT2Zz9wYnJvAAAIdklEQVRjypQpbNu2jZNOOokzL/1J3ffwVlXCli+gc68Wv583Rf/1TQqzptAO\n+Bg4gciJfjpwgbvPi1nmdOBHRHofHQHc5e6HN7XdRGsKItK2NHQ1LBFZWVNw9yoz+xHwEpEE96C7\nzzOzy4P544DniSSERUS6pCZ+KS8iIkkL9TkFd3+eyIk/tmxczLQDV4YZg4i0Xe6esebcbJVs648e\n8xWRVqmoqIh169YlfRJsS9yddevWUVRUlPA2cmaYCxFpW/r27cvy5ctZu7aFg122cUVFRfTt2zfh\n9ZUURKRVKigoYMCA5HuXSV1qPhIRkSglBRERiVJSEBGRqNAeXguLma0FPk9w9R7AFykMJ1WyNS7I\n3tgUV8sorpZpi3Ht7e7FzS3U6pJCMsxsRjxP9KVbtsYF2Rub4moZxdUyuRyXmo9ERCRKSUFERKJy\nLSncn+kAGpGtcUH2xqa4WkZxtUzOxpVT9xRERKRpuVZTEBGRJuRMUjCzU8xsoZktMrMxIe+rn5m9\nbmbzzWyemf04KB9rZivMbHbwdVrMOtcFsS00s5Njyg81sw+DeXdZCoaENLMlwTZnm9mMoKybmb1i\nZp8E3/dIZ2xmtn/McZltZuVmdk0mjpmZPWhma8xsbkxZyo6PmbU3syeD8mlm1j+JuH5nZh+Z2Rwz\ne9bMugbl/c1sa8xxGxezTjriStnvLcVxPRkT0xIzm52B49XY+SHjf2NAZFS9tv5F5H0OnwL7AIXA\nB8CQEPfXCzgkmO5M5GVDQ4CxwM8aWH5IEFN7YEAQa34w7z1gOGDAC8CpKYhvCdCjXtlvgTHB9Bjg\n9kzEFvP7WgXsnYljBhwLHALMDeP4AD8ExgXT5wFPJhHXSUC7YPr2mLj6xy5XbzvpiCtlv7dUxlVv\n/h+AGzNwvBo7P2T8b8zdc6amcDiwyN0Xu3sl8AQwMqyduXupu88KpiuABUCfJlYZCTzh7tvd/TMi\nLx063Mx6Abu7+7se+e0+DJwZUtgjgYeC6Ydi9pOJ2E4APnX3ph5SDC0ud38TWN/A/lJ1fGK39RRw\nQjy1mYbicveX3b0q+PgukVfaNipdcTUho8erVrD+OcCEprYRUlyNnR8y/jcGudN81AdYFvN5OU2f\npFMmqLYdDEwLiq4KqvoPxlQPG4uvTzBdvzxZDkwxs5lmNjoo28vdS4PpVcBeGYoNIlc2sf+s2XDM\nUnl8ousEJ/QyoHsKYryUyNVirQFBU8i/zeyYmH2nK65U/d7COF7HAKvd/ZOYsrQfr3rnh6z4G8uV\npJARZtYJeBq4xt3LgT8TacIaCpQSqb5mwtHuPhQ4FbjSzI6NnRlcdWSkW5qZFQLfBP4RFGXLMYvK\n5PFpjJndAFQBjwVFpUBJ8Hv+CfC4me2expCy7vdWz/nUvfBI+/Fq4PwQlcm/sVxJCiuAfjGf+wZl\noTGzAiK/8Mfc/RkAd1/t7tXuXgM8QKRZq6n4VlC3OSAlcbv7iuD7GuDZII7VQXW0tsq8JhOxEUlU\ns9x9dRBjVhwzUnt8ouuYWTugC7Au0cDM7HvAGcCFwcmEoKlhXTA9k0g79H7piivFv7dUH692wNnA\nkzHxpvV4NXR+IEv+xnIlKUwHBpnZgOBK9DxgUlg7C9ru/goscPc7Ysp7xSx2FlDbK2IScF7QY2AA\nMAh4L6hKlpvZ8GCbFwP/TDK2jmbWuXaayI3KuUEM3w0W+27MftIWW6DOFVw2HLOY/aXq+MRuaxTw\nWu3JvKXM7BTgv4FvuvuWmPJiM8sPpvcJ4lqcxrhS+XtLWVyBrwMfuXu06SWdx6ux8wPZ8jcW7x3p\n1v4FnEbkLv+nwA0h7+toIlW/OcDs4Os04BHgw6B8EtArZp0bgtgWEtNbBhhG5B/qU+AeggcOk4ht\nHyI9GT4A5tUeCyLtja8CnwBTgG4ZiK0jkauZLjFlaT9mRJJSKbCDSDvtZak8PkARkeaxRUR6j+yT\nRFyLiLQd1/6d1fY4+Vbw+50NzAK+kea4UvZ7S2VcQfl44PJ6y6bzeDV2fsj435i764lmERHZKVea\nj0REJA5KCiIiEqWkICIiUUoKIiISpaQgIiJRSgqSM8ys2uqOxNrkaLlmdrmZXZyC/S4xsx4JrHey\nmd1skdEzX2h+DZHktct0ACJptNUjwxjExd3HNb9UqI4BXg++v5XhWCRHqKYgOS+4kv+tRcalf8/M\n9g3Kx5rZz4Lpqy0y/v0cM3siKOtmZhODsnfN7KCgvLuZvWyRsfL/QmRY49p9fSfYx2wzu6/2Kdp6\n8ZxrkXH+rwb+SGSYiEvMLLSn8EVqKSlILulQr/no3Jh5Ze5+IJGnQv/YwLpjgIPd/SDg8qDsZuD9\noOx6IkMXA9wEvOXuBxAZW6oEwMwGA+cCRwU1lmrgwvo7cvcniYycOTeI6cNg399M5ocXiYeajySX\nNNV8NCHm+50NzJ8DPGZmE4GJQdnRRIZHwN1fC2oIuxN5ucvZQflkM9sQLH8CcCgwPTJUDR3YOehZ\nffsBi4Ppjh4Zd18kdEoKIhHeyHSt04mc7L8B3GBmByawDwMecvfrmlwo8orUHkA7M5sP9Aqak65y\n96kJ7Fckbmo+Eok4N+b7O7EzzCwP6OfurwO/IDIMcSdgKkHzj5mNAL7wyLj4bwIXBOWnArUvmHkV\nGGVmewbzupnZ3vUDcfdhwGQib8/6LZFBC4cqIUg6qKYguaRDcMVd60V3r+2WuoeZzQG2Exm+O1Y+\n8KiZdSFytX+Xu280s7HAg8F6W9g5VPHNwAQzmwe8DSwFcPf5ZvZL4OUg0ewArgQaeu3oIURuNP8Q\nuKOB+SKh0CipkvPMbAkwzN2/yHQsIpmm5iMREYlSTUFERKJUUxARkSglBRERiVJSEBGRKCUFERGJ\nUlIQEZEoJQUREYn6f8ZWHg/5sFSSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x179b5390128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# train multi agent with D4PG\n",
    "multiagent=MAD4PG(24,2,32,BUFFER_SIZE = int(1e6),BATCH_SIZE = 64,GAMMA = 0.99,TAU = 1e-2,LR_ACTOR = 1e-4,LR_CRITIC = 1e-4,WEIGHT_DECAY = 0,UPDATE_EVERY = 6,N_step=7)\n",
    "scores,scores1,scores2 = train(check_pt_path='multiagents_d4pg',n_episodes=20000)\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores1)), scores1)\n",
    "plt.plot(np.arange(len(scores2)), scores2)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.legend(['agent1', 'agent2'], loc='lower right')\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (max over agents) from episode 1: 2.0000000298023224\n",
      "Score (max over agents) from episode 2: 0.0\n",
      "Score (max over agents) from episode 3: 2.600000038743019\n"
     ]
    }
   ],
   "source": [
    "#Watch a mad4pg trained Agent\n",
    "multiagent=MAD4PG(24,2,32,BUFFER_SIZE = int(1e6),BATCH_SIZE = 64,GAMMA = 0.99,TAU = 1e-2,LR_ACTOR = 1e-4,LR_CRITIC = 1e-4,WEIGHT_DECAY = 0,UPDATE_EVERY = 6,N_step=7)\n",
    "multiagent.mad4pg_agent[0].actor_local.load_state_dict(torch.load('multiagents_d4pg_actor1_last.pth'))\n",
    "multiagent.mad4pg_agent[1].actor_local.load_state_dict(torch.load('multiagents_d4pg_actor2_last.pth'))\n",
    "\n",
    "env = UnityEnvironment(file_name=\".\\Tennis_Windows_x86_64\\Tennis.exe\")\n",
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]\n",
    "\n",
    "for i in range(1, 4):                                      # play game for 3 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(2)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = multiagent.acts(states) # select an action (for each agent)\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Score (max over agents) from episode {}: {}'.format(i, np.max(scores)))\n",
    "    \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}