tutorials/05-Electrophysiology Computed Tables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with automated computations: Computed tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome back! In this session, we are going to continue working with the pipeline for the mouse electrophysiology example. \n",
    "\n",
    "In this session, we will learn to:\n",
    "\n",
    "* compute various statistics for each neuron by defining a `Computed` table\n",
    "* define a `Lookup` table to store parameters for computation\n",
    "* define another `Computed` table to perform spike detection and store the detected spikes\n",
    "* automatically trigger computations for all missing entries with `populate`\n",
    "* define a `Part` table to save the results computed with the master `Computed` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing first, let's import `datajoint` again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datajoint as dj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are going to perform some computations, let's go ahead and import NumPy as well as Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly as before, to continue working with the tables we defined in the previous notebook, we can either redefine the classes for each table `Mouse`, `Session`, `Neuron` and populate them. Or, again for your convenience, we can import them from the `tutorial_pipeline.ephys_cell_activity` module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tutorial_pipeline.ephys_cell_activity import schema, Mouse, Session, Neuron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Neuron()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ephys_cell_activity.py` also fills each table with data and import the neuron activity data from `.npy` files to make sure we are all on the same page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computations in data pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have successfully imported all data we have into our pipeline, it's time for us to start analyzing them! \n",
    "\n",
    "When you perform computations in the DataJoint data pipeline, you focus and design tables in terms of **what** is it that you are computing rather than the **how**. You should think in terms of the \"things\" that you are computing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's say that we want to compute the satististics, such as mean, standard deviation, and maximum value of each of our neuron's activity traces. Hence we want to compute the neuron's **activity statistics** for each neuron!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the new \"thing\" or entity here is `ActivityStatistics`, where each entry corresponds to the statistics of a single `Neuron`. Let's start designing the table, paying special attention to the dependencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistics of neuron activities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we create the table to store the statistics, let's think about how we might go about computing interesting statistics for a single neuron.\n",
    "\n",
    "Let's start by fetching one neuron to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = Neuron().fetch('KEY')\n",
    "\n",
    "# pick one key\n",
    "key = keys[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Neuron() & key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets go ahead and grab the `activity` data stored as numpy array. As we learned in the last session, we can `fetch` it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity = (Neuron() & key).fetch('activity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a bit subtle, but `fetch` returns a NumPy array of the attribute, even if the attribute contains a NumPy array. So here, we actually got a NumPy array of NumPy array. We can of course just index into it,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but if we knew that there was only one item, we can use `fetch1` instead to save some trouble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity = (Neuron() & key).fetch1('activity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compute some statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - compute the mean of the activity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - compute the standard deviation of the activity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - compute the maximum of the activity\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a good idea on how to:\n",
    "1. fetch the activity of a neuron knowing its primary key, and\n",
    "2. compute interesting statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Armed with this knowledge, let's go ahead and define the table for `ActivityStatistics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining `ActivityStatistics` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and work out the definition of the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class ActivityStatistics(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> Neuron\n",
    "    ---\n",
    "    mean: float    # mean activity\n",
    "    stdev: float   # standard deviation of activity\n",
    "    max: float     # maximum activity\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you notice that we are now inheriting from `dj.Computed`?  `Computed` is yet another table tier that signifies that **the entries of this table are computed using data in other tables**. `Computed` tables are represented as red circles in the Diagram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like the `Imported` tables, `Computed` tables make use of the same `make` and `populate` logic for defining new entries in the table. Let's go ahead and implement `make` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - complete the `make` method\n",
    "\n",
    "@schema\n",
    "class ActivityStatistics(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> Neuron\n",
    "    ---\n",
    "    mean: float    # mean activity\n",
    "    stdev: float   # standard deviation of activity\n",
    "    max: float     # maximum activity\n",
    "    \"\"\"\n",
    "    \n",
    "    def make(self, key):\n",
    "        activity = (Neuron() & key).fetch1('activity')    # fetch activity as NumPy array\n",
    "\n",
    "        # compute various statistics on activity\n",
    "        key['mean'] =                 # compute mean\n",
    "        key['stdev'] =                # compute standard deviation\n",
    "        key['max'] =                  # compute max\n",
    "        self.insert1(key)\n",
    "        print('Computed statistics for mouse_id {mouse_id} session_date {session_date}'.format(**key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and populate the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ActivityStatistics.populate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ActivityStatistics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila!! You have computed statistics for each neuron activity!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spike detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's go ahead and tackle a more challenging computation. While having raw neural traces in itself can be quite interesting, nothing is as exciting as spikes! Let's take a look at the neurons activities and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all keys\n",
    "keys = Neuron.fetch('KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch all activities - returned as NumPy array of NumPy arrays\n",
    "activities = (Neuron & keys).fetch('activity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, len(activities), figsize=(16, 4))\n",
    "for activity, ax in zip(activities, axs.ravel()):\n",
    "    ax.plot(activity)\n",
    "    ax.set_xlabel('Time')\n",
    "    ax.set_ylabel('Activity')\n",
    "\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now focus on one trace instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "key = keys[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity = (Neuron & key).fetch1('activity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(activity)\n",
    "plt.xlabel('Time')\n",
    "plt.ylabel('Activity')\n",
    "plt.xlim([0, 300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps we can use threshold to detect when a spike occurs. Threshold of `0.5` may be a good start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 0.5\n",
    "\n",
    "# find activity above threshold\n",
    "above_thrs = (activity > threshold).astype(int)   \n",
    "\n",
    "plt.plot(activity)\n",
    "plt.plot(above_thrs)\n",
    "plt.xlabel('Time')\n",
    "plt.xlim([0, 300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to find out **when** it crossed the threshold. That is, find time bins where `above_thrs` goes from 0 (`False`) to 1 (`True`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rising = (np.diff(above_thrs) > 0).astype(int)   # find rising edge of crossing threshold\n",
    "spikes = np.hstack((0, rising))    # prepend 0 to account for shortening due to np.diff\n",
    "\n",
    "plt.plot(activity)\n",
    "plt.plot(above_thrs)\n",
    "plt.plot(np.where(spikes>0), 1,  'ro'); # plot only spike points\n",
    "plt.xlabel('Time')\n",
    "plt.xlim([0, 300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's also compute the spike counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = spikes.sum()   # compute total spike counts\n",
    "count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is our complete spike detection algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - try different values of threshold!\n",
    "\n",
    "threshold =     # enter different threshold values here\n",
    "\n",
    "# find activity above threshold\n",
    "above_thrs = (activity > threshold).astype(int) \n",
    "\n",
    "rising = (np.diff(above_thrs) > 0).astype(int)   # find rising edge of crossing threshold\n",
    "spikes = np.hstack((0, rising))    # prepend 0 to account for shortening due to np.diff\n",
    "\n",
    "count = spikes.sum()   # compute total spike counts\n",
    "\n",
    "\n",
    "plt.plot(activity)\n",
    "plt.plot(above_thrs)\n",
    "plt.plot(np.where(spikes>0), 1,  'ro'); # plot only spike points\n",
    "plt.xlabel('Time')\n",
    "plt.title('Total spike counts: {}'.format(count));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now notice that the exact spikes you detect depend on the value of the `threshold`. Therefore, the `threshold` is a parameter for our spike detection computation. Rather than fixing the value of the threshold, we might want to try different values and see what works well.\n",
    "\n",
    "In other words, you want to compute `Spikes` for a **combination** of `Neuron`s and different `threshold` values. To do this while still taking advantage of the `make` and `populate` logic, you would want to define a table to house parameters for spike detection in a `Lookup` table!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter `Lookup` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define `SpikeDetectionParam` table to hold different parameter configuration for our spike detection algorithm. We are going to define this table as a `Lookup` table, rather than a `Manual` table. By now, you know that `Lookup` must be yet another **table tier** in DataJoint. `Lookup` tables are depicted by gray boxes in the Diagram.\n",
    "\n",
    "This tier indicates that the table will contain information:\n",
    "* that will be referenced by other tables\n",
    "* that doesn't change much - usually contains a few pre-known entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class SpikeDetectionParam(dj.Lookup):\n",
    "    definition = \"\"\"\n",
    "    sdp_id: int      # unique id for spike detection parameter set\n",
    "    ---\n",
    "    threshold: float   # threshold for spike detection\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining `Spikes` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's take everything together and define the `Spikes` table. Here each entry of the table will be *a set of spikes* for a single neuron, using a particular value of the `SpikeDetectionParam`. In other words, any particular entry of the `Spikes` table is determined by **a combination of a neuron and spike detection parameters**.\n",
    "\n",
    "We capture this by depending on both `Neuron` and `SpikeDetectionParam`. For each spike set, we want to store the detected spikes and the total number of spikes. The table definition will look something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Spikes(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> Neuron\n",
    "    -> SpikeDetectionParam\n",
    "    ---\n",
    "    spikes: longblob     # detected spikes\n",
    "    count: int           # total number of detected spikes\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the `Waveform` part table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would also like to store the waveform at each detected spike, for a particular neuron. For simplicity, let's define the waveform to be 40 sample points before and after the onset of the detected spike. \n",
    "To accomplish this, we will define a `dj.Part` table `Waveform` which will contain the waveform per spike. The table definition is something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Spikes(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> Neuron\n",
    "    -> SpikeDetectionParam\n",
    "    ---\n",
    "    spikes: longblob     # detected spikes\n",
    "    count: int           # total number of detected spikes\n",
    "    \"\"\"\n",
    "    \n",
    "    class Waveform(dj.Part):\n",
    "        definition = \"\"\"\n",
    "        -> master\n",
    "        spike_id: int\n",
    "        ---\n",
    "        waveform: longblob  # waveform extracted from this spike\n",
    "        \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `-> master` syntax denotes that the `Waveform` part table is foreign key constrained by `Spike` table - i.e. the master table. The master table drives the ***populate*** logic, and the content of the part table is generally ingested together with the content of the master table, all in one step (i.e. one `make()` call)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Diagram, we see that `Spikes` is a computed table (red circle) that depends on **both Neuron and SpikeDetectionParam**. Finally, let's go ahead and implement the `make` method for the `Spikes` table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Spikes(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> Neuron\n",
    "    -> SpikeDetectionParam\n",
    "    ---\n",
    "    spikes: longblob     # detected spikes\n",
    "    count: int           # total number of detected spikes\n",
    "    \"\"\"\n",
    "    \n",
    "    class Waveform(dj.Part):\n",
    "        definition = \"\"\"\n",
    "        -> master\n",
    "        spike_id: int\n",
    "        ---\n",
    "        waveform: longblob  # waveform extracted from this spike\n",
    "        \"\"\"\n",
    "        \n",
    "    def make(self, key):\n",
    "        print('Populating for: ', key)\n",
    "\n",
    "        activity = (Neuron() & key).fetch1('activity')\n",
    "        threshold = (SpikeDetectionParam() & key).fetch1('threshold')\n",
    "\n",
    "        above_thrs = (activity > threshold).astype(int)   # find activity above threshold\n",
    "        rising = (np.diff(above_thrs) > 0).astype(int)   # find rising edge of crossing threshold\n",
    "        spikes = np.hstack((0, rising))    # prepend 0 to account for shortening due to np.diff\n",
    "\n",
    "        count = spikes.sum()   # compute total spike counts\n",
    "        print('Detected {} spikes!\\n'.format(count))\n",
    "\n",
    "        # create and insert a new dictionary containing `key` and additionally `spikes` and `count`\n",
    "        self.insert1(dict(key, spikes=spikes, count=count))  \n",
    "        \n",
    "        # extract waveform for the `Waveform` part-table\n",
    "        before_spk, after_spk = 40, 40  # extract 40 sample points before and after a spike as the waveform\n",
    "        for spk_id, spk in enumerate(np.where(spikes==1)[0]):\n",
    "            \n",
    "            # For simplicity, skip the spikes too close to the beginning or the end\n",
    "            if spk - before_spk < 0 or spk + after_spk > len(activity) + 1: \n",
    "                continue\n",
    "                \n",
    "            wf = activity[spk - before_spk: spk + after_spk] \n",
    "            \n",
    "            # create and insert a new dictionary containing `key` and additionally `spike_id` and `waveform`\n",
    "            self.Waveform.insert1(dict(key, spike_id=spk_id, waveform=wf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The implementation of the spike detection is pretty much what we had above, except that we now fetch the value of `threshold` from the `SpikeDetectionParam` table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the `Spikes` table, we see that it indeed inherits the primary key attributes from **both Neuron (`mouse_id`, `session_date`) and SpikeDetectionParam (`sdp_id`)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Spikes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Populating `Spikes` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to populate! When we call `populate` on `Spikes`, DataJoint will automatically call `make` on **every valid combination of the parent tables - Neuron and SpikeDetectionParam**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the Spikes table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm... `populate` doesn't seem to be doing anything... What could be the cause?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at `SpikeDetectionParam` reveals the issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's right! We have not added a detection parameter set yet. Let's go ahead and add one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam.insert1((0, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we should really be ready to perform the computation..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the Spikes table for real!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Spikes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and we now have spike detection running!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the waveform(s) from the Waveform part table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - Now, build a query for the waveforms from mouse 100, session on \"2017-05-25\", with detection param 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - try fetching all the waveforms\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - and plot the average waveform\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trying out other parameter values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how different thresholds affect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam.insert1((1, 0.9))  # add another threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the \"missing\" entry in Spikes table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Spikes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the results of spike detection under different parameter settings can live happily next to each other, without any confusion as to what is what."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting entries \"upstream\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's say that we decided that we don't like the first spike threshold of `0.5`. While there is really nothing wrong keeping those results around, you might decide that you'd rather delete all computations performed with that threshold to keep your tables clean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While you can restrict `Spikes` table to the specific parameter id (i.e. `sdp_id = 0`) and delete the entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(Spikes & 'sdp_id = 0').delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can simply delete the unwanted parameter from the `SpikeDetectionParam` table, and let DataJoint cascade the deletion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpikeDetectionParam() & 'sdp_id = 0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(SpikeDetectionParam() & 'sdp_id = 0').delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Spikes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully extended your pipeline with a table to represent recorded data (`Neuron` as `Imported` table), tables that performs and represents computation results (`ActivityStatistics` and `Spikes` as `Computed` tables) and a table to hold computation parameters (`SpikeDetectionParam` as `Lookup` table)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our pipeline is still fairly simple but completely capable of handling analysis!"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:light"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}