Workbook.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OpenCV - used for image loading and processing\n",
    "import cv2\n",
    "# Numpy - used for calculations\n",
    "import numpy as np\n",
    "# Glob - for converting file lists to arrays\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a file list using glob\n",
    "\n",
    "With glob you can pass paths to datasets and use wildcards to return a list of all the files into a useful array. Make an array containing the file names of all the galaxy images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read an image using OpenCV2\n",
    "\n",
    "To start this task you will need to load images into python arrays. One way of doing this is using OpenCV. Here is a link to the OpenCV introduction to image manipulation:\n",
    "\n",
    "https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_gui/py_image_display/py_image_display.html\n",
    "\n",
    "Load one of the images from the file name array you just created (in greyscale)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the image using matplotlib\n",
    "\n",
    "We can use OpenCV to plot images, however, it can be rather cumbersome for our needs. I would suggest using matplotlib to plot the image. Use matplotlib to display the image you have loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Log the flux\n",
    "\n",
    "In astronomy we often rescale the flux of an image to show details that would normally be washed out by the high intensity regions. One popular way to do this is to instead take the logarithm of the flux in each pixel. Rescale the image by taking the logarithm of each pixel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply a gaussian blur\n",
    "\n",
    "You might notice that this image has some background flux. To counteract this we can apply a random gaussian blur to distribute the flux between pixels more evenly. Apply a gaussian blur to the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Any further preprocessing\n",
    "\n",
    "Consider any further preprocessing steps you could make to increase the usefulness of the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine the previous steps into a single function\n",
    "\n",
    "Now you have the makings of a preprocessing pipeline. Combine the previous steps into a single function that takes a file name as an input and returns a log scaled and blurred image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a fuction to calculate features\n",
    "\n",
    "You can now load and preprocess images. The next step is to consider possible features you might want to extract from the images. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Circular mask\n",
    "\n",
    "Before we can do anything we need to create a mask that we can use for selecting groups of pixels. Create a function that takes a shape array and a radius and returns a boolean array of the given shape with indexes inside the radius equal to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gini coefficient\n",
    "\n",
    "As mentioned an interesting variable to consider is the Gini coefficient. Create a function that takes an array as an input and returns the Gini coefficient of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gini profile\n",
    "\n",
    "You can now calculate the Gini coefficient for a given array. Now using the circlar mask, calculate the Gini coefficient at different radii and produce a plot of radius versus Gini coefficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gini gradient\n",
    "\n",
    "The next thing to consider is the derivative of this profile. Calculate the gradient of the profile you just produced and display it using matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating features\n",
    "\n",
    "Using the Gini profiles you have just created create a function that takes a file name as an input and outputs some features of the Gini profle. For example: the minimum values of the Gini gradient, the radius of the mask at this point, the maximum gini coeffient, the radius at this point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a feature dataset\n",
    "\n",
    "Now modify the function to write the values to a file and loop over all images in the dataset. Make sure you include the file name as one of your outputs (you will need this!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the output into a pandas dataframe\n",
    "\n",
    "Now you have created your dataset, load it as a dataframe using pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised clustering\n",
    "\n",
    "Start by trying to use unsupervised clustering algorithms to try to seperate barred and unbarred galaxies using your features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a dataframe without the file names\n",
    "\n",
    "First, you will need to drop the file names from the dataframe. Make a new dataframe which only contains the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standardize features\n",
    "\n",
    "Because the features all have different scales you will need to standardize them, otherwise the fitting will be biased. Use the inbuilt scikit-learn StandardScaler function to standardize your features.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pick a clustering algorthm\n",
    "\n",
    "Go ahead and pick a clustering algorthm to use. You might want to do some plotting to see how which ones might be best to use. \n",
    "\n",
    "You can see a comparison of different methods on the scikit-learn website:\n",
    "https://scikit-learn.org/stable/modules/clustering.html\n",
    "\n",
    "Once you have picked your algorthm(s), import and initialize them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spectral \n",
    "\n",
    "https://scikit-learn.org/stable/modules/clustering.html#spectral-clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Birch \n",
    "\n",
    "https://scikit-learn.org/stable/modules/clustering.html#birch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GMM <-\n",
    "\n",
    "https://scikit-learn.org/stable/modules/mixture.html#mixture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit using your selected classifier\n",
    "\n",
    "Now go ahead and fit your chosen algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use your model to predict the labels\n",
    "\n",
    "Use your model to predict the different classes from your feature space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the clusters\n",
    "\n",
    "Use matplotlib to compare the clusters your algorithm has identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare to human classification\n",
    "\n",
    "In this task we actually have human classifications for all the galaxies. Let's compare the clusters that you have found to the actual distribution of barred and unbarred galaxies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the labeled dataframe\n",
    "\n",
    "The barred_unbarred.csv file contains the file names and flags for each file, 1 denotes barred and 0 unbarred. Load the csv file into a new pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change the index to the file name\n",
    "\n",
    "You will need to join the two pandas dataframes. To do this first you need to change the index of each dataframe to the file name column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Join the two dataframes\n",
    "\n",
    "Next you can simply join the two dataframes into one new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the real distribution\n",
    "\n",
    "You now have your features and the actual label for the galaxy (barred/unbarred). Now using matplotlib to compare the distribution of barred and unbarred galaxies in your feature space and compare the the clusters you found before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised machine learning\n",
    "\n",
    "Clearly for this take unsupervised machine learning can only get us so far. But because we have the actual labels we can go ahead and use supervised machine learning instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop any NaNs\n",
    "\n",
    "Start by dropping all the NaNs in your dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seperate observations from features\n",
    "\n",
    "Now seperate your dataframe into observations (features) and targets (actual labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standardize features \n",
    "\n",
    "Again you will need to standardize your features to avoid bias.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data into training and test data\n",
    "\n",
    "Next split your datasets into a training set and a test set. You can hard code this if you want, but I would suggest using the scikit-learn function train_test_split\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression\n",
    "\n",
    "Let's start with a vary widely used algorithm, logistic regression. Initialize, train and evaluate a logistic regression model.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SGDC\n",
    "\n",
    "Compare to annother widely used classifier, Stochastic Gradient Decient Classifier.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Suport vector machine\n",
    "\n",
    "Next, compare to a suport vector machine approach.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/svm.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick one more\n",
    "\n",
    "Pick a further classifier, train a model and compare to LG, SGDC and SVM.\n",
    "\n",
    "https://scikit-learn.org/stable/supervised_learning.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a GridSearch pipeline\n",
    "\n",
    "With machine learning techniques come hyperparameters. These parameters need to be tuned to get the optimal model. Create a grid search pipeline to explore teh hyperparameter space and find the optimal model.\n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Futher model evaulation\n",
    "\n",
    "One of the most important steps in machine learning is evaluation. Does a model really do a good job? Is it overfit? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a confusion matrix\n",
    "\n",
    "One quick tool is the confusion matrix. This is often used for spare feature spaces, but can still be a useful tool in this project. Create a confusion matrix for your best model.\n",
    "\n",
    "https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "Now you have a way to create models for the feature space you have generated. As is always the case with machine learning, the quality of the feature space is the most important part of the process. Go back and try to find better features and get better and better models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "174px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}