pyspark-MLlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Spark MLlib:</h1>\n",
    "\n",
    "Go to the Apache Spark page at https://spark.apache.org/. \n",
    "<UL>\n",
    "<LI>Documentation>Latest Release>Programming Guides> MLlib provides available algorithms are listed here, e.g., classification, regression, clustering.\n",
    "\n",
    "**NB.** Extracting features algorithms are also available, e.g., vectorIndexer, PCA.  </LI>\n",
    "    \n",
    "    \n",
    "<LI>\n",
    "Documentation>Latest Release>API Docs>Python, provides detail information about the functions available in the MLlib package.</LI>\n",
    " </UL>\n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Input format:</h2>\n",
    "\n",
    "The input dataset should have only one/two columns. One for 'label' and one for all 'features'. See section 'Formatting MLlib input' for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression Example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName('mllib').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In this example, the input data is already in the correct format (i.e., has two columns of features and label)\n",
    "We can call the below read function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data = spark.read.format('libsvm').load('sample_linear_regression_data.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------------+\n",
      "|              label|            features|\n",
      "+-------------------+--------------------+\n",
      "| -9.490009878824548|(10,[0,1,2,3,4,5,...|\n",
      "| 0.2577820163584905|(10,[0,1,2,3,4,5,...|\n",
      "| -4.438869807456516|(10,[0,1,2,3,4,5,...|\n",
      "|-19.782762789614537|(10,[0,1,2,3,4,5,...|\n",
      "| -7.966593841555266|(10,[0,1,2,3,4,5,...|\n",
      "| -7.896274316726144|(10,[0,1,2,3,4,5,...|\n",
      "| -8.464803554195287|(10,[0,1,2,3,4,5,...|\n",
      "| 2.1214592666251364|(10,[0,1,2,3,4,5,...|\n",
      "| 1.0720117616524107|(10,[0,1,2,3,4,5,...|\n",
      "|-13.772441561702871|(10,[0,1,2,3,4,5,...|\n",
      "| -5.082010756207233|(10,[0,1,2,3,4,5,...|\n",
      "|  7.887786536531237|(10,[0,1,2,3,4,5,...|\n",
      "| 14.323146365332388|(10,[0,1,2,3,4,5,...|\n",
      "|-20.057482615789212|(10,[0,1,2,3,4,5,...|\n",
      "|-0.8995693247765151|(10,[0,1,2,3,4,5,...|\n",
      "| -19.16829262296376|(10,[0,1,2,3,4,5,...|\n",
      "|  5.601801561245534|(10,[0,1,2,3,4,5,...|\n",
      "|-3.2256352187273354|(10,[0,1,2,3,4,5,...|\n",
      "| 1.5299675726687754|(10,[0,1,2,3,4,5,...|\n",
      "| -0.250102447941961|(10,[0,1,2,3,4,5,...|\n",
      "+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "all_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, test_data = all_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|              label|\n",
      "+-------+-------------------+\n",
      "|  count|                368|\n",
      "|   mean| 0.3449742030841025|\n",
      "| stddev| 10.002903086492546|\n",
      "|    min|-28.571478869743427|\n",
      "|    max|  27.78383192005107|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an instance of Linear Regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# featuresCol and labelCol are the feature and label col name, respectively.\n",
    "lr = LinearRegression(featuresCol='features',labelCol='label',predictionCol='prediction',regParam=0.0, elasticNetParam=0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NB. Regularizing the Linear Regression model:\n",
    "\n",
    "***Ordinary Least square loss:***  regParam = 0  and   elasticNetParam = 0\n",
    "\n",
    "***Ridge (L2):***                  regParam > 0  and   elasticNetParam = 0\n",
    "\n",
    "***LASSO (L1):***                  regParam > 0  and   elasticNetParam = 1\n",
    "\n",
    "***Elastic net (L1+L2):***         regParam > 0  and   elasticNetParam &isin; (0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit the model (train the model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "lrModel = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DenseVector([0.304, 1.1027, -0.9633, 2.2453, -0.0762, 1.2084, 0.2702, -0.7544, -0.6322, 0.2282])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrModel.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27827499500963354"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrModel.intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_summary = lrModel.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.02963794930112995"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_summary.r2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.840158316048619"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_summary.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the model (test the model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11.07457218916498"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "| -27.95456795978699|\n",
      "|-24.036702338156424|\n",
      "|-26.787671379819635|\n",
      "| -20.47985032261661|\n",
      "|  -19.3821758322171|\n",
      "| -19.32343899120233|\n",
      "| -16.61414553142395|\n",
      "|-16.922863567109914|\n",
      "|-15.880851617675004|\n",
      "|-17.404932746758213|\n",
      "|-16.240651231038118|\n",
      "|-19.897900090192714|\n",
      "|-14.869980008755615|\n",
      "| -17.20311403285779|\n",
      "|-15.339977390253512|\n",
      "|-14.555161340950864|\n",
      "|-14.469243135335658|\n",
      "|-16.718023956084227|\n",
      "| -17.51813627408178|\n",
      "|-13.301707287750471|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.residuals.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model (transform the unlabeled data)\n",
    "At deployment time, data is unlabeled, so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = lrModel.transform(test_data.select('features'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|            features|          prediction|\n",
      "+--------------------+--------------------+\n",
      "|(10,[0,1,2,3,4,5,...|   1.149084531303916|\n",
      "|(10,[0,1,2,3,4,5,...|   0.549262217219912|\n",
      "|(10,[0,1,2,3,4,5,...|  3.8378454436235607|\n",
      "|(10,[0,1,2,3,4,5,...|  0.5952895483431853|\n",
      "|(10,[0,1,2,3,4,5,...|-0.49081520585130517|\n",
      "|(10,[0,1,2,3,4,5,...|  1.0482254251977003|\n",
      "|(10,[0,1,2,3,4,5,...| -1.1894806572405667|\n",
      "|(10,[0,1,2,3,4,5,...| -0.4038571655660324|\n",
      "|(10,[0,1,2,3,4,5,...| -1.1456406465345437|\n",
      "|(10,[0,1,2,3,4,5,...|  0.7127257254471082|\n",
      "|(10,[0,1,2,3,4,5,...| 0.15499219001662898|\n",
      "|(10,[0,1,2,3,4,5,...|   4.117215057569414|\n",
      "|(10,[0,1,2,3,4,5,...| -0.5674047846756034|\n",
      "|(10,[0,1,2,3,4,5,...|  1.8683465529354497|\n",
      "|(10,[0,1,2,3,4,5,...|  0.2834944157110795|\n",
      "|(10,[0,1,2,3,4,5,...|-0.26699156880032465|\n",
      "|(10,[0,1,2,3,4,5,...| -0.2935151175954693|\n",
      "|(10,[0,1,2,3,4,5,...|   2.389045447008785|\n",
      "|(10,[0,1,2,3,4,5,...|  3.7456947123789095|\n",
      "|(10,[0,1,2,3,4,5,...| 0.26177922364585543|\n",
      "+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#predictions.join(test_data, predictions.features == test_data.features, 'inner').head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression evaluation metrics:\n",
    "\n",
    "***Mean Absolute Error (MAE)***:\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/gif.latex?\\frac{1}{n}\\sum_{i=1}^{n}\\left&space;|&space;y_{i}-\\hat{y}_{i}&space;\\right&space;|\" title=\"\\frac{1}{n}\\sum_{i=1}^{n}\\left | y_{i}-\\hat{y}_{i} \\right |\" />\n",
    "\n",
    "***Mean Squared Error (MSE)***: Larger errors are noted more than with MAE (we are computing square of errors, so larger errors become even larger!)\n",
    "Cons: MSE is not in the same unit as y any more.\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/gif.latex?\\frac{1}{n}\\sum_{i=1}^{n}(y_{i}-\\hat{y}_{i})^2\" title=\"\\frac{1}{n}\\sum_{i=1}^{n}(y_{i}-\\hat{y}_{i})^2\" />\n",
    "\n",
    "***Root Mean Squared Error (RMSE)***: Root of MSE, so it has the same unit as y. More popular than the above error metrics.\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/gif.latex?\\sqrt{\\frac{1}{n}\\sum_{i=1}^{n}(y_{i}-\\hat{y}_{i})^2&space;}\" title=\"\\sqrt{\\frac{1}{n}\\sum_{i=1}^{n}(y_{i}-\\hat{y}_{i})^2 }\" />\n",
    "\n",
    "***R-Squared***: It is not quite an error metric! It is more of a statistical measure of your regression model. A measure of how much variance your model accounts for. Between [0-1]\n",
    "\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/gif.latex?\\large&space;R^2=1-\\frac{\\sum_{i=1}^{n}(y_i-\\hat{y}_i)^2}{\\sum_{i=1}^{n}(y_i-\\bar{y})^2}\" title=\"\\large R^2=1-\\frac{\\sum_{i=1}^{n}(y_i-\\hat{y}_i)^2}{\\sum_{i=1}^{n}(y_i-\\bar{y})^2}\" />\n",
    "\n",
    "***Adjusted R-Squared***: Adjusted R-Squared takes into account the number of independent variables you employ in your model and can help indicate if a variable is useless or not.\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/gif.latex?\\large&space;Adjusted\\&space;R^2=1-\\frac{(n-1)}{[n-(k&plus;1)]}(1-R^2)\" title=\"\\large Adjusted\\ R^2=1-\\frac{(n-1)}{[n-(k+1)]}(1-R^2)\" />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Formatting MLlib input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = spark.read.csv('Ecommerce_Customers.csv', inferSchema=True, header = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mstephenson@fernandez.com\n",
      "835 Frank TunnelWrightmouth, MI 82180-9605\n",
      "Violet\n",
      "34.49726772511229\n",
      "12.65565114916675\n",
      "39.57766801952616\n",
      "4.0826206329529615\n",
      "587.9510539684005\n"
     ]
    }
   ],
   "source": [
    "for item in data.head(1)[0]:\n",
    "    print (item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set feature column:\n",
    "We want to only use 'Avg Session Length','Time on App','Time on Website', and 'Length of Membership' columns as features. \n",
    "\n",
    "**VectorAssembler** sets what columns are our features and assign a name to the feature column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a VectorAssembler instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(inputCols=['Avg Session Length','Time on App','Time on Website',\n",
    "                                       'Length of Membership'],\n",
    "                            outputCol = 'features')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the VectorAssembler instance to ransform the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = assembler.transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------------------+\n",
      "|            features|Yearly Amount Spent|\n",
      "+--------------------+-------------------+\n",
      "|[34.4972677251122...|  587.9510539684005|\n",
      "|[31.9262720263601...|  392.2049334443264|\n",
      "|[33.0009147556426...| 487.54750486747207|\n",
      "|[34.3055566297555...|  581.8523440352177|\n",
      "|[33.3306725236463...|  599.4060920457634|\n",
      "|[33.8710378793419...|   637.102447915074|\n",
      "|[32.0215955013870...|  521.5721747578274|\n",
      "|[32.7391429383803...|  549.9041461052942|\n",
      "|[33.9877728956856...|  570.2004089636196|\n",
      "|[31.9365486184489...|  427.1993848953282|\n",
      "|[33.9925727749537...|  492.6060127179966|\n",
      "|[33.8793608248049...|  522.3374046069357|\n",
      "|[29.5324289670579...|  408.6403510726275|\n",
      "|[33.1903340437226...|  573.4158673313865|\n",
      "|[32.3879758531538...|  470.4527333009554|\n",
      "|[30.7377203726281...|  461.7807421962299|\n",
      "|[32.1253868972878...| 457.84769594494855|\n",
      "|[32.3388993230671...| 407.70454754954415|\n",
      "|[32.1878120459321...|  452.3156754800354|\n",
      "|[32.6178560628234...|   605.061038804892|\n",
      "+--------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_data = output.select('features','Yearly Amount Spent')\n",
    "final_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "#split to train-test\n",
    "train_data , test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                346|\n",
      "|   mean| 499.89733812560945|\n",
      "| stddev|  81.56403887266724|\n",
      "|    min| 256.67058229005585|\n",
      "|    max|  765.5184619388373|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                154|\n",
      "|   mean|  498.0035073885354|\n",
      "| stddev|  74.25293922502175|\n",
      "|    min|  275.9184206503857|\n",
      "|    max|  712.3963268096637|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LinearRegression(labelCol = 'Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# StringIndexer vs. OneHotEncoder:\n",
    "\n",
    "##### StringIndexer:\n",
    "StringIndexer (equivalent to LabelEncoder in sklearn) assigns index [0-num_labels] to each string.\n",
    "The problem here is, since there are different numbers in the same column, the model will misunderstand the data to be in some kind of order. But this is not the case at all. To overcome this problem, we use One Hot Encoder.\n",
    "\n",
    "##### OneHotEncoder:\n",
    "**Note:** pyspark OneHotEncoder is different from scikit-learn’s OneHotEncoder.\n",
    "\n",
    "A one-hot encoder that maps a column of **category indices** to a column of binary vectors. So, we should encode categorical features to categorical indices using **StringIndexer** first then apply pyspark OneHotEncoder.\n",
    "\n",
    "**Note:** The last category is not included by default (configurable via dropLast), because it makes the vector entries sum up to one, and hence linearly dependent. So an input value of 4.0 maps to [0.0, 0.0, 0.0, 0.0].\n",
    "\n",
    "\n",
    "\n",
    "Check Linear_Regression_Consulting_Project.ipynb for an example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GridSearch and CrossValidation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "from pyspark.ml.evaluation import RegressionEvaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see an example check Linear_Regression_Consulting_Project.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluator: \n",
    "Once we applied model.transform() on test data, we can evaluate the predicted values using an **Evaluator** instance.\n",
    "We can define an evaluator instance and then call the available metrics the predicted DataFrame.\n",
    "\n",
    "#### Different evaluators exist depending on our algorithm:\n",
    "\n",
    "\n",
    "***RegressionEvaluator***: rmse (default), mse, r2, mae, var(explained variance).\n",
    "\n",
    "***BinaryClassificationEvaluator***: \"areaUnderROC\" (default), \"areaUnderPR\"\n",
    "\n",
    "***MulticlassClassificationEvaluator***: \"f1\" (default), \"accuracy\", \"weightedPrecision\", \"weightedRecall\", \"weightedTruePositiveRate\", \"weightedFalsePositiveRate\", \"weightedFMeasure\", \"truePositiveRateByLabel\", \"falsePositiveRateByLabel\", \"precisionByLabel\", \"recallByLabel\", \"fMeasureByLabel\", \"logLoss\", \"hammingLoss\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check an example, check Linear_Regression_Consulting_Project.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipelines:\n",
    "Define multiple stages as a pipeline. Then, run the pipeline afterward. \n",
    "\n",
    "To see an example please check Logistic_Regression_Consulting_Project.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalizing features:\n",
    "\n",
    "Examples of normalizers:\n",
    "\n",
    "**StandardScaler:** Normalizing each feature to have unit standard deviation and/or zero mean\n",
    "\n",
    "**MinMaxScaler:** Rescaling each feature to a specific range (default [0, 1]). It takes min and max parameters.\n",
    "\n",
    "Other algorithms can be found at this <a href=\"https://spark.apache.org/docs/latest/ml-features.html#normalizer\"> link </a>.\n",
    "\n",
    "NB. Input should be in libsvm format (two columns of 'features' and 'label')\n",
    "See an example in Logistic_Regression_Consulting_Project.ipynb\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}