catboost_bring_your_own_container_training_with_dvc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Training a CatBoost regression model with data from DVC\n",
    "\n",
    "With Amazon SageMaker, you can package your own algorithms that can than be trained and deployed in the SageMaker environment. This notebook will guide you through an example that shows you how to build a Docker container for SageMaker and use it for training and inference.\n",
    "\n",
    "By packaging an algorithm in a container, you can bring almost any code to the Amazon SageMaker environment, regardless of programming language, environment, framework, or dependencies. \n",
    "\n",
    "### California Housing dataset\n",
    "We use the California Housing dataset, present in [Scikit-Learn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html). \n",
    "\n",
    "The California Housing dataset was originally published in:\n",
    "\n",
    "Pace, R. Kelley, and Ronald Barry. \"Sparse spatial auto-regressions.\" Statistics & Probability Letters 33.3 (1997): 291-297.\n",
    "\n",
    "\n",
    "### DVC\n",
    "DVC is built to make ML models shareable and reproducible. It is designed to handle large files, data sets, machine learning models, and metrics as well as code.\n",
    "\n",
    "[DVC Official Site](https://dvc.org/)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Part 1: Packaging and Uploading your Algorithm for use with Amazon SageMaker\n",
    "\n",
    "### An overview of Docker\n",
    "\n",
    "If you're familiar with Docker already, you can skip ahead to the next section.\n",
    "\n",
    "For many data scientists, Docker containers are a new concept, but they are not difficult, as you'll see here. \n",
    "\n",
    "Docker provides a simple way to package arbitrary code into an _image_ that is totally self-contained. Once you have an image, you can use Docker to run a _container_ based on that image. Running a container is just like running a program on the machine except that the container creates a fully self-contained environment for the program to run. Containers are isolated from each other and from the host environment, so the way you set up your program is the way it runs, no matter where you run it.\n",
    "\n",
    "Docker is more powerful than environment managers like conda or virtualenv because (a) it is completely language independent and (b) it comprises your whole operating environment, including startup commands, environment variable, etc.\n",
    "\n",
    "In some ways, a Docker container is like a virtual machine, but it is much lighter weight. For example, a program running in a container can start in less than a second and many containers can run on the same physical machine or virtual machine instance.\n",
    "\n",
    "Docker uses a simple file called a `Dockerfile` to specify how the image is assembled. We'll see an example of that below. You can build your Docker images based on Docker images built by yourself or others, which can simplify things quite a bit.\n",
    "\n",
    "Docker has become very popular in the programming and devops communities for its flexibility and well-defined specification of the code to be run. It is the underpinning of many services built in the past few years, such as [Amazon ECS].\n",
    "\n",
    "Amazon SageMaker uses Docker to allow users to train and deploy arbitrary algorithms.\n",
    "\n",
    "In Amazon SageMaker, Docker containers are invoked in a certain way for training and a slightly different way for hosting. The following sections outline how to build containers for the SageMaker environment.\n",
    "\n",
    "Some helpful links:\n",
    "\n",
    "* [Docker home page](http://www.docker.com)\n",
    "* [Getting started with Docker](https://docs.docker.com/get-started/)\n",
    "* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)\n",
    "* [`docker run` reference](https://docs.docker.com/engine/reference/run/)\n",
    "\n",
    "[Amazon ECS]: https://aws.amazon.com/ecs/\n",
    "\n",
    "### How Amazon SageMaker runs your Docker container\n",
    "\n",
    "Because you can run the same image in training or hosting, Amazon SageMaker runs your container with the argument `train` or `serve`. How your container processes this argument depends on the container:\n",
    "\n",
    "* In the example here, we don't define an `ENTRYPOINT` in the Dockerfile so Docker will run the command `train` at training time and `serve` at serving time. In this example, we define these as executable Python scripts, but they could be any program that we want to start in that environment.\n",
    "* If you specify a program as an `ENTRYPOINT` in the Dockerfile, that program will be run at startup and its first argument will be `train` or `serve`. The program can then look at that argument and decide what to do.\n",
    "* If you are building separate containers for training and hosting (or building only for one or the other), you can define a program as an `ENTRYPOINT` in the Dockerfile and ignore (or verify) the first argument passed in. \n",
    "\n",
    "#### Running your container during training\n",
    "\n",
    "When Amazon SageMaker runs training, your `train` script is run just like a regular Python program. A number of files are laid out for your use, under the `/opt/ml` directory:\n",
    "\n",
    "    /opt/ml\n",
    "    |-- input\n",
    "    |   |-- config\n",
    "    |   |   |-- hyperparameters.json\n",
    "    |   |   `-- resourceConfig.json\n",
    "    |   `-- data\n",
    "    |       `-- <channel_name>\n",
    "    |           `-- <input data>\n",
    "    |-- model\n",
    "    |   `-- <model files>\n",
    "    `-- output\n",
    "        `-- failure\n",
    "\n",
    "##### The input\n",
    "\n",
    "* `/opt/ml/input/config` contains information to control how your program runs. `hyperparameters.json` is a JSON-formatted dictionary of hyperparameter names to values. These values will always be strings, so you may need to convert them. `resourceConfig.json` is a JSON-formatted file that describes the network layout used for distributed training. Since scikit-learn doesn't support distributed training, we'll ignore it here.\n",
    "* `/opt/ml/input/data/<channel_name>/` (for File mode) contains the input data for that channel. The channels are created based on the call to CreateTrainingJob but it's generally important that channels match what the algorithm expects. The files for each channel will be copied from S3 to this directory, preserving the tree structure indicated by the S3 key structure. \n",
    "* `/opt/ml/input/data/<channel_name>_<epoch_number>` (for Pipe mode) is the pipe for a given epoch. Epochs start at zero and go up by one each time you read them. There is no limit to the number of epochs that you can run, but you must close each pipe before reading the next epoch.\n",
    "\n",
    "##### The output\n",
    "\n",
    "* `/opt/ml/model/` is the directory where you write the model that your algorithm generates. Your model can be in any format that you want. It can be a single file or a whole directory tree. SageMaker will package any files in this directory into a compressed tar archive file. This file will be available at the S3 location returned in the `DescribeTrainingJob` result.\n",
    "* `/opt/ml/output` is a directory where the algorithm can write a file `failure` that describes why the job failed. The contents of this file will be returned in the `FailureReason` field of the `DescribeTrainingJob` result. For jobs that succeed, there is no reason to write this file as it will be ignored.\n",
    "\n",
    "#### Running your container during hosting\n",
    "\n",
    "Hosting has a very different model than training because hosting is responding to inference requests that come in via HTTP. In this example, we use our recommended Python serving stack to provide robust and scalable serving of inference requests:\n",
    "\n",
    "![Request serving stack](stack.png)\n",
    "\n",
    "This stack is implemented in the sample code here and you can mostly just leave it alone. \n",
    "\n",
    "Amazon SageMaker uses two URLs in the container:\n",
    "\n",
    "* `/ping` will receive `GET` requests from the infrastructure. Your program returns 200 if the container is up and accepting requests.\n",
    "* `/invocations` is the endpoint that receives client inference `POST` requests. The format of the request and the response is up to the algorithm. If the client supplied `ContentType` and `Accept` headers, these will be passed in as well. \n",
    "\n",
    "The container will have the model files in the same place they were written during training:\n",
    "\n",
    "    /opt/ml\n",
    "    `-- model\n",
    "        `-- <model files>\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### The parts of the sample container\n",
    "\n",
    "In the `container` directory are all the components you need to package the sample algorithm for Amazon SageMager:\n",
    "\n",
    "    .\n",
    "    |-- Dockerfile\n",
    "    |-- build_and_push.sh\n",
    "    `-- catboost_regressor\n",
    "        |-- nginx.conf\n",
    "        |-- predictor.py\n",
    "        |-- serve\n",
    "        |-- train\n",
    "        `-- wsgi.py\n",
    "\n",
    "Let's discuss each of these in turn:\n",
    "\n",
    "* __`Dockerfile`__ describes how to build your Docker container image. More details below.\n",
    "* __`build_and_push.sh`__ is a script that uses the Dockerfile to build your container images and then pushes it to ECR. We'll invoke the commands directly later in this notebook, but you can just copy and run the script for your own algorithms.\n",
    "* __`catboost_regressor`__ is the directory which contains the files that will be installed in the container.\n",
    "* __`local_test`__ is a directory that shows how to test your new container on any computer that can run Docker, including an Amazon SageMaker notebook instance. Using this method, you can quickly iterate using small datasets to eliminate any structural bugs before you use the container with Amazon SageMaker. We'll walk through local testing later in this notebook.\n",
    "\n",
    "In this simple application, we only install five files in the container. You may only need that many or, if you have many supporting routines, you may wish to install more. These five show the standard structure of our Python containers, although you are free to choose a different toolset and therefore could have a different layout. If you're writing in a different programming language, you'll certainly have a different layout depending on the frameworks and tools you choose.\n",
    "\n",
    "The files that we'll put in the container are:\n",
    "\n",
    "* __`nginx.conf`__ is the configuration file for the nginx front-end. Generally, you should be able to take this file as-is.\n",
    "* __`predictor.py`__ is the program that actually implements the Flask web server and the decision tree predictions for this app. You'll want to customize the actual prediction parts to your application. Since this algorithm is simple, we do all the processing here in this file, but you may choose to have separate files for implementing your custom logic.\n",
    "* __`serve`__ is the program started when the container is started for hosting. It simply launches the gunicorn server which runs multiple instances of the Flask app defined in `predictor.py`. You should be able to take this file as-is.\n",
    "* __`train`__ is the program that is invoked when the container is run for training. You will modify this program to implement your training algorithm.\n",
    "* __`wsgi.py`__ is a small wrapper used to invoke the Flask app. You should be able to take this file as-is.\n",
    "\n",
    "In summary, the two files you will probably want to change for your application are `train` and `predictor.py`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!cat container/Dockerfile"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Building and registering the container"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "%%sh\n",
    "\n",
    "# The name of our algorithm\n",
    "algorithm_name=sagemaker-catboost-dvc\n",
    "\n",
    "cd container\n",
    "\n",
    "chmod +x catboost_regressor/train\n",
    "chmod +x catboost_regressor/serve\n",
    "\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "\n",
    "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n",
    "region=$(aws configure get region)\n",
    "region=${region:-us-west-2}\n",
    "\n",
    "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n",
    "\n",
    "# If the repository doesn't exist in ECR, create it.\n",
    "aws ecr describe-repositories --repository-names \"${algorithm_name}\" > /dev/null 2>&1\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    aws ecr create-repository --repository-name \"${algorithm_name}\" > /dev/null\n",
    "fi\n",
    "\n",
    "# Get the login command from ECR and execute it directly\n",
    "aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${fullname}\n",
    "\n",
    "# Build the docker image locally with the image name and then push it to ECR\n",
    "# with the full name.\n",
    "\n",
    "docker build  -t ${algorithm_name} .\n",
    "docker tag ${algorithm_name} ${fullname}\n",
    "\n",
    "docker push ${fullname}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Prepare data for model inference"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data = fetch_california_housing()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import pandas as pd\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data.data, data.target, test_size=0.25, random_state=42\n",
    ")\n",
    "\n",
    "# we don't train a model, so we will need only the testing data\n",
    "testX = pd.DataFrame(X_test, columns=data.feature_names)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Using your Algorithm in Amazon SageMaker"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# Define IAM role\n",
    "import boto3\n",
    "import re\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import sagemaker as sage\n",
    "from time import gmtime, strftime\n",
    "\n",
    "sess = sage.Session()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Create an estimator and fit the model"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "To use DVC integration, pass a `dvc_repo_url` and `dvc_branch_name` as parameters when you create the Estimator object.\n",
    "\n",
    "We will train on the `dev_dataset_1` branch first.\n",
    "\n",
    "When doing `dvc pull`, this is the dataset structure:\n",
    "\n",
    "```\n",
    "dataset\n",
    "    |-- train\n",
    "    |   |-- california_train.csv\n",
    "    |-- test\n",
    "    |   |-- california_test.csv\n",
    "    |-- validation\n",
    "    |   |-- california_validation.csv\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "dvc_repo_url = \"https://github.com/eitansela/sagemaker-dvc-catboost-demo.git\"\n",
    "dvc_branch_name = \"dev_dataset_1\""
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "account = sess.boto_session.client(\"sts\").get_caller_identity()[\"Account\"]\n",
    "region = sess.boto_session.region_name\n",
    "image = \"{}.dkr.ecr.{}.amazonaws.com/sagemaker-catboost-dvc:latest\".format(account, region)\n",
    "\n",
    "estimator = sage.estimator.Estimator(\n",
    "    image,\n",
    "    role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    sagemaker_session=sess,\n",
    "    hyperparameters={\n",
    "            \"dvc-repo-url\": dvc_repo_url,\n",
    "            \"dvc-branch-name\":  dvc_branch_name\n",
    "        },\n",
    ")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "estimator.fit()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "On the logs above you can see those lines, indicating about the files pulled by dvc:\n",
    "\n",
    "```\n",
    "Running dvc pull command\n",
    "A       train/california_train.csv\n",
    "A       test/california_test.csv\n",
    "A       validation/california_validation.csv\n",
    "3 files added and 3 files fetched\n",
    "Starting the training.\n",
    "Found train files: ['/opt/ml/input/data/dataset/train/california_train.csv']\n",
    "Found validation files: ['/opt/ml/input/data/dataset/train/california_train.csv']\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now train on the `dev_dataset_2` branch.\n",
    "\n",
    "When doing `dvc pull`, this is the dataset structure:\n",
    "\n",
    "```\n",
    "dataset\n",
    "    |-- train\n",
    "    |   |-- california_train_1.csv\n",
    "    |   |-- california_train_2.csv\n",
    "    |   |-- california_train_3.csv\n",
    "    |   |-- california_train_4.csv\n",
    "    |   |-- california_train_5.csv\n",
    "    |-- test\n",
    "    |   |-- california_test.csv\n",
    "    |-- validation\n",
    "    |   |-- california_validation_1.csv`\n",
    "    |   |-- california_validation_2.csv`\n",
    "    |   |-- california_validation_3.csv`\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "dvc_branch_name = \"dev_dataset_2\""
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "estimator = sage.estimator.Estimator(\n",
    "    image,\n",
    "    role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.large\",\n",
    "    sagemaker_session=sess,\n",
    "    hyperparameters={\n",
    "            \"dvc-repo-url\": dvc_repo_url,\n",
    "            \"dvc-branch-name\":  dvc_branch_name\n",
    "        },\n",
    ")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "estimator.fit()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "On the logs above you can see those lines, indicating about the files pulled by dvc:\n",
    "\n",
    "```\n",
    "Running dvc pull command\n",
    "A       validation/california_validation_2.csv\n",
    "A       validation/california_validation_1.csv\n",
    "A       validation/california_validation_3.csv\n",
    "A       train/california_train_4.csv\n",
    "A       train/california_train_5.csv\n",
    "A       train/california_train_2.csv\n",
    "A       train/california_train_3.csv\n",
    "A       train/california_train_1.csv\n",
    "A       test/california_test.csv\n",
    "9 files added and 9 files fetched\n",
    "Starting the training.\n",
    "Found train files: ['/opt/ml/input/data/dataset/train/california_train_2.csv', '/opt/ml/input/data/dataset/train/california_train_5.csv', '/opt/ml/input/data/dataset/train/california_train_4.csv', '/opt/ml/input/data/dataset/train/california_train_1.csv', '/opt/ml/input/data/dataset/train/california_train_3.csv']\n",
    "Found validation files: ['/opt/ml/input/data/dataset/validation/california_validation_2.csv', '/opt/ml/input/data/dataset/validation/california_validation_1.csv', '/opt/ml/input/data/dataset/validation/california_validation_3.csv']\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Hosting your model"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "\n",
    "predictor = estimator.deploy(1, \"ml.t2.medium\", serializer=csv_serializer)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Invoke endpoint with the Python SDK"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "predicted = predictor.predict(testX[data.feature_names].to_csv(header=False, index=False)).decode('utf-8')\n",
    "print(predicted)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## (Optional) Delete the Endpoint"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "predictor.delete_endpoint()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}