Variation4.MTL.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36m$file.$     \n",
       "\u001b[39m\n",
       "\u001b[32mimport \u001b[39m\u001b[36mcommon._\n",
       "\u001b[39m\n",
       "\u001b[32mimport \u001b[39m\u001b[36mcats._, cats.implicits._, cats.data._\n",
       "\u001b[39m\n",
       "\u001b[32mimport \u001b[39m\u001b[36mfs2.Stream\n",
       "\u001b[39m\n",
       "\u001b[32mimport \u001b[39m\u001b[36mdoobie._, doobie.implicits._\u001b[39m"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import $file.common\n",
    "import common._\n",
    "import cats._, cats.implicits._, cats.data._\n",
    "import fs2.Stream\n",
    "import doobie._, doobie.implicits._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variation 4. MTL-based Repositories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DAO approach is doomed. It forces us to commit to particular computation types: `Id[_]` (i.e. synchronous), `cats.effect.IO[_]`, `scala.concurrent.Future[_]`, etc. (i.e. asynchronous), `cats.StateT[_, _]` (if we want to do unit testing without mocking), etc. All these computation types demand particular DAO APIs, which would, in turn, demand particular business logic implementations. Clearly, this is a no-good. What if we had a truly generic DAO API that could be accommodated to any single computation type? This is what the MTL-style offers to us: the possibility of programming DAO APIs which are parameterised by any computation type we like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mclass\u001b[39m \u001b[36mCountry\u001b[39m\n",
       "defined \u001b[32mclass\u001b[39m \u001b[36mCity\u001b[39m\n",
       "defined \u001b[32mtrait\u001b[39m \u001b[36mCityRepo\u001b[39m\n",
       "defined \u001b[32mtrait\u001b[39m \u001b[36mCountryRepo\u001b[39m\n",
       "defined \u001b[32mtrait\u001b[39m \u001b[36mWorldRepo\u001b[39m"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Case classes, as before\n",
    "\n",
    "case class Country(code: String, name: String, capital: Option[Int])\n",
    "case class City(id: Int, name: String, countryCode: String, population: Int)\n",
    "\n",
    "// DAO APIs as type constructor classes\n",
    "\n",
    "trait CityRepo[F[_]]{\n",
    "    def findCity(id: Int): F[City]\n",
    "    def findAllCities: F[City]\n",
    "}\n",
    "\n",
    "trait CountryRepo[F[_]]{\n",
    "    def findCountry(code: String): F[Country]\n",
    "    def findAllCountries: F[Country]\n",
    "}\n",
    "\n",
    "trait WorldRepo[F[_]] extends CityRepo[F] with CountryRepo[F]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the type constructor parameter `F[_]` represents a generic computation type, and these APIs represent classes of computations, namely those computation types which can allow us to access world data. Queries are then programmed much in the same way as before, only that we need extra APIs to compose the instructions of the domain repository models: `Monad[_[_]]` and `FunctionFilter[_[_]]`. These are also computation classes as well: the class of imperative computations and the computations that can be filtered, respectively. This is all we need in order to write our `largeCapitals` query, once and for all, in the same readable way as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mfunction\u001b[39m \u001b[36mlargeCapitals\u001b[39m"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def largeCapitals[F[_]: Monad: FunctorFilter](implicit \n",
    "        W: WorldRepo[F]): F[(String, String)] = \n",
    "    for {\n",
    "        Country(_, name, Some(capital)) <- W.findAllCountries\n",
    "        city <- W.findCity(capital)\n",
    "        if city.population > 8000000\n",
    "    } yield (city.name, name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we really run this business logic for any kind of computation? Let's try it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stream-based Doobie implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first try to run our query against the world database. In order to do so, we choose to translate `WorldRepo` instructions in terms of `Stream[ConnectionIO, T]]` computations, i.e. computations that eventually (when run) return a stream of values of type `T` obtained through JDBC `ConnectionIO` programs. This is the instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mtype\u001b[39m \u001b[36mDoobieStr\u001b[39m\n",
       "defined \u001b[32mobject\u001b[39m \u001b[36mDoobieStrWorldRepo\u001b[39m"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type DoobieStr[T] = Stream[ConnectionIO, T]\n",
    "\n",
    "implicit object DoobieStrWorldRepo extends WorldRepo[DoobieStr]{\n",
    "\n",
    "    def findCity(id: Int): Stream[ConnectionIO, City] = \n",
    "        sql\"select id, name, countryCode, population from city where id = $id\"\n",
    "            .query[City].stream\n",
    "    \n",
    "    def findCountry(code: String): Stream[ConnectionIO, Country] = \n",
    "        sql\"select code, name, capital from country where code = $code\"\n",
    "            .query[Country].stream\n",
    "    \n",
    "    def findAllCountries: Stream[ConnectionIO, Country] = \n",
    "        sql\"select code, name, capital from country\"\n",
    "            .query[Country].stream\n",
    "    \n",
    "    def findAllCities: Stream[ConnectionIO, City] = \n",
    "        sql\"select id, name, countryCode, population from country\"\n",
    "            .query[City].stream\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run the query, we just need to specify the desired computation type (all the required dependencies will be injected automatically through the implicit mechanism); then, we compile the stream and the JDBC program, and, last, interpret the resulting IO program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "844 millis\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[36mres5\u001b[39m: \u001b[32mList\u001b[39m[(\u001b[32mString\u001b[39m, \u001b[32mString\u001b[39m)] = \u001b[33mList\u001b[39m(\n",
       "  (\u001b[32m\"Jakarta\"\u001b[39m, \u001b[32m\"Indonesia\"\u001b[39m),\n",
       "  (\u001b[32m\"Seoul\"\u001b[39m, \u001b[32m\"South Korea\"\u001b[39m),\n",
       "  (\u001b[32m\"Ciudad de M\\u00e9xico\"\u001b[39m, \u001b[32m\"Mexico\"\u001b[39m),\n",
       "  (\u001b[32m\"Moscow\"\u001b[39m, \u001b[32m\"Russian Federation\"\u001b[39m)\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "largeCapitals[DoobieStr] // Stream[ConnectionIO, (String, String)]\n",
    "    .compile.toList      // ConnectionIO[List[(String, String)]]\n",
    "    .transact(xa)        // IO[List[(String, String)]]\n",
    "    .unsafeRunSync\n",
    "    .timed(1).millis// List[(String, String)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It works! Can we also do unit testing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unit testing with `StateT`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unit testing can be done in a purely functional way, i.e. without mocking libraries, using a particular type of computation: state transformers. The basic idea is to interpret domain instructions in terms of transformations or queries over the `World` state (which is represented as an in-memory data type). In our simplified case, we don't have transformations, so a computation `World => List[T]` suffices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mclass\u001b[39m \u001b[36mWorld\u001b[39m\n",
       "defined \u001b[32mobject\u001b[39m \u001b[36mWorld\u001b[39m"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "case class World(\n",
    "    countries: Map[String, Country],\n",
    "    cities: Map[Int, City])\n",
    "\n",
    "object World{\n",
    "    \n",
    "    type State[T] = StateT[List, World, T]\n",
    "\n",
    "    implicit object StateTWorldRepo extends WorldRepo[State]{\n",
    "\n",
    "        // Cities\n",
    "        \n",
    "        def findCity(id: Int): State[City] = \n",
    "            StateT.inspectF(_.cities.get(id).toList)\n",
    "        \n",
    "        def findAllCities: State[City] = \n",
    "            StateT.inspectF(_.cities.values.toList)\n",
    "        \n",
    "        //  Countries\n",
    "        \n",
    "        def findCountry(code: String): State[Country] =\n",
    "            StateT.inspectF(_.countries.get(code).toList)\n",
    "\n",
    "        def findAllCountries: State[Country] = \n",
    "            StateT.inspectF(_.countries.values.toList)\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The very same scalatest specification than before will serve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36morg.scalatest._\n",
       "\n",
       "\u001b[39m\n",
       "defined \u001b[32mclass\u001b[39m \u001b[36mLargeCapitalsSpec\u001b[39m"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.scalatest._\n",
    "\n",
    "class LargeCapitalsSpec(largeCapitals: World => List[(String, String)])\n",
    "extends FlatSpec with Matchers{\n",
    "    \n",
    "    val smallWorld: World =         \n",
    "        World(Map(\"ES\" -> Country(\"ES\",\"Spain\",Some(0)),\n",
    "                \"USA\" -> Country(\"USA\", \"United States\", Some(1)),\n",
    "                \"UK\" -> Country(\"UK\", \"United Kingdom\", Some(2)),\n",
    "                \"UNK\" -> Country(\"UNK\", \"Unknown\", None)),\n",
    "        Map(0->City(0,\"Madrid\",\"ES\",9000000),\n",
    "            1->City(1,\"Washington\", \"USA\", 10000000),\n",
    "            2->City(2,\"London\", \"UK\", 500000)))    \n",
    "    \n",
    "    \"large capitals\" should \"be right\" in {\n",
    "        largeCapitals(smallWorld).toSet shouldBe \n",
    "            Set((\"Madrid\", \"Spain\"), (\"Washington\", \"United States\"))\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in order to unit test our query, we just compile to the required type discarding the resulting state, using `runA`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mcmd6$Helper$LargeCapitalsSpec:\u001b[0m\n",
      "\u001b[32mlarge capitals\u001b[0m\n",
      "\u001b[32m- should be right\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "run(new LargeCapitalsSpec(largeCapitals[World.State].runA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### However ... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were so obsessed with modularity that we didn't pay attention to the performance of our interpreters. This is not so important in the unit testing interpreter, but the doobie one ... really matters. Let's obtain some figures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[36mmtlTime\u001b[39m: \u001b[32mLong\u001b[39m = \u001b[32m96790157L\u001b[39m\n",
       "\u001b[36msqlTime\u001b[39m: \u001b[32mLong\u001b[39m = \u001b[32m16623119L\u001b[39m"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val mtlTime: Long = \n",
    "    largeCapitals[DoobieStr] \n",
    "        .compile.toList      \n",
    "        .transact(xa)        \n",
    "        .unsafeRunSync       \n",
    "        .timed(50)\n",
    "        ._2\n",
    "\n",
    "val sqlTime: Long = \n",
    "    sql\"\"\"\n",
    "        | select C.name, X.name \n",
    "        | from city as C, country as X \n",
    "        | where C.id = X.capital and C.population > 8000000\"\"\".stripMargin\n",
    "        .query[(String, String)]\n",
    "        .to[List]\n",
    "        .transact(xa)\n",
    "        .unsafeRunSync\n",
    "        .timed(50)\n",
    "        ._2\n",
    "\n",
    "println(s\"ratio: ${mtlTime/sqlTime}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Around five to ten times more than the plain sql query ... Something's going on here. Effectively, the postgres log tells us that we are suffering from the so-called _avalance query_ problem. Each world model instruction from the repositories is compiled into an independent SQL query, and the interpreter does nothing to reassemble those queries into the optimum one. And the worst thing is that the more data we have (the more countries, in this case), the more inefficient our query will be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are we really lost here? Can't we somehow implement a smart interpreter that generates the optimum query? Yes, enter the field of [Quoted Domain Specific Languages](Variation5.Quill.ipynb)!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Scala",
   "language": "scala",
   "name": "scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".sc",
   "mimetype": "text/x-scala",
   "name": "scala",
   "nbconvert_exporter": "script",
   "version": "2.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}