clipdraw.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "CLIPDraw AI Telephone.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e7DCzf-EHzL_"
      },
      "source": [
        "# CLIPDraw\n",
        "Synthesize drawings to match a text prompt!\n",
        "\n",
        "\n",
        "![](https://kvfrans.com/content/images/2021/06/Screen-Shot-2021-06-10-at-8.47.23-PM.png)\n",
        "\n",
        "\n",
        "> This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, a constraint that biases drawings towards simpler human-recognizable shapes. Results compare between CLIPDraw and other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse artistic styles, and scaling from simple to complex visual representations as stroke count is increased. \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "This Colab notebook goes along with the paper: [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://arxiv.org/abs/2106.14843) \n",
        "\n",
        "by **Kevin Frans**, L.B. Soros, Olaf Witkowski\n",
        "\n",
        "Read the blog post for cool results and analysis! [https://kvfrans.com/clipdraw-exploring-text-to-drawing-synthesis/](https://kvfrans.com/clipdraw-exploring-text-to-drawing-synthesis/)\n",
        "\n",
        "Feel free to tweet me any cool creations, [@kvfrans](https://twitter.com/kvfrans)\n",
        "\n",
        "Code adapted from diffvg: https://github.com/BachiLi/diffvg/blob/master/apps/painterly_rendering.py\n",
        "\n",
        "--- \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "**STEPS:**\n",
        "\n",
        "\n",
        "1. Click \"Connect\" in the top right corner\n",
        "2. Runtime -> Change runtime type -> Hardware accelerator -> GPU\n",
        "2. Click the run button on \"Pre Installation\". This will install dependencies, it may take a while.\n",
        "2. **Important:** Runtime -> Restart Runtime\n",
        "3. Run the \"Imports and Notebook Utilities\" and \"Load CLIP\" sections.\n",
        "\n",
        "\n",
        "5. \"Curve Optimizer\" will synthesize a drawing to match your text. You can edit the text prompt at the top of the code block.\n",
        "6. \"Video Renderer\" can create videos that show the optimization process, and videos that render a drawing stroke-by-stroke."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5dyyH781qzIC",
        "cellView": "form"
      },
      "source": [
        "#@title Pre Installation {vertical-output: true}\n",
        "\n",
        "import subprocess\n",
        "\n",
        "CUDA_version = [s for s in subprocess.check_output([\"nvcc\", \"--version\"]).decode(\"UTF-8\").split(\", \") if s.startswith(\"release\")][0].split(\" \")[-1]\n",
        "print(\"CUDA version:\", CUDA_version)\n",
        "\n",
        "if CUDA_version == \"10.0\":\n",
        "    torch_version_suffix = \"+cu100\"\n",
        "elif CUDA_version == \"10.1\":\n",
        "    torch_version_suffix = \"+cu101\"\n",
        "elif CUDA_version == \"10.2\":\n",
        "    torch_version_suffix = \"\"\n",
        "else:\n",
        "    torch_version_suffix = \"+cu110\"\n",
        "\n",
        "# !pip install torch==1.7.1{torch_version_suffix} torchvision==0.8.2{torch_version_suffix} -f https://download.pytorch.org/whl/torch_stable.html ftfy regex\n",
        "%cd /content/\n",
        "!pip install svgwrite\n",
        "!pip install svgpathtools\n",
        "!pip install cssutils\n",
        "!pip install numba\n",
        "!pip install torch-tools\n",
        "!pip install visdom\n",
        "\n",
        "!git clone https://github.com/BachiLi/diffvg\n",
        "%cd diffvg\n",
        "# !ls\n",
        "!git submodule update --init --recursive\n",
        "\n",
        "data = []\n",
        "with open(\"/content/diffvg/CMakeLists.txt\", \"r+\") as inFile:\n",
        "    for line in inFile:\n",
        "        if \"find_package(TensorFlow)\" in line:\n",
        "            pass\n",
        "        else:\n",
        "            data.append(line)\n",
        "    inFile.seek(0)\n",
        "    for d in data:\n",
        "        inFile.write(d)\n",
        "    inFile.truncate()\n",
        "!python setup.py install\n",
        "\n",
        "!pip install ftfy regex tqdm\n",
        "!pip install git+https://github.com/openai/CLIP.git --no-deps"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hjt9T3ARukAg",
        "cellView": "form",
        "outputId": "add047b2-7848-40d8-f7de-589706feab3e"
      },
      "source": [
        "#@title Imports and Notebook Utilities {vertical-output: true}\n",
        "%tensorflow_version 2.x\n",
        "\n",
        "import os\n",
        "import io\n",
        "import PIL.Image, PIL.ImageDraw\n",
        "import base64\n",
        "import zipfile\n",
        "import json\n",
        "import requests\n",
        "import numpy as np\n",
        "import matplotlib.pylab as pl\n",
        "import glob\n",
        "\n",
        "from IPython.display import Image, HTML, clear_output\n",
        "from tqdm import tqdm_notebook, tnrange\n",
        "\n",
        "os.environ['FFMPEG_BINARY'] = 'ffmpeg'\n",
        "import moviepy.editor as mvp\n",
        "from moviepy.video.io.ffmpeg_writer import FFMPEG_VideoWriter\n",
        "\n",
        "\n",
        "def imread(url, max_size=None, mode=None):\n",
        "  if url.startswith(('http:', 'https:')):\n",
        "    r = requests.get(url)\n",
        "    f = io.BytesIO(r.content)\n",
        "  else:\n",
        "    f = url\n",
        "  img = PIL.Image.open(f)\n",
        "  if max_size is not None:\n",
        "    img = img.resize((max_size, max_size))\n",
        "  if mode is not None:\n",
        "    img = img.convert(mode)\n",
        "  img = np.float32(img)/255.0\n",
        "  return img\n",
        "\n",
        "def np2pil(a):\n",
        "  if a.dtype in [np.float32, np.float64]:\n",
        "    a = np.uint8(np.clip(a, 0, 1)*255)\n",
        "  return PIL.Image.fromarray(a)\n",
        "\n",
        "def imwrite(f, a, fmt=None):\n",
        "  a = np.asarray(a)\n",
        "  if isinstance(f, str):\n",
        "    fmt = f.rsplit('.', 1)[-1].lower()\n",
        "    if fmt == 'jpg':\n",
        "      fmt = 'jpeg'\n",
        "    f = open(f, 'wb')\n",
        "  np2pil(a).save(f, fmt, quality=95)\n",
        "\n",
        "def imencode(a, fmt='jpeg'):\n",
        "  a = np.asarray(a)\n",
        "  if len(a.shape) == 3 and a.shape[-1] == 4:\n",
        "    fmt = 'png'\n",
        "  f = io.BytesIO()\n",
        "  imwrite(f, a, fmt)\n",
        "  return f.getvalue()\n",
        "\n",
        "def im2url(a, fmt='jpeg'):\n",
        "  encoded = imencode(a, fmt)\n",
        "  base64_byte_string = base64.b64encode(encoded).decode('ascii')\n",
        "  return 'data:image/' + fmt.upper() + ';base64,' + base64_byte_string\n",
        "\n",
        "def imshow(a, fmt='jpeg'):\n",
        "  display(Image(data=imencode(a, fmt)))\n",
        "\n",
        "\n",
        "def tile2d(a, w=None):\n",
        "  a = np.asarray(a)\n",
        "  if w is None:\n",
        "    w = int(np.ceil(np.sqrt(len(a))))\n",
        "  th, tw = a.shape[1:3]\n",
        "  pad = (w-len(a))%w\n",
        "  a = np.pad(a, [(0, pad)]+[(0, 0)]*(a.ndim-1), 'constant')\n",
        "  h = len(a)//w\n",
        "  a = a.reshape([h, w]+list(a.shape[1:]))\n",
        "  a = np.rollaxis(a, 2, 1).reshape([th*h, tw*w]+list(a.shape[4:]))\n",
        "  return a\n",
        "\n",
        "from torchvision import utils\n",
        "def show_img(img):\n",
        "    img = np.transpose(img, (1, 2, 0))\n",
        "    img = np.clip(img, 0, 1)\n",
        "    img = np.uint8(img * 254)\n",
        "    # img = np.repeat(img, 4, axis=0)\n",
        "    # img = np.repeat(img, 4, axis=1)\n",
        "    pimg = PIL.Image.fromarray(img, mode=\"RGB\")\n",
        "    imshow(pimg)\n",
        "\n",
        "def zoom(img, scale=4):\n",
        "  img = np.repeat(img, scale, 0)\n",
        "  img = np.repeat(img, scale, 1)\n",
        "  return img\n",
        "\n",
        "class VideoWriter:\n",
        "  def __init__(self, filename='_autoplay.mp4', fps=30.0, **kw):\n",
        "    self.writer = None\n",
        "    self.params = dict(filename=filename, fps=fps, **kw)\n",
        "\n",
        "  def add(self, img):\n",
        "    img = np.asarray(img)\n",
        "    if self.writer is None:\n",
        "      h, w = img.shape[:2]\n",
        "      self.writer = FFMPEG_VideoWriter(size=(w, h), **self.params)\n",
        "    if img.dtype in [np.float32, np.float64]:\n",
        "      img = np.uint8(img.clip(0, 1)*255)\n",
        "    if len(img.shape) == 2:\n",
        "      img = np.repeat(img[..., None], 3, -1)\n",
        "    self.writer.write_frame(img)\n",
        "\n",
        "  def close(self):\n",
        "    if self.writer:\n",
        "      self.writer.close()\n",
        "\n",
        "  def __enter__(self):\n",
        "    return self\n",
        "\n",
        "  def __exit__(self, *kw):\n",
        "    self.close()\n",
        "    if self.params['filename'] == '_autoplay.mp4':\n",
        "      self.show()\n",
        "\n",
        "  def show(self, **kw):\n",
        "      self.close()\n",
        "      fn = self.params['filename']\n",
        "      display(mvp.ipython_display(fn, **kw))\n",
        "\n",
        "!nvidia-smi -L\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import os\n",
        "\n",
        "# torch.set_default_tensor_type('torch.cuda.FloatTensor')\n",
        "\n",
        "print(\"Torch version:\", torch.__version__)\n",
        "\n",
        "# !pip install DALL-E"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GPU 0: Tesla V100-SXM2-16GB (UUID: GPU-b3a207b3-8786-f04d-e1da-c70d87550aff)\n",
            "Torch version: 1.9.0+cu102\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z-Wt7UjTi8Le",
        "outputId": "73f1514b-5e05-4275-a54a-789ba6213998"
      },
      "source": [
        "#@title Load CLIP {vertical-output: true}\n",
        "\n",
        "# os.environ['CUDA_LAUNCH_BLOCKING'] = '1'\n",
        "\n",
        "import os\n",
        "import clip\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "import torchvision\n",
        "from torchvision import transforms\n",
        "from torchvision.datasets import CIFAR100\n",
        "\n",
        "# Load the model\n",
        "device = torch.device('cuda')\n",
        "model, preprocess = clip.load('ViT-B/32', device, jit=False)\n",
        "\n",
        "nouns = \"aardvark abyssinian accelerator accordion account accountant acknowledgment acoustic acrylic act action active activity actor actress adapter addition address adjustment adult advantage advertisement advice afghanistan africa aftermath afternoon aftershave afterthought age agenda agreement air airbus airmail airplane airport airship alarm albatross alcohol algebra algeria alibi alley alligator alloy almanac alphabet alto aluminium aluminum ambulance america amount amusement anatomy anethesiologist anger angle angora animal anime ankle answer ant antarctica anteater antelope anthony anthropology apartment apology apparatus apparel appeal appendix apple appliance approval april aquarius arch archaeology archeology archer architecture area argentina argument aries arithmetic arm armadillo armchair armenian army arrow art ash ashtray asia asparagus asphalt asterisk astronomy athlete atm atom attack attempt attention attic attraction august aunt australia australian author authorisation authority authorization avenue babies baboon baby back backbone bacon badge badger bag bagel bagpipe bail bait baker bakery balance balinese ball balloon bamboo banana band bandana bangladesh bangle banjo bank bankbook banker bar barbara barber barge baritone barometer base baseball basement basin basket basketball bass bassoon bat bath bathroom bathtub battery battle bay beach bead beam bean bear beard beast beat beautician beauty beaver bed bedroom bee beech beef beer beet beetle beggar beginner begonia behavior belgian belief believe bell belt bench bengal beret berry bestseller betty bibliography bicycle bike bill billboard biology biplane birch bird birth birthday bit bite black bladder blade blanket blinker blizzard block blood blouse blow blowgun blue board boat bobcat body bolt bomb bomber bone bongo bonsai book bookcase booklet boot border botany bottle bottom boundary bow bowl bowling box boy bra brace bracket brain brake branch brand brandy brass brazil bread break breakfast breath brian brick bridge british broccoli brochure broker bronze brother brother-in-law brow brown brush bubble bucket budget buffer buffet bugle building bulb bull bulldozer bumper bun burglar burma burn burst bus bush business butane butcher butter button buzzard cabbage cabinet cable cactus cafe cake calculator calculus calendar calf call camel camera camp can canada canadian cancer candle cannon canoe canvas cap capital cappelletti capricorn captain caption car caravan carbon card cardboard cardigan care carnation carol carp carpenter carriage carrot cart cartoon case cast castanet cat catamaran caterpillar cathedral catsup cattle cauliflower cause caution cave c-clamp cd ceiling celery celeste cell cellar cello celsius cement cemetery cent centimeter century ceramic cereal certification chain chair chalk chance change channel character chard charles chauffeur check cheek cheese cheetah chef chemistry cheque cherries cherry chess chest chick chicken chicory chief child children chill chime chimpanzee chin china chinese chive chocolate chord christmas christopher chronometer church cicada cinema circle circulation cirrus citizenship city clam clarinet class claus clave clef clerk click client climb clipper cloakroom clock close closet cloth cloud cloudy clover club clutch coach coal coast coat cobweb cockroach cocktail cocoa cod coffee coil coin coke cold collar college collision colombia colon colony color colt column columnist comb comfort comic comma command commission committee community company comparison competition competitor composer composition computer condition condor cone confirmation conga congo conifer connection consonant continent control cook cooking copper copy copyright cord cork cormorant corn cornet correspondent cost cotton couch cougar cough country course court cousin cover cow cowbell crab crack cracker craftsman crate crawdad crayfish crayon cream creator creature credit creditor creek crib cricket crime criminal crocodile crocus croissant crook crop cross crow crowd crown crush cry cub cuban cucumber cultivator cup cupboard cupcake curler currency current curtain curve cushion custard customer cut cuticle cycle cyclone cylinder cymbal dad daffodil dahlia daisy damage dance dancer danger daniel dash dashboard database date daughter david day dead deadline deal death deborah debt debtor decade december decimal decision decrease dedication deer defense deficit degree delete delivery den denim dentist deodorant department deposit description desert design desire desk dessert destruction detail detective development dew diamond diaphragm dibble dictionary dietician difference digestion digger digital dill dime dimple dinghy dinner dinosaur diploma dipstick direction dirt disadvantage discovery discussion disease disgust dish distance distribution distributor diving division divorced dock doctor dog dogsled doll dollar dolphin domain donald donkey donna door dorothy double doubt downtown dragon dragonfly drain drake drama draw drawbridge drawer dream dredger dress dresser dressing drill drink drive driver driving drizzle drop drug drum dry dryer duck duckling dugout dungeon dust eagle ear earth earthquake ease east edge edger editor editorial education edward eel effect egg eggnog eggplant egypt eight elbow element elephant elizabeth ellipse emery employee employer encyclopedia end enemy energy engine engineer engineering english enquiry entrance environment epoch epoxy equinox equipment era error estimate ethernet ethiopia euphonium europe evening event examination example exchange exclamation exhaust ex-husband existence expansion experience expert explanation ex-wife eye eyebrow eyelash eyeliner face facilities fact factory fahrenheit fairies fall family fan fang farm farmer fat father father-in-law faucet fear feast feather feature february fedelini feedback feeling feet felony female fender ferry ferryboat fertilizer fiber fiberglass fibre fiction field fifth fight fighter file find fine finger fir fire fired fireman fireplace firewall fish fisherman flag flame flare flat flavor flax flesh flight flock flood floor flower flugelhorn flute fly foam fog fold font food foot football footnote force forecast forehead forest forgery fork form format fortnight foundation fountain fowl fox foxglove fragrance frame france freckle freeze freezer freighter french freon friction friday fridge friend frog front frost frown fruit fuel fur furniture galley gallon game gander garage garden garlic gas gasoline gate gateway gauge gazelle gear gearshift geese gemini gender geography geology geometry george geranium german germany ghana ghost giant giraffe girdle girl gladiolus glass glider gliding glockenspiel glove glue goal goat gold goldfish golf gondola gong good-bye goose gore-tex gorilla gosling government governor grade grain gram granddaughter grandfather grandmother grandson grape graphic grass grasshopper gray grease great-grandfather great-grandmother greece greek green grenade grey grill grip ground group grouse growth guarantee guatemalan guide guilty guitar gum gun gym gymnast hacksaw hail hair haircut half-brother half-sister halibut hall hallway hamburger hammer hamster hand handball handicap handle handsaw harbor hardboard hardcover hardhat hardware harmonica harmony harp hat hate hawk head headlight headline health hearing heart heat heaven hedge height helen helicopter helium hell helmet help hemp hen heron herring hexagon hill himalayan hip hippopotamus history hobbies hockey hoe hole holiday home honey hood hook hope horn horse hose hospital hot hour hourglass house hovercraft hub hubcap humidity humor hurricane hyacinth hydrant hydrofoil hydrogen hyena hygienic ice icebreaker icicle icon idea ikebana illegal imprisonment improvement impulse inch income increase index india indonesia industry ink innocent input insect instruction instrument insulation insurance interactive interest internet interviewer intestine invention inventory invoice iran iraq iris iron island israel italian italy jacket jaguar jail jam james january japan japanese jar jasmine jason jaw jeans jeep jeff jelly jellyfish jennifer jet jewel jogging john join joke joseph journey judge judo juice july jumbo jump jumper june jury justice jute kale kamikaze kangaroo karate karen kayak kendo kenneth kenya ketchup kettle kettledrum kevin key keyboard keyboarding kick kidney kilogram kilometer kimberly kiss kitchen kite kitten kitty knee knickers knife knight knot knowledge kohlrabi korean laborer lace ladybug lake lamb lamp lan land landmine language larch lasagna latency latex lathe laugh laundry laura law lawyer layer lead leaf learning leather leek leg legal lemonade lentil leo leopard letter lettuce level libra library license lier lift light lightning lilac lily limit linda line linen link lion lip lipstick liquid liquor lisa list literature litter liver lizard llama loaf loan lobster lock locket locust look loss lotion love low lumber lunch lunchroom lung lunge lute luttuce lycra lynx lyocell lyre lyric macaroni machine macrame magazine magic magician maid mail mailbox mailman makeup malaysia male mall mallet man manager mandolin manicure manx map maple maraca marble march margaret margin maria marimba mark mark market married mary mascara mask mass match math mattock may mayonnaise meal measure meat mechanic medicine meeting melody memory men menu mercury message metal meteorology meter methane mexican mexico mice michael michelle microwave middle mile milk milkshake millennium millimeter millisecond mimosa mind mine minibus mini-skirt minister mint minute mirror missile mist mistake mitten moat modem mole mom monday money monkey month moon morning morocco mosque mosquito mother mother-in-law motion motorboat motorcycle mountain mouse moustache mouth move multi-hop multimedia muscle museum music musician mustard myanmar nail name nancy napkin narcissus nation neck need needle neon nepal nephew nerve nest net network news newsprint newsstand nic nickel niece nigeria night nitrogen node noise noodle north north america north korea norwegian nose note notebook notify novel november number numeric nurse nut nylon oak oatmeal objective oboe observation occupation ocean ocelot octagon octave october octopus odometer offence offer office oil okra olive onion open opera operation ophthalmologist opinion option orange orchestra orchid order organ organisation organization ornament ostrich otter ounce output outrigger oval oven overcoat owl owner ox oxygen oyster package packet page pail pain paint pair pajama pakistan palm pamphlet pan pancake pancreas panda pansy panther panties pantry pants panty pantyhose paper paperback parade parallelogram parcel parent parentheses park parrot parsnip part particle partner partridge party passbook passenger passive pasta paste pastor pastry patch path patient patio patricia paul payment pea peace peak peanut pear pedestrian pediatrician peen peer-to-peer pelican pen penalty pencil pendulum pentagon peony pepper perch perfume period periodical peripheral permission persian person peru pest pet pharmacist pheasant philippines philosophy phone physician piano piccolo pickle picture pie pig pigeon pike pillow pilot pimple pin pine ping pink pint pipe pisces pizza place plain plane planet plant plantation plaster plasterboard plastic plate platinum play playground playroom pleasure plier plot plough plow plywood pocket poet point poison poland police policeman polish politician pollution polo polyester pond popcorn poppy population porch porcupine port porter position possibility postage postbox pot potato poultry pound powder power precipitation preface prepared pressure price priest print printer prison probation process processing produce product production professor profit promotion propane property prose prosecution protest protocol pruner psychiatrist psychology ptarmigan puffin pull puma pump pumpkin punch punishment puppy purchase purple purpose push pvc pyjama pyramid quail quality quart quarter quartz queen question quicksand quiet quill quilt quince quit quiver quotation rabbi rabbit racing radar radiator radio radish raft rail railway rain rainbow raincoat rainstorm rake ramie random range rat rate raven ravioli ray rayon reaction reading reason receipt recess record recorder rectangle red reduction refrigerator refund regret reindeer relation relative religion relish reminder repair replace report representative request resolution respect responsibility rest restaurant result retailer revolve revolver reward rhinoceros rhythm rice richard riddle rifle ring rise risk river riverbed road roadway roast robert robin rock rocket rod roll romania romanian ronald roof room rooster root rose rotate route router rowboat rub rubber rugby rule run russia russian rutabaga ruth sack sagittarius sail sailboat sailor salad salary sale salesman salmon salt sampan samurai sand sandra sandwich santa sarah sardine satin saturday sauce saudi arabia sausage save saw saxophone scale scallion scanner scarecrow scarf scene scent schedule school science scissors scooter scorpio scorpion scraper screen screw screwdriver sea seagull seal seaplane search seashore season seat second secretary secure security seed seeder segment select selection self semicircle semicolon sense sentence separated september servant server session sex shade shadow shake shallot shame shampoo shape share shark sharon shears sheep sheet shelf shell shield shingle ship shirt shock shoe shoemaker shop shorts shoulder shovel show shrimp shrine siamese siberian side sideboard sidecar sidewalk sign signature silica silk silver sing singer single sink sister sister-in-law size skate skiing skill skin skirt sky slash slave sled sleep sleet slice slime slip slipper slope smash smell smile smoke snail snake sneeze snow snowboarding snowflake snowman snowplow snowstorm soap soccer society sociology sock soda sofa softball softdrink software soil soldier son song soprano sort sound soup sousaphone south africa south america south korea soy soybean space spade spaghetti spain spandex spark sparrow spear specialist speedboat sphere sphynx spider spike spinach spleen sponge spoon spot spring sprout spruce spy square squash squid squirrel stage staircase stamp star start starter state statement station statistic steam steel stem step step-aunt step-brother stepdaughter step-daughter step-father step-grandfather step-grandmother stepmother step-mother step-sister stepson step-son step-uncle steven stew stick stinger stitch stock stocking stomach stone stool stop stopsign stopwatch store storm story stove stranger straw stream street streetcar stretch string structure study sturgeon submarine substance subway success sudan suede sugar suggestion suit summer sun sunday sundial sunflower sunshine supermarket supply support surfboard surgeon surname surprise susan sushi swallow swamp swan sweater sweatshirt sweatshop swedish sweets swim swimming swing swiss switch sword swordfish sycamore syria syrup system table tablecloth tabletop tachometer tadpole tail tailor taiwan talk tank tanker tanzania target taste taurus tax taxi taxicab tea teacher teaching team technician teeth television teller temper temperature temple tempo tendency tennis tenor tent territory test text textbook texture thailand theater theory thermometer thing thistle thomas thought thread thrill throat throne thumb thunder thunderstorm thursday ticket tie tiger tights tile timbale time timer timpani tin tip tire titanium title toad toast toe toenail toilet tomato tom-tom ton tongue tooth toothbrush toothpaste top tornado tortellini tortoise touch tower town toy tractor trade traffic trail train tramp transaction transmission transport trapezoid tray treatment tree trial triangle trick trigonometry trip trombone trouble trousers trout trowel truck trumpet trunk t-shirt tsunami tub tuba tuesday tugboat tulip tuna tune turkey turkey turkish turn turnip turnover turret turtle tv twig twilight twine twist typhoon tyvek uganda ukraine ukrainian umbrella uncle underclothes underpants undershirt underwear unit united kingdom unshielded use utensil uzbekistan vacation vacuum valley value van vase vault vegetable vegetarian veil vein velvet venezuela venezuelan verdict vermicelli verse vessel vest veterinarian vibraphone vietnam view vinyl viola violet violin virgo viscose vise vision visitor voice volcano volleyball voyage vulture waiter waitress walk wall wallaby wallet walrus war warm wash washer wasp waste watch watchmaker water waterfall wave wax way wealth weapon weasel weather wedge wednesday weed weeder week weight whale wheel whip whiskey whistle white wholesaler whorl wilderness william willow wind windchime window windscreen windshield wine wing winter wire wish witch withdrawal witness wolf woman women wood wool woolen word work workshop worm wound wrecker wren wrench wrinkle wrist writer xylophone yacht yak yam yard yarn year yellow yew yogurt yoke yugoslavian zebra zephyr zinc zipper zone zoo zoology\"\n",
        "nouns = nouns.split(\" \")\n",
        "noun_prompts = [\"a drawing of a \" + x for x in nouns]\n",
        "\n",
        "# Calculate features\n",
        "with torch.no_grad():\n",
        "    nouns_features = model.encode_text(torch.cat([clip.tokenize(noun_prompts).to(device)]))\n",
        "print(nouns_features.shape, nouns_features.dtype)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([2343, 512]) torch.float16\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4XIVMSJuWgxG"
      },
      "source": [
        "#@title Curve Optimizer {vertical-output: true}\n",
        "\n",
        "%cd /content/diffvg/apps/\n",
        "\n",
        "prompt = \"Watercolor painting of an underwater submarine.\"\n",
        "neg_prompt = \"A badly drawn sketch.\"\n",
        "neg_prompt_2 = \"Many ugly, messy drawings.\"\n",
        "text_input = clip.tokenize(prompt).to(device)\n",
        "text_input_neg1 = clip.tokenize(neg_prompt).to(device)\n",
        "text_input_neg2 = clip.tokenize(neg_prompt_2).to(device)\n",
        "use_negative = False # Use negative prompts?\n",
        "\n",
        "# Thanks to Katherine Crowson for this. \n",
        "# In the CLIPDraw code used to generate examples, we don't normalize images\n",
        "# before passing into CLIP, but really you should. Turn this to True to do that.\n",
        "use_normalized_clip = False \n",
        "\n",
        "# Calculate features\n",
        "with torch.no_grad():\n",
        "    text_features = model.encode_text(text_input)\n",
        "    text_features_neg1 = model.encode_text(text_input_neg1)\n",
        "    text_features_neg2 = model.encode_text(text_input_neg2)\n",
        "\n",
        "import pydiffvg\n",
        "import torch\n",
        "import skimage\n",
        "import skimage.io\n",
        "import random\n",
        "import ttools.modules\n",
        "import argparse\n",
        "import math\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "\n",
        "pydiffvg.set_print_timing(False)\n",
        "\n",
        "gamma = 1.0\n",
        "\n",
        "# ARGUMENTS. Feel free to play around with these, especially num_paths.\n",
        "args = lambda: None\n",
        "args.num_paths = 256\n",
        "args.num_iter = 1000\n",
        "args.max_width = 50\n",
        "\n",
        "# Use GPU if available\n",
        "pydiffvg.set_use_gpu(torch.cuda.is_available())\n",
        "device = torch.device('cuda')\n",
        "pydiffvg.set_device(device)\n",
        "\n",
        "canvas_width, canvas_height = 224, 224\n",
        "num_paths = args.num_paths\n",
        "max_width = args.max_width\n",
        "\n",
        "# Image Augmentation Transformation\n",
        "augment_trans = transforms.Compose([\n",
        "    transforms.RandomPerspective(fill=1, p=1, distortion_scale=0.5),\n",
        "    transforms.RandomResizedCrop(224, scale=(0.7,0.9)),\n",
        "])\n",
        "\n",
        "if use_normalized_clip:\n",
        "    augment_trans = transforms.Compose([\n",
        "    transforms.RandomPerspective(fill=1, p=1, distortion_scale=0.5),\n",
        "    transforms.RandomResizedCrop(224, scale=(0.7,0.9)),\n",
        "    transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))\n",
        "])\n",
        "\n",
        "\n",
        "# Initialize Random Curves\n",
        "shapes = []\n",
        "shape_groups = []\n",
        "for i in range(num_paths):\n",
        "    num_segments = random.randint(1, 3)\n",
        "    num_control_points = torch.zeros(num_segments, dtype = torch.int32) + 2\n",
        "    points = []\n",
        "    p0 = (random.random(), random.random())\n",
        "    points.append(p0)\n",
        "    for j in range(num_segments):\n",
        "        radius = 0.1\n",
        "        p1 = (p0[0] + radius * (random.random() - 0.5), p0[1] + radius * (random.random() - 0.5))\n",
        "        p2 = (p1[0] + radius * (random.random() - 0.5), p1[1] + radius * (random.random() - 0.5))\n",
        "        p3 = (p2[0] + radius * (random.random() - 0.5), p2[1] + radius * (random.random() - 0.5))\n",
        "        points.append(p1)\n",
        "        points.append(p2)\n",
        "        points.append(p3)\n",
        "        p0 = p3\n",
        "    points = torch.tensor(points)\n",
        "    points[:, 0] *= canvas_width\n",
        "    points[:, 1] *= canvas_height\n",
        "    path = pydiffvg.Path(num_control_points = num_control_points, points = points, stroke_width = torch.tensor(1.0), is_closed = False)\n",
        "    shapes.append(path)\n",
        "    path_group = pydiffvg.ShapeGroup(shape_ids = torch.tensor([len(shapes) - 1]), fill_color = None, stroke_color = torch.tensor([random.random(), random.random(), random.random(), random.random()]))\n",
        "    shape_groups.append(path_group)\n",
        "\n",
        "# Just some diffvg setup\n",
        "scene_args = pydiffvg.RenderFunction.serialize_scene(\\\n",
        "    canvas_width, canvas_height, shapes, shape_groups)\n",
        "render = pydiffvg.RenderFunction.apply\n",
        "img = render(canvas_width, canvas_height, 2, 2, 0, None, *scene_args)\n",
        "points_vars = []\n",
        "stroke_width_vars = []\n",
        "color_vars = []\n",
        "for path in shapes:\n",
        "    path.points.requires_grad = True\n",
        "    points_vars.append(path.points)\n",
        "    path.stroke_width.requires_grad = True\n",
        "    stroke_width_vars.append(path.stroke_width)\n",
        "for group in shape_groups:\n",
        "    group.stroke_color.requires_grad = True\n",
        "    color_vars.append(group.stroke_color)\n",
        "\n",
        "# Optimizers\n",
        "points_optim = torch.optim.Adam(points_vars, lr=1.0)\n",
        "width_optim = torch.optim.Adam(stroke_width_vars, lr=0.1)\n",
        "color_optim = torch.optim.Adam(color_vars, lr=0.01)\n",
        "\n",
        "# Run the main optimization loop\n",
        "for t in range(args.num_iter):\n",
        "\n",
        "    # Anneal learning rate (makes videos look cleaner)\n",
        "    if t == int(args.num_iter * 0.5):\n",
        "        for g in points_optim.param_groups:\n",
        "            g['lr'] = 0.4\n",
        "    if t == int(args.num_iter * 0.75):\n",
        "        for g in points_optim.param_groups:\n",
        "            g['lr'] = 0.1\n",
        "    \n",
        "    points_optim.zero_grad()\n",
        "    width_optim.zero_grad()\n",
        "    color_optim.zero_grad()\n",
        "    scene_args = pydiffvg.RenderFunction.serialize_scene(\\\n",
        "        canvas_width, canvas_height, shapes, shape_groups)\n",
        "    img = render(canvas_width, canvas_height, 2, 2, t, None, *scene_args)\n",
        "    img = img[:, :, 3:4] * img[:, :, :3] + torch.ones(img.shape[0], img.shape[1], 3, device = pydiffvg.get_device()) * (1 - img[:, :, 3:4])\n",
        "    if t % 5 == 0:\n",
        "        pydiffvg.imwrite(img.cpu(), '/content/res/iter_{}.png'.format(int(t/5)), gamma=gamma)\n",
        "    img = img[:, :, :3]\n",
        "    img = img.unsqueeze(0)\n",
        "    img = img.permute(0, 3, 1, 2) # NHWC -> NCHW\n",
        "\n",
        "    loss = 0\n",
        "    NUM_AUGS = 4\n",
        "    img_augs = []\n",
        "    for n in range(NUM_AUGS):\n",
        "        img_augs.append(augment_trans(img))\n",
        "    im_batch = torch.cat(img_augs)\n",
        "    image_features = model.encode_image(im_batch)\n",
        "    for n in range(NUM_AUGS):\n",
        "        loss -= torch.cosine_similarity(text_features, image_features[n:n+1], dim=1)\n",
        "        if use_negative:\n",
        "            loss += torch.cosine_similarity(text_features_neg1, image_features[n:n+1], dim=1) * 0.3\n",
        "            loss += torch.cosine_similarity(text_features_neg2, image_features[n:n+1], dim=1) * 0.3\n",
        "\n",
        "    # Backpropagate the gradients.\n",
        "    loss.backward()\n",
        "\n",
        "    # Take a gradient descent step.\n",
        "    points_optim.step()\n",
        "    width_optim.step()\n",
        "    color_optim.step()\n",
        "    for path in shapes:\n",
        "        path.stroke_width.data.clamp_(1.0, max_width)\n",
        "    for group in shape_groups:\n",
        "        group.stroke_color.data.clamp_(0.0, 1.0)\n",
        "    \n",
        "    if t % 10 == 0:\n",
        "        show_img(img.detach().cpu().numpy()[0])\n",
        "        # show_img(torch.cat([img.detach(), img_aug.detach()], axis=3).cpu().numpy()[0])\n",
        "        print('render loss:', loss.item())\n",
        "        print('iteration:', t)\n",
        "        with torch.no_grad():\n",
        "            im_norm = image_features / image_features.norm(dim=-1, keepdim=True)\n",
        "            noun_norm = nouns_features / nouns_features.norm(dim=-1, keepdim=True)\n",
        "            similarity = (100.0 * im_norm @ noun_norm.T).softmax(dim=-1)\n",
        "            values, indices = similarity[0].topk(5)\n",
        "            print(\"\\nTop predictions:\\n\")\n",
        "            for value, index in zip(values, indices):\n",
        "                print(f\"{nouns[index]:>16s}: {100 * value.item():.2f}%\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Eru5XUuOCi6c",
        "cellView": "form"
      },
      "source": [
        "#@title Video Renderer {vertical-output: true}\n",
        "\n",
        "# Render a picture with each stroke.\n",
        "with torch.no_grad():\n",
        "    for i in range(args.num_paths):\n",
        "        print(i)\n",
        "        scene_args = pydiffvg.RenderFunction.serialize_scene(\\\n",
        "            canvas_width, canvas_height, shapes[:i+1], shape_groups[:i+1])\n",
        "        img = render(canvas_width, canvas_height, 2, 2, t, None, *scene_args)\n",
        "        img = img[:, :, 3:4] * img[:, :, :3] + torch.ones(img.shape[0], img.shape[1], 3, device = pydiffvg.get_device()) * (1 - img[:, :, 3:4])\n",
        "        pydiffvg.imwrite(img.cpu(), '/content/res/stroke_{}.png'.format(i), gamma=gamma)\n",
        "print(\"ffmpeging\")\n",
        "\n",
        "# Convert the intermediate renderings to a video.\n",
        "from subprocess import call\n",
        "call([\"ffmpeg\", \"-y\", \"-framerate\", \"60\", \"-i\",\n",
        "    \"/content/res/iter_%d.png\", \"-vb\", \"20M\",\n",
        "    \"/content/res/out.mp4\"])\n",
        "\n",
        "call([\"ffmpeg\", \"-y\", \"-framerate\", \"60\", \"-i\",\n",
        "    \"/content/res/stroke_%d.png\", \"-vb\", \"20M\",\n",
        "    \"/content/res/out_strokes.mp4\"])\n",
        "\n",
        "call([\"ffmpeg\", \"-y\", \"-i\", \"/content/res/out.mp4\", \"-filter_complex\",\n",
        "    \"[0]trim=0:2[hold];[0][hold]concat[extended];[extended][0]overlay\",\n",
        "    \"/content/res/out_longer.mp4\"])\n",
        "\n",
        "call([\"ffmpeg\", \"-y\", \"-i\", \"/content/res/out_strokes.mp4\", \"-filter_complex\",\n",
        "    \"[0]trim=0:2[hold];[0][hold]concat[extended];[extended][0]overlay\",\n",
        "    \"/content/res/out_strokes_longer.mp4\"])\n",
        "\n",
        "\n",
        "display(mvp.ipython_display(\"/content/res/out_longer.mp4\"))\n",
        "display(mvp.ipython_display(\"/content/res/out_strokes_longer.mp4\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b2jqNT0VYPWp",
        "cellView": "form"
      },
      "source": [
        "#@title Pixel Optimizer (Ignore) {vertical-output: true}\n",
        "\n",
        "%cd /content/diffvg/apps/\n",
        "\n",
        "prompt = \"Underwater\"\n",
        "text_input = clip.tokenize(prompt).to(device)\n",
        "\n",
        "# Calculate features\n",
        "with torch.no_grad():\n",
        "    text_features = model.encode_text(text_input)\n",
        "\n",
        "import torch\n",
        "import skimage\n",
        "import skimage.io\n",
        "import random\n",
        "import ttools.modules\n",
        "import argparse\n",
        "import math\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "\n",
        "\n",
        "\n",
        "class ImageBase(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.p = torch.nn.Parameter(torch.ones(224, 224, 3))\n",
        "    def forward(self):\n",
        "        return torch.nn.functional.sigmoid(self.p)\n",
        "\n",
        "device = torch.device('cuda')\n",
        "canvas_width, canvas_height = 224, 224\n",
        "\n",
        "augment_trans = transforms.Compose([\n",
        "    transforms.RandomPerspective(fill=1, p=1),\n",
        "    transforms.RandomResizedCrop(224, scale=(0.7,0.9)),\n",
        "])\n",
        "\n",
        "ib = ImageBase().to(device)\n",
        "\n",
        "t_img = imread('https://lh5.googleusercontent.com/mjvIYutjtOGEEU2cBYuFMvCrBCg4-MGh3DqCRlLqwn5I6VvdKdtwWvAYlndQbv-VUudPcecQ_TEGFjYaTuS_r0LNI83Sp8MlXJb6OarJ9mu-IkmKPlg9Gaw3gOjQvvgvuUB5ghJjlaE')\n",
        "target = torch.from_numpy(t_img).to(torch.float32)\n",
        "ib.p = target\n",
        "\n",
        "# Optimize\n",
        "optim = torch.optim.Adam(ib.parameters(), lr=0.01)\n",
        "# Adam iterations.\n",
        "for t in range(args.num_iter):\n",
        "    optim.zero_grad()\n",
        "    img = ib()\n",
        "    # Convert img from HWC to NCHW\n",
        "    img = img.unsqueeze(0)\n",
        "    img = img.permute(0, 3, 1, 2) # NHWC -> NCHW\n",
        "\n",
        "    loss = 0\n",
        "    for n in range(16):\n",
        "        img_aug = augment_trans(img)\n",
        "        image_features = model.encode_image(img_aug)\n",
        "        loss -= torch.cosine_similarity(text_features, image_features, dim=1)\n",
        "        # loss += torch.abs(torch.mean(1-img_aug)) * 0.1\n",
        "\n",
        "    # Backpropagate the gradients.\n",
        "    loss.backward()\n",
        "\n",
        "    # Take a gradient descent step.\n",
        "    optim.step()\n",
        "    \n",
        "    if t % 10 == 0:\n",
        "        show_img(img.detach().cpu().numpy()[0])\n",
        "        show_img(img_aug.detach().cpu().numpy()[0])\n",
        "        print('render loss:', loss.item())\n",
        "        print('iteration:', t)\n",
        "        with torch.no_grad():\n",
        "            im_norm = image_features / image_features.norm(dim=-1, keepdim=True)\n",
        "            noun_norm = nouns_features / nouns_features.norm(dim=-1, keepdim=True)\n",
        "            similarity = (100.0 * im_norm @ noun_norm.T).softmax(dim=-1)\n",
        "            values, indices = similarity[0].topk(5)\n",
        "            print(\"\\nTop predictions:\\n\")\n",
        "            for value, index in zip(values, indices):\n",
        "                print(f\"{nouns[index]:>16s}: {100 * value.item():.2f}%\")\n",
        "\n",
        "# Convert the intermediate renderings to a video.\n",
        "from subprocess import call\n",
        "call([\"ffmpeg\", \"-framerate\", \"24\", \"-i\",\n",
        "    \"results/painterly_rendering/iter_%d.png\", \"-vb\", \"20M\",\n",
        "    \"results/painterly_rendering/out.mp4\"])"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}