03_noninteractive/01_noninteractive_notebook_on_hpc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run demanding OpenMM notebooks non-interactively on an HPC\n",
    "\n",
    "**<span style=\"color:#A03;font-size:14pt\">\n",
    "&#x270B; HANDS-ON! &#x1F528;\n",
    "</span>**\n",
    "\n",
    "A non-interactive job can be configured and submitted to the queue as follows:\n",
    "\n",
    "1. If you have not used the HPC yet for other parts of the tutorial, copy the following files to a directory on the HPC:\n",
    "   - `01_noninteractive_notebook_on_hpc.ipynb`\n",
    "   - `alanine-dipeptide.pdb`\n",
    "   - `job_openmm_hpc_cpu.sh` (job script for generic HPC setup, may need tweaking)\n",
    "   - `job_openmm_vsc_cpu.sh` (works on VSC clusters)\n",
    "   - `job_openmm_vsc_gpu.sh` (works on VSC clusters with GPUs)\n",
    "\n",
    "1. The files starting with `job_` are job scripts that define the calculation that needs to be performed on a cluster.\n",
    "   We provide basic job scripts that may require some additional modifications to better suit your needs.\n",
    "   If the cluster you intend to use has GPUs, start from `job_openmm_vsc_gpu.sh`, or use `job_openmm_vsc_cpu.sh` otherwise.\n",
    "   (A more generic `job_openmm_hpc_cpu.sh` can be used, if you are not working with the VSC clusters.)\n",
    "   Both VSC scripts use only software installed on the cluster,\n",
    "   and not the Python environment set up for the course.\n",
    "\n",
    "1. When you reuse a job script for another notebook, you should edit it and change `01_noninteractive_notebook_on_hpc.ipynb` to the name of your notebook.\n",
    "   You can also change the `SBATCH` arguments to suit your needs:\n",
    "  \n",
    "   - Number of CPU cores is set by `--cpus-per-task`.\n",
    "   - Number of GPU cores is set by `--gpus-per-task`. (Not more than 1)\n",
    "   - Maximum wall time is set by `--time`. (Your job will be killed in case your calculation does not stop in time.)\n",
    "   - Maximum memory usage is set by `--mem`. (Your job will be killed when it uses more. If you don't specify this, the default is very low.)\n",
    "  \n",
    "   The `sbatch` program supports many other options to control the execution of the job.\n",
    "   Consult the [`sbatch` documentation](https://slurm.schedmd.com/sbatch.html) for more details.\n",
    "\n",
    "1. Finally, when your job script is ready, you can submit it by emtering the command `sbatch job_openmm_vsc_cpu.sh` in a virtual terminal.\n",
    "   This will put your job on the queue.\n",
    "   As soon as resources are available to run your job, it will be executed.\n",
    "   \n",
    "   > For **VSC** users:\n",
    "   > To debug a job script, it is recommended to test on the Donphan cluster first.\n",
    "   Once it is working as expected, you can switch to a production cluster.\n",
    "   > \n",
    "   > The default cluster is `doduo` when submitting jobs.\n",
    "   > If you want to use another cluster, you need to run the following command, before calling `sbatch`:\n",
    "   > \n",
    "   > ```bash\n",
    "   > module swap cluster/donphan\n",
    "   > ```\n",
    "   > \n",
    "   > To see the available clusters:\n",
    "   > \n",
    "   > ```bash\n",
    "   > module av cluster/\n",
    "   > ```\n",
    "\n",
    "1. You can check the status of your job by entering the `squeue` command in a virtual terminal.\n",
    "   VSC users can also check their job queue on [login.hpc.ugent.be](https://login.hpc.ugent.be).\n",
    "   Note that in a virtual terminal, you only get to see your jobs running on the current cluster.\n",
    "   Also for this, you need to switch to the right cluster, e.g. with `module swap cluster/...`.\n",
    "\n",
    "1. Once the job has completed, a copy of this notebook is created with all the outputs \n",
    "   (`01_run_openmm_on_a_hpc.nbconvert.ipynb`), and also all other output files can be found in the same directory.\n",
    "\n",
    "**Notes for the GPU job script:**\n",
    "\n",
    "- The GPU job script sets a variable `OPENMM_DEFAULT_PLATFORM=CUDA`, which tells OpenMM to use GPUs.\n",
    "\n",
    "- For VSC users:\n",
    "    - It only makes sense to submit `job_openmm_vsc_gpu.sh` on `joltik` or `accelgor` Tier-2 clusters:\n",
    "\n",
    "        - On `joltik`, it is recommended to use 8 CPUs per GPU.\n",
    "        - On `accelgor`, this becomes 12 CPUs per GPU.\n",
    "\n",
    "  - The GPU speedup for this notebook (on Joltik) is about a factor 30."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all the modules we need.\n",
    "from sys import stdout\n",
    "\n",
    "from openmm import *\n",
    "from openmm.app import *\n",
    "from openmm.unit import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code was taken from [../02/02_alanine_dipeptide.ipynb](../02/02_alanine_dipeptide.ipynb), example 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdb = PDBFile(\"alanine-dipeptide.pdb\")\n",
    "modeller = Modeller(pdb.topology, pdb.positions)\n",
    "forcefield = ForceField(\"amber14-all.xml\", \"amber14/tip3pfb.xml\")\n",
    "modeller.addSolvent(forcefield, model=\"tip3p\", padding=1.2 * nanometer)\n",
    "print(modeller.topology)\n",
    "# Write a PDB file to provide a topology of the solvated\n",
    "# system to MDTraj below.\n",
    "with open(\"init3.pdb\", \"w\") as outfile:\n",
    "    PDBFile.writeFile(modeller.topology, modeller.positions, outfile)\n",
    "\n",
    "# The modeller builds a periodic box with the solute and solvent molecules.\n",
    "# PME is the method to compute long-range electristatic interactions in\n",
    "# periodic systems.\n",
    "system = forcefield.createSystem(modeller.topology, nonbondedMethod=PME, constraints=HBonds)\n",
    "temperature = 300 * kelvin\n",
    "pressure = 1 * bar\n",
    "integrator = LangevinIntegrator(temperature, 1 / picosecond, 2 * femtoseconds)\n",
    "system.addForce(MonteCarloBarostat(pressure, temperature))\n",
    "simulation = Simulation(modeller.topology, system, integrator)\n",
    "simulation.context.setPositions(modeller.positions)\n",
    "simulation.minimizeEnergy()\n",
    "simulation.reporters.append(DCDReporter(\"traj3.dcd\", 100))\n",
    "simulation.reporters.append(\n",
    "    StateDataReporter(stdout, 1000, step=True, temperature=True, elapsedTime=True)\n",
    ")\n",
    "simulation.reporters.append(\n",
    "    StateDataReporter(\n",
    "        \"scalars3.csv\",\n",
    "        100,\n",
    "        time=True,\n",
    "        potentialEnergy=True,\n",
    "        totalEnergy=True,\n",
    "        temperature=True,\n",
    "    )\n",
    ")\n",
    "simulation.step(100000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}