-ESGF Cookbook
-
-
-
This Project Pythia Cookbook covers how to access and analyze datasets that can be accessed from Earth System Grid Federation (ESGF) cyberinfrastructure.
-Motivation
-This cookbook focuses on highlighting analysis recipes, as well as data acccess methods, all accesible within the Python programming language. This cookbook also spans beyond the scope of a single Climate Model Intercomparison Project (ex. CMIP6), expanding to other experiments/datasets such as CMIP5 and obs4MIPs.
-Authors
-Max Grover, Nathan Collier, Carsten Ehbrecht, Jacqueline Nugent, Gerardo Rivera Tello
-Contributors
- -Structure
- - -Running the Notebooks
-You can either run the notebook using the NIMBUS Juptyerhub or on your local machine.
-Running on Binder
-The simplest way to interact with a Jupyter Notebook is through -the NIMBUS Juptyerhub, which enables the execution of a -Jupyter Book in the cloud-like infrastructure. The details of how this works are not -important for now. All you need to know is how to launch a Pythia -Cookbooks chapter via Binder. Simply navigate your mouse to -the top right corner of the book chapter you are viewing and click -on the rocket ship icon, (see figure below), and be sure to select -“launch Binder”. After a moment you should be presented with a -notebook that you can interact with. I.e. you’ll be able to execute -and even change the example programs. You’ll see that the code cells -have no output at first, until you execute them by pressing -Shift+Enter. Complete details on how to interact with -a live Jupyter notebook are described in Getting Started with -Jupyter.
-Running on Your Own Machine
-If you are interested in running this material locally on your computer, you will need to follow this workflow:
-(Replace “cookbook-example” with the title of your cookbooks)
--
-
Clone the
-https://github.com/esgf2-us/esgf-cookbookrepository:--git clone https://github.com/esgf2-us/esgf-cookbook.git -
-Move into the
-cookbook-exampledirectory--cd esgf-cookbook -
-Create and activate your conda environment from the
-environment.ymlfile--conda env create -f environment.yml -conda activate esgf-cookbook-dev -
-Move into the
-notebooksdirectory and start up Jupyterlab--cd notebooks/ -jupyter lab -
-
-
-# ESGF Cookbook
-
-[](https://github.com/ProjectPythia/esgf-cookbook/actions/workflows/nightly-build.yaml)
-[](https://binder.projectpythia.org/v2/gh/ProjectPythia/esgf-cookbook/main?labpath=notebooks)
-[](https://doi.org/10.5281/zenodo.11663067)
-
-This Project Pythia Cookbook covers how to access and analyze datasets that can be accessed from Earth System Grid Federation (ESGF) cyberinfrastructure.
-
-## Motivation
-
-This cookbook focuses on highlighting analysis recipes, as well as data acccess methods, all accesible within the Python programming language. This cookbook also spans beyond the scope of a single Climate Model Intercomparison Project (ex. CMIP6), expanding to other experiments/datasets such as CMIP5 and obs4MIPs.
-
-## Authors
-
-[Max Grover](@mgrover1), [Nathan Collier](@nocollier), [Carsten Ehbrecht](@cehbrecht), [Jacqueline Nugent](@jacnugent), [Gerardo Rivera Tello](@griverat)
-
-### Contributors
-
-![\"ESGF](\"images/logos/esgf2-us.png\")
"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Complex Searching with `intake-esgf`\n",
- "\n",
- "## Overview\n",
- "\n",
- "In this tutorial we will present an interface under design to facilitate complex searching using [intake-esgf](https://github.com/esgf2-us/intake-esgf). `intake-esgf` is a small `intake` and `intake-esm` *inspired* package under development in ESGF2. Please note that there is a name collison with an existing package in PyPI and conda. You will need to install the package from [source](https://github.com/esgf2-us/intake-esgf).\n",
- "\n",
- "## Prerequisites\n",
- "\n",
- "| Concepts | Importance | Notes |\n",
- "| --- | --- | --- |\n",
- "| [Install Package](https://github.com/esgf2-us/intake-esgf) | Necessary | |\n",
- "| [Understanding of NetCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html) | Helpful | Familiarity with metadata structure |\n",
- "| Familiar with [intake-esm](https://intake-esm.readthedocs.io/en/stable/) | Helpful | Similar interface |\n",
- "| [Transient climate response](https://doi.org/10.1029/2008JD010405) | Background | |\n",
- "- **Time to learn**: 30 minutes\n",
- "\n",
- "## Imports"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "import intake_esgf\n",
- "from intake_esgf import ESGFCatalog"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Initializing the Catalog\n",
- "\n",
- "As with `intake-esm` we first instantiate the catalog. However, since we will populate the catalog with search results, the catalog starts empty. Internally, we query different ESGF index nodes for information about what datasets you wish to include in your analysis. As ESGF2 is actively working on an index redesign, our catlogs by default point to a Globus (ElasticSearch) based index at ALCF (Argonne Leadership Computing Facility)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Perform a search() to populate the catalog.\n",
- "GlobusESGFIndex('anl-dev')\n",
- "GlobusESGFIndex('ornl-dev')\n"
- ]
- }
- ],
- "source": [
- "cat = ESGFCatalog()\n",
- "print(cat)\n",
- "for ind in cat.indices: # Which indices are included?\n",
- " print(ind)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We also provide support for connecting to the ESGF1 Solr-based indices. You may specify a server in the dictionary or multiple servers - just make sure to include `True`.\n",
- "\n",
- "Uncommend the line setting `all_indices=True` to include all available indices."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "GlobusESGFIndex('anl-dev')\n",
- "GlobusESGFIndex('ornl-dev')\n",
- "SolrESGFIndex('esgf.ceda.ac.uk')\n",
- "SolrESGFIndex('esgf-data.dkrz.de')\n",
- "SolrESGFIndex('esgf-node.ipsl.upmc.fr')\n",
- "SolrESGFIndex('esg-dn1.nsc.liu.se')\n",
- "SolrESGFIndex('esgf-node.llnl.gov')\n",
- "SolrESGFIndex('esgf.nci.org.au')\n",
- "SolrESGFIndex('esgf-node.ornl.gov')\n",
- "SolrESGFIndex('esgf-fedtest.llnl.gov')\n"
- ]
- }
- ],
- "source": [
- "intake_esgf.conf.set(indices={\"esgf-node.llnl.gov\": True,\n",
- " \"esgf-node.ornl.gov\": True,\n",
- " \"esgf.ceda.ac.uk\": True})\n",
- "\n",
- "intake_esgf.conf.set(all_indices=True) # all federated indices\n",
- "\n",
- "cat = ESGFCatalog()\n",
- "for ind in cat.indices:\n",
- " print(ind)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Populate the catalog\n",
- "\n",
- "Many times, an analysis will require several variables across multiple experiments. For example, if one were to compute the transient climate response (TCRE), you would need tempererature (`tas`) and carbon emissions from land (`nbp`) and ocean (`fgco2`) for a 1% CO2 increase experiment (`1pctCO2`) as well as the control experiment (`piControl`). If TCRE is not in your particular science, that is ok for this notebook. It is a motivating example and the specifics are less important than the search concepts. First, we perform a search in a familiar syntax."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "7d80531e054547bdb6c6bfc22e94137d",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- " Searching indices: 0%| |0/5 [ ?index/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Summary information for 390 results:\n",
- "mip_era [CMIP6]\n",
- "activity_drs [CMIP]\n",
- "institution_id [MOHC, MRI, MPI-M, NCAR, NOAA-GFDL, NCC, NIMS-...\n",
- "source_id [UKESM1-0-LL, MRI-ESM2-0, MPI-ESM1-2-LR, CESM2...\n",
- "experiment_id [piControl, 1pctCO2]\n",
- "member_id [r1i1p1f2, r1i2p1f1, r1i1p1f1, r2i1p1f1, r3i1p...\n",
- "table_id [Lmon, Omon, Amon]\n",
- "variable_id [nbp, fgco2, tas]\n",
- "grid_label [gn, gr1, gr]\n",
- "dtype: object\n"
- ]
- }
- ],
- "source": [
- "cat.search(\n",
- " experiment_id=[\"piControl\", \"1pctCO2\"],\n",
- " variable_id=[\"tas\", \"fgco2\", \"nbp\"],\n",
- " table_id=[\"Amon\", \"Omon\", \"Lmon\"],\n",
- ")\n",
- "print(cat)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Internally, this launches simultaneous searches that are combined locally to provide a global view of what datasets are available. While the Solr indices themselves can be searched in distributed fashion, they will not report if an index has failed to return a response. As index nodes go down from time to time, this can leave you with a false impression that you have found all the datasets of interest. By managing the searches locally, `intake-esgf` can report back to you that an index has failed and that your results may be incomplete.\n",
- "\n",
- "If you would like details about what `intake-esgf` is doing, look in the local cache directory (`${HOME}/.esgf/`) for a `esgf.log` file. This is a full history of everything that `intake-esgf` has searched, downloaded, or accessed. You can also look at just this session by calling `session_log()`. In this case you will see how long each index took to return a response and if any failed"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[36;20m2024-12-06 15:50:33 \u001b[0m└─GlobusESGFIndex('anl-dev') results=329 response_time=1.24\n",
- "\u001b[36;20m2024-12-06 15:50:34 \u001b[0m└─GlobusESGFIndex('ornl-dev') results=650 response_time=1.94\n",
- "\u001b[36;20m2024-12-06 15:50:42 \u001b[0m└─SolrESGFIndex('esgf.ceda.ac.uk') results=231 response_time=10.25\n",
- "\u001b[36;20m2024-12-06 15:50:54 \u001b[0m└─SolrESGFIndex('esgf-node.ornl.gov') results=650 response_time=21.72\n",
- "\u001b[36;20m2024-12-06 15:51:16 \u001b[0m└─SolrESGFIndex('esgf-node.llnl.gov') results=680 response_time=44.04\n",
- "\u001b[36;20m2024-12-06 15:51:16 \u001b[0mcombine_time=0.16\n",
- "\u001b[36;20m2024-12-06 15:51:16 \u001b[0m\u001b[36;32msearch end\u001b[0m total_time=44.27\n",
- "\n"
- ]
- }
- ],
- "source": [
- "print(cat.session_log())"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "At this stage of the search you have a catalog full of possibly relevant datasets for your analysis, stored in a `pandas` dataframe. You are free to view and manipulate this dataframe to help hone these results down. It is available to you as the `df` member of the `ESGFCatalog`. You should be careful to only remove rows as internally we could use any column in the downloading of the data. Also note that we have removed the user-facing notion of *where* the data is hosted. The `id` column of this dataframe is a list of full `dataset_ids` which includes the location information. At the point when you are ready to download data, we will choose locations automatically that are fastest for you."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "![\"ESGF](\"images/esgf.png\")
\n",
- "![\"Rooki](\"images/logos/rooki_logo.png\"){kind=logo}
\n",
- "![\"Cartopy](\"images/logos/cartopy_logo.png\"){kind=logo}
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "50b5d7e7-df4e-4992-a29b-8804b081a320",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Demo: Regridding and Plotting with Rooki and Cartopy \n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "abd4b497-cdbf-4c29-857c-3017abf9e927",
- "metadata": {
- "tags": []
- },
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0f79862d-7181-4f04-966c-19b5e03a22a5",
- "metadata": {},
- "source": [
- "## Overview\n",
- "In this notebook, we demonstrate how to use Rooki to regrid CMIP model data and plot it in Cartopy for two examples:\n",
- "\n",
- "1. Regrid two CMIP models onto the same grid \n",
- "1. Coarsen the output for one model"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4f1db315-fb2d-466d-bd6e-8a4ef18b6cf1",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Prerequisites\n",
- "\n",
- "| Concepts | Importance | Notes |\n",
- "| --- | --- | --- |\n",
- "| [Intro to intake-esgf](https://projectpythia.org/esgf-cookbook/notebooks/intro-search.html) | Necessary | |\n",
- "| [Intro to Cartopy](https://foundations.projectpythia.org/core/cartopy/cartopy.html) | Necessary | |\n",
- "| [Using Rooki to access CMIP6 data](https://projectpythia.org/esgf-cookbook/notebooks/rooki.html) | Helpful | Familiarity with rooki |\n",
- "| [Understanding of NetCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html) | Helpful | Familiarity with metadata structure |\n",
- "\n",
- "- **Time to learn**: 15 minutes\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7cbc5d91-db3f-4afd-9093-c3abc7dec82b",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2582d535-9b99-4115-b0ee-7459acd76ec0",
- "metadata": {},
- "source": [
- "## Imports\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ad4562c9-f5eb-496e-9e17-6453f426e910",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "import os\n",
- "import intake_esgf\n",
- "\n",
- "# Run this on the DKRZ node in Germany, using the ESGF1 index node at LLNL\n",
- "os.environ[\"ROOK_URL\"] = \"http://rook.dkrz.de/wps\"\n",
- "intake_esgf.conf.set(indices={\"anl-dev\": False,\n",
- " \"ornl-dev\": False,\n",
- " \"esgf-node.llnl.gov\": True})\n",
- "\n",
- "import rooki.operators as ops\n",
- "import matplotlib.pyplot as plt\n",
- "import matplotlib.colors as mcolors\n",
- "import cartopy.crs as ccrs\n",
- "import cartopy.feature as cfeature\n",
- "\n",
- "import intake_esgf\n",
- "from intake_esgf import ESGFCatalog\n",
- "from rooki import rooki\n",
- "from matplotlib.gridspec import GridSpec\n",
- "from mpl_toolkits.axes_grid1.inset_locator import inset_axes\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c2b47b1d-db2d-4074-8c92-bb71fa0459a7",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Example 1: Regrid two CMIP6 models onto the same grid"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cc1d512a-68d3-43cf-aac7-6ca233d9ef73",
- "metadata": {},
- "source": [
- "In this example, we want to compare the historical precipitation output between two CMIP models, CESM2 and CanESM5. Here will will look at the annual mean precipitation for 2010. "
- ]
- },
- {
- "cell_type": "markdown",
- "id": "46f5fba3-7410-465c-abdf-4e338855284c",
- "metadata": {},
- "source": [
- "### Access the desired datasets using intake-esgf and rooki"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c5f4dc65-0dff-4023-880c-f511cbc58666",
- "metadata": {
- "tags": []
- },
- "source": [
- "The function and workflow to read in CMPI6 data using `intake-esgf` and `rooki` in the next few cells are adapted from [intake-esgf-with-rooki.ipynb](https://github.com/ProjectPythia/esgf-cookbook/blob/cf69015a464b68ee28cfdd4a27cee4e9d6ca2ca9/notebooks/use-intake-esgf-with-rooki.ipynb). Essentially, we use `intake-esgf` to find the dataset IDs we want and then subset and average them using `rooki`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d053a676-2a27-4be0-93c0-eafb9671c0bc",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def separate_dataset_id(id_list):\n",
- " rooki_id = id_list[0]\n",
- " rooki_id = rooki_id.split(\"|\")[0]\n",
- " #rooki_id = f\"css03_data.{rooki_id}\" # <-- just something you have to know for now :(\n",
- " return rooki_id\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "095db615-275a-4dbc-8467-833fd7992aed",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "cat = ESGFCatalog()\n",
- "cat.search(\n",
- " activity_id='CMIP',\n",
- " experiment_id=[\"historical\",],\n",
- " variable_id=[\"pr\"],\n",
- " member_id='r1i1p1f1',\n",
- " grid_label='gn',\n",
- " table_id=\"Amon\",\n",
- " source_id = [ \"CESM2\", \"CanESM5\"]\n",
- " )\n",
- "\n",
- "dsets = [separate_dataset_id(dataset) for dataset in list(cat.df.id.values)]\n",
- "dsets\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "777f6bc4-f3a8-4110-bc2a-82cbf227ec4e",
- "metadata": {},
- "source": [
- "Subset the data to get the precipitation variable for 2010 and then average by time:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "bf653879-96b5-48e0-be9b-0f0cc08152e2",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "dset_list = [[]]*len(dsets)\n",
- "\n",
- "for i, dset_id in enumerate(dsets):\n",
- " wf = ops.AverageByTime(\n",
- " ops.Subset(\n",
- " ops.Input('pr', [dset_id]),\n",
- " time='2010/2010'\n",
- " )\n",
- " )\n",
- "\n",
- " resp = wf.orchestrate()\n",
- "\n",
- " # if it worked, add the dataset to our list\n",
- " if resp.ok:\n",
- " dset_list[i] = resp.datasets()[0]\n",
- " \n",
- " # if it failed, tell us why\n",
- " else:\n",
- " print(resp.status)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e040d078-3981-4246-a10b-c50cf104d8ed",
- "metadata": {},
- "source": [
- "Print the dataset list to get an overview of the metadata structure:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b2ed096a-2cfc-4e51-9b2a-43b9ee4f103e",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(dset_list)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "776f84fd-e329-42e8-bab4-54253636aefc",
- "metadata": {},
- "source": [
- "### Compare the precipitation data between models"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ee469ea1-e402-4e55-b709-0de01e7875b3",
- "metadata": {},
- "source": [
- "First, let's quickly plot the 2010 annual mean precipitation for each model to see what we're working with. Since precipitation values vary greatly in magnitude, using a log-normalized colormap makes the data easier to visualize. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e49b55e3-1970-4410-8557-9328f31853fb",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "for dset in dset_list:\n",
- " dset.pr.plot(norm=mcolors.LogNorm())\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6cb2aca3-16b4-4bc3-986b-3b1c3e3b4c51",
- "metadata": {
- "jp-MarkdownHeadingCollapsed": true,
- "tags": []
- },
- "source": [
- "Uncomment and run the following cell. If we try to take the difference outright, it fails! "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4276c97e-d798-42b7-846b-98f6460ce897",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# pr_diff = dset_list[0].pr - dset_list[1].pr"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5c7745fe-2ddd-4232-ad1c-c76601909db7",
- "metadata": {
- "tags": []
- },
- "source": [
- "The models have different grids so we can't directly subtract the data. We can use the `grid` attribute to get information on which grid each uses."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "11beadd3-beef-4337-a6cf-1fdd1d657ddf",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(dset_list[0].grid)\n",
- "print(dset_list[1].grid)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4947c0a2-e796-4a57-aaaf-d4ba07fb26a6",
- "metadata": {},
- "source": [
- "### Regrid the models onto the same grid with Rooki"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4823272f-614b-437c-8331-95c24c267b47",
- "metadata": {
- "tags": []
- },
- "source": [
- "Look at the documentation on the `regrid` operator to see the available grid types and regrid methods:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "423e4302-6aa6-42e7-9686-d99c1b8cd3af",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "rooki.regrid?"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8f6d5022-7973-4133-b321-5b4804f5eb9b",
- "metadata": {
- "tags": []
- },
- "source": [
- "Here we'll do the same process as before to read in and subset the datasets with rooki, but now we **regrid using `ops.Regrid` before averaging over time**. In this example, we use `method=nearest_s2d` to regrid each model onto the target grid using a nearest neighbors method. The target grid is a 1.25° grid, specified by `grid='1pt25deg'`.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4993b311-f18b-4d79-8902-a6040aff3271",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "rg_list = [[]]*len(dsets)\n",
- "\n",
- "for i, dset_id in enumerate(dsets):\n",
- " wf = ops.AverageByTime(\n",
- " ops.Regrid(\n",
- " ops.Subset(\n",
- " ops.Input('pr', [dset_id]),\n",
- " time='2010/2010'\n",
- " ),\n",
- " method='nearest_s2d',\n",
- " grid='1pt25deg'\n",
- " )\n",
- " )\n",
- "\n",
- "\n",
- " resp = wf.orchestrate()\n",
- " \n",
- " # if it worked, add the regridded dataset to our list\n",
- " if resp.ok:\n",
- " rg_list[i] = resp.datasets()[0]\n",
- " \n",
- " # if it failed, tell us why\n",
- " else:\n",
- " print(resp.status)\n",
- " "
- ]
- },
- {
- "cell_type": "markdown",
- "id": "dfb8efaf-a4ab-4709-b1eb-a38c6f4bade2",
- "metadata": {},
- "source": [
- "Print the list of regridded datasets to get an overview of the metadata structure. Note how `lat` and `lon` are now the same and each dataset has additional attributes, including `grid_original` and `regrid_operation`, which all keep track of the regridding operations we just completed."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e38e4a39-0465-4ed8-8a5c-6bdcd61d77c1",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(rg_list)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d048801f-fb36-46b2-84af-f1b61768727c",
- "metadata": {},
- "source": [
- "Now they are on the same grid!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "59b6fefa-90e7-49c7-b816-27351a0f51f4",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(rg_list[0].grid)\n",
- "print(rg_list[1].grid)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "168c41e8-7ae2-4ee6-9020-94f9de00500d",
- "metadata": {},
- "source": [
- "### Quick plot the before and after for each model\n",
- "The plots largely look the same, as they should - with the nearest neighbors method, we are just shifting the precipitation data onto a different grid without averaging between grid cells."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a9d5213f-65a1-415f-b3fe-6860d70fd14d",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(dset_list[0].source_id)\n",
- "for ds in [dset_list[0], rg_list[0]]:\n",
- " ds.pr.plot(norm=mcolors.LogNorm())\n",
- " plt.show()\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0fabcf8b-c9e2-40c4-b7e1-ce7de3809d29",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "print(dset_list[1].source_id)\n",
- "for ds in [dset_list[1], rg_list[1]]:\n",
- " ds.pr.plot(norm=mcolors.LogNorm())\n",
- " plt.show()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "36031d82-c46f-4da5-8cbb-82067ade465b",
- "metadata": {},
- "source": [
- "#### Take the difference between precipitation datasets and plot it\n",
- "Now that both models are on the same grid, we can subtract the precipitation datasets and plot the difference!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4f781e26-0c43-45e9-be89-31d3575f4c99",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "pr_diff = rg_list[0] - rg_list[1]\n",
- "\n",
- "pr_diff.pr.plot(cmap=\"bwr\")\n",
- "plt.show()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e1fc1957-c334-4431-9e0a-15715106b5d6",
- "metadata": {},
- "source": [
- "### Plot everything together\n",
- "Plot the regridded precipitation data as well as the difference between models on the same figure. We can use `Cartopy` to make it pretty. With `GridSpec`, we can also split up the figure and organize it to use the same colorbar for more than one panel."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "86ab0c4a-ce76-48ab-843d-e79333bdd58c",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# set up figure\n",
- "fig = plt.figure(figsize=(6, 8))\n",
- "gs = GridSpec(3, 2, width_ratios=[1, 0.1], hspace=0.2)\n",
- "\n",
- "# specify the projection\n",
- "proj = ccrs.Mollweide()\n",
- "\n",
- "# set up plots for each model\n",
- "axpr_1 = plt.subplot(gs[0, 0], projection=proj)\n",
- "axpr_2 = plt.subplot(gs[1, 0], projection=proj)\n",
- "axdiff = plt.subplot(gs[2, 0], projection=proj)\n",
- "\n",
- "# axes where the colorbar will go \n",
- "axcb_pr = plt.subplot(gs[:2, 1]) \n",
- "axcb_diff = plt.subplot(gs[2, 1])\n",
- "axcb_pr.axis(\"off\")\n",
- "axcb_diff.axis(\"off\")\n",
- "\n",
- "# plot the precipitation for both models\n",
- "for i, ax in enumerate([axpr_1, axpr_2]):\n",
- " ds_rg = rg_list[i]\n",
- " pcm = ax.pcolormesh(ds_rg.lon, ds_rg.lat, ds_rg.pr.isel(time=0), norm=mcolors.LogNorm(vmin=1e-7, vmax=3e-4),\n",
- " transform=ccrs.PlateCarree()\n",
- " )\n",
- " ax.set_title(ds_rg.parent_source_id)\n",
- " ax.add_feature(cfeature.COASTLINE)\n",
- " \n",
- "# now plot the difference\n",
- "pcmd = axdiff.pcolormesh(pr_diff.lon, pr_diff.lat, pr_diff.pr.isel(time=0), cmap=\"bwr\", vmin=-3e-4, vmax=3e-4,\n",
- " transform=ccrs.PlateCarree()\n",
- " )\n",
- "axdiff.set_title(\"{a} - {b}\".format(a=rg_list[0].parent_source_id, b=rg_list[1].parent_source_id))\n",
- "axdiff.add_feature(cfeature.COASTLINE)\n",
- "\n",
- "# set the precipitation colorbar\n",
- "axcb_pr_ins = inset_axes(axcb_pr, width=\"50%\", height=\"75%\", loc=\"center\")\n",
- "cbar_pr = plt.colorbar(pcm, cax=axcb_pr_ins, orientation=\"vertical\", extend=\"both\")\n",
- "cbar_pr.set_label(\"{n} ({u})\".format(n=rg_list[0].pr.long_name, u=rg_list[0].pr.units))\n",
- "\n",
- "# set the difference colorbar\n",
- "axcb_diff_ins = inset_axes(axcb_diff, width=\"50%\", height=\"100%\", loc=\"center\")\n",
- "cbar_diff = plt.colorbar(pcmd, cax=axcb_diff_ins, orientation=\"vertical\", extend=\"both\")\n",
- "cbar_diff.set_label(\"Difference ({u})\".format(u=pr_diff.pr.units))\n",
- "\n",
- "plt.show()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a77d6b7b-5729-4ba3-9e72-6f960dbe3253",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Example 2: Coarsen the output for one model"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "919dd269-6a7a-48d9-a477-f1e76ed48d11",
- "metadata": {},
- "source": [
- "We can also use `Rooki` to regrid the data from one model onto a coarser grid. In this case, it may make more sense to use a conservative regridding method, which will conserve the physical fluxes between grid cells, rather than the nearest neighbors method we used in Example 1. "
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d32f8ab0-05ef-41a6-a599-f4a289103c2e",
- "metadata": {},
- "source": [
- "### Get the data using intake-esgf and Rooki again"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "81cdf527-cec0-4353-98d7-29cc3effa014",
- "metadata": {},
- "source": [
- "In this example, we'll look at the annual mean near-surface air temperature for CESM2 in 2010. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d64f5c8d-6355-4225-b5c8-8f2c344fa241",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "cat = ESGFCatalog()\n",
- "cat.search(\n",
- " activity_id='CMIP',\n",
- " experiment_id=[\"historical\",],\n",
- " variable_id=[\"tas\"],\n",
- " member_id='r1i1p1f1',\n",
- " grid_label='gn',\n",
- " table_id=\"Amon\",\n",
- " source_id = [ \"CESM2\"]\n",
- " )\n",
- "\n",
- "dsets = [separate_dataset_id(dataset) for dataset in list(cat.df.id.values)]\n",
- "dsets\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a647dc7d-e107-49ef-87de-08dbf9024cd2",
- "metadata": {},
- "source": [
- "First, get the dataset with the original grid:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9c70d292-6e0e-43dd-b64e-5b143e38590e",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "wf = ops.AverageByTime(\n",
- " ops.Subset(\n",
- " ops.Input('tas', [dsets[0]]),\n",
- " time='2010/2010'\n",
- " )\n",
- ")\n",
- "\n",
- "resp = wf.orchestrate()\n",
- "\n",
- "if resp.ok:\n",
- " ds_og = resp.datasets()[0]\n",
- "else:\n",
- " print(resp.status)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "138de51a-727e-4f47-8876-eccbfc6d74bd",
- "metadata": {},
- "source": [
- "Use the `.grid` attribute to get information on the native grid:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b43c0357-ef9e-469d-b2e7-1635d0387ee5",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "ds_og.grid"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a52f11e3-a17f-4a74-849e-a4219de6a2c9",
- "metadata": {},
- "source": [
- "The native grid is 0.9°x1.25°, so let's try coarsening to a 1.25°x1.25° grid using the conservative method:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "710606ac-1c41-4b27-b2c4-2ae4957357ea",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "wf = ops.AverageByTime(\n",
- " ops.Regrid(\n",
- " ops.Subset(\n",
- " ops.Input('tas', [dsets[0]]),\n",
- " time='2010/2010'\n",
- " ),\n",
- " method='conservative',\n",
- " grid='1pt25deg'\n",
- " )\n",
- ")\n",
- "\n",
- "resp = wf.orchestrate()\n",
- "\n",
- "if resp.ok:\n",
- " ds_125 = resp.datasets()[0]\n",
- "else:\n",
- " print(resp.status)\n",
- " "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f51dd55a-a419-4a24-bc2f-4bdaeb435318",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "ds_125.grid"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a8556572-54a6-42b0-94d8-cc562d40ccd3",
- "metadata": {},
- "source": [
- "We can also make it even coarser by regridding to a 2.5°x2.5° grid:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "71a84338-90cf-49ec-b120-8f071213cf3e",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "wf = ops.AverageByTime(\n",
- " ops.Regrid(\n",
- " ops.Subset(\n",
- " ops.Input('tas', [dsets[0]]),\n",
- " time='2010/2010'\n",
- " ),\n",
- " method='conservative',\n",
- " grid='2pt5deg'\n",
- " )\n",
- ")\n",
- "\n",
- "resp = wf.orchestrate()\n",
- "\n",
- "if resp.ok:\n",
- " ds_25 = resp.datasets()[0]\n",
- "else:\n",
- " print(resp.status)\n",
- " "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "412c50c9-0a30-4f28-8c97-c8a3d957de51",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "ds_25.grid"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "087503d7-e4ac-42f8-82a6-770a92795780",
- "metadata": {},
- "source": [
- "### Plot each dataset to look at the coarsened grids"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "020b10a4-95ff-4ecf-a203-32db9f3beaf9",
- "metadata": {},
- "source": [
- "Make a quick plot first:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "96f1af51-6836-436f-a09f-1d6375614f98",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "for ds in [ds_og, ds_125, ds_25]:\n",
- " ds[\"tas\"].plot()\n",
- " plt.show()\n",
- " "
- ]
- },
- {
- "cell_type": "markdown",
- "id": "919dc7a1-ad2a-4291-aa90-c2d622f1d433",
- "metadata": {},
- "source": [
- "### Plot the coarsened datsets together using Cartopy"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e4a00e56-4b5c-47da-ae9c-ae4b1ce0686d",
- "metadata": {
- "tags": []
- },
- "source": [
- "Now let's zoom in on a smaller region, the continental US, to get a clear view of the difference in grid resolution. Here we can also decrease the colorbar limits to better see how the variable `tas` varies within the smaller region."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0679c3e5-f2bc-4e50-8b4d-f0cefa54d65a",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# set up the figure\n",
- "fig = plt.figure(figsize=(6, 8))\n",
- "gs = GridSpec(3, 2, width_ratios=[1, 0.1], height_ratios=[1, 1, 1], hspace=0.3, wspace=0.2)\n",
- "\n",
- "# specify the projection\n",
- "proj = ccrs.PlateCarree()\n",
- "\n",
- "# set up plot axes\n",
- "ax1 = plt.subplot(gs[0, 0], projection=proj)\n",
- "ax2 = plt.subplot(gs[1, 0], projection=proj)\n",
- "ax3 = plt.subplot(gs[2, 0], projection=proj)\n",
- "axes_list = [ax1, ax2, ax3]\n",
- "\n",
- "# set up colorbar axis\n",
- "axcb = plt.subplot(gs[:, 1])\n",
- "\n",
- "# loop through each dataset and its corresponding axis\n",
- "for i, dset in enumerate([ds_og, ds_125, ds_25]):\n",
- " plot_ds = dset.tas.isel(time=0)\n",
- " ax = axes_list[i]\n",
- " pcm = ax.pcolormesh(plot_ds.lon, plot_ds.lat, plot_ds, vmin=270, vmax=302.5, transform=proj)\n",
- " \n",
- " # add borders and coastlines\n",
- " ax.add_feature(cfeature.BORDERS)\n",
- " ax.coastlines()\n",
- " \n",
- " # limit to CONUS for this example\n",
- " ax.set_xlim(-130, -60)\n",
- " ax.set_ylim(22, 52)\n",
- " \n",
- " # add grid labels on bottom & left only\n",
- " gl = ax.gridlines(color=\"None\", draw_labels=True)\n",
- " gl.top_labels = False\n",
- " gl.right_labels = False\n",
- " \n",
- " # label with the regrid type; if it fails, that means it hasn't been regridded\n",
- " # (so label with the grid attribute instead)\n",
- " try:\n",
- " ax.set_title(dset.regrid_operation)\n",
- " except:\n",
- " ax.set_title(dset.grid)\n",
- " \n",
- "# use the same colorbar for all plots\n",
- "axcb.axis(\"off\")\n",
- "axcb_ins = inset_axes(axcb, width=\"50%\", height=\"75%\", loc=\"center\")\n",
- "cbar = plt.colorbar(pcm, cax=axcb_ins, orientation=\"vertical\", extend=\"both\")\n",
- "cbar.set_label(\"{n} ({u})\".format(n=plot_ds.long_name, u=plot_ds.units))\n",
- " \n",
- "plt.show()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e57aee30-caff-4916-bb95-efa00ff15ba4",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9f883994-e2f8-4ce9-8ca2-56fedb2e1a58",
- "metadata": {},
- "source": [
- "## Summary\n",
- "\n",
- "`Rooki` offers a quick and easy way to regrid CMIP model data that can be located using `intake-esgf`. `Cartopy` lets us easily customize the plot to neatly display the geospatial data. "
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6c9caac9-3de5-4842-90c5-e4de995c06ef",
- "metadata": {},
- "source": [
- "## Resources and references\n",
- "* [Regridding overview from NCAR](https://climatedataguide.ucar.edu/climate-tools/regridding-overview), including descriptions of various regridding methods\n",
- "* [Rooki regridding example notebook](https://github.com/roocs/rooki/blob/master/notebooks/demo/demo-rooki-regrid-cmip6.ipynb)\n",
- "* [Rooki documentation](https://rooki.readthedocs.io/en/latest/)\n",
- "* [Cartopy logo image source](https://scitools.org.uk/cartopy/docs/v0.16/gallery/logo.html)\n",
- "* [Rooki logo image source](https://rooki.readthedocs.io/en/latest/#)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/_preview/44/_sources/notebooks/how-to-cite.md b/_preview/44/_sources/notebooks/how-to-cite.md
deleted file mode 100644
index 7fea855..0000000
--- a/_preview/44/_sources/notebooks/how-to-cite.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# How to Cite This Cookbook
-
-The material in this Project Pythia Cookbook is licensed for free and open consumption and reuse. All code is served under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), while all non-code content is licensed under [Creative Commons BY 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Effectively, this means you are free to share and adapt this material so long as you give appropriate credit to the Cookbook authors and the Project Pythia community.
-
-The source code for the book is [released on GitHub](https://github.com/ProjectPythia/esgf-cookbook) and archived on Zenodo. This DOI will always resolve to the latest release of the book source:
-
-[](https://doi.org/10.5281/zenodo.11663067)
diff --git a/_preview/44/_sources/notebooks/intro-search.ipynb b/_preview/44/_sources/notebooks/intro-search.ipynb
deleted file mode 100644
index 1ac8147..0000000
--- a/_preview/44/_sources/notebooks/intro-search.ipynb
+++ /dev/null
@@ -1,275 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![\"Project](\"images/ProjectPythia_Logo_Final-01-Blue.svg\"){kind=logo}
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Project Pythia Notebook Template\n",
- "\n",
- "Next, title your notebook appropriately with a top-level Markdown header, `#`. Do not use this level header anywhere else in the notebook. Our book build process will use this title in the navbar, table of contents, etc. Keep it short, keep it descriptive. Follow this with a `---` cell to visually distinguish the transition to the prerequisites section."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview\n",
- "If you have an introductory paragraph, lead with it here! Keep it short and tied to your material, then be sure to continue into the required list of topics below,\n",
- "\n",
- "1. This is a numbered list of the specific topics\n",
- "1. These should map approximately to your main sections of content\n",
- "1. Or each second-level, `##`, header in your notebook\n",
- "1. Keep the size and scope of your notebook in check\n",
- "1. And be sure to let the reader know up front the important concepts they'll be leaving with"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Prerequisites\n",
- "This section was inspired by [this template](https://github.com/alan-turing-institute/the-turing-way/blob/master/book/templates/chapter-template/chapter-landing-page.md) of the wonderful [The Turing Way](https://the-turing-way.netlify.app) Jupyter Book.\n",
- "\n",
- "Following your overview, tell your reader what concepts, packages, or other background information they'll **need** before learning your material. Tie this explicitly with links to other pages here in Foundations or to relevant external resources. Remove this body text, then populate the Markdown table, denoted in this cell with `|` vertical brackets, below, and fill out the information following. In this table, lay out prerequisite concepts by explicitly linking to other Foundations material or external resources, or describe generally helpful concepts.\n",
- "\n",
- "Label the importance of each concept explicitly as **helpful/necessary**.\n",
- "\n",
- "| Concepts | Importance | Notes |\n",
- "| --- | --- | --- |\n",
- "| [Intro to Cartopy](https://foundations.projectpythia.org/core/cartopy/cartopy.html) | Necessary | |\n",
- "| [Understanding of NetCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html) | Helpful | Familiarity with metadata structure |\n",
- "| Project management | Helpful | |\n",
- "\n",
- "- **Time to learn**: estimate in minutes. For a rough idea, use 5 mins per subsection, 10 if longer; add these up for a total. Safer to round up and overestimate.\n",
- "- **System requirements**:\n",
- " - Populate with any system, version, or non-Python software requirements if necessary\n",
- " - Otherwise use the concepts table above and the Imports section below to describe required packages as necessary\n",
- " - If no extra requirements, remove the **System requirements** point altogether"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Imports\n",
- "Begin your body of content with another `---` divider before continuing into this section, then remove this body text and populate the following code cell with all necessary Python imports **up-front**:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import sys"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Your first content section"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This is where you begin your first section of material, loosely tied to your objectives stated up front. Tie together your notebook as a narrative, with interspersed Markdown text, images, and more as necessary,"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# as well as any and all of your code cells\n",
- "print(\"Hello world!\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### A content subsection\n",
- "Divide and conquer your objectives with Markdown subsections, which will populate the helpful navbar in Jupyter Lab and here on the Jupyter Book!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# some subsection code\n",
- "new = \"helpful information\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Another content subsection\n",
- "Keep up the good work! A note, *try to avoid using code comments as narrative*, and instead let them only exist as brief clarifications where necessary."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Your second content section\n",
- "Here we can move on to our second objective, and we can demonstrate"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Subsection to the second section\n",
- "\n",
- "#### a quick demonstration\n",
- "\n",
- "##### of further and further\n",
- "\n",
- "###### header levels"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "as well $m = a * t / h$ text! Similarly, you have access to other $\\LaTeX$ equation [**functionality**](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html) via MathJax (demo below from link),\n",
- "\n",
- "\\begin{align}\n",
- "\\dot{x} & = \\sigma(y-x) \\\\\n",
- "\\dot{y} & = \\rho x - y - xz \\\\\n",
- "\\dot{z} & = -\\beta z + xy\n",
- "\\end{align}"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Check out [**any number of helpful Markdown resources**](https://www.markdownguide.org/basic-syntax/) for further customizing your notebooks and the [**Jupyter docs**](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html) for Jupyter-specific formatting information. Don't hesitate to ask questions if you have problems getting it to look *just right*."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Last Section\n",
- "\n",
- "If you're comfortable, and as we briefly used for our embedded logo up top, you can embed raw html into Jupyter Markdown cells (edit to see):"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![\"Alpha](\"images/alpha_output.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cffd29c8",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "81f6c01b-1e08-463d-90d5-b9e7be5a61ac",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Overview\n",
- "\n",
- "In this demo we combine multiple multiple tools described in previous cookbooks to subset, regrid and process CMIP6 data. We will be computing a measure of ENSO nonlinearity by computing the EOFs of the pacific sea surface temperature anomalies. This measure is particularly useful for characterizing models by their ability to represent different ENSO extremes (Karamperidou et al., 2017).\n",
- "\n",
- "The process we are going to follow in this demo is:\n",
- "\n",
- "1. Find the CMIP6 data we need using intake-esgf\n",
- "2. Subset the data and regrid it to a common grid using Rooki\n",
- "3. Load the datasets into xarray and perform the computations\n",
- "4. Plot the results\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "31d3693d-4e01-4982-b1d0-dffcd2a13157",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Prerequisites\n",
- "\n",
- "| Concepts | Importance | Notes |\n",
- "| --- | --- | --- |\n",
- "| [Intro to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Necessary | How to use xarray to work with NetCDF data |\n",
- "| [Intro to Intake-ESGF](intro-search) | Necessary | How to configure a search and use output |\n",
- "| [Intro to Rooki](rooki) | Helpful | How to initialize and run rooki |\n",
- "| [Intro to EOFs](https://projectpythia.org/eofs-cookbook/notebooks/eof-intro.html) | Helpful | Understanding of EOFs |\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "- **Time to learn**: 20 minutes"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "288086a4",
- "metadata": {},
- "source": [
- "## Imports"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a2339b90",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "import os\n",
- "import intake_esgf\n",
- "\n",
- "# Run this on the DKRZ node in Germany, using the ESGF1 index node at LLNL\n",
- "os.environ[\"ROOK_URL\"] = \"http://rook.dkrz.de/wps\"\n",
- "intake_esgf.conf.set(indices={\"anl-dev\": False,\n",
- " \"ornl-dev\": False,\n",
- " \"esgf-node.llnl.gov\": True})\n",
- "\n",
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "import numpy.polynomial.polynomial as poly\n",
- "import xarray as xr\n",
- "import xeofs as xe\n",
- "from intake_esgf import ESGFCatalog\n",
- "from rooki import operators as ops\n",
- "from rooki import rooki"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d6ed87c2",
- "metadata": {},
- "source": [
- "## Retrieve subset of CMIP6 data\n",
- "\n",
- "The CMIP6 dataset is identified by a dataset-id. Using intake-esgf we can query the ESGF database for the variables and models we are interested in. For this demo we are interested in the tos (sea surface temperature) variable for the historical runs. Also, for sake of simplicity we will only query a subset of the models available."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "66b3b3c0-6aa0-465b-bc17-86ae2ce5f25b",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "cat = ESGFCatalog()\n",
- "cat.search(\n",
- " experiment_id=[\"historical\"],\n",
- " variable_id=[\"tos\"],\n",
- " table_id=[\"Omon\"],\n",
- " project=[\"CMIP6\"],\n",
- " grid_label=[\"gn\"],\n",
- " source_id=[\n",
- " \"CAMS-CSM1-0\",\n",
- " \"FGOALS-g3\",\n",
- " \"CMCC-CM2-SR5\",\n",
- " \"CNRM-CM6-1\",\n",
- " \"CNRM-ESM2-1\",\n",
- " \"CESM2\",\n",
- " ],\n",
- ")\n",
- "cat.remove_ensembles()\n",
- "print(cat)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4aea426b",
- "metadata": {},
- "source": [
- "Once the catalog has been queried, we have to do some manipulation in pandas to keep only the dataset_id. This has to be done because the same data has multiple locations online, and these get appended at the end of the dataset_id. Rookie only accepts the dataset_id without the online location, so we get rid of it in the next step."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9482b7d7",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def keep_ds_id(ds):\n",
- " return ds[0].split(\"|\")[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "46726e56-030d-4e54-a1a4-5e2f2ca11b43",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "collections = cat.df.id.apply(keep_ds_id).to_list()\n",
- "collections"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "513d3941",
- "metadata": {},
- "source": [
- "We are left with a list of dataset_ids that Rookie can accept as input for the next step."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "674a3b8b",
- "metadata": {},
- "source": [
- "## Subset and regrid the data\n",
- "\n",
- "We define a function that will do the subset and regridding for us for each of the dataset_ids we have. The function will take the dataset_id as input and then use Rookie functions to select 100 years of data for the tos variable in the Pacific Ocean region. We don't need high resolution data for this particular use, so 2.5 degree resolution is enough."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "30e8c66b",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def get_pacific_ocean(dataset_id):\n",
- " wf = ops.Regrid(\n",
- " ops.Subset(\n",
- " ops.Input(\"tos\", [dataset_id]),\n",
- " time=\"1900-01-01/2000-01-31\",\n",
- " area=\"100,-20,280,20\",\n",
- " ),\n",
- " method=\"nearest_s2d\",\n",
- " grid=\"2pt5deg\",\n",
- " )\n",
- " resp = wf.orchestrate()\n",
- " if resp.ok:\n",
- " print(f\"{resp.size_in_mb=}\")\n",
- " ds = resp.datasets()[0]\n",
- " else:\n",
- " ds = xr.Dataset()\n",
- " return ds"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "eacbecbd",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "sst_data = {dset: get_pacific_ocean(dset) for dset in collections}"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "46301d38",
- "metadata": {},
- "source": [
- "## ENSO nonlinearity measure: `alpha` value"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "788b135d",
- "metadata": {},
- "source": [
- "This part of the demo is computation heavy. You can refer to Takahashi et al. (2011) and Karamperidou et al. (2017) for more details on the usefulness and computation of the `alpha` parameter.\n",
- "\n",
- "The `alpha` parameter is computed by doing a quadratic fit to the first two EOFs for the DJF season of the SST anomalies in the Pacific region. We are looking to obtain two EOFs modes that represent the Eastern and central pacific SST patterns, which is why we include a correction factor to account for the fact the sometimes the EOFs come with the opposite sign.\n",
- "\n",
- "The higher the value of `alpha`, the more nonlinear (or extreme) ENSO events can be represented by the model. Likewise, a model with lower `alpha` values will have a harder time representing extreme ENSO events, making it not suitable for climate studies of ENSO in a warming climate (Cai et al., 2018, 2021)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "99631cca",
- "metadata": {},
- "source": [
- "We are looking to obtain data that can reproduce a figure similar to the one below (taken from Karamperiou et al., 2017):"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4266bc6",
- "metadata": {},
- "source": [
- "![\"Alpha](\"images/alpha_example.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "31e2e06a",
- "metadata": {},
- "source": [
- "Each of the \"wings\" of this boomerang-shaped distribution represents a different ENSO extreme, with the left (right) wing representing the extreme central (eastern) pacific El Niño events. More details on Takahashi et al. (2011)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f43be532-c565-45e7-84d8-21be6e4e351e",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def compute_alpha(pc1, pc2):\n",
- " coefs = poly.polyfit(pc1, pc2, deg=2)\n",
- " xfit = np.arange(pc1.min(), pc1.max() + 0.1, 0.1)\n",
- " fit = poly.polyval(xfit, coefs)\n",
- " return coefs[-1], xfit, fit\n",
- "\n",
- "\n",
- "def correction_factor(model):\n",
- " _eofs = model.components()\n",
- " _subset = dict(lat=slice(-5, 5), lon=slice(140, 180))\n",
- " corr_factor = np.zeros(2)\n",
- " corr_factor[0] = 1 if _eofs.sel(mode=1, **_subset).mean() > 0 else -1\n",
- " corr_factor[1] = 1 if _eofs.sel(mode=2, **_subset).mean() > 0 else -1\n",
- " return xr.DataArray(corr_factor, coords=[(\"mode\", [1, 2])])\n",
- "\n",
- "\n",
- "def compute_index(ds):\n",
- " tos = ds.tos.sel(lat=slice(-20, 20), lon=slice(100, 280))\n",
- " tos_anom = tos.groupby(\"time.month\").apply(lambda x: x - x.mean(\"time\"))\n",
- "\n",
- " # Compute Eofs\n",
- " model = xe.models.EOF(n_modes=2, use_coslat=True)\n",
- " model.fit(tos_anom, dim=\"time\")\n",
- " corr_factor = correction_factor(model)\n",
- " # eofs = s_model.components()\n",
- " scale_factor = model.singular_values() / np.sqrt(model.explained_variance())\n",
- " pcs = (\n",
- " model.scores().convert_calendar(\"standard\", align_on=\"date\")\n",
- " * scale_factor\n",
- " * corr_factor\n",
- " )\n",
- "\n",
- " pc1 = pcs.sel(mode=1)\n",
- " pc1 = pc1.sel(time=pc1.time.dt.month.isin([12, 1, 2]))\n",
- " pc1 = pc1.resample(time=\"QS-DEC\").mean().dropna(\"time\")\n",
- "\n",
- " pc2 = pcs.sel(mode=2)\n",
- " pc2 = pc2.sel(time=pc2.time.dt.month.isin([12, 1, 2]))\n",
- " pc2 = pc2.resample(time=\"QS-DEC\").mean().dropna(\"time\")\n",
- "\n",
- " alpha, xfit, fit = compute_alpha(pc1, pc2)\n",
- "\n",
- " return pc1, pc2, alpha, xfit, fit"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2334677a",
- "metadata": {},
- "source": [
- "Now we can compute the `alpha` parameter for each of the models we have selected."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "140ee71c-01ad-4df5-a4af-d3f7f28c622a",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "alpha_fits = {}\n",
- "for key, item in sst_data.items():\n",
- " if len(item.variables) == 0:\n",
- " continue\n",
- " alpha_fits[key] = compute_index(item)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b8714f85",
- "metadata": {},
- "source": [
- "## Plot the results\n",
- "\n",
- "Finally, we can plot the results of the `alpha` parameter for each of the models we have selected. This will give us an idea of how well the models represent different ENSO extremes."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6b327ec5-f261-4ae8-9ee8-19fff28b62dc",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "fig, axs = plt.subplots(nrows=3, ncols=2, figsize=(8, 12))\n",
- "axs = axs.ravel()\n",
- "for num, (ds, (pc1, pc2, alpha, xfit, fit)) in enumerate(alpha_fits.items()):\n",
- " ax = axs[num]\n",
- " ax.axhline(0, color=\"k\", linestyle=\"--\", alpha=0.2)\n",
- " ax.axvline(0, color=\"k\", linestyle=\"--\", alpha=0.2)\n",
- "\n",
- " # draw a line 45 degrees\n",
- " x = np.linspace(-6, 6, 100)\n",
- " y = x\n",
- " ax.plot(x, y, color=\"k\", alpha=0.5, lw=1)\n",
- " ax.plot(-x, y, color=\"k\", alpha=0.5, lw=1)\n",
- "\n",
- " ax.scatter(\n",
- " pc1,\n",
- " pc2,\n",
- " s=8,\n",
- " marker=\"o\",\n",
- " c=\"w\",\n",
- " edgecolors=\"k\",\n",
- " linewidths=0.5,\n",
- " )\n",
- "\n",
- " ax.plot(xfit, fit, c=\"r\", label=f\"$\\\\alpha=${alpha:.2f}\")\n",
- "\n",
- " ax.set_xlabel(\"PC1\")\n",
- " ax.set_ylabel(\"PC2\")\n",
- "\n",
- " ax.set_title(ds.split(\".\")[3])\n",
- "\n",
- " ax.set_xlim(-4, 4)\n",
- " ax.set_ylim(-4, 4)\n",
- " ax.legend()\n",
- "fig.subplots_adjust(hspace=0.3)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9f67612b",
- "metadata": {},
- "source": [
- "From this example, we can see that from the subset of models we have selected, the `alpha` parameter is higher for CMCC-CM2-SR5 compared to the other models as the \"boomerang\" shape is better represented in this model. This indicates that this model is better at representing extreme ENSO events compared to the other models."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f3cc1404-0030-4e7c-98bb-498c354301d2",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Summary\n",
- "In this notebook, we used intake-esgf with Rooki Python client to retrieve a subset of a CMIP6 dataset. The subset and regrid operations are executed remotely on a Rook subsetting service (using OGC API and xarray/clisops). The dataset is analyzed using xeofs to extract a measurement used in ENSO research. We also showed that remote operators can be chained to be executed in a single workflow operation.\n",
- "\n",
- "## Resources\n",
- "- [Roocs on GitHub](https://github.com/roocs)\n",
- "- [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/)\n",
- "- [STAC](https://stacspec.org/en)\n",
- "\n",
- "## References\n",
- "- Cai, W., Santoso, A., Collins, M., Dewitte, B., Karamperidou, C., Kug, J.-S., Lengaigne, M., McPhaden, M. J., Stuecker, M. F., Taschetto, A. S., Timmermann, A., Wu, L., Yeh, S.-W., Wang, G., Ng, B., Jia, F., Yang, Y., Ying, J., Zheng, X.-T., … Zhong, W. (2021). Changing El Niño–Southern Oscillation in a warming climate. Nature Reviews Earth & Environment, 2(9), 628–644. https://doi.org/10.1038/s43017-021-00199-z\n",
- "- Cai, W., Wang, G., Dewitte, B., Wu, L., Santoso, A., Takahashi, K., Yang, Y., Carréric, A., & McPhaden, M. J. (2018). Increased variability of eastern Pacific El Niño under greenhouse warming. Nature, 564(7735), 201–206. https://doi.org/10.1038/s41586-018-0776-9\n",
- "- Karamperidou, C., Jin, F.-F., & Conroy, J. L. (2017). The importance of ENSO nonlinearities in tropical pacific response to external forcing. Climate Dynamics, 49(7), 2695–2704. https://doi.org/10.1007/s00382-016-3475-y\n",
- "- Takahashi, K., Montecinos, A., Goubanova, K., & Dewitte, B. (2011). ENSO regimes: Reinterpreting the canonical and Modoki El Niño. Geophysical Research Letters, 38(10). https://doi.org/10.1029/2011GL047364\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e92000c2-4af7-407f-ac39-69988b083cfe",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/_preview/44/_sources/notebooks/running-on-nimbus.md b/_preview/44/_sources/notebooks/running-on-nimbus.md
deleted file mode 100644
index 1257cdb..0000000
--- a/_preview/44/_sources/notebooks/running-on-nimbus.md
+++ /dev/null
@@ -1,42 +0,0 @@
-# Running Notebooks on Nimbus
-
-Interested in running your notebooks on ESGF infrastructure? Please follow these steps!
-
-## 1. Apply for access to the Nimbus Access
-
-Please fill out [this form](https://forms.gle/h8y14hfefcaCraEJ8) to request access to the Nimbus Jupyterhub!
-
-You will be added to the [Nimbus User Group](https://github.com/orgs/esgf-nimbus/people), which is used for authentication!
-
-## 2. Clone this repository
-
-Once you log into the Jupyterhub (https://nimbus.llnl.gov/), go to your home directory (shown by default) and clone this repository
-
-```bash
-git clone https://github.com/ProjectPythia/esgf-cookbook.git
-```
-
-## 3. Build your Execution Environment
-The cookbook environment is slightly different than the base environment available on the hub. You will need to build the required environment, using the `environment.yml` file in the repository.
-
-```bash
-conda env create -f esgf-cookbook/environment.yml
-```
-
-## Activate Your Environment
-Once you build the enivronment, you will need to activate it. You will need to follow the following steps:
-
-```bash
-# Make sure you can activate the environment
-source .bashrc
-
-# Activate the environment
-conda activate esgf-cookbook-dev
-```
-
-## Open a Notebook and Select the `esgf-cookbook-dev` environment
-Now that you have an environment, you can select this when opening a notebook. Select in the top right corner the kernel options, and select `esgf-cookbook-dev`.
-
-Wait a second for the notebook to pick up on this, then execute your cells!
-
-If you need to install more packages, or update versions, you can do so by updating your `environment.yml` file or by installing via the command line.
\ No newline at end of file
diff --git a/_preview/44/_sources/notebooks/use-intake-esgf-with-rooki.ipynb b/_preview/44/_sources/notebooks/use-intake-esgf-with-rooki.ipynb
deleted file mode 100644
index 0f6b06c..0000000
--- a/_preview/44/_sources/notebooks/use-intake-esgf-with-rooki.ipynb
+++ /dev/null
@@ -1,382 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "5191269c-944c-4516-9f51-6cdfc704852a",
- "metadata": {},
- "source": [
- "![\"Intake](\"images/intake-rooki-demo.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fa96801d-4d1a-4264-94f9-9bf12a77421a",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "545103ee-abac-4da8-af3b-c8877a3d2d6c",
- "metadata": {},
- "source": [
- "# Using intake-esgf with rooki\n",
- "\n",
- "## Overview\n",
- "\n",
- "In this notebook we will demonstrate how to use intake-esgf and rooki to perform server-side operations and return the result to the user. This will occur in several steps.\n",
- "\n",
- "1. We use intake-esgf to find data which is local to the ORNL server and then form an id which rooki uses to load the data remotely.\n",
- "2. We build a rooki workflow which uses these ids (`rooki_id`) to subset and average the data remotely.\n",
- "3. The results are downloaded locally and we visualize them interactively using hvplot.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0b633dab-4c9d-482a-b148-8f9b09102e78",
- "metadata": {},
- "source": [
- "## Prerequisites\n",
- "\n",
- "| Concepts | Importance | Notes |\n",
- "| --- | --- | --- |\n",
- "| [Intro to Intake-ESGF](intro-search) | Necessary | How to configure a search and use output |\n",
- "| [Intro to Rooki](rooki) | Helpful | How to initialize and run rooki |\n",
- "| [Intro to hvPlot](https://hvplot.holoviz.org/user_guide/Geographic_Data.html) | Necessary | How to plot interactive visualizations |\n",
- "\n",
- "- **Time to learn**: 30 minutes"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "64a5b6f9-eeee-4726-abdc-686e96dfc3cf",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f7571ba4-43cf-4f61-b74a-a51cdee373aa",
- "metadata": {},
- "source": [
- "## Imports"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b3de6244-5afc-4cb4-aabe-7508423944ae",
- "metadata": {},
- "source": [
- "Before importing rooki, we need to set an environment variable that will signal the rooki client to use the web processing service (WPS) deployment located at Oak Ridge National Lab (ORNL)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "c12d0875-9794-4101-b74f-346148cb36c0",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "import os\n",
- "\n",
- "# Configuration line to set the wps node - in this case, use ORNL in the USA\n",
- "url = \"https://esgf-node.ornl.gov/wps\"\n",
- "os.environ[\"ROOK_URL\"] = url\n",
- "\n",
- "from rooki import operators as ops\n",
- "from rooki import rooki"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d19a5944-12d7-401c-a9cb-8e92f1f95f96",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Other imports\n",
- "import holoviews as hv\n",
- "import hvplot.xarray\n",
- "import intake_esgf\n",
- "import matplotlib.pyplot as plt\n",
- "import panel as pn\n",
- "import xarray as xr\n",
- "from intake_esgf import ESGFCatalog\n",
- "\n",
- "hv.extension(\"bokeh\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7dd9b81a-9aeb-4769-8c67-4db88a089859",
- "metadata": {},
- "source": [
- "## Search and Find Data for Surface Temperature on the ORNL Node\n",
- "\n",
- "Let's start with refining which index we would like to search from. For this analysis, we are remotely computing on the ORNL node since this is where rooki is running. We know this from checking the `._url` method of rooki!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d3cdea76-9a74-4001-91b4-6c1b664835b2",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "rooki._url"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4a4442f9-6c41-4a56-ac66-4299d0c4ddda",
- "metadata": {},
- "source": [
- "### Set the Index Node and Search\n",
- "\n",
- "Because we are using the ORNL-based WPS, we only need information about ORNL holdings. So here we configure intake-esgf to only look at the ORNL index for data information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8d78dd66-16d0-4573-b94d-b7c52200cee8",
- "metadata": {},
- "outputs": [],
- "source": [
- "intake_esgf.conf.set(indices={\"anl-dev\": False,\n",
- " \"ornl-dev\": True})"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "37d7a372-8db7-4a02-9d3f-6d8fc54e7e88",
- "metadata": {},
- "source": [
- "Now we instantiate the catalog and perform a search for surface air temperature (tas) data from a few institution's models. Note that we have also included specificity of the data node. The ORNL index contains information about holdings beyond the ORNL data node and so we give this to force the catalog to only return information about holdings which are local to ORNL."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d0f5ede4-feb0-4896-9e65-e100cbfadee4",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "cat = ESGFCatalog().search(\n",
- " experiment_id=\"historical\",\n",
- " variable_id=\"tas\",\n",
- " member_id=\"r1i1p1f1\",\n",
- " table_id=\"Amon\",\n",
- " institution_id=[\"MIROC\", \"NCAR\", \"NASA-GISS\", \"CMCC\"],\n",
- ")\n",
- "cat.df"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b00d07ff-58a7-4141-922c-9f7a97772f1e",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Extract IDs to Pass to Rooki\n",
- "\n",
- "The catalog returns a lot of information about the datasets that were found, but the rooki WPS interface just needs an ID that looks similar to what we find in the `id` column of the dataframe. We need to remove the `|esgf-node.ornl.gov` on the end and prepend a `ccs03_data`. To do this we will write a function and apply it to the dataframe.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d77e9325-3594-4454-a330-e6d3b139c32b",
- "metadata": {},
- "outputs": [],
- "source": [
- "def build_rooki_id(id_list):\n",
- " rooki_id = id_list[0]\n",
- " rooki_id = rooki_id.split(\"|\")[0]\n",
- " rooki_id = f\"css03_data.{rooki_id}\" # <-- just something you have to know for now :(\n",
- " return rooki_id\n",
- "\n",
- "rooki_ids = cat.df.id.apply(build_rooki_id).to_list()\n",
- "rooki_ids"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4a470bea-581c-4796-a716-a02fd02e1531",
- "metadata": {},
- "source": [
- "### Compute with Rooki\n",
- "Now that we have a list of IDs to pass to rooki, let's compute! In our case we are interested in the annual temperature from 1990-2000 over an area that includes India (latitude from 0 to 35, longitude from 65 to 100). The following function will construct a rooki workflow that uses operators (functions in the `ops` namespace) that rooki uses to:\n",
- "\n",
- "- read in data (`ops.Input`)\n",
- "- subset in time and space (`ops.Subset`), and\n",
- "- average in time (`ops.AverageByTime`) on a yearly frequency.\n",
- "\n",
- "We then check to make sure the response is okay, and if it is, return the processed dataset to the user! If something went wrong, the function will raise an error and show you the message that rooki sent back."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "38704069-b1ad-474f-99f6-3273554df831",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "def india_annual_temperature(rooki_id):\n",
- " workflow = ops.AverageByTime(\n",
- " ops.Subset(\n",
- " ops.Input(\"tas\", [rooki_id]),\n",
- " time=\"1990-01-01/2000-01-01\",\n",
- " area=\"65,0,100,35\",\n",
- " ),\n",
- " freq=\"year\",\n",
- " )\n",
- " response = workflow.orchestrate()\n",
- " if not response.ok:\n",
- " raise ValueError(response)\n",
- " return response.datasets()[0]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cf88a09b-8178-4798-afc6-744242e65ca6",
- "metadata": {},
- "source": [
- "Now let's test a single rooki_id to demonstrate successful functionality. The rooki_id let's the WPS know on which dataset we are intersted in operating and then the data is loaded remotely, subset, and then averaged. After this computation is finished on the server, the result is transferred to you and loaded into a xarray dataset. Inspect the dataset header to see that there are 10 times, one for each year and the latitude and longitude range spans our input values."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "94609894-705d-4fdc-a235-b2f1f5747305",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "india_annual_temperature(rooki_ids[0])"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4ecacf51-e35f-4810-8250-3e90a1b5888a",
- "metadata": {},
- "source": [
- "Now that we have some confidence in our workflow function, we can iterate over rooki_id's running for each and saving into a dictionary whose keys are the different models. You should see messages print to the screen which inform you where the temporary output is being downloaded. This location can be configured in rooki, but for now we will just load them into datasets."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a52e7d71-18fd-4bbe-91a7-fa86725ad4de",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "dsd = {\n",
- " rooki_id.split(\".\")[4]: india_annual_temperature(rooki_id)\n",
- " for rooki_id in rooki_ids\n",
- "}"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4a77cda9-536f-4fc2-8e89-9ce2e4937810",
- "metadata": {},
- "source": [
- "## Visualize the Output\n",
- "Let's use hvPlot to visualize. The datasets are stored in a dictionary of datasets, we need to:\n",
- "- Extract a single key\n",
- "- Plot a contour filled visualization, with some geographic features"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1b08b0a0-7add-4a50-8e62-9bb88b98cc7c",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "tas = dsd[\"MIROC6\"].tas\n",
- "tas.hvplot.contourf(\n",
- " x=\"lon\",\n",
- " y=\"lat\",\n",
- " cmap=\"Reds\",\n",
- " levels=20,\n",
- " clim=(250, 320),\n",
- " features=[\"land\", \"ocean\"],\n",
- " alpha=0.7,\n",
- " widget_location=\"bottom\",\n",
- " clabel=\"Yearly Average Temperature (K)\",\n",
- " geo=True,\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "33c9fce3-722a-4cc9-8b4c-5d2038cc981f",
- "metadata": {},
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "86e3979e-505a-455c-8753-6cd5c7f0b219",
- "metadata": {},
- "source": [
- "## Summary\n",
- "Within this notebook, we learned how to specify a specific index node to search from, pass discovered datasets to rooki, and chain remote-compute with several operations using rooki. We then visualized the output using hvPlot, leading to an interactive plot!\n",
- "\n",
- "### What's next?\n",
- "More adaptations of the intake-esgf + rooki to remotely compute on ESGF data."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bbda4631-017e-4295-b2a5-e97fe5197cf8",
- "metadata": {},
- "source": [
- "## Resources and references\n",
- " - [intake-esgf documentation](https://intake-esgf.readthedocs.io/en/latest/)\n",
- " - [rooki documentation](https://rooki.readthedocs.io/en/latest/)\n",
- " - [Working with geographic data with hvPlot](https://hvplot.holoviz.org/user_guide/Geographic_Data.html)\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/_preview/44/_sphinx_design_static/design-tabs.js b/_preview/44/_sphinx_design_static/design-tabs.js
deleted file mode 100644
index b25bd6a..0000000
--- a/_preview/44/_sphinx_design_static/design-tabs.js
+++ /dev/null
@@ -1,101 +0,0 @@
-// @ts-check
-
-// Extra JS capability for selected tabs to be synced
-// The selection is stored in local storage so that it persists across page loads.
-
-/**
- * @type {Record