palmpy Installation

The palmpy python package consists of multiple submodules, which contain python functions dedicated to one specific type of tasks to be performed regarding PALM static file generation and beyond. These submodules and their corresponding nesting are:

• palmpy.staticcreation
  • geodatatools
  • makestatictools
    • dictfolder
• palmpy.postprocessing

palmpy.staticcreation.geodatatools provides functions that can be used with resprect to handling geodata. This includes functions to clip data to specific extents, rasterize shapefiles and perform operations on shapefiles.

palmpy.staticcreation.makestatictools provides functions that are required to generate a static driver file. These code sections are called repeatedly, hence a modularization as functions was appropriate. It contains functions to perform consistency checks, load geodata, perform operations on datasets and others, in total over 30 functions.

palmpy.staticcreation.dictfolder.dialect contains various files with dictionaries, that govern the mapping of source data classes to palm classes for vegetation, streets, pavements, water and others. Keeping them separates allows an extension of palmpy with an infinite set of possible mappings.

palmpy.postprocessing.posttools contains only one basic plotting function at the moment. Initially it was planned to include a whole set of plotting functions for PALM. However, a new plotting module was announced to be proposed by the PALM team that leverages python plotting functions, which is why this plan was abandoned. The new plotting mechanisms are not available though (as of May 2020), but their publication is imminent.

Installation of palmpy

TLDR: It is recommented to create a conda environment. Create a new env by issuing the following command:
conda create -n palmpyenv -c conda-forge pandas geopandas numpy scipy matplotlib pillow netcdf4 gdal rasterio xarray
To run palmpy, activate the environment, execute the script with python make_static.py and a file request popup will let you select your namelist.ini file.

Python Environment

In order for palmpy to operate, you need (most of) the following packages:

Package Name	Version used in developing palmpy
numpy	1.18.1
scipy	1.4.1
matplotlib	3.1.3
pillow	7.0.0
netcdf4	1.4.2
gdal	3.0.2
xarray	0.20.1
rasterio	1.1.0
(pynco)	(1.0.0)
(pandas)	1.0.3
(geopandas)	0.6.1

It is strongly recommended to set up python with the conda package management ecosystem. There are many ways to do it. Users relatively unfamiliar with python can install the full Anaconda software, which comes with the most relevant python packages for doing science, but also with some baggage in form of software. More experienced users, who know what packages they need and are able to install them easily, can install python with a Miniconda installer, which only installs the bare minimum of packages and python on the machine (much smaller download size). Most importantly, the conda package manager is installed as well.

Once a running conda environment (usually named base, you can check it with entering conda env list into the Anaconda Prompt (in Windows) or your regular shell (in Linux)) is present. we set up a new environment that contains all necessary packages to run palmpy. Environment are an essential tool when it comes to software development and allows to "freeze" an environment to have defined package versions and dependencies for a particular project. After the following process, an environment "palm" (or however you name it) shall be present in the conda environment list.

Different conda environments on a computer.

A one liner to install a new env and all necessary packages is: conda create -n palmpyenv -c conda-forge pandas geopandas numpy scipy matplotlib pillow netcdf4 gdal rasterio xarray (tested May 2022).

Depending on your system, choose the appropriate .yml file from the env folder. Try to use the one with package version info first. This may not work on systems other than Windows 10 x86.

  conda env create -f palmenv-versinfo.yml -n <envname>

The string after -f represents the path to the .yml file, the string after -n the name of the environment that you create. If it does not work because some packages are not found, use the .yml file without version numbers attached to the packages. If there are errors relating to "invalid name" or similar, check your path to the .yml file.

Should there be a need for a new export of a working environment from conda, this can be done with the following command (while the environment in question is activated):

conda env export --nobuilds --from-history > out.yml

A Word on Environments and Packages

This concept of having various environments with the same packages over and over again may be confusing to new python users at first. However, it makes perfect sense to have controlled package versions for a project. As python modules are updated, some functions may stop working, and new functions may be added that make life much easier. Having a python environment for everyone working on a project ensures that everyone is able to execute this particular code and it can be shared among coworkers. Therefore, and because palmpy is the result of a student project, continuous maintenance will not be available. Should therefore e.g. numpy be updated and a specific function be made obsolete, on which palmpy relied on heavily, the code will simply stop working. Therefore, installing a python environment from a .yml file ensures that the environment is set up correctly with correct version numbers. However, this process can be prone to errors, especially when installing an environment on different operating systems. Therefore, a .yml file without version names is included, that should work on any given operating system.

For this project, Spyder was used to write palmpy. Spyder is a package like any other and can be installed with conda. When installing multiple environments and you intend to use spyder for every one of them, it seems that you have to include the spyder package in these environments as well. In the end, when you start up spyder to work on a particular project, make sure to open the correct one - you can recognize that on the Shortcut name "Spyder (palm)" or "Spyder" (no env name for the base environment) or "Spyder (randomproject)".

Package Installation

When importing a package in python with import package, python automatically scans its $PATH variable for the required package. Therefore, if we want palmpy to be fount, we need to move the palmpy folder into one of those locations. In order to know, where you should put it, open a python console in your newly created environment (eg. open Spyder (envname) to open Spyder with your desired environment or open Anaconda promt, run conda activate envname, followed by python to enter the python interpreter). Run the following:

import sys
print(sys.path)

This will output a list of paths, that are searched for the module. Your best bet is to put the palmpy module folder into your ...\\Miniconda3\\envs\envname\\lib folder.

If you are able to import the module palmpy with import palmpy and no error appears, the package was installed correctly.

Importing palmpy

It is recommended to load the different modules of the palmpy package as follows:

#static file generation
import palmpy.staticcreation.geodatatools as gdt      	# geodata modification tools
import palmpy.staticcreation.makestatictools as mst   	# static generation tools
import palmpy.staticcreation.mapdicts.<dialect> as mpd	# mapping dictionaries for mst

#post processing
import palmpy.postprocessing.tools as ppt             	# postprocessing tools

To create a static file, you need the palmpy geodatatools, makestatictools and the dictionaries for mapping shp-file classes to PALM classes - which are separately available in mapdicts. Should a default mapping be not suitable for a particular project, this mapdicts file can be modified accordingly. For individual application of palmpy, these dicts can be accessed through python individually.

When importing the modules like outlined above, a lot of code can be omitted in the project, as it is not necessary to include the specific hierarchy to each submodule over and over again. Of course, palmpy can also be imported with the basic import palmpy command.