Merge remote-tracking branch 'origin/features/#450-add-metadata' into…

… features/#450-add-metadata-kh

CHANGELOG.rst

@@ -458,6 +458,8 @@ Changed
   `#882 <https://github.com/openego/eGon-data/issues/882>`_
 * Insert crossboarding gas pipeline with Germany in eGon100RE
   `#881 <https://github.com/openego/eGon-data/issues/881>`_
+* Harmonize H2 carrier names in eGon100RE
+  `#929 <https://github.com/openego/eGon-data/issues/929>`_
 * Rename noflex to lowflex scenario for motorized individual travel
   `#921 <https://github.com/openego/eGon-data/issues/921>`_
 * Update creation of heat demand timeseries
@@ -477,6 +479,8 @@ Changed
   `#519 <https://github.com/openego/eGon-data/issues/519>`_
 * Add missing VOM costs for heat sector components
   `#942 <https://github.com/openego/eGon-data/issues/942>`_
+* Add sanity checks for gas sector in eGon2035
+  `#864 <https://github.com/openego/eGon-data/issues/864>`_
 * Desaggregate industry demands to OSM areas and industrial sites
   `#1001 <https://github.com/openego/eGon-data/issues/1001>`_
 * Add gas generator in Norway
@@ -486,6 +490,8 @@ Changed
   created for a single process. This fixes issue `#799`_.
 * Insert rural heat per supply technology
   `#1026 <https://github.com/openego/eGon-data/issues/1026>`_
+* Insert lifetime for components from p-e-s in eGon100RE
+  `#1073 <https://github.com/openego/eGon-data/issues/1073>`_
 * Change hgv data source to use database
   `#1086 <https://github.com/openego/eGon-data/issues/1086>`_
 * Rename eMob MIT carrier names (use underscores)
@@ -731,13 +737,17 @@ Bug Fixes
 * Fix URL of TYNDP scenario dataset
 * Automatically generated tasks now get unique :code:`task_id`\s.
   Fixes issue `#985`_ via PR `#986`_.
+* Adjust capcities of German CH4 stores
+  `#1096 <https://github.com/openego/eGon-data/issues/1096>`_
 * Fix faulty DSM time series
   `#1088 <https://github.com/openego/eGon-data/issues/1088>`_
 * Set upper limit on commissioning date for units from MaStR
   dataset
   `#1098 <https://github.com/openego/eGon-data/issues/1098>`_
 * Fix conversion factor for CH4 loads abroad in eGon2035
   `#1104 <https://github.com/openego/eGon-data/issues/1104>`_
+* Change structure of documentation in rtd
+  `#11126 <https://github.com/openego/eGon-data/issues/1126>`_
 
 .. _PR #692: https://github.com/openego/eGon-data/pull/692
 .. _#343: https://github.com/openego/eGon-data/issues/343

docs/about.rst

@@ -0,0 +1,111 @@
+***************
+About eGon-data
+***************
+
+Project background
+==================
+
+egon-data provides a transparent and reproducible open data-based data processing pipeline for generating data models suitable for energy system modeling. The data is customized for the requirements of the research project eGo_n. The research project aims to develop tools for open and cross-sectoral planning of transmission and distribution grids. For further information please visit the `eGo_n project website <https://ego-n.org/>`_.
+egon-data is a further development of the `Data processing <https://github.com/openego/data_processing>`_ developed in the former research project `open_eGo <https://openegoproject.wordpress.com/>`_. It aims to extend the data models as well as improve the replicability and manageability of the data preparation and processing. 
+The resulting data set serves as an input for the optimization tools `eTraGo <https://github.com/openego/eTraGo>`_, `ding0 <https://github.com/openego/ding0>`_ and `eDisGo <https://github.com/openego/eDisGo>`_ and delivers, for example, data on grid topologies, demands/demand curves and generation capacities in a high spatial resolution. The outputs of egon-data are published under open-source and open-data licenses.  
+
+
+Objectives of the project
+=========================
+
+Driven by the expansion of renewable generation capacity and the progressing electrification of other energy sectors, the electrical grid increasingly faces new challenges: fluctuating supply of renewable energy and simultaneously a changing demand pattern caused by sector coupling. However, the integration of non-electric sectors such as gas, heat, and e-mobility enables more flexibility options. The eGo_n project aims to investigate the effects of sector coupling on the electrical grid and the benefits of new flexibility options. This requires the creation of a spatially and temporally highly resolved database for all sectors considered. 
+
+Project consortium and funding
+==================================
+
+The following universities and research institutes were involved in the creation of eGon-data: 
+
+* University of Applied Sciences Flensburg
+* Reiner Lemoine Institut
+* Otto von Guericke University Magdeburg
+* DLR Institute of Networked Energy Systems
+* Europa-Universität Flensburg 
+
+The eGo_n project (FKZ: 03EI1002) is supported by the Federal Ministry for Economic Affairs and Climate Action (BMWK) on the basis of a decision by the German Bundestag.
+
+.. image:: images/Logos_Projektpartner_egon_data.png
+  :width: 400
+  :alt: Logos of project partners
+
+
+eGon-data as one element of the eGo-Toolchain
+=============================================
+
+In the eGo_n project different tools were developed, which are in exchange with each other and have to serve the respective requirements on data scope, resolution, and format. The results of the data model creation have to be especially adapted to the requirements of the tools eTraGo and eDisGo for power grid optimization on different grid levels. 
+A PostgreSQL database serves as an interface between the data model creation and the optimization tools.
+The figure below visualizes the interdependencies between the different tools. 
+
+.. image:: images/Toolchain_web_desktop.svg
+  :width: 800
+  :alt: eGon-data tool chain
+
+.. _concept-and-scenarios-ref:
+
+Modeling concept and scenarios
+===============================
+
+eGon-data provides a data model suitable for calculations and optimizations with the tools eTraGo, eDisGo and eGo and therefore aims to satisfy all requirements regarding the scope and temporal as well as spatial granularity of the resulting data model.
+The following image visualizes the different components considered in scenario ``eGon2035``.
+
+.. image:: images/egon-modell-szenario-egon2035.png
+  :width: 800
+  :alt: Components of the data models
+
+eGon-data aims to create different scenarios, which differ in terms of RE penetration or the availability of flexibility options. Currently, the following scenarios are available or in progress. 
+
+* ``eGon2035`` Mid-termin scenario based on assumptions from the German network expansion plan 'scenario C2035', version 2021 and TYNDP
+* ``eGon2035_lowflex`` Mid-termin scenario similar to 'eGon2035', but with a limited availability of flexibility options
+* ``eGon100RE`` Long-term scenario with a 100% RE penetration, based on optimization results with PyPSA-Eur-Sec and additional data inputs (work-in-progress)
+
+.. list-table:: Installed capacities of German power park in scenario ``eGon2035`` and ``eGon2035_lowflex``
+   :widths: 50 50
+   :header-rows: 1
+
+   * - carrier
+     - Installed capacities
+   * - gas
+     - 46.7 GW
+   * - oil
+     - 1.3 GW
+   * - pumped hydro
+     - 10.2 GW
+   * - wind onshore
+     - 90.9 GW
+   * - wind offshore
+     - 34.0 GW
+   * - solar
+     - 120.1 GW
+   * - biomass
+     - 8.7 GW
+   * - others
+     - 5.4 GW
+
+
+.. list-table:: German energy demands in scenarios ``eGon2035`` and ``eGon2035_lowflex``
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Demand sector
+     - Energy demand
+   * - MIT transport
+     - 41.4 TWh :subscript:`el`
+   * - central heat
+     - 68.9 TWh :subscript:`th`
+   * - rural heat
+     - 423.2 TWh :subscript:`th`
+   * - electricity
+     - 498.1 TWh :subscript:`el`
+   * - Methane industry
+     - 196.0 TWh :subscript:`CH4`
+   * - Hydrogen industry
+     - 16.1 TWh :subscript:`H2`
+   * - Hydrogen transport
+     - 26.5 TWh :subscript:`H2`
+
+
+

docs/conf.py

@@ -47,3 +47,8 @@
 
 add_module_names = False
 modindex_common_prefix = ["egon.data.", "egon.data.datasets."]
+
+autodoc_type_aliases = {
+    "Dependencies": "egon.data.datasets.Dependencies",
+    "Tasks": "egon.data.datasets.Tasks"
+}

docs/data.rst

@@ -1,21 +1,124 @@
 ****
 Data
 ****
+The description of the methods, input data and results of the eGon-data pipeline is given in the following section.
+References to datasets and functions are integrated if more detailed information is required.
+
+Main input data and their processing
+====================================
+
+All methods in the eGon-data workflow rely on public and freely available data from different external sources. The most important data sources
+and their processing within the eGon-data pipeline are described here.
+
+.. include:: data/input_data.rst
+
+Grid models
+===========
+
+Power grid models of different voltage levels form a central part of the eGon data model, which is required for cross-grid-level optimization.
+In addition, sector coupling necessitates the representation of the gas grid infrastructure, which is also described in this section.
+
+Electricity grid
+----------------
+
+.. include:: data/electricity_grids.rst
+
+Gas grid
+--------
+
+.. include:: data/gas_grids.rst
+
+Demand
+======
+
+Electricity, heat and gas demands from different consumption sectors are taken into account in eGon-data. The related methods to distribute and
+process the demand data are described in the following chapters for the different consumption sectors separately.
+
+.. _electricity-demand-ref:
+
+Electricity
+-----------
+
+.. include:: data/electricity_demand.rst
+
+Heat
+----
+
+.. include:: data/heat_demand.rst
+
+Gas
+---
+
+.. include:: data/gas_demand.rst
+
+.. _mobility-demand-ref:
+
+Mobility
+--------
+
+.. include:: data/mobility_demand.rst
+
+
+Supply
+======
+
+The distribution and assignment of supply capacities or potentials are carried out technology-specific. The different methods are described in the
+following chapters.
+
+Electricity
+-----------
+
+.. include:: data/electricity_supply.rst
+
+Heat
+----
+
+.. include:: data/heat_supply.rst
+
+Gas
+---
+.. include:: data/gas_supply.rst
+
+
+Flexibility options
+===================
+
+Different flexibility options are part of the model and can be utilized in the optimization of the energy system. Therefore detailed information about
+flexibility potentials and their distribution are needed. The considered technologies described in the following chapters range from different storage units,
+through dynamic line rating to Demand-Side-Management measures.
+
+Demand-Side-Management
+----------------------
+
+.. include:: data/DSM.rst
+
+Dynamic line rating
+-------------------
+
+.. include:: data/DLR.rst
+
+E-Mobility
+----------
+
+.. include:: data/e-mobility.rst
+
+Battery stores
+----------------
+
+.. include:: data/batteries.rst
+
+Gas stores
+-----------------
+
+.. include:: data/gas_stores.rst
+
+Heat stores
+-------------
+
+.. include:: data/heat_stores.rst
 
-Scenarios
-=========
 
 Published data
 ==============
 
-Data bundle
------------
 
-The data bundle is published on
-`zenodo <https://sandbox.zenodo.org/record/1167119>`_. It contains several data
-sets, which serve as a basis for egon-data. One such data set is the geocoding
-for the `MaStR data set <https://sandbox.zenodo.org/record/1132987>`_ which is
-used for eGon-data as well. Whenever the MaStR data set is updated it is
-necessary to redo the geocoding with the new data set and update the data
-bundle accordingly. The geocoding can be done based on the
-`mastr-geocoding repository <https://github.com/RLI-sandbox/mastr-geocoding>`_.

docs/data/DLR.rst

@@ -0,0 +1,47 @@
+====================================================
+Methods to include dynamic line rating in our model
+====================================================
+
+To calculate the transmission capacity of each transmission line in the model, 
+the procedure suggested in the **Principles for the Expansion Planning of the
+German Transmission Network** [NEP2021] where used:
+
+1. Import the temperature and wind temporal raster layers from ERA-5. Hourly
+resolution data from the year 2011 was used. Raster resolution
+latitude-longitude grids at 0.25° x 0.25°.
+
+2. Import shape file for the 9 regions proposed by the Principles for
+the Expansion Planning. See Figure 1.
+
+.. image:: images/regions_DLR.png
+  :width: 400
+  :alt: regions DLR
+
+Figure 1: Representative regions in Germany for DLR analysis [NEP2021]
+
+3. Find the lowest wind speed in each region. To perform this, for each
+independent region, the wind speed of every cell in the raster layer should be
+extracted and compared. This procedure is repeated for each hour in the
+year 2011. The results are the 8760 lowest wind speed per region.
+
+4. Find the highest temperature in each region. To perform this, for each
+independent region, the temperature of every cell in the raster layer should
+be extracted and compared. This procedure is repeated for each hour in the
+year 2011. The results are the 8760 maximum temperature per region.
+
+5. Calculate the maximum capacity for each region using the parameters shown in
+Figure 2.
+
+.. image:: images/table_max_capacity_DLR.png
+  :width: 400
+  :alt: table_max_capacity_DLR
+
+Figure 2: transmission capacity based on max temperature and min wind speed [NEP2021]
+
+6. Assign the maximum capacity of the corresponding region to each transmission
+line inside each one of them. Crossborder lines and underground lines receive
+no values. It means that their capacities are static and equal to their nominal
+values. Lines that cross borders between regions receive the lowest 
+capacity per hour of the regions containing the line.
+
+.. [NEP2021] Principles for the Expansion Planning of the German Transmission Network https://www.netzentwicklungsplan.de/

docs/data/DSM.rst

@@ -0,0 +1 @@
+How did we implement DSM? Results etc. 

docs/data/batteries.rst

@@ -0,0 +1,37 @@
+Battery storage units comprise home batteries and larger, grid-supportive batteries. National capacities for home batteries arise from external sources, e.g. the Grid Development Plan for the scenario ``eGon2035``, whereas the capacities of large-scale batteries are a result of the grid optimization tool `eTraGo <https://github.com/openego/eTraGo>`_.
+
+Home battery capacities are first distributed to medium-voltage grid districts (MVGD) and based on that further disaggregated to single buildings. The distribution on MVGD level is done proportional to the installed capacities of solar rooftop power plants, assuming that they are used as solar home storage. 
+
+Potential large-scale batteries are included in the data model at every substation. The data model includes technical and economic parameters, such as efficiencies and investment costs. The energy-to-power ratio is set to a fixed value of 6 hours. Other central parameters are given in the following table
+
+.. list-table:: Parameters of batteries for scenario eGon2035
+   :widths: 40 30 30
+   :header-rows: 1
+
+   * - 
+     - Value
+     - Sources
+
+   * - Efficiency store
+     - 98 % 
+     - [DAE_store]_
+
+   * - Efficiency dispatch
+     - 98 %
+     - [DAE_store]_
+
+   * - Standing loss
+     - 0 %
+     - [DAE_store]_
+
+   * - Investment costs
+     - 838 €/kW
+     - [DAE_store]_
+
+   * - Home storage units
+     - 16.8 GW
+     - [NEP2021]_
+
+
+On transmission grid level, distinguishing between home batteries and large-scale batteries was not possible. Therefore, the capacities of home batteries were set as a lower boundary of the large-scale battery capacities. 
+This is implemented in the dataset :py:class:`StorageEtrago <egon.data.datasets.storages_etrago.StorageEtrago>`, the data for batteries in the transmission grid is stored in the database table :py:class:`grid.egon_etrago_storage <egon.data.datasets.etrago_setup.EgonPfHvStorage>`.

docs/data/e-mobility.rst

@@ -0,0 +1 @@
+What flexibilities does e-mobility provide to the system. How did we implement it?

docs/data/electricity_demand.rst

@@ -0,0 +1 @@
+Information about electricity demands and their spatial and temporal aggregation