diff --git a/CHANGELOG.rst b/CHANGELOG.rst index a7c653334..57a7f4f0c 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -494,12 +494,12 @@ Changed `#1073 `_ * Change hgv data source to use database `#1086 `_ +* Rename eMob MIT carrier names (use underscores) + `#1105 `_ * Change desposit ID for data_bundle download from zenodo sandbox `#1110 `_ * Use MaStR geocoding results for pv rooftop to buildings mapping workflow `#1095 `_ -* Rename eMob MIT carrier names (use underscores) - `#1105 `_ .. _#799: https://github.com/openego/eGon-data/issues/799 @@ -747,7 +747,12 @@ Bug Fixes * Fix conversion factor for CH4 loads abroad in eGon2035 `#1104 `_ * Change structure of documentation in rtd - `#11126 `_ + `#1126 `_ +* Fix URL of eGon data-bundle dataset + `#1154 `_ +* Fix URLs of MaStR datasets +* Fix CRS in ERA5 transformation + `#1159 `_ .. _PR #692: https://github.com/openego/eGon-data/pull/692 .. _#343: https://github.com/openego/eGon-data/issues/343 diff --git a/docs/about.rst b/docs/about.rst index 28f7d3525..561f87bdb 100644 --- a/docs/about.rst +++ b/docs/about.rst @@ -50,12 +50,27 @@ Modeling concept and scenarios =============================== eGon-data provides a data model suitable for calculations and optimizations with the tools eTraGo, eDisGo and eGo and therefore aims to satisfy all requirements regarding the scope and temporal as well as spatial granularity of the resulting data model. + + + +System boundaries and general assumptions +----------------------------------------- + +* Sectors +* Focus on Germany +* Neighbouring countries (which ones and why) +* Spatial resolution / aggregartion levels +* Temporal resolution incl. assumptions on weather year + The following image visualizes the different components considered in scenario ``eGon2035``. .. image:: images/egon-modell-szenario-egon2035.png :width: 800 :alt: Components of the data models +Scenarios +--------- + eGon-data aims to create different scenarios, which differ in terms of RE penetration or the availability of flexibility options. Currently, the following scenarios are available or in progress. * ``eGon2035`` Mid-termin scenario based on assumptions from the German network expansion plan 'scenario C2035', version 2021 and TYNDP diff --git a/docs/data.rst b/docs/data.rst index 4bbf47357..b529e7938 100644 --- a/docs/data.rst +++ b/docs/data.rst @@ -87,7 +87,7 @@ Different flexibility options are part of the model and can be utilized in the o flexibility potentials and their distribution are needed. The considered technologies described in the following chapters range from different storage units, through dynamic line rating to Demand-Side-Management measures. -Demand-Side-Management +Demand-Side Management ---------------------- .. include:: data/DSM.rst diff --git a/docs/data/DSM.rst b/docs/data/DSM.rst index 79d4a96ef..e1a114b53 100644 --- a/docs/data/DSM.rst +++ b/docs/data/DSM.rst @@ -1 +1,50 @@ -How did we implement DSM? Results etc. +Demand-side management (DSM) potentials are calculated in function :func:`dsm_cts_ind_processing`. +Potentials relevant for the high and extra-high voltage grid are identified in the function :func:`dsm_cts_ind`, +potentials within the medium- and low-voltage grids are determined within the function :func:`dsm_cts_ind_individual` +in a higher spatial resolution. All this is part of the dataset :py:class:`DsmPotential `. +The implementation is documented in detail within the following student work (in German): [EsterlDentzien]_. + +Loads eligible to be shifted are assumed within industrial loads and loads from Commercial, Trade and Service (CTS). +Therefore, load time series from these sectors are used as input data (see section ref:`elec_demand-ref`). +Shiftable shares of loads mainly derive from heating and cooling processes and selected energy-intensive +industrial processes (cement production, wood pulp, paper production, recycling paper). Technical and sociotechnical +constraints are considered using the parametrization elaborated in [Heitkoetter]_. An overview over the +resulting potentials for scenario ``eGon2035`` can be seen in figure :ref:`dsm_potential`. The table below summarizes the +aggregated potential for Germany per scenario. As the annual conventional electrical loads are assumed to be lower in the +scenario ``eGon100RE``, also the DSM potential decreases compared to the scenario ``eGon2035``. + +.. figure:: /images/DSM_potential.png + :name: dsm_potential + :width: 600 + + Aggregated DSM potential in Germany for scenario ``eGon2035`` + +.. list-table:: Aggregated DSM Potential for Germany + :widths: 20 20 20 + :header-rows: 1 + + * - + - CTS + - Industry + + * - eGon2035 + - 1.2 GW + - 150 MW + + * - eGon100RE + - 900 MW + - 150 MW + +DSM is modelled following the approach of [Kleinhans]_. DSM components are created wherever +respective loads are seen. Minimum and maximum shiftable power per time step depict time-dependent +charging and discharging power of a storage-equivalent buffers. Time-dependent capacities +of those buffers account for the time frame of management bounding the period within which +the shifting can be conducted. Figure :ref:`dsm_shifted_p-example` shows the resulting potential at one exemplary bus. + +.. figure:: /images/shifted_dsm-example.png + :name: dsm_shifted_p-example + :width: 600 + + Time-dependent DSM potential at one exemplary bus + + diff --git a/docs/data/electricity_demand.rst b/docs/data/electricity_demand.rst index d375e954c..3e82adbe2 100644 --- a/docs/data/electricity_demand.rst +++ b/docs/data/electricity_demand.rst @@ -1 +1,91 @@ -Information about electricity demands and their spatial and temporal aggregation +.. _elec_demand_ref: +The electricity demand considered includes demand from the residential, commercial and industrial sector. +The target values for scenario *eGon2035* are taken from the German grid development plan from 2021 [NEP2021]_, +whereas the distribution on NUTS3-levels corresponds to the data from the research project *DemandRegio* [demandregio]_. +The following table lists the electricity demands per sector: + +.. list-table:: Electricity demand per sector + :widths: 25 50 + :header-rows: 1 + + * - Sector + - Annual electricity demand in TWh + * - residential + - 115.1 + * - commercial + - 123.5 + * - industrial + - 259.5 + +A further spatial and temporal distribution of the electricity demand is needed to fullfil all requirements of the +subsequent grid optimization. Therefore different, sector-specific distributions methods were developed and applied. + +Residential electricity demand +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The annual electricity demands of households on NUTS3-level from *DemandRegio* are scaled to meet the national target +values for the respective scenario in dataset :py:class:`DemandRegio `. +A further spatial and temporal distribution of residential electricity demands is performed in +:py:class:`HouseholdElectricityDemand ` as described +in [Buettner2022]_. +The result is a consistent dataset across aggregation levels with an hourly resolution. + +.. figure:: /images/S27-3.png + :name: spatial_distribution_electricity_demand + :width: 400 + + Electricity demand on NUTS 3-level (upper left); Exemplary MVGD (upper right); Study region in Flensburg (20 Census cells, bottom) from [Buettner2022]_ + + +.. figure:: /images/S27-4a.png + :name: aggregation_level_electricity_demand + :width: 400 + + Electricity demand time series on different aggregation levels from [Buettner2022]_ + + + +Commercial electricity demand +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The distribution of electricity demand from the commercial, trade and service (CTS) sector is also based on data from +*DemandRegio*, which provides annual electricity demands on NUTS3-level for Germany. In dataset +:py:class:`CtsElectricityDemand ` the annual electricity +demands are further distributed to census cells (100x100m cells from [Census]_) based on the distribution of heat demands, +which is taken from the Pan-European Thermal Altlas version 5.0.1 [Peta]_. For further information refer to section +ref:`heat_demand`. +The applied methods for a futher spatial and temporal distribution to buildings is described in [Buettner2022]_ and +performed in dataset :py:class:`CtsDemandBuildings ` + +Industrial electricity demand +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To distribute the annual industrial electricity demand OSM landuse data as well as information on industrial sites are +taken into account. +In a first step (:py:class:`CtsElectricityDemand `) +different sources providing information about specific sites and further information on the industry sector in which +the respective industrial site operates are combined. Here, the three data sources [Hotmaps]_, [sEEnergies]_ and +[Schmidt2018]_ are aligned and joined. +Based on the resulting list of industrial sites in Germany and information on industrial landuse areas from OSM [OSM]_ +which where extracted and processed in :py:class:`OsmLanduse ` the annual demands +were distributed. +The spatial and temporal distribution is performed in +:py:class:`IndustrialDemandCurves `. +For the spatial distribution of annual electricity demands from *DemandRegio* [demandregio]_ which are available on +NUTS3-level are in a first step evenly split 50/50 between industrial sites and OSM-polygons tagged as industrial areas. +Per NUTS-3 area the respective shares are then distributed linearily based on the area of the corresponding landuse polygons +and evenly to the identified industrial sites. +In a next step the temporal disaggregation of the annual demands is carried out taking information about the industrial +sectors and sector-specific standard load profiles from [demandregio]_ into account. +Based on the resulting time series and their peak loads the corresponding grid level and grid connections point is +identified. + +Electricity demand in neighbouring countries +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The neighbouring countries considered in the model are represented in a lower spatial resolution of one or two buses per +country. The national demand timeseries in an hourly resolution of the respective countries is taken from the Ten-Year +Network Development Plan, Version 2020 [TYNDP]_. In case no data for the target year is available the data is is +interpolated linearly. +Refer to the corresponding dataset for detailed information: +:py:class:`ElectricalNeighbours ` diff --git a/docs/data/electricity_grids.rst b/docs/data/electricity_grids.rst index 0d288e836..8c94693c8 100644 --- a/docs/data/electricity_grids.rst +++ b/docs/data/electricity_grids.rst @@ -3,18 +3,32 @@ High and extra-high voltage grids ++++++++++++++++++++++++++++++++++ -The model of the German extra-high (eHV) and high voltage (HV) grid is based -on data retrieved from OpenStreetMap (status January 2021) [OSM]_ and additional -parameters for standard transmission lines from [Brakelmann2004]_. To gather all -required information, such as line topology, voltage level, substation locations, -and electrical parameters, to create a calculable power system model, the `osmTGmod -tool `_ was used. The corresponding dataset -:py:class:`Osmtgmod ` executes osmTGmod +The model of the German extra-high (eHV) and high voltage (HV) grid is based +on data retrieved from OpenStreetMap (OSM) (status January 2021) [OSM]_ and additional +parameters for standard transmission lines from [Brakelmann2004]_. To gather all +required information, such as line topology, voltage level, substation locations, +and electrical parameters, to create a calculable power system model, the `*osmTGmod* +tool `_ was used. The corresponding dataset +:py:class:`Osmtgmod ` executes osmTGmod and writes the resulting data to the database. The resulting grid model includes the voltage levels 380, 220 and 110 kV and -all substations interconnecting these grid levels. For further information on the -generation of the grid topology please refer to [Mueller2018]_. +all substations interconnecting the different grid levels. Information about +border crossing lines are as well extracted from *OSM* data by *osmTGmod*. +For further information on the generation of the grid topology please refer to [Mueller2018]_. +The neighbouring countries are included in the model in a significantly lower +spatial resolution with one or two nodes per country. The border crossing lines +extracted by *osmTGmod* are extended to representative nodes of the respective +country in dataset +:py:class:`ElectricalNeighbours `. The +resulting grid topology is shown in the following figure. + +..figure:: images/Stromnetz.png + :scale: 50 % + :name: gridtopology_ehv_hv + :alt: Grid topology extra-high and high voltage grid + + .. _ding0-grids: diff --git a/docs/data/electricity_supply.rst b/docs/data/electricity_supply.rst index fffd7379c..f8e490429 100644 --- a/docs/data/electricity_supply.rst +++ b/docs/data/electricity_supply.rst @@ -19,13 +19,68 @@ The final distribution in the eGon2035 scenario is shown in figure :ref:`generat Onshore wind +++++++++++++ +The allocation of onshore wind power plants is implemented in the function :func:`insert` +which is part of the dataset :class:`PowerPlants`. +The following steps are conducted: + +#. The sites and capacities of exisitng onshore wind parks are imported using MaStR data (see :ref:`mastr-ref`). + +#. Potential areas for onshore wind parks are assumed to be areas With high mean wind speed, at the same time that some locations like protected natural areas or zones close to urban centers are discarted. Those areas are imported through the data bundle, see :ref:`data-bundle-ref`). + +#. The locations of existing parks and the potential areas are intersected with each other while considering a buffer around the locations of existing parks to find out where there are already parks at or close to potential areas. This results in a selection of potential areas. + +#. The capacities of the existing parks matching potential areas are summed up and compared to the target values for the specific scenario per federal state (see :ref:`concept-and-scenarios-ref`). The required expansion capacity is derived. + +#. If expansion of wind onshore capacity is required, capacities are calculated depending on the area size of the formerly selected potential areas. 21.05 MW/km² and 16.81 MW/km² are used for federal states in the north and in the south of the country respectively. The resulting parks are therefore located on the selected potential areas. + +#. The resulting capacities are compared to the target values for the specific scenario per federal state. If the target value is exceeded, a linear downscaling is conducted. If the target value is not reached yet, the remaining capacity is distributed linearly among the rest of the potential areas within the state. + Offshore wind ++++++++++++++ +The allocation of offshore wind power plants is implemented in the function :func:`insert` +which is part of the dataset :class:`PowerPlants`. +The following steps are conducted: + +#. A compilation of offshore wind parks for different scenarios created by NEP are extracted from the data bundle. See :ref:`data-bundle-ref`. This data includes installed capacities, connection points (or a potential one for future power plants) and location. See figure :ref:`offshore_power_plants_areas`. + +#. Each connection point is matched to one of the substations previously created. Despite the fact that the generators are located in the sea, all the power generated by them will be injected into the grid through these substations, that in some cases can be several kilometers in land. + +#. For the eGon100RE scenario, the installed capacities are scaled up in order to achieve the targed in :ref:`concept-and-scenarios-ref`. + +#. Each offshore wind power plant receives an hourly maximal generation capacity based on weather data for its own geographical location. Weather data provided by ERA5. + +.. figure:: /images/offshore_power_plants_areas.png + :name: offshore_power_plants_areas + :width: 400 + + Areas for offshore wind park in North and Baltic sea. Source: NEP + + PV ground mounted ++++++++++++++++++ -.. _pv-rooftop-ref: +The distribution of PV ground mounted is implemented in function :func:`insert` +which is part of the dataset :class:`PowerPlants`. +The following steps are conducted: + +#. The sites and capacities of exisitng PV parks are imported using MaStR data (see :ref:`mastr-ref`). + +#. Potential areas for PV ground mounted are assumed to be areas next to highways and railways as well as on agricultural land with a low degree of utilisation, as it can be seen in figure :ref:`pv_ground_mounted-example`. Those areas (provided through the data bundle, see :ref:`data-bundle-ref`) are imported while merging or disgarding small areas. + +#. The locations of existing parks and the potential areas are intersected with each other while considering a buffer around the locations of existing parks to find out where there already are parks at or close to potential areas. This results in a selection of potential areas. + +#. The capacities of the existing parks are considered and compared to the target values for the specific scenario per federal state (see :ref:`concept-and-scenarios-ref`). The required expansion capacity is derived. + +#. If expansion of PV ground mounted capacity is required, capacities are calculated depending on the area size of the formerly selected potential areas. The resulting parks are therefore located on the selected potential areas. + +#. The resulting capacities are compared to the target values for the specific scenario per federal state. If the target value is exceeded, a linear downscaling is conducted. If the target value is not reached yet, the remaining capacity is distributed linearly among the rest of the potential areas within the state. + +.. figure:: /images/PV_freiflaeche.png + :name: pv_ground_mounted-example + :width: 400 + + Example: sites of existing PV ground mounted parks and potential areas PV rooftop +++++++++++ @@ -83,6 +138,45 @@ Disaggregation of PV rooftop scenario capacities: using MaStR data as basis. Hydro -++++++ ++++++ + +In the case of hydropower plants, a distinction is made between the carrier run-of-river +and reservoir. +The methods to distribute and allocate are the same for both carriers. +In a first step all suitable power plants (correct carrier, valid geolocation, information +about federal state) are selected and their installed capacity is scaled to meet the target +values for the respective federal state and scenario. +Information about the voltage level the power plants are connected to is obtained. In case +no information is availabe the voltage level is identified using threshold values for the +installed capacity (see :func:`assign_voltage_level `). +In a next step the correct grid connection point is identified based on the voltage level +and geolocation of the power plants (see :func:`assign_bus_id `) +The resulting list of power plants it added to table +:py:class:`EgonPowerPlants `. + +Biomass ++++++++ + +The allocation of biomass-based power plants follows the same method as the one for hydro +power plants and is performed in function :func:`insert_biomass_plants ` + + + +Conventional +++++++++++++ + +**CHP** + + +**non-chp** + + +In function :func:`allocate_conventional_non_chp_power_plants ` +capacities for conventional power plants, which are no chp plants, with carrier *oil* and +*gas* are allocated. + + + + diff --git a/docs/data/gas_demand.rst b/docs/data/gas_demand.rst index 072c66394..561130936 100644 --- a/docs/data/gas_demand.rst +++ b/docs/data/gas_demand.rst @@ -1 +1,20 @@ -Information about gas demands and their spatial and temporal aggregation, including hydrogen and methane demands +In the scenario eGon2035 the gas demand data in Germany stems from the eXtremOS project +(https://openaccess.ffe.de/10.34805/ffe-24-21/) where demands are +provided on a NUTS-3 level and hourly resolution for the year 2035. +These include methane and hydrogen demands for industry. +For the neighboring countries there are no hydrogen nodes available. Instead respective +hydrogen demands for industry are indirectly modelled as electricity demands. The total +hydrogen demand is derived from the *Distributed Energy Scenario* of the *TYNDP 2020* and +has been linearly interpolated between the years 2030 and 2040. The demands are temporarily +constant. +The methane demands of neighboring countries accounts for industry demands and +heating demands. The total demand data stems also from the *Distributed Energy Scenario* of the *TYNDP 2020* +interpolated between 2030 and 2040 while ites temporal profile is derived from the *PyPSA-eur-sec* run +because of the high share of rural heating in total methane demand. + +For the scenario eGon100RE the methane and hydrogen demands for industry in Germany +have been calculated in a *PyPSA-eur-sec* run. The spatial and temporal distribution +accords to the hydrogen industry demands of the eXtremOS project for the year 2050. +In eGon100RE no industrial methane demand is assumed. +For the neighboring countries the industrial gas demands (methane and hydrogen) stem from +the *PyPSA-eur-sec* run. \ No newline at end of file diff --git a/docs/data/gas_grids.rst b/docs/data/gas_grids.rst index f3df18ea2..471218c0a 100644 --- a/docs/data/gas_grids.rst +++ b/docs/data/gas_grids.rst @@ -1,7 +1,20 @@ -Information about the gas grids and how they were created +The gas grid data stems from the *SciGRID_gas project* (https://www.gas.scigrid.de/) which covers the European Gas +Transmission Grid. All data generated in the *SciGRID_gas* project is licenced under +*Creative Commons Attribution 4.0 International Public License*. +The specific dataset version is IGGIELGN and can be downloaded at https://zenodo.org/record/4767098. +*SciGRID_gas* contains extensive data on pipelines, storages, LNGs, productions, consumers and more. +Further information can be obtained in the IGGIELGN documentation. +For eGon-data, *SciGRID_gas* infrastructure data in Germany has been extracted and used in full resolution +while data of neighboring countries has been aggregated. -Methane grid -++++++++++++++ + +Methane grid +~~~~~~~~~~~~ +In the eGon2035 scenario the methane grid is, apart from minor adjustments, equivalent to the gas grid described in the +*SciGRID_gas IGGIELGN* dataset. Hydrogen grid -++++++++++++++ +~~~~~~~~~~~~~ +In the eGon2035 scenario H2 nodes are present at every methane bus (H2_grid) and at locations where there +are possible H2 cavern storages available (H2_saltcavern). There is no explicit H2 pipeline grid available but H2 can +be transported using the methane grid. diff --git a/docs/data/gas_stores.rst b/docs/data/gas_stores.rst index 35c10d070..eca676c65 100644 --- a/docs/data/gas_stores.rst +++ b/docs/data/gas_stores.rst @@ -1,7 +1,13 @@ -Description of methods and assumptions to include potential h2 stores in the system - Hydrogen stores --------------- +There are two types of hydrogen stores available: Underground respectively saltcavern stores +and overground stores respectively steel tank stores. The steel tank stores are available at every hydrogen bus +and have no restrictions regarding the possible build-up potential. On the other hand saltcavern stores are capped +by the respective methane underground storage capacity stemming from the *SciGRID_gas IGGIELGN* dataset. Methane stores -------------- +The data of the methane stores stems from the *SciGRID_gas IGGIELGN* dataset picturing the exisiting CH4 cavern stores. +Additionally the CH4 grid has a storage capacity of 130 GWh which is an estimation of the Bundesnetzagentur. For the +scenario eGon100RE this storage capacity is split between H2 and CH4 stores, with the same share as the pipes capacity. +These capacities have been calculated with a *PyPSA-eur-sec* run. \ No newline at end of file diff --git a/docs/data/gas_supply.rst b/docs/data/gas_supply.rst index e5cc586f9..2eb072961 100644 --- a/docs/data/gas_supply.rst +++ b/docs/data/gas_supply.rst @@ -1 +1,11 @@ -Information on gas supply - hydrogen and methane. +The scenario eGon2035 includes the production of fossil methane and biogas. +The potential supply of fossil methane in Germany stems from the *SciGRID_gas* dataset in the iteration +*IGGIELGN* (https://zenodo.org/record/4767098). The potential for biogas production accords +to the *Biogaspartner Einspeiseatlas* (https://www.biogaspartner.de/einspeiseatlas/). The supply cap +for both, fossil methane and biogas, is derived from the *Netzentwicklungsplan (NEP) Gas 2020-2030* +(https://fnb-gas.de/wp-content/uploads/2021/09/fnb_gas_nep_gas_2020_de-1.pdf). +For the neighboring countries gas supply has been linearly interpolated from the *TYNDP 2020* between the years 2030 and 2040. + +For the eGon100RE scenario only biogas can be produced. The source for its potential production in Germany +is also the *Biogaspartner Einspeiseatlas* but its maximum allowed production is derived from a +*PyPSA-eur-sec* run. For the neighboring countries all data stems from the *PyPSA-eur-sec* run. \ No newline at end of file diff --git a/docs/data/heat_demand.rst b/docs/data/heat_demand.rst index 6e004ae28..d03dac90c 100644 --- a/docs/data/heat_demand.rst +++ b/docs/data/heat_demand.rst @@ -1,4 +1,5 @@ -Heat demands comprise space heating and dirinking hot water demands from +.. _heat_demand: +Heat demands comprise space heating and drinking hot water demands from residential and comertial trade and service (CTS) buildings. Process heat demands from the industry are, depending on the required temperature level, modelled as electrcity, hydrogen or methane demand. diff --git a/docs/data/input_data.rst b/docs/data/input_data.rst index b1dd53cb6..c86ce0c82 100644 --- a/docs/data/input_data.rst +++ b/docs/data/input_data.rst @@ -40,8 +40,8 @@ The data are obtained from zenodo, where raw MaStR data, downloaded with the too is provided. It contains all data from the MaStR, including possible duplicates. Currently, two versions are used: -* `2021-05-03 `_ -* `2022-11-17 `_ +* `2021-04-30 `_ +* `2022-11-17 `_ The download is implemented in :class:`MastrData`. diff --git a/docs/images/DSM_potential.png b/docs/images/DSM_potential.png new file mode 100644 index 000000000..1312cfda0 Binary files /dev/null and b/docs/images/DSM_potential.png differ diff --git a/docs/images/PV_freiflaeche.png b/docs/images/PV_freiflaeche.png new file mode 100644 index 000000000..353f25c83 Binary files /dev/null and b/docs/images/PV_freiflaeche.png differ diff --git a/docs/images/Stromnetz.png b/docs/images/Stromnetz.png new file mode 100644 index 000000000..47034f110 Binary files /dev/null and b/docs/images/Stromnetz.png differ diff --git a/docs/images/offshore_power_plants_areas.png b/docs/images/offshore_power_plants_areas.png new file mode 100644 index 000000000..9b874b1a6 Binary files /dev/null and b/docs/images/offshore_power_plants_areas.png differ diff --git a/docs/images/shifted_dsm-example.png b/docs/images/shifted_dsm-example.png new file mode 100644 index 000000000..2ca2622b4 Binary files /dev/null and b/docs/images/shifted_dsm-example.png differ diff --git a/docs/literature.rst b/docs/literature.rst index e44a0dc0f..9bb2b9fff 100644 --- a/docs/literature.rst +++ b/docs/literature.rst @@ -16,16 +16,24 @@ Literature .. [Energiereferenzprognose] Prognos AG, Energiewirtschaftliches Institut an der Universität zu Köln, Gesellschaft für Wirtschaftliche Strukturforschung mbH: Entwicklung der Energiemärkte – Energiereferenzprognose (2014) +.. [EsterlDentzien] Katharina Esterl, Hannah Dentzien, Integration von Demand Side Management in eTraGo, Student Work, Hochschule Flensburg, URL https://ego-n.org/theses/2021_SP_Esterl_Dentzien_DSM-eTraGo.pdf + .. [eXtremOS] A. Guminski, C. Fiedler, S. Kigle, C. Pellinger, P. Dossow, K. Ganz, F. Jetter, T. Kern, T. Limmer, A. Murmann, J. Reinhard, T. Schmid, T. Schmidt-Achert, S. von Roon, eXtremOS Summary Report (2021). doi:https://doi.org/10.34805/ffe-24-21. .. [FfE2017] Flexibilisierung der Kraft-Wärme-Kopplung; 2017; Forschungsstelle für Energiewirtschaft e.V. (FfE) +.. [Heitkoetter] Wilko Heitkoetter, Bruno U. Schyska, Danielle Schmidt, Wided Medjroubi, Thomas Vogt, Carsten Agert, Assessment of the regionalised demand response potential in Germany using an open source tool and dataset, Advances in Applied Energy (2021), URL https://www.sciencedirect.com/science/article/pii/S2666792420300019 + .. [Helfenbein2021] K. Helfenbein, Analyse des Einflusses netzdienlicher Ladestrategien auf Verteilnetze aufgrund der zunehmenden Netzintegration von Elektrofahrzeugen, Master’s thesis, Hochschule für Technik und Wirtschaft Berlin, URL https://reiner-lemoine-institut.de/analyse-einflussesnetzdienlicher-ladestrategien-verteilnetze-zunehmender-netzintegration-elektrofahrzeugen-helfenbein-2021/ .. [Hotmaps] S. Pezzutto, S. Zambotti, S. Croce, P. Zambelli, G. Garegnani, C. Scaramuzzino, R. P. Pascuas, A. Zubaryeva, F. Haas, D. Exner, A. Mueller, M. Hartner, T. Fleiter, A.-L. Klingler, M. Kuehnbach, P. Manz, S. Marwitz, M. Rehfeldt, J. Steinbach, E. Popovski, Hotmaps project, d2.3 wp2 report – open data set for the eu28 (2018). URL www.hotmaps-project.eu .. [Huelk2017] L. Hülk, L. Wienholt, I. Cußmann, U.P. Müller, C. Matke, E. Kötter, Allocation of annual electricity consumption and power generation capacities across multiple voltage levels in a high spatial resolution, International Journal of Sustainable Energy Planning and Management Vol. 13 2017 79–92. URL https://journals.aau.dk/index.php/sepm/article/view/1833 +.. [Kleinhans] D. Kleinhans, Towards a systematic characterization of the potential of demand side management, arXiv (2014), doi: 10.48550/ARXIV.1401.4121. URL https://arxiv.org/abs/1401.4121 + +.. [MaStR] Bundesnetzagentur für Elektrizität, Gas, Telekommunikation, Post und Eisenbahnen, Marktstammdatenregister - Datendownload (Nov. 2022). URL https://www.marktstammdatenregister.de/MaStR/Datendownload + .. [MiD2017] Bundesministerium für Digitales und Verkehr, Mobilität in Deutschland 2017 (2017). URL https://daten.clearingstelle-verkehr.de/279/ .. [Mueller2018] U. Mueller, L. Wienholt, D. Kleinhans, I. Cussmann, W.-D. Bunke, G. Pleßmann, J. Wendiggensen 2018 J. Phys.: Conf. Ser. 977 012003, DOI 10.1088/1742-6596/977/1/012003 @@ -43,3 +51,5 @@ Literature .. [Schmidt2018] D. Schmidt, Supplementary material to the masters thesis: NUTS-3 Regionalization of Industrial Load Shifting Potential in Germany using a Time-Resolved Model (Nov. 2019). doi:10.5281/zenodo.3613767. URL https://doi.org/10.5281/zenodo.3613767 .. [sEEnergies] T. Fleiter, P. Manz, N. Neuwirth, F. Mildner, K. Persson, U.AND Kermeli, W. Crijns-Graus, C. Rutten, seenergies d5.1 dataset web-app.seenergies arcgis online web-apps hosted by europa-universität flensburg (2020). URL https://tinyurl.com/sEEnergies-D5-1 + +.. [TYNDP] European Network of Transmission System Operators for Electricity, European Network of Transmission System Operators for Gas, Ten-Year Network Development Plans - “TYNDP 2020 Scenarios” (2020) diff --git a/src/egon/data/datasets.yml b/src/egon/data/datasets.yml index ddf7246db..314747388 100755 --- a/src/egon/data/datasets.yml +++ b/src/egon/data/datasets.yml @@ -295,7 +295,7 @@ mastr: - "gsgk" - "storage" file_basename: "bnetza_mastr" - deposit_id: 808086 + deposit_id: 10480930 mastr_new: technologies: @@ -308,7 +308,7 @@ mastr_new: - "storage" - "wind" file_basename: "bnetza_mastr" - deposit_id: 1132987 + deposit_id: 10480958 egon2021_date_max: "2021-12-31 23:59:00" geocoding_path: - "data_bundle_egon_data" @@ -540,7 +540,7 @@ solar_rooftop: data-bundle: sources: zenodo: - deposit_id: 1167119 + deposit_id: 10226009 targets: file: 'data_bundle_egon_data.zip' diff --git a/src/egon/data/datasets/DSM_cts_ind.py b/src/egon/data/datasets/DSM_cts_ind.py index a74ae11da..7b21acbdb 100644 --- a/src/egon/data/datasets/DSM_cts_ind.py +++ b/src/egon/data/datasets/DSM_cts_ind.py @@ -8,6 +8,10 @@ sanity checks. See [#1120](https://github.com/openego/eGon-data/issues/1120) for updates. """ +import datetime +import json + +from omi.dialects import get_dialect from sqlalchemy import ARRAY, Column, Float, Integer, String from sqlalchemy.ext.declarative import declarative_base import geopandas as gpd @@ -18,6 +22,15 @@ from egon.data.datasets import Dataset from egon.data.datasets.electricity_demand.temporal import calc_load_curve from egon.data.datasets.industry.temporal import identify_bus +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_odbl, + meta_metadata, + meta_metadata, + sources, +) # CONSTANTS # TODO: move to datasets.yml @@ -128,7 +141,7 @@ def __init__(self, dependencies): name=self.name, version=self.version, dependencies=self.dependencies, - tasks=(dsm_cts_ind_processing), + tasks=(dsm_cts_ind_processing,), ) @@ -205,6 +218,195 @@ class EgonSitesIndLoadCurvesIndividualDsmTimeseries(Base): e_min = Column(ARRAY(Float)) +def add_metadata_individual(): + targets = config.datasets()["DSM_CTS_industry"]["targets"] + + targets = { + k: v for k, v in targets.items() if "dsm_timeseries" in v["table"] + } + + title_dict = { + "egon_etrago_electricity_cts_dsm_timeseries": ( + "DSM flexibility band time series for CTS" + ), + "egon_osm_ind_load_curves_individual_dsm_timeseries": ( + "DSM flexibility band time series for OSM industry sites" + ), + "egon_demandregio_sites_ind_electricity_dsm_timeseries": ( + "DSM flexibility band time series for demandregio industry sites" + ), + "egon_sites_ind_load_curves_individual_dsm_timeseries": ( + "DSM flexibility band time series for other industry sites" + ), + } + + description_dict = { + "egon_etrago_electricity_cts_dsm_timeseries": ( + "DSM flexibility band time series for CTS in 1 h resolution " + "including available store capacity and power potential" + ), + "egon_osm_ind_load_curves_individual_dsm_timeseries": ( + "DSM flexibility band time series for OSM industry sites in 1 h " + "resolution including available store capacity and power potential" + ), + "egon_demandregio_sites_ind_electricity_dsm_timeseries": ( + "DSM flexibility band time series for demandregio industry sites " + "in 1 h resolution including available store capacity and power " + "potential" + ), + "egon_sites_ind_load_curves_individual_dsm_timeseries": ( + "DSM flexibility band time series for other industry sites in 1 h " + "resolution including available store capacity and power potential" + ), + } + + keywords_dict = { + "egon_etrago_electricity_cts_dsm_timeseries": ["cts"], + "egon_osm_ind_load_curves_individual_dsm_timeseries": [ + "osm", + "industry", + ], + "egon_demandregio_sites_ind_electricity_dsm_timeseries": [ + "demandregio", + "industry", + ], + "egon_sites_ind_load_curves_individual_dsm_timeseries": ["industry"], + } + + primaryKey_dict = { + "egon_etrago_electricity_cts_dsm_timeseries": ["bus"], + "egon_osm_ind_load_curves_individual_dsm_timeseries": ["osm_id"], + "egon_demandregio_sites_ind_electricity_dsm_timeseries": [ + "industrial_sites_id", + ], + "egon_sites_ind_load_curves_individual_dsm_timeseries": ["site_id"], + } + + sources_dict = { + "egon_etrago_electricity_cts_dsm_timeseries": [ + sources()["nep2021"], + sources()["zensus"], + ], + "egon_osm_ind_load_curves_individual_dsm_timeseries": [ + sources()["hotmaps_industrial_sites"], + sources()["schmidt"], + sources()["seenergies"], + ], + "egon_demandregio_sites_ind_electricity_dsm_timeseries": [ + sources()["openstreetmap"], + ], + "egon_sites_ind_load_curves_individual_dsm_timeseries": [ + sources()["hotmaps_industrial_sites"], + sources()["openstreetmap"], + sources()["schmidt"], + sources()["seenergies"], + ], + } + + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-17" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + for t_dict in targets.values(): + schema = t_dict["schema"] + table = t_dict["table"] + name = f"{schema}.{table}" + + meta = { + "name": name, + "title": title_dict[table], + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": description_dict[table], + "language": "en-US", + "keywords": ["dsm", "timeseries"] + keywords_dict[table], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "none", + }, + "temporal": { + "referenceDate": "2011-01-01", + "timeseries": { + "start": "2011-01-01", + "end": "2011-12-31", + "resolution": "1 h", + "alignment": "left", + "aggregationType": "average", + }, + }, + "sources": [ + sources()["egon-data"], + sources()["vg250"], + sources()["demandregio"], + ] + + sources_dict[table], + "licenses": [license_odbl("© eGon development team")], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": name, + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + schema, + table, + ), + "primaryKey": ["scn_name"] + primaryKey_dict[table], + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://" + "github.com/OpenEnergyPlatform/oemetadata/blob/master/" + "metadata/v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + schema, + table, + ) + + # Code def cts_data_import(cts_cool_vent_ac_share): """ @@ -999,10 +1201,10 @@ def delete_dsm_entries(carrier): # buses - sql = f""" - DELETE FROM {targets["bus"]["schema"]}.{targets["bus"]["table"]} b - WHERE (b.carrier LIKE '{carrier}'); - """ + sql = ( + f"DELETE FROM {targets['bus']['schema']}.{targets['bus']['table']} b " + f"WHERE (b.carrier LIKE '{carrier}');" + ) db.execute_sql(sql) # links @@ -1621,3 +1823,5 @@ def dsm_cts_ind_processing(): dsm_cts_ind() dsm_cts_ind_individual() + + add_metadata_individual() diff --git a/src/egon/data/datasets/ch4_storages.py b/src/egon/data/datasets/ch4_storages.py index 8c9e9b60b..18e1db935 100755 --- a/src/egon/data/datasets/ch4_storages.py +++ b/src/egon/data/datasets/ch4_storages.py @@ -35,7 +35,7 @@ class CH4Storages(Dataset): *Dependencies* * :py:class:`GasAreaseGon2035 ` - * :py:class:`GasAreaseGon2035 ` + * :py:class:`GasAreaseGon100RE ` * :py:class:`GasNodesAndPipes ` *Resulting tables* diff --git a/src/egon/data/datasets/data_bundle/__init__.py b/src/egon/data/datasets/data_bundle/__init__.py index 928547fde..76db4c650 100644 --- a/src/egon/data/datasets/data_bundle/__init__.py +++ b/src/egon/data/datasets/data_bundle/__init__.py @@ -25,7 +25,7 @@ def download(): # Get parameters from config and set download URL sources = config.datasets()["data-bundle"]["sources"]["zenodo"] url = ( - f"https://sandbox.zenodo.org/record/{sources['deposit_id']}/files/" + f"https://zenodo.org/record/{sources['deposit_id']}/files/" "data_bundle_egon_data.zip" ) target_file = config.datasets()["data-bundle"]["targets"]["file"] diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py b/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py index b73454fe4..cc5fca4e7 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel/__init__.py @@ -31,6 +31,7 @@ EgonEvMvGridDistrict, EgonEvPool, EgonEvTrip, + add_metadata, ) from egon.data.datasets.emobility.motorized_individual_travel.ev_allocation import ( # noqa: E501 allocate_evs_numbers, @@ -475,5 +476,6 @@ def generate_model_data_tasks(scenario_name): *generate_model_data_tasks(scenario_name="eGon2035"), *generate_model_data_tasks(scenario_name="eGon100RE"), }, + add_metadata, ), ) diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py b/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py index 95744835b..42396f538 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel/db_classes.py @@ -1,7 +1,10 @@ """ DB tables / SQLAlchemy ORM classes for motorized individual travel """ +import datetime +import json +from omi.dialects import get_dialect from sqlalchemy import ( Boolean, Column, @@ -15,11 +18,23 @@ from sqlalchemy.dialects.postgresql import REAL from sqlalchemy.ext.declarative import declarative_base +from egon.data import db +from egon.data.datasets.emobility.motorized_individual_travel.helpers import ( + read_simbev_metadata_file, +) from egon.data.datasets.mv_grid_districts import MvGridDistricts from egon.data.datasets.scenario_parameters import EgonScenario - -# from sqlalchemy.orm import relationship - +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_agpl, + license_ccby, + license_odbl, + meta_metadata, + meta_metadata, + sources, +) Base = declarative_base() @@ -270,7 +285,7 @@ class EgonEvMetadata(Base): __tablename__ = "egon_ev_metadata" __table_args__ = {"schema": "demand"} - scenario = Column(String, primary_key=True) + scenario = Column(String, primary_key=True, index=True) eta_cp = Column(Float) stepsize = Column(Integer) start_date = Column(DateTime) @@ -278,3 +293,373 @@ class EgonEvMetadata(Base): soc_min = Column(Float) grid_timeseries = Column(Boolean) grid_timeseries_by_usecase = Column(Boolean) + + +def add_metadata(): + """ + Add metadata to tables egon_ev_metadata, egon_ev_mv_grid_district, + egon_ev_trip in schema demand + """ + # egon_ev_metadata + schema = "demand" + meta_run_config = read_simbev_metadata_file("eGon100RE", "config").loc[ + "basic" + ] + + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-17" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + table = "egon_ev_metadata" + name = f"{schema}.{table}" + + meta = { + "name": name, + "title": "eGon EV metadata", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ( + "Metadata regarding the generation of EV trip profiles with SimBEV" + ), + "language": "en-US", + "keywords": ["ev", "mit", "simbev", "metadata", "parameters"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "none", + "resolution": "none", + }, + "temporal": { + "referenceDate": f"{meta_run_config.start_date}", + "timeseries": {}, + }, + "sources": [ + sources()["egon-data"], + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_ccby(attribution="© Reiner Lemoine Institut") + ], + }, + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_agpl(attribution="© Reiner Lemoine Institut") + ], + }, + ], + "licenses": [license_ccby()], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": name, + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + schema, + table, + ), + "primaryKey": "scenario", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github." + "com/OpenEnergyPlatform/oemetadata/blob/master/metadata/" + "v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + schema, + table, + ) + + table = "egon_ev_mv_grid_district" + name = f"{schema}.{table}" + + meta = { + "name": name, + "title": "eGon EV MV grid district", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ("EV mapping to MV grids"), + "language": "en-US", + "keywords": ["ev", "mit", "simbev", "mv", "grid"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "Grid district", + }, + "temporal": { + "referenceDate": f"{meta_run_config.start_date}", + "timeseries": {}, + }, + "sources": [ + sources()["bgr_inspee"], + sources()["bgr_inspeeds"], + sources()["bgr_inspeeds_data_bundle"], + sources()["bgr_inspeeds_report"], + sources()["demandregio"], + sources()["dsm-heitkoetter"], + sources()["egon-data"], + sources()["era5"], + sources()["hotmaps_industrial_sites"], + sources()["mastr"], + sources()["nep2021"], + sources()["openffe_gas"], + sources()["openstreetmap"], + sources()["peta"], + sources()["pipeline_classification"], + sources()["SciGRID_gas"], + sources()["schmidt"], + sources()["technology-data"], + sources()["tyndp"], + sources()["vg250"], + sources()["zensus"], + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_ccby(attribution="© Reiner Lemoine Institut") + ], + }, + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_agpl(attribution="© Reiner Lemoine Institut") + ], + }, + ], + "licenses": [license_odbl()], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": name, + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + schema, + table, + ), + "primaryKey": "id", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github." + "com/OpenEnergyPlatform/oemetadata/blob/master/metadata/" + "v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + schema, + table, + ) + + table = "egon_ev_trip" + name = f"{schema}.{table}" + + meta = { + "name": name, + "title": "eGon EV trip profiles", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ("EV trip profiles generated with SimBEV"), + "language": "en-US", + "keywords": ["ev", "mit", "simbev", "trip", "profiles"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "none", + }, + "temporal": { + "referenceDate": f"{meta_run_config.start_date}", + "timeseries": {}, + }, + "sources": [ + sources()["bgr_inspee"], + sources()["bgr_inspeeds"], + sources()["bgr_inspeeds_data_bundle"], + sources()["bgr_inspeeds_report"], + sources()["demandregio"], + sources()["dsm-heitkoetter"], + sources()["egon-data"], + sources()["era5"], + sources()["hotmaps_industrial_sites"], + sources()["mastr"], + sources()["nep2021"], + sources()["openffe_gas"], + sources()["openstreetmap"], + sources()["peta"], + sources()["pipeline_classification"], + sources()["SciGRID_gas"], + sources()["schmidt"], + sources()["technology-data"], + sources()["tyndp"], + sources()["vg250"], + sources()["zensus"], + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_ccby(attribution="© Reiner Lemoine Institut") + ], + }, + { + "title": "SimBEV", + "description": ( + "Simulation of electric vehicle charging demand" + ), + "path": "https://github.com/rl-institut/simbev", + "licenses": [ + license_agpl(attribution="© Reiner Lemoine Institut") + ], + }, + ], + "licenses": [license_odbl()], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": name, + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + schema, + table, + ), + "primaryKey": ["scenario", "event_id"], + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github." + "com/OpenEnergyPlatform/oemetadata/blob/master/metadata/" + "v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + schema, + table, + ) diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py index 96a16e042..ae8532b82 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/__init__.py @@ -17,6 +17,7 @@ from egon.data.datasets import Dataset from egon.data.datasets.emobility.motorized_individual_travel_charging_infrastructure.db_classes import ( # noqa: E501 EgonEmobChargingInfrastructure, + add_metadata, ) from egon.data.datasets.emobility.motorized_individual_travel_charging_infrastructure.infrastructure_allocation import ( # noqa: E501 run_tracbev, @@ -148,5 +149,6 @@ def __init__(self, dependencies): get_tracbev_data, }, run_tracbev, + add_metadata, ), ) diff --git a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py index 6f4606353..0a4502019 100644 --- a/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py +++ b/src/egon/data/datasets/emobility/motorized_individual_travel_charging_infrastructure/db_classes.py @@ -2,11 +2,23 @@ DB tables / SQLAlchemy ORM classes for charging infrastructure """ +import datetime +import json + from geoalchemy2 import Geometry +from omi.dialects import get_dialect from sqlalchemy import Column, Float, Integer, String from sqlalchemy.ext.declarative import declarative_base -from egon.data import config +from egon.data import config, db +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_odbl, + meta_metadata, + meta_metadata, +) Base = declarative_base() DATASET_CFG = config.datasets()["charging_infrastructure"] @@ -31,3 +43,121 @@ class EgonEmobChargingInfrastructure(Base): srid=DATASET_CFG["original_data"]["sources"]["tracbev"]["srid"] ) ) + + +def add_metadata(): + """ + Add metadata to table grid.egon_emob_charging_infrastructure + """ + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-14" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + meta = { + "name": "grid.egon_emob_charging_infrastructure", + "title": "eGon Electromobility Charging Infrastructure", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ( + "Identified sites for charging infrastructure for motorized " + "individual travel using TracBEV" + ), + "language": "en-US", + "keywords": [ + "mit", + "charging", + "infrastructure", + "electromobility", + "tracbev", + ], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": None, + "extent": "Germany", + "resolution": "1 m", + }, + "temporal": { + "referenceDate": "2022-04-21", + "timeseries": {}, + }, + "sources": [ + { + "title": "TracBEV input data", + "description": ( + "This data set is used with the software tool TracBEV to " + "calculate locations for charging infrastructure from " + "SimBEV results." + ), + "path": "https://zenodo.org/record/6466480#.YmE9xtPP1hE", + "licenses": [license_odbl(attribution="© Schiel, Moritz")], + } + ], + "licenses": [ + license_odbl(attribution="© eGon development team"), + ], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": "grid.egon_emob_charging_infrastructure", + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + DATASET_CFG["targets"]["charging_infrastructure"][ + "schema" + ], + DATASET_CFG["targets"]["charging_infrastructure"][ + "table" + ], + ), + "primaryKey": "cp_id", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github.com/Op" + "enEnergyPlatform/oemetadata/blob/master/metadata/v141/metadat" + "a_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time zone " + "(YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, en-US, " + "de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/OpenEnergyP" + "latform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + DATASET_CFG["targets"]["charging_infrastructure"]["schema"], + DATASET_CFG["targets"]["charging_infrastructure"]["table"], + ) diff --git a/src/egon/data/datasets/era5.py b/src/egon/data/datasets/era5.py index b32061373..ffd6085fe 100644 --- a/src/egon/data/datasets/era5.py +++ b/src/egon/data/datasets/era5.py @@ -104,7 +104,7 @@ def import_cutout(boundary="Europe"): "SELECT geometry as geom FROM boundaries.vg250_sta_bbox", db.engine(), ) - .to_crs(4623) + .to_crs(4326) .geom ) xs = slice(geom_de.bounds.minx[0], geom_de.bounds.maxx[0]) diff --git a/src/egon/data/datasets/gas_neighbours/__init__.py b/src/egon/data/datasets/gas_neighbours/__init__.py index 7732553b1..303b5a3bb 100755 --- a/src/egon/data/datasets/gas_neighbours/__init__.py +++ b/src/egon/data/datasets/gas_neighbours/__init__.py @@ -14,6 +14,22 @@ class GasNeighbours(Dataset): + """ + Inserts generation, demand, grid, OCGTs and gas neighbors into database. + + *Dependencies* + * :py:class:`GasNodesAndPipes ` + * :py:class:`PypsaEurSec ` + * :py:class:`ElectricalNeighbours ` + * :py:class:`HydrogenBusEtrago ` + * :py:class:`GasAreaseGon100RE ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_link ` is extended + * :py:class:`grid.egon_etrago_load ` is extended + * :py:class:`grid.egon_etrago_generator ` is extended + + """ def __init__(self, dependencies): super().__init__( name="GasNeighbours", diff --git a/src/egon/data/datasets/mastr.py b/src/egon/data/datasets/mastr.py index 0881e9dd3..eef2bfe51 100644 --- a/src/egon/data/datasets/mastr.py +++ b/src/egon/data/datasets/mastr.py @@ -22,7 +22,7 @@ def download(dataset_name, download_dir): # Get parameters from config and set download URL data_config = egon.data.config.datasets()[dataset_name] zenodo_files_url = ( - f"https://sandbox.zenodo.org/record/" + f"https://zenodo.org/record/" f"{data_config['deposit_id']}/files/" ) @@ -58,15 +58,15 @@ class mastr_data_setup(Dataset): The downloaded data incorporates two different datasets: - Dump 2021-05-03 - * Source: https://sandbox.zenodo.org/record/808086 + Dump 2021-04-30 + * Source: https://zenodo.org/records/10480930 * Used technologies: PV plants, wind turbines, biomass, hydro plants, combustion, nuclear, gsgk, storage * Data is further processed in the :py:class:`PowerPlants ` dataset Dump 2022-11-17 - * Source: https://sandbox.zenodo.org/record/1132839 + * Source: https://zenodo.org/records/10480958 * Used technologies: PV plants, wind turbines, biomass, hydro plants * Data is further processed in module :py:mod:`mastr ` and :py:class:`PowerPlants @@ -79,7 +79,7 @@ class mastr_data_setup(Dataset): #: name: str = "MastrData" #: - version: str = "0.0.1" + version: str = "0.0.2" #: tasks = (download_mastr_data,) diff --git a/src/egon/data/datasets/power_etrago/__init__.py b/src/egon/data/datasets/power_etrago/__init__.py index 862271fa6..e43058619 100755 --- a/src/egon/data/datasets/power_etrago/__init__.py +++ b/src/egon/data/datasets/power_etrago/__init__.py @@ -7,6 +7,17 @@ class OpenCycleGasTurbineEtrago(Dataset): + """ + Insert open cycle gas turbines (OCGT) into the database. + + *Dependencies* + * :py:class:`GasAreaseGon2035 ` + * :py:class:`PowerPlants ` + + *Resulting tables* + * :py:class:`grid.egon_etrago_link ` is extended + + """ def __init__(self, dependencies): super().__init__( name="OpenCycleGasTurbineEtrago", diff --git a/src/egon/data/datasets/power_plants/__init__.py b/src/egon/data/datasets/power_plants/__init__.py index 8a293a678..c9645c201 100755 --- a/src/egon/data/datasets/power_plants/__init__.py +++ b/src/egon/data/datasets/power_plants/__init__.py @@ -95,7 +95,7 @@ class PowerPlants(Dataset): #: name: str = "PowerPlants" #: - version: str = "0.0.18" + version: str = "0.0.19" def __init__(self, dependencies): super().__init__( diff --git a/src/egon/data/datasets/power_plants/mastr.py b/src/egon/data/datasets/power_plants/mastr.py index 225c7c685..dabb20008 100644 --- a/src/egon/data/datasets/power_plants/mastr.py +++ b/src/egon/data/datasets/power_plants/mastr.py @@ -33,6 +33,7 @@ from egon.data import config, db from egon.data.datasets.mastr import WORKING_DIR_MASTR_NEW from egon.data.datasets.power_plants.mastr_db_classes import ( + add_metadata, EgonMastrGeocoded, EgonPowerPlantsBiomass, EgonPowerPlantsCombustion, @@ -528,3 +529,5 @@ def import_mastr() -> None: if_exists="append", schema=target_tables[tech].__table_args__["schema"], ) + + add_metadata() diff --git a/src/egon/data/datasets/power_plants/mastr_db_classes.py b/src/egon/data/datasets/power_plants/mastr_db_classes.py index 7891dd796..f95c6d1fd 100644 --- a/src/egon/data/datasets/power_plants/mastr_db_classes.py +++ b/src/egon/data/datasets/power_plants/mastr_db_classes.py @@ -1,4 +1,8 @@ +import datetime +import json + from geoalchemy2 import Geometry +from omi.dialects import get_dialect from sqlalchemy import ( Boolean, Column, @@ -10,6 +14,17 @@ ) from sqlalchemy.ext.declarative import declarative_base +from egon.data import config, db +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_dedl, + meta_metadata, + meta_metadata, + sources, +) + Base = declarative_base() @@ -280,3 +295,154 @@ class EgonPowerPlantsStorage(Base): geometry_geocoded = Column(Boolean) geom = Column(Geometry("POINT", 4326), index=True, nullable=True) + + +def add_metadata(): + technologies = config.datasets()["mastr_new"]["technologies"] + + target_tables = { + "pv": EgonPowerPlantsPv, + "wind": EgonPowerPlantsWind, + "biomass": EgonPowerPlantsBiomass, + "hydro": EgonPowerPlantsHydro, + "combustion": EgonPowerPlantsCombustion, + "gsgk": EgonPowerPlantsGsgk, + "nuclear": EgonPowerPlantsNuclear, + "storage": EgonPowerPlantsStorage, + } + + deposit_id_data_bundle = config.datasets()["data-bundle"]["sources"][ + "zenodo" + ]["deposit_id"] + deposit_id_mastr = config.datasets()["mastr_new"]["deposit_id"] + + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-15" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + for technology in technologies: + target_table = target_tables[technology] + + meta = { + "name": ( + f"{target_table.__table_args__['schema']}." + f"{target_table.__tablename__}" + ), + "title": f"eGon {technology} power plants", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ( + f"eGon {technology} power plants status quo derived from MaStR" + ), + "language": "en-US", + "keywords": [technology, "mastr"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "1 m", + }, + "temporal": { + "referenceDate": ( + config.datasets()["mastr_new"]["egon2021_date_max"].split( + " " + )[0] + ), + "timeseries": {}, + }, + "sources": [ + { + "title": "Data bundle for egon-data", + "description": ( + "Data bundle for egon-data: A transparent and " + "reproducible data processing pipeline for energy " + "system modeling" + ), + "path": ( + "https://zenodo.org/record/" + f"{deposit_id_data_bundle}#.Y_dWM4CZMVM" + ), + "licenses": [license_dedl(attribution="© Cußmann, Ilka")], + }, + { + "title": ( + "open-MaStR power unit registry for eGo^n project" + ), + "description": ( + "Data from Marktstammdatenregister (MaStR) data using " + "the data dump from 2022-11-17 for eGon-data." + ), + "path": ( + f"https://zenodo.org/record/{deposit_id_mastr}" + ), + "licenses": [license_dedl(attribution="© Amme, Jonathan")], + }, + sources()["egon-data"], + ], + "licenses": [license_dedl(attribution="© eGon development team")], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": ( + f"{target_table.__table_args__['schema']}." + f"{target_table.__tablename__}" + ), + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + target_table.__table_args__["schema"], + target_table.__tablename__, + geom_columns=["geom"], + ), + "primaryKey": "id", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github." + "com/OpenEnergyPlatform/oemetadata/blob/master/metadata/" + "v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + target_table.__table_args__["schema"], + target_table.__tablename__, + ) diff --git a/src/egon/data/datasets/power_plants/metadata.py b/src/egon/data/datasets/power_plants/metadata.py index 91fe353e6..703fd3f33 100644 --- a/src/egon/data/datasets/power_plants/metadata.py +++ b/src/egon/data/datasets/power_plants/metadata.py @@ -42,7 +42,7 @@ def metadata(): { "title": '"open-MaStR power unit registry"', "description": "Raw data download Marktstammdatenregister (MaStR) data using the webservice. All data from the Marktstammdatenregister is included. There are duplicates included. For further information read in the documentationg of the original data source: https://www.marktstammdatenregister.de/MaStRHilfe/subpages/statistik.html", - "path": "https://sandbox.zenodo.org/record/808086", + "path": "https://zenodo.org/record/10480958", "licenses": [ { "name": licenses_datenlizenz_deutschland( @@ -121,7 +121,7 @@ def metadata(): { "title": '"Data bundle for egon-data"', "description": "Zenodo repository to provide several different input data sets for eGon-data", - "path": "https://sandbox.zenodo.org/record/1167119", + "path": "https://zenodo.org/record/10226009", "licenses": [license_ccby("© eGon development team")], }, ], diff --git a/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py b/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py index 07c43489d..51858a59f 100644 --- a/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py +++ b/src/egon/data/datasets/power_plants/pv_rooftop_buildings.py @@ -10,10 +10,13 @@ from collections import Counter from functools import wraps from time import perf_counter +import datetime +import json from geoalchemy2 import Geometry from loguru import logger from numpy.random import RandomState, default_rng +from omi.dialects import get_dialect from pyproj.crs.crs import CRS from sqlalchemy import BigInteger, Column, Float, Integer, String from sqlalchemy.dialects.postgresql import HSTORE @@ -29,6 +32,16 @@ from egon.data.datasets.power_plants.mastr_db_classes import EgonPowerPlantsPv from egon.data.datasets.scenario_capacities import EgonScenarioCapacities from egon.data.datasets.zensus_vg250 import Vg250Gem +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_dedl, + license_odbl, + meta_metadata, + meta_metadata, + sources, +) engine = db.engine() Base = declarative_base() @@ -2082,6 +2095,138 @@ class EgonPowerPlantPvRoofBuilding(Base): weather_cell_id = Column(Integer) +def add_metadata(): + schema = "supply" + table = "egon_power_plants_pv_roof_building" + name = f"{schema}.{table}" + deposit_id_mastr = config.datasets()["mastr_new"]["deposit_id"] + deposit_id_data_bundle = config.datasets()["data-bundle"]["sources"][ + "zenodo" + ]["deposit_id"] + + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-16" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + meta = { + "name": name, + "title": "eGon power plants rooftop solar", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": ( + "eGon power plants rooftop solar systems allocated to buildings" + ), + "language": "en-US", + "keywords": ["photovoltaik", "solar", "pv", "mastr", "status quo"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "building", + }, + "temporal": { + "referenceDate": ( + config.datasets()["mastr_new"]["egon2021_date_max"].split(" ")[ + 0 + ] + ), + "timeseries": {}, + }, + "sources": [ + { + "title": "Data bundle for egon-data", + "description": ( + "Data bundle for egon-data: A transparent and " + "reproducible data processing pipeline for energy " + "system modeling" + ), + "path": ( + "https://zenodo.org/record/" + f"{deposit_id_data_bundle}#.Y_dWM4CZMVM" + ), + "licenses": [license_dedl(attribution="© Cußmann, Ilka")], + }, + { + "title": ("open-MaStR power unit registry for eGo^n project"), + "description": ( + "Data from Marktstammdatenregister (MaStR) data using " + "the data dump from 2022-11-17 for eGon-data." + ), + "path": ( + f"https://zenodo.org/record/{deposit_id_mastr}" + ), + "licenses": [license_dedl(attribution="© Amme, Jonathan")], + }, + sources()["openstreetmap"], + sources()["era5"], + sources()["vg250"], + sources()["egon-data"], + ], + "licenses": [license_odbl("© eGon development team")], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": name, + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + schema, + table, + ), + "primaryKey": "index", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github." + "com/OpenEnergyPlatform/oemetadata/blob/master/metadata/" + "v141/metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time " + "zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, " + "en-US, de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata()["metadataVersion"])() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + schema, + table, + ) + + def create_scenario_table(buildings_gdf): """Create mapping table pv_unit <-> building for scenario""" EgonPowerPlantPvRoofBuilding.__table__.drop(bind=engine, checkfirst=True) @@ -2095,6 +2240,8 @@ def create_scenario_table(buildings_gdf): index=False, ) + add_metadata() + def add_weather_cell_id(buildings_gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame: sql = """ diff --git a/src/egon/data/datasets/saltcavern/__init__.py b/src/egon/data/datasets/saltcavern/__init__.py index ac1f040cc..848495a87 100644 --- a/src/egon/data/datasets/saltcavern/__init__.py +++ b/src/egon/data/datasets/saltcavern/__init__.py @@ -89,6 +89,17 @@ def to_postgres(): class SaltcavernData(Dataset): + """Inserts Saltcavern shapes into database + + *Dependencies* + * :py:class:`DataBundle ` + * :py:class:`Vg250 ` + + *Resulting tables* + * :py:class:`EgonPfHvGasVoronoi ` + + """ + def __init__(self, dependencies): super().__init__( name="SaltcavernData", diff --git a/src/egon/data/datasets/scenario_capacities.py b/src/egon/data/datasets/scenario_capacities.py index 835e54358..b8b6470ce 100755 --- a/src/egon/data/datasets/scenario_capacities.py +++ b/src/egon/data/datasets/scenario_capacities.py @@ -788,7 +788,6 @@ def eGon100_capacities(): ) - def add_metadata(): """Add metdata to supply.egon_scenario_capacities @@ -825,7 +824,9 @@ def add_metadata(): "name": "supply.egon_scenario_capacities", "title": "eGon scenario capacities", "id": "WILL_BE_SET_AT_PUBLICATION", - "description": "Installed capacities of scenarios used in the eGon project", + "description": ( + "Installed capacities of scenarios used in the eGon project" + ), "language": ["de-DE"], "publicationDate": datetime.date.today().isoformat(), "context": context(), diff --git a/src/egon/data/datasets/storages/home_batteries.py b/src/egon/data/datasets/storages/home_batteries.py index 40d19c1ea..fbd631bd8 100644 --- a/src/egon/data/datasets/storages/home_batteries.py +++ b/src/egon/data/datasets/storages/home_batteries.py @@ -32,14 +32,28 @@ The selection of buildings is done randomly until a result is reached which is close to achieving the sizing specification. """ +import datetime +import json + from loguru import logger from numpy.random import RandomState +from omi.dialects import get_dialect from sqlalchemy import Column, Float, Integer, String from sqlalchemy.ext.declarative import declarative_base import numpy as np import pandas as pd from egon.data import config, db +from egon.data.metadata import ( + context, + contributors, + generate_resource_fields_from_db_table, + license_dedl, + license_odbl, + meta_metadata, + meta_metadata, + sources, +) Base = declarative_base() @@ -173,6 +187,8 @@ def allocate_home_batteries_to_buildings(): create_table(pd.concat(df_list, ignore_index=True)) + add_metadata() + class EgonHomeBatteries(Base): targets = config.datasets()["home_batteries"]["targets"] @@ -188,6 +204,142 @@ class EgonHomeBatteries(Base): capacity = Column(Float) +def add_metadata(): + """ + Add metadata to table supply.egon_home_batteries + """ + targets = config.datasets()["home_batteries"]["targets"] + deposit_id_mastr = config.datasets()["mastr_new"]["deposit_id"] + deposit_id_data_bundle = config.datasets()["data-bundle"]["sources"][ + "zenodo" + ]["deposit_id"] + + contris = contributors(["kh", "kh"]) + + contris[0]["date"] = "2023-03-15" + + contris[0]["object"] = "metadata" + contris[1]["object"] = "dataset" + + contris[0]["comment"] = "Add metadata to dataset." + contris[1]["comment"] = "Add workflow to generate dataset." + + meta = { + "name": ( + f"{targets['home_batteries']['schema']}." + f"{targets['home_batteries']['table']}" + ), + "title": "eGon Home Batteries", + "id": "WILL_BE_SET_AT_PUBLICATION", + "description": "Home storage systems allocated to buildings", + "language": "en-US", + "keywords": ["battery", "batteries", "home", "storage", "building"], + "publicationDate": datetime.date.today().isoformat(), + "context": context(), + "spatial": { + "location": "none", + "extent": "Germany", + "resolution": "building", + }, + "temporal": { + "referenceDate": "2021-12-31", + "timeseries": {}, + }, + "sources": [ + { + "title": "Data bundle for egon-data", + "description": ( + "Data bundle for egon-data: A transparent and " + "reproducible data processing pipeline for energy " + "system modeling" + ), + "path": ( + "https://zenodo.org/record/" + f"{deposit_id_data_bundle}#.Y_dWM4CZMVM" + ), + "licenses": [license_dedl(attribution="© Cußmann, Ilka")], + }, + { + "title": ("open-MaStR power unit registry for eGo^n project"), + "description": ( + "Data from Marktstammdatenregister (MaStR) data using " + "the data dump from 2022-11-17 for eGon-data." + ), + "path": ( + f"https://zenodo.org/record/{deposit_id_mastr}" + ), + "licenses": [license_dedl(attribution="© Amme, Jonathan")], + }, + sources()["openstreetmap"], + sources()["era5"], + sources()["vg250"], + sources()["egon-data"], + sources()["nep2021"], + sources()["mastr"], + sources()["technology-data"], + ], + "licenses": [license_odbl("© eGon development team")], + "contributors": contris, + "resources": [ + { + "profile": "tabular-data-resource", + "name": ( + f"{targets['home_batteries']['schema']}." + f"{targets['home_batteries']['table']}" + ), + "path": "None", + "format": "PostgreSQL", + "encoding": "UTF-8", + "schema": { + "fields": generate_resource_fields_from_db_table( + targets["home_batteries"]["schema"], + targets["home_batteries"]["table"], + ), + "primaryKey": "index", + }, + "dialect": {"delimiter": "", "decimalSeparator": ""}, + } + ], + "review": {"path": "", "badge": ""}, + "metaMetadata": meta_metadata(), + "_comment": { + "metadata": ( + "Metadata documentation and explanation (https://github.com/" + "OpenEnergyPlatform/oemetadata/blob/master/metadata/v141/" + "metadata_key_description.md)" + ), + "dates": ( + "Dates and time must follow the ISO8601 including time zone " + "(YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)" + ), + "units": "Use a space between numbers and units (100 m)", + "languages": ( + "Languages must follow the IETF (BCP47) format (en-GB, en-US, " + "de-DE)" + ), + "licenses": ( + "License name must follow the SPDX License List " + "(https://spdx.org/licenses/)" + ), + "review": ( + "Following the OEP Data Review (https://github.com/" + "OpenEnergyPlatform/data-preprocessing/wiki)" + ), + "none": "If not applicable use (none)", + }, + } + + dialect = get_dialect(meta_metadata())() + + meta = dialect.compile_and_render(dialect.parse(json.dumps(meta))) + + db.submit_comment( + f"'{json.dumps(meta)}'", + targets["home_batteries"]["schema"], + targets["home_batteries"]["table"], + ) + + def create_table(df): """Create mapping table home battery <-> building id""" engine = db.engine() diff --git a/src/egon/data/metadata/__init__.py b/src/egon/data/metadata/__init__.py index dff660905..25e267217 100755 --- a/src/egon/data/metadata/__init__.py +++ b/src/egon/data/metadata/__init__.py @@ -8,6 +8,8 @@ from egon.data.datasets import Dataset from egon.data.db import engine +EGON_ATTRIBUTION: str = "© eGon development team" + def context(): """ @@ -55,7 +57,7 @@ def meta_metadata(): } -def licenses_datenlizenz_deutschland(attribution): +def licenses_datenlizenz_deutschland(attribution=EGON_ATTRIBUTION): """ License information for Datenlizenz Deutschland @@ -103,7 +105,7 @@ def licenses_datenlizenz_deutschland(attribution): } -def license_odbl(attribution): +def license_odbl(attribution=EGON_ATTRIBUTION): """ License information for Open Data Commons Open Database License (ODbL-1.0) @@ -128,7 +130,7 @@ def license_odbl(attribution): } -def license_ccby(attribution): +def license_ccby(attribution=EGON_ATTRIBUTION): """ License information for Creative Commons Attribution 4.0 International (CC-BY-4.0) @@ -153,7 +155,7 @@ def license_ccby(attribution): } -def license_geonutzv(attribution): +def license_geonutzv(attribution=EGON_ATTRIBUTION): """ License information for GeoNutzV @@ -184,7 +186,7 @@ def license_geonutzv(attribution): } -def license_agpl(attribution): +def license_agpl(attribution=EGON_ATTRIBUTION): """ License information for GNU Affero General Public License v3.0 @@ -214,7 +216,7 @@ def license_agpl(attribution): } -def license_dedl(attribution): +def license_dedl(attribution=EGON_ATTRIBUTION): """ License information for Data licence Germany – attribution – version 2.0 @@ -234,7 +236,7 @@ def license_dedl(attribution): "path": "https://www.govdata.de/dl-de/by-2-0", "instruction": ( "Any use will be permitted provided it fulfils the requirements of" - " this 'Data licence Germany – attribution – Version 2.0'. The " + ' this "Data licence Germany – attribution – Version 2.0". The ' "data and meta-data provided may, for commercial and " "non-commercial use, in particular be copied, printed, presented, " "altered, processed and transmitted to third parties; be merged " @@ -243,8 +245,8 @@ def license_dedl(attribution): "external business processes, products and applications in public " "and non-public electronic networks. The user must ensure that the" " source note contains the following information: the name of the " - "provider, the annotation 'Data licence Germany – attribution – " - "Version 2.0' or 'dl-de/by-2-0' referring to the licence text " + 'provider, the annotation "Data licence Germany – attribution – ' + 'Version 2.0" or "dl-de/by-2-0" referring to the licence text ' "available at www.govdata.de/dl-de/by-2-0, and a reference to the " "dataset (URI). This applies only if the entity keeping the data " "provides the pieces of information 1-3 for the source note. " @@ -413,7 +415,7 @@ def sources(): " developed requirements and criteria for the" " assessment of suitable sites even if their" " exploration is still at an early stage and there is" - " little knowledge of the salinaries' structures." + ' little knowledge of the salinaries structures.' " Scientists at DEEP.KBB GmbH in Hanover, worked" " together with their project partners at the Federal" " Institute for Geosciences and Natural Resources and" @@ -499,7 +501,7 @@ def sources(): " project eGon (https://ego-n.org/)" ), "path": "https://github.com/openego/eGon-data", - "licenses": [license_agpl("© eGon development team")], + "licenses": [license_agpl(EGON_ATTRIBUTION)], }, "egon-data_bundle": { "title": "Data bundle for egon-data", @@ -507,7 +509,7 @@ def sources(): "Zenodo repository to provide several different input" " data sets for eGon-data" ), - "path": "https://sandbox.zenodo.org/record/1167119", + "path": "https://zenodo.org/record/10226009", "licenses": [license_ccby("© eGon development team")], }, "Einspeiseatlas": { @@ -638,7 +640,7 @@ def sources(): " https://www.marktstammdatenregister.de/MaStRHilfe" "/subpages/statistik.html" ), - "path": "https://sandbox.zenodo.org/record/808086", + "path": "https://zenodo.org/record/10480930", "licenses": [ licenses_datenlizenz_deutschland( "© 2021 Bundesnetzagentur für Elektrizität, Gas," @@ -689,7 +691,7 @@ def sources(): "title": "OpenStreetMap Data Extracts (Geofabrik)", "description": ( "Full data extract of OpenStreetMap data for defined" - " spatial extent at ''referenceDate''" + ' spatial extent at "referenceDate"' ), "path": ( "https://download.geofabrik.de/europe/germany-210101.osm.pbf"