Raster Time Series Integration #23

SoerenHoffstedt · 2019-04-03T08:45:13Z

I integrated the raster time series (RTS) processing into mapping without changing the old processing of rasters. This leads to some restrictions detailed below that do not apply when the old raster data type would be replaced by RTS.

This PR contains the following:

RasterTimeSeries data type and operator result
RasterTimeSeriesOperator as an abstract base class for operators that can create RTS
The RTS creates Descriptor objects
OperatorState as the base class for storing the state variables of a RTS (see below)
RasterTimeSeriesSource operator with a GDAL Source backend
RTS expression operator added (separate from the old expression operator because I ported it from my bachelor thesis for testing)
RTS support added to Projection operator
Integration of RTS as result types into GenericOperator, QueryProcessor, etc.
Integration into WCS/WMS services and mapping manager queries

Raster Time Series

Because the tiles of a RTS are processed successively by calling the nextDescriptor method, operators have to keep state variables that are necessary for their work (the easiest example is the source operator that has to track the spatial and temporal state of the last returned tile). In my thesis, these state variables were simply member variables of the operators. But in mapping the result for a query rectangle is requested when calling getRasterTimeSeries and that could be done multiple times for different queries. Therefore, having the state variables in the operator is not suitable.

An operator creates a RasterTimeSeries object that is used to create the tile descriptors and iterate over them. The RasterTimeSeries class contains a pointer to the operator that created it and an OperatorState object. In its nextDescriptor method it passes the state to the nextDescriptor method of the operator. Additionally, RasterTimeSeries has the method getAsRaster that puts the descriptors of the next raster together into one raster. Using this method allows easy integration into the current services but it makes, for example, more sense for the WCS service to write an exporter that writes the tile data directly into the GDAL data structure.

WCS/WMS Service Integration

Service must know if a RTS or a raster is requested. The info is not provided in the layers/coverageid strings to the WCS/WMS services because it was not needed until now. When the query is created, it is assumed that a raster is requested.
Therefore, I check if a RTS is requested by looking for a URL parameter to the service: result_type with the content raster_time_series. For normal raster requests, no parameter must be provided.

https://github.com/SoerenHoffstedt/mapping-core/blob/8cdf1209594991e4fff09f08299158c76ca69256/src/services/wcs.cpp#L138-L149

Tile Processing Order

One challenge is the handling of the tile processing order. In the future it might be useful to determine the processing order algorithmically based on the operator tree. For now I assume that the request to mapping contains information about the processing order.

The solution for now is that the last operator assumes that the processing order is provided in the parameters. Therefore, the GenericOperator class has two getRasterTimeSeries methods: one takes the additional processing order argument, the other one does not. The abstract RasterTimeSeriesOperator class implements the version without the processing order argument, reads it from the parameters, and calls the other version. The actual RTS operators only have to implement the version with the processing order argument. If only one version would exist, each operator would have to check if it has to read the processing order from the parameters or not.

As we discussed, this is not an optimal solution because the parameter does not change the result of the operator but only the technicality of how it is processing.
Putting the processing order into QueryTools is not possible the QueryTools are passed as const references and thus an operator could not change the processing order of an input operator. Another possibility is adding the processing order to the query rectangle as an optional value.

Query Processor

The query processor is an intermediate step in processing a query that wraps the result and abstracts where and how the result is processed. The problem for a RTS is that it is a data structure for creating the actual results. Because it is processing the RTS successively, the result can not and should not contain all the finished data.
For the local query processor backend (it is the only one available right now) it does not make a difference but if, for example, a distributed backend would be added it would not work RTS.

As a possible solution, the query processor could act as an operator, returning not the descriptor of the last operator of the query but a descriptor that is wrapping the result and is created by the query processor. The query processor can load/process the tile data from the input descriptor and move the result into the closure of the new descriptor. Thus, the query processor could decide where to process the tiles. This could also incorporate moving futures to the tile data into the closure if needed.
All this is not implemented or tested at the moment and would require to analyze the thread safety of the operators and getRaster closures.

SoerenHoffstedt · 2019-04-03T09:26:40Z

Travis CI seems to fail because it uses g++ 4.8 as the compiler but that version does not fully support C++17 and does not have the std::optional header.

michaelmattig · 2019-04-08T09:32:07Z

src/datatypes/descriptor.cpp

+          nodata(desc->nodata),
+          _isOnlyNodata(desc->_isOnlyNodata),
+          dataType(desc->dataType),
+          op(newOperator) //assuming this copies an input descriptor, the operator pointer should not be copied.


Why is this a raw pointer and what do you mean by descriptor?

michaelmattig · 2019-04-08T09:42:31Z

src/datatypes/descriptor.h

+
+    /**
+    * The order of returning the tile descriptors by an operator.
+    * @Spatial Operator returns one tile for raster, then advances to the next tile and starts with the first raster again.


michaelmattig · 2019-04-08T09:49:36Z

src/datatypes/raster_time_series.h

+        TileIterator end();
+
+        /**
+         * For temporal order this skips the tiles of the current raster that are not yet processed, advancing time


michaelmattig · 2019-04-08T09:49:45Z

src/datatypes/raster_time_series.h

+        void skipCurrentRaster(const uint32_t skipCount = 1);
+
+        /**
+         * For spatial order this skips the other rasters not yet processed for the current tile. For the next tile they will appear again.


michaelmattig · 2019-04-08T09:53:02Z

src/raster_time_series/operator_state/operator_state.h

+     * A RasterTimeSeries object contains of an operator state and a pointer to the operator.
+     * The operator creates tile descriptors based on the operator state it gets passed by the RasterTimeSeries object.
+     *
+     * The default class contains a vector of input RasterTimeSeries and methods for skipping the current raster or tile


why multiple input rts?

michaelmattig · 2019-04-08T09:58:13Z

src/operators/raster_time_series/raster_time_series_source.cpp

+    //skipCurrentRaster/Tile is called it will also be set already.
+    //else it must be increased here and for next tile increaseDimensions will be set true.
+    if(s.increaseDimensions){
+        if(order == ProcessingOrder::Temporal){


is this intuitive?

michaelmattig · 2019-04-08T10:00:33Z

src/raster_time_series/operator_state/source_state.h

+
+        /**
+         * Increases the time to the next rasters starting time in the time series.
+         * @return true when increasing the time to the next raster went over the end of the time series.


more intuitive other way around?

michaelmattig · 2019-04-08T11:23:47Z

src/raster_time_series/gdal_source_backend.cpp

+
+double GDALSourceBackend::getCurrentTimeEnd(double currTime) const {
+    ptime currPTime = from_time_t((time_t)currTime);
+    timeStep.increase(currPTime);


why increase?

michaelmattig · 2019-04-08T11:23:54Z

src/raster_time_series/gdal_source_backend.cpp

+    return o == ProcessingOrder::Temporal || o == ProcessingOrder::Spatial;
+}
+
+void GDALSourceBackend::increaseCurrentTime(double &currTime) {


by how much?

jdroenner · 2019-04-08T11:24:08Z

src/datatypes/descriptor.h

+     * negative values can be used in the context of tiles that cover space outside of the actual raster
+     * because of the fixed tile alignment at the projections origin.
+     */
+    class Resolution {


jdroenner · 2019-04-08T11:27:56Z

src/datatypes/descriptor.h

+    * This allows only copying the metadata of a descriptor because the getRaster closure stored in the
+    * descriptor class can not be copied.
+    */
+    class DescriptorInfo {


A descriptor could also contain the offset of a tile to the origin and optional min max mean values

michaelmattig · 2019-04-08T11:28:08Z

src/operators/raster_time_series/raster_time_series_source.cpp

+void RasterTimeSeriesSource::getProvenance(ProvenanceCollection &pc) {
+    //TODO: this is a point of friction because the provenance data must come from the backend but i don't have
+    // access to the backend here (it is in the OperatorState).
+    // Solution idea: make backend statically available by the backend name?


but how to get the correct source from the backend?

SoerenHoffstedt added 5 commits April 1, 2019 17:27

Basic integration of raster time series processing.

c5d1d7e

Raster Time Series: GDAL source backend.

2de7138

Raster Time Series: Expression operator added.

3cb6f08

Raster Time Series: Projection operator added.

8cdf120

Header for std::optional added

a84adc2

michaelmattig reviewed Apr 8, 2019

View reviewed changes

jdroenner reviewed Apr 8, 2019

View reviewed changes

michaelmattig reviewed Apr 8, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raster Time Series Integration #23

Raster Time Series Integration #23

SoerenHoffstedt commented Apr 3, 2019

SoerenHoffstedt commented Apr 3, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

michaelmattig Apr 8, 2019

jdroenner Apr 8, 2019

jdroenner Apr 8, 2019

michaelmattig Apr 8, 2019

Raster Time Series Integration #23

Are you sure you want to change the base?

Raster Time Series Integration #23

Conversation

SoerenHoffstedt commented Apr 3, 2019

Raster Time Series

WCS/WMS Service Integration

Tile Processing Order

Query Processor

SoerenHoffstedt commented Apr 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment