[Бенчмаркинг PaddlePaddle] Реализация пайплайна для PaddlePaddle

README.md

@@ -30,6 +30,7 @@ DLI supports inference using the following frameworks:
 - [Spektral][spektral] (Python API).
 - [RKNN][rknn] (C++ API).
 - [ncnn][ncnn] (Python API).
+- [PaddlePaddle][PaddlePaddle] (Python API).
 
 More information about DLI is available on the web-site
 ([here][dli-ru-web-page] (in Russian)
@@ -95,6 +96,7 @@ p Learning Inference Benchmark //
   - `TVM` is a directory of Dockerfiles for Apache TVM.
   - `TensorFlow` is a directory of Dockerfiles for Intel® Optimizations
     for TensorFlow.
+  - `PaddlePaddle` is a directory of Dockerfiles for PaddlePaddle.
 
 - `docs` directory contains auxiliary documentation. Please, find
   complete documentation at the [Wiki page][dli-wiki].
@@ -262,6 +264,7 @@ Report questions, issues and suggestions, using:
 [spektral]: https://graphneural.network
 [rknn]: https://github.com/rockchip-linux/rknn-toolkit2
 [ncnn]: https://github.com/Tencent/ncnn
+[PaddlePaddle]: https://www.paddlepaddle.org.cn/en
 [benchmark-app]: https://github.com/openvinotoolkit/openvino/tree/master/samples/cpp/benchmark_app
 [dli-ru-web-page]: http://hpc-education.unn.ru/dli-ru
 [dli-web-page]: http://hpc-education.unn.ru/dli

docker/PaddlePaddle/Dockerfile

@@ -0,0 +1,27 @@
+FROM ubuntu_for_dli
+
+WORKDIR /root/
+ARG PADDLE_VERSION=2.5.2
+RUN pip3 install --upgrade pip && \
+    pip3 install opencv-python requests PyYAML docker
+RUN export LD_LIBRARY_PATH=/root/miniconda3/lib:${LD_LIBRARY_PATH}
+
+WORKDIR /tmp/
+RUN mkdir resnet50_paddle
+WORKDIR /tmp/resnet50_paddle/
+RUN wget -q https://github.com/itlab-vision/itlab-vision-dl-benchmark-models/tree/main/paddlepaddle_models/resnet50/resnet50/inference.pdmodel
+RUN wget -q https://github.com/itlab-vision/itlab-vision-dl-benchmark-models/tree/main/paddlepaddle_models/resnet50/resnet50/inference.pdiparams
+WORKDIR /tmp/
+RUN wget https://paddle-inference-dist.bj.bcebos.com/Paddle-Inference-Demo/resnet50.tgz
+RUN tar xzf resnet50.tgz
+RUN python3 -m pip install paddlepaddle==${PADDLE_VERSION} -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+WORKDIR /tmp/open_model_zoo/tools/accuracy_checker
+COPY config_tf2.yml config_tf2.yml
+
+RUN python3 setup.py install_core
+
+RUN accuracy_check -c config_tf2.yml -m ../../../resnet50 -s sample
+
+WORKDIR /tmp/
+

docker/PaddlePaddle/config_tf2.yml

@@ -0,0 +1,30 @@
+models:
+  - name: resnet50
+    launchers:
+      - framework: paddle_paddle
+        device: cpu
+        model: inference.pdmodel
+        params: inference.pdiparams
+        adapter: classification
+
+    datasets:
+      - name: sample_dataset
+        data_source: sample_dataset/test
+        annotation_conversion:
+          converter: cifar
+          data_batch_file: cifar-10-batches-py/test_batch
+          convert_images: True
+          converted_images_dir: sample_dataset/test
+          num_classes: 10
+
+        preprocessing:
+          - type: resize
+            size: 224
+          - type: bgr_to_rgb
+          - type: normalization
+            mean: (127.5, 127.5, 127.5)
+            std: (127.5, 127.5, 127.5)
+
+        metrics:
+          - type: accuracy
+            top_k: 1

requirements_frameworks.txt

@@ -15,5 +15,10 @@ dgl==2.0.0
 --find-links https://data.dgl.ai/wheels-test/repo.html
 dglgo==0.0.2
 
+tflite
+
+paddlepaddle==2.6.0
+--extra-index-url https://mirror.baidu.com/pypi/simple
+
 ncnn
 spektral==1.3.0

results/validation/validation_results_paddlepaddle.md

@@ -0,0 +1,54 @@
+# Validation results for the models inferring using PaddlePaddle
+
+## Image classification
+
+### Test image #1
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 709 x 510
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00000023.JPEG"></img>
+</div>
+
+Model | Parameters | Python API |
+-|-|-|
+resnet-50 |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.9931559 Granny Smith<br>0.0009120 piggy bank, penny bank<br>0.0007721 bell pepper<br>0.0007689 tennis ball<br>0.0005548 candle, taper, wax light|
+PPLCNet_x1_0_infer |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.2785943 Granny Smith<br>0.2241544 piggy bank, penny bank<br>0.0404602 saltshaker, salt shaker<br>0.0131707 soap dispenser<br>0.0114298 lemon
+|
+
+### Test image #2
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 500 x 500
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00000247.JPEG">
+</div>
+
+Model | Parameters | Python API |
+-|-|-|
+resnet-50 |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.9891654 junco, snowbird<br>0.0044086 chickadee<br>0.0033522 water ouzel, dipper<br>0.0014910 brambling, Fringilla montifringilla<br>0.0003624 indigo bunting, indigo finch, indigo bird, Passerina cyanea|
+PPLCNet_x1_0_infer |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.8259031 junco, snowbird<br>0.0340593 brambling, Fringilla montifringilla<br>0.0055266 chickadee<br>0.0050722 house finch, linnet, Carpodacus mexicanus<br>0.0034595 bulbul|
+
+### Test image #3
+
+Data source: [ImageNet][imagenet]
+
+Image resolution: 333 x 500
+
+<div style='float: center'>
+<img width="150" src="images\ILSVRC2012_val_00018592.JPEG">
+</div>
+
+Model | Parameters | Python API |
+-|-|-|
+resnet-50 |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.3656897 liner, ocean liner<br>0.1008371 container ship, containership, container vessel<br>0.0759774 dock, dockage, docking facility<br>0.0707850 lifeboat<br>0.0556011 breakwater, groin, groyne, mole, bulwark, seawall, jetty|
+PPLCNet_x1_0_infer |Mean values - [123.675,116.28,103.53],<br>scale value - [58.395,57.12,57.375]|0.1249109 submarine, pigboat, sub, U-boat<br>0.1198353 breakwater, groin, groyne, mole, bulwark, seawall, jetty <br>0.0568103 liner, ocean liner<br>0.0351734 lifeboat<br>0.0326451 dock, dockage, docking facility|
+
+<!-- LINKS -->
+[imagenet]: http://www.image-net.org
+[ms_coco]: http://cocodataset.org
+[PASCAL_VOC_2012]: http://host.robots.ox.ac.uk/pascal/VOC/voc2012

src/benchmark/README.md

@@ -21,6 +21,7 @@ the following frameworks:
 - [Apache TVM][tvm] (Python API).
 - [RKNN][rknn].
 - [Spektral][spektral] (Python API).
+- [PaddlePaddle][paddlepaddle] (PythonAPI).
 
 ### Implemented algorithm
 
@@ -115,7 +116,7 @@ For **the throughput mode**, performance metrics are provided below.
 **Note:** published tables contain only FPS indicators (until 06.2023
 the table contained Batch FPS indicators).
 
-### Metrics for Intel® Optimization for Caffe, Intel® Optimizations for TensorFlow, TensorFlow Lite, OpenCV, MXNet, PyTorch and ONNX Runtime
+### Metrics for Intel® Optimization for Caffe, Intel® Optimizations for TensorFlow, TensorFlow Lite, OpenCV, MXNet, PyTorch, PaddlePaddle and ONNX Runtime
 
 Inference performance evaluations for Intel® Optimization for Caffe,
 Intel® Optimizations for TensorFlow, TensorFlow Lite, OpenCV, MXNet,
@@ -272,3 +273,4 @@ pip install openvino_dev[mxnet,caffe,caffe2,onnx,pytorch,tensorflow2]==<your ver
 [tvm]: https://tvm.apache.org
 [rknn]: https://github.com/rockchip-linux/rknpu2
 [spektral]: https://graphneural.network
+[paddlepaddle]: https://www.paddlepaddle.org.cn/en

src/benchmark/config_parser_factory.py

@@ -2,6 +2,7 @@
 from frameworks.known_frameworks import KnownFrameworks
 from frameworks.openvino.openvino_parameters_parser import OpenVINOParametersParser
 from frameworks.tensorflow.tensorflow_parameters_parser import TensorFlowParametersParser
+from frameworks.paddlepaddle.paddlepaddle_parameters_parser import PaddlePaddleParametersParser
 from frameworks.tensorflow_lite.tensorflow_lite_parameters_parser import TensorFlowLiteParametersParser
 from frameworks.mxnet.mxnet_parameters_parser import MXNetParametersParser
 from frameworks.opencv_dnn_python.opencv_dnn_python_parameters_parser import OpenCVDNNPythonParametersParser
@@ -19,6 +20,8 @@ def get_parameters_parser(framework):
         return IntelCaffeParametersParser()
     if framework == KnownFrameworks.tensorflow:
         return TensorFlowParametersParser()
+    if framework == KnownFrameworks.paddlepaddle:
+        return PaddlePaddleParametersParser()
     if framework == KnownFrameworks.openvino_dldt:
         return OpenVINOParametersParser()
     if framework == KnownFrameworks.onnx_runtime:

src/benchmark/frameworks/framework_wrapper_registry.py

@@ -7,6 +7,7 @@
 from .onnx_runtime.onnx_runtime_wrapper import OnnxRuntimeWrapper
 from .onnx_runtime_python.onnx_runtime_python_wrapper import ONNXRuntimePythonWrapper
 from .tensorflow_lite.tensorflow_lite_wrapper import TensorFlowLiteWrapper
+from .paddlepaddle.paddlepaddle_wrapper import PaddlePaddleWrapper
 from .tensorflow_lite_cpp.tensorflow_lite_cpp_wrapper import TensorFlowLiteCppWrapper
 from .opencv_dnn_python.opencv_dnn_python_wrapper import OpenCVDNNPythonWrapper
 from .mxnet.mxnet_wrapper import MXNetWrapper
@@ -42,6 +43,7 @@ def _get_wrappers(self):
         self._framework_wrappers[IntelCaffeWrapper.framework_name] = IntelCaffeWrapper()
         self._framework_wrappers[TensorFlowWrapper.framework_name] = TensorFlowWrapper()
         self._framework_wrappers[OpenVINOWrapper.framework_name] = OpenVINOWrapper()
+        self._framework_wrappers[PaddlePaddleWrapper.framework_name] = PaddlePaddleWrapper()
         self._framework_wrappers[OnnxRuntimeWrapper.framework_name] = OnnxRuntimeWrapper()
         self._framework_wrappers[ONNXRuntimePythonWrapper.framework_name] = ONNXRuntimePythonWrapper()
         self._framework_wrappers[TensorFlowLiteWrapper.framework_name] = TensorFlowLiteWrapper()

src/benchmark/frameworks/known_frameworks.py

@@ -1,4 +1,5 @@
 class KnownFrameworks:
+    paddlepaddle = 'PaddlePaddle'
     openvino_dldt = 'OpenVINO DLDT'
     tensorflow = 'TensorFlow'
     caffe = 'Caffe'

src/benchmark/frameworks/paddlepaddle/paddlepaddle_parameters_parser.py

@@ -0,0 +1,81 @@
+from ..config_parser.dependent_parameters_parser import DependentParametersParser
+from ..config_parser.framework_parameters_parser import FrameworkParameters
+
+
+class PaddlePaddleParametersParser(DependentParametersParser):
+    def parse_parameters(self, curr_test):
+        CONFIG_FRAMEWORK_DEPENDENT_TAG = 'FrameworkDependent'
+        CONFIG_FRAMEWORK_DEPENDENT_CHANNEL_SWAP_TAG = 'ChannelSwap'
+        CONFIG_FRAMEWORK_DEPENDENT_MEAN_TAG = 'Mean'
+        CONFIG_FRAMEWORK_DEPENDENT_INPUT_SCALE_TAG = 'InputScale'
+        CONFIG_FRAMEWORK_DEPENDENT_INPUT_SHAPE_TAG = 'InputShape'
+        CONFIG_FRAMEWORK_DEPENDENT_INPUT_NAMES_TAG = 'InputName'
+        CONFIG_FRAMEWORK_DEPENDENT_OUTPUT_NAMES_TAG = 'OutputNames'
+        CONFIG_FRAMEWORK_DEPENDENT_THREAD_COUNT_TAG = 'ThreadCount'
+
+        dep_parameters_tag = curr_test.getElementsByTagName(CONFIG_FRAMEWORK_DEPENDENT_TAG)[0]
+
+        _channel_swap = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_CHANNEL_SWAP_TAG)[0].firstChild
+        _mean = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_MEAN_TAG)[0].firstChild
+        _input_scale = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_INPUT_SCALE_TAG)[0].firstChild
+        _input_shape = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_INPUT_SHAPE_TAG)[0].firstChild
+        _input_names = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_INPUT_NAMES_TAG)[0].firstChild
+        _output_names = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_OUTPUT_NAMES_TAG)[0].firstChild
+        _thread_count = dep_parameters_tag.getElementsByTagName(
+            CONFIG_FRAMEWORK_DEPENDENT_THREAD_COUNT_TAG)[0].firstChild
+
+        return PaddlePaddleParameters(
+            channel_swap=_channel_swap.data if _channel_swap else None,
+            mean=_mean.data if _mean else None,
+            input_scale=_input_scale.data if _input_scale else None,
+            input_shapes=_input_shape.data if _input_shape else None,
+            input_name=_input_names.data if _input_names else None,
+            output_names=_output_names.data if _output_names else None,
+            thread_count=_thread_count.data if _thread_count else None,
+        )
+
+
+class PaddlePaddleParameters(FrameworkParameters):
+    def __init__(self, channel_swap, mean, input_scale, input_shapes, output_names, input_name,
+                 thread_count):
+        self.channel_swap = None
+        self.mean = None
+        self.input_scale = None
+        self.input_shape = None
+        self.input_name = None
+        self.output_names = None
+        self.nthreads = None
+        self.delegate = None
+        self.delegate_options = None
+
+        if self._parameter_is_not_none(channel_swap):
+            self.channel_swap = self._process_sequence_arg(channel_swap)
+        if self._parameter_is_not_none(mean):
+            self.mean = self._process_sequence_arg(mean)
+        if self._parameter_is_not_none(input_scale):
+            self.input_scale = self._process_sequence_arg(input_scale)
+        if self._parameter_is_not_none(input_shapes):
+            self.input_shape = self._process_sequence_arg(input_shapes)
+        if self._parameter_is_not_none(input_name):
+            self.input_name = input_name
+        if self._parameter_is_not_none(output_names):
+            self.output_names = output_names
+        if self._parameter_is_not_none(thread_count):
+            if self._int_value_is_correct(thread_count):
+                self.nthreads = thread_count
+            else:
+                raise ValueError('Threads count can only take integer value')
+
+    @staticmethod
+    def _process_sequence_arg(sequence):
+        if '[' not in sequence:
+            sequence = f'[{sequence}]'
+        if ',' not in sequence:
+            sequence = sequence.replace(' ', ',')
+        return sequence

src/benchmark/frameworks/paddlepaddle/paddlepaddle_process.py

@@ -0,0 +1,63 @@
+from pathlib import Path
+
+from ..processes import ProcessHandler
+
+
+class PaddlePaddleProcess(ProcessHandler):
+    benchmark_app_name = 'paddlepaddle_benchmark'
+    launcher_latency_units = 'seconds'
+
+    def __init__(self, test, executor, log):
+        super().__init__(test, executor, log)
+
+    @staticmethod
+    def create_process(test, executor, log):
+        return PaddlePaddleProcess(test, executor, log)
+
+    def get_performance_metrics(self):
+        return self.get_performance_metrics_from_json_report()
+
+    def _fill_command_line(self):
+        path_to_paddlepaddle_script = Path.joinpath(self.inference_script_root, 'inference_paddlepaddle.py')
+        python = ProcessHandler.get_cmd_python_version()
+
+        model = self._test.model.model
+        params = self._test.model.weight
+        dataset = self._test.dataset.path if self._test.dataset else None
+        batch = self._test.indep_parameters.batch_size
+        device = self._test.indep_parameters.device
+
+        common_params = (f'-m {model} -p {params} -i {dataset} -b {batch} -d {device} '
+                         f'--report_path {self.report_path}')
+
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '-i', dataset)
+
+        time_limit = self._test.indep_parameters.test_time_limit
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--time', time_limit)
+
+        channel_swap = self._test.dep_parameters.channel_swap
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--channel_swap', channel_swap)
+
+        mean = self._test.dep_parameters.mean
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--mean', mean)
+
+        input_scale = self._test.dep_parameters.input_scale
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--input_scale', input_scale)
+
+        input_shape = self._test.dep_parameters.input_shape
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--input_shapes', input_shape)
+
+        input_name = self._test.dep_parameters.input_name
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--input_names', input_name)
+
+        output_names = self._test.dep_parameters.output_names
+        common_params = self._add_optional_argument_to_cmd_line(common_params, '--output_names', output_names)
+
+        common_params = self._add_argument_to_cmd_line(common_params, '--raw_output', 'true')
+
+        command_line = f'{python} {path_to_paddlepaddle_script} {common_params}'
+
+        nthreads = self._test.dep_parameters.nthreads
+        command_line = self._add_optional_argument_to_cmd_line(command_line, '-nthreads', nthreads)
+
+        return command_line

src/benchmark/frameworks/paddlepaddle/paddlepaddle_wrapper.py

@@ -0,0 +1,15 @@
+from .paddlepaddle_process import PaddlePaddleProcess
+from ..config_parser.test_reporter import Test
+from ..framework_wrapper import FrameworkWrapper
+
+
+class PaddlePaddleWrapper(FrameworkWrapper):
+    framework_name = 'PaddlePaddle'
+
+    @staticmethod
+    def create_process(test, executor, log, cpp_benchmark_path=None, **kwargs):
+        return PaddlePaddleProcess.create_process(test, executor, log)
+
+    @staticmethod
+    def create_test(model, dataset, indep_parameters, dep_parameters):
+        return Test(model, dataset, indep_parameters, dep_parameters)

src/configs/README.md

@@ -225,6 +225,22 @@
   - `TimeoutOverhead` - опциональный тег для задания дополнительного времени поверх установленного timeout.
     Например, для скачивания тяжелой модели из внешних репозиториев, по умолчанию 300 секунд. 
 
+- Набор тегов для тестирования вывода средствами PaddlePaddle:
+
+  - `ChannelSwap` - тег, необязательный для заполнения. Описывает изменение порядка каналов на
+    входном изображении.
+  - `Mean` - тег, необязательный для заполнения. Определяет средние значения, которые будут вычитаться
+    по каждому из каналов входного изображения.
+  - `InputScale` - тег, необязательный для заполнения. Определяет коэффициент масштабирования входного
+    изображения.
+  - `InputShape` - тег, необязательный для заполнения. Определяет размеры входного тензора в формате NHWC.
+    По умолчанию не установлен.
+  - `InputName` - тег, необязательный для заполнения. Определяет название входного узла модели.
+    По умолчанию не установлен.
+  - `ThreadCount` - опциональный тег. Описывает максимальное количество физических
+    потоков для выполнения вывода. По умолчанию будет выставлено число потоков, равное
+    физическому количеству ядер в системе.
+
 - Набор тегов для тестирования вывода средствами ONNX Runtime Python API:
 
   - `ChannelSwap` - тег, необязательный для заполнения. Описывает изменение порядка каналов на