Skip to content

Latest commit

 

History

History
287 lines (230 loc) · 21 KB

README.md

File metadata and controls

287 lines (230 loc) · 21 KB

Table of Contents

Introduction

Running benchmarks on the cloud is not only useful to compare different providers but also to measure the differences when changing the operating conditions of the cluster (e.g., updating the underlying physical hardware, replacing the CNI provider, adding more nodes, running different workloads in background, etc.).

kubemarks focuses on the latter aspect, specifically on Kubernetes: it can run a large set of benchmarks and expose their results to the Prometheus monitoring system.

Existing work

Adapting existing benchmarks to run on Kubernetes may not be straight-forward, especially when dealing with distributed ones that, by definition, need to involve multiple pods. Yet there are some attempts online that try to do so. On the other hand, retrieving benchmark results from different pods requires way more work than just adapting them to Kubernetes.

This is where PerfKit Benchmarker comes into play, an open-source tool by Google Cloud Platform that contains a set of benchmarks that are ready to be run on several cloud offerings, including Kubernetes. In short, PerfKit Benchmarker:

  • creates and configures as many Kubernetes pods as needed by the benchmark,
  • handles their lifecycle,
  • installs dependencies,
  • runs benchmarks,
  • retrieves results from all pods and, lastly,
  • makes it easy to add additional "results writers" so that results can be exported in different ways.

Implementation

This section aims to describe some system and implementation details needed to accomplish what has been described in the introduction.

Supported benchmarks

There are four main categories of PerfKit Benchmarker-supported benchmarks runnable on Kubernetes:

Due to the shortage of time, benchmarks belonging to the latter group still do not run properly. However, the current bugs seem to be pretty trivial to solve. For instance, MongoDB does not start because of a missing release file:

kubemarks-pkb STDERR: E: The repository 'https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/3.0 Release' does not have a Release file.

And Redis simply exceeds Resource Quotas:

kubemarks-pkb STDERR: Error from server (Forbidden): error when creating "/tmp/tmpaIV4bj": pods "pkb-77f34c4d-9" is forbidden: exceeded quota: default-quota, requested: limits.cpu=5, used: limits.cpu=50, limited: limits.cpu=54

Non-running benchmarks are tracked in issue #5.

Please note that PerfKit Benchmarker has to be extended so that benchmarks can expose additional information to the Prometheus Pushgateway (e.g., Node IDs). Besides tracking benchmarks that still have to be extended, issue #21 also lists a few previous commits that show how to do this.

PerfKit Benchmarker fork changes

Besides minor bug fixes, the current PerfKit Benchmarker fork has been extended with an additional "results writer", i.e., endpoint to which results are exported at the end of a single benchmark execution. More specifically, it now includes a Prometheus Pushgateway exporter, which exposes results according to the OpenMetrics format. The official Prometheus Python client has been used to implement this result writer. Furthermore, the CSV writer can now write results in "append" mode, allowing it to gradually add entries to the same CSV file as soon as benchmarks finish.

Including node IDs in benchmark results

While PerfKit Benchmarker does include physical node information in benchmark results (e.g., lscpu command output), it does not include Kubernetes node IDs. This information is essential to make a comparison between different hardware solutions. Since PerfKit Benchmarker is in charge of creating and configuring pods, its source code had to be extended to make pods aware of the node ID they were running on. To do this, the Kubernetes Downward API comes in handy: it makes it possible to expose pod and container fields to a running container: here is the JSON snippet which made that possible:

'env': [{
    'name': 'KUBE_NODE',
    'valueFrom': {
        'fieldRef': {
        'fieldPath': 'spec.nodeName'
        }
    }
}]

This way, each Kubernetes pod can retrieve the node ID of the physical machine on which it is running, and PerfKit Benchmarker can successfully include this information in the results.

Running benchmarks periodically

Benchmarks are run periodically as a Kubernetes CronJob. It periodically executes a shell script (scripts/pkb/start.sh) that cycles through all the benchmarks to be executed and, for each one of them, it

  1. checks whether it is compatible with Kubernetes, and
  2. builds a proper argument list to be passed to PerfKit Benchmarker.

Docker images

A Kubernetes CronJob launches periodic jobs in Docker containers. The base CronJob file of this repository yaml/base/kubemarks-pkb-cronjob.yaml mainly executes PerfKit Benchmarker, which in turn needs to launch benchmarks in Docker containers so that the Kubernetes scheduler can allocate those onto pods. marcomicera/kubemarks-cronjob and marcomicera/kubemarks-pkb are the Docker images launched by the CronJob and PerfKit Benchmarker, respectively.

CronJob files and their structure

Kustomize is a tool included in kubectl that allows users to customize Kubernetes objects. In this project, it is mainly used to:

  1. combine multiple objects in the right order of declaration into a single YAML file, and
  2. enable object inheritance.

Objects combination is particularly useful when creating the base CronJob file that runs an user-defined list of benchmarks, while inheritance makes it possible to create dedicated CronJob files for single benchmarks, making it simpler for the user to launch single benchmarks.

The base CronJob file running an user-defined benchmarks list

Kustomize needs a kustomization file as a starting point. The base kustomization.yaml file simply lists all the Kubernetes objects needed to launch kubemarks:

resources:
  - kubemarks-conf.yaml # benchmarks list, pushgateway address
  - kubemarks-role.yaml # PerfKit Benchmarker permissions
  - kubemarks-role-binding.yaml # associating Role to ServiceAccount
  - kubemarks-pkb-cronjob.yaml # PerfKit Benchmarker base CronJob file
  - kubemarks-git-creds.yaml # Git credentials for downloading private repo
  - kubemarks-sa.yaml # ServiceAccount
  - kubemarks-num-pods.yaml # number of pods to be used for each benchmark

All these files can be combined together and applied with a single command:

$ kubectl kustomize yaml/base | kubectl apply -f -

Before launching this command, yaml/base/kubemarks-num-pods.yaml, yaml/base/kubemarks-conf.yaml, and yaml/base/kubemarks-pkb-cronjob.yaml should be modified accordingly, as described in the Guide section.

Repo cloner InitContainer

An InitContainer takes care of cloning this repository first. If this repository is private, then it needs Git credentials mounted as a Secret: please take a look at the corresponding developer guide section to learn more.

Dedicated CronJob files for benchmarks

When launching single benchmarks, Kustomize can be used to override some Kubernetes object fields of the base CronJob file. Looking at a benchmark-specific kustomization file is enough to determine which fields are actually overridden. The following kustomization.yaml file depicts all changes needed to run the fio benchmark:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
nameSuffix: -fio
patchesStrategicMerge:
  - benchmarks-list.yaml
  - schedule.yaml

This file inherits all Kubernetes objects of the base kustomization.yaml file, and:

  • adds a metadata-name suffix to all its Kubernetes objects (i.e., -fio),
  • defines the list of benchmarks to be executed (a ConfigMap in yaml/benchmarks/fio/benchmarks-list.yaml containing only fio in the list) and,
  • overrides the frequency with which this benchmark is run (based on its average completion time).

The nameSuffix field causes all resources to have this suffix prepended to their name, hence each benchmark will run in a dedicated CronJob whose name contains the benchmark one.

It is worth noticing that since every benchmark's Kustomize folder (e.g., yaml/benchmarks/fio) contains its own schedule.yaml file, every benchmark will be launched at different frequencies. Benchmarks already have pre-determined frequency values based on their average completion time (i.e., the longer they take to complete, the less frequent their corresponding CronJob will be launched), but this can be overridden by simply editing their schedule.yaml file before launching. Here is an example of schedule.yaml file (for the fio benchmark):

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: kubemarks-pkb
spec:
  schedule: '0 */4 * * *'
status: {}

According to the Cron format, fio will be launched every 4 hours. Modifying such files while a benchmark is running in a CronJob will not cause any change.

Permissions

Upon launching a benchmark, Kustomize uses the base kustomization.yaml file (as described here) to assemble together various Kubernetes objects. Between these objects, there are a couple ones used for Role-based access control, such as a ServiceAccount yaml/base/kubemarks-sa.yaml, a Role yaml/base/kubemarks-role.yaml and a RoleBinding object yaml/base/kubemarks-role-binding.yaml.

User guide

This section examines the How to run it section of the main README.md file in depth.

Configuration

This section describes all configuration steps to be made before launching benchmarks.

Number of Kubernetes pods

The number of Kubernetes pods to be used for every benchmark is defined in the yaml/base/kubemarks-num-pods.yaml ConfigMap. Here is an extract:

kind: ConfigMap
metadata:
  name: kubemarks-num-pods
apiVersion: v1
data:
  kubemarks-num-pods.yaml: |
    flags:
      cloud: Kubernetes
      kubernetes_anti_affinity: false

    block_storage_workload:
      description: >
        Runs FIO in sequential, random, read and
        write modes to simulate various scenarios.
      vm_groups:
        default:
          vm_count: 1

It is worth noticing that PerfKit Benchmarker uses the term VM as a generalization of Kubernetes pod since it supports multiple cloud providers.

This ConfigMap will be then applied by Kustomize upon launching a benchmark, and it will then be mounted as a file in the container running PerfKit Benchmarker.

CronJob frequency

Next, the general CronJob frequency can be adjusted in the yaml/base/kubemarks-pkb-cronjob.yaml file:

schedule: '*/30 * * * *'

The schedule follows the Cron format. There is no need to specify this for the one-benchmark-only CronJob files, as the frequency will be automatically set by Kustomize.

Benchmarks list and Pushgateway address

The yaml/base/kubemarks-conf.yaml file contains two experiment options, namely

# `kubectl kustomize yaml/base | kubectl apply -f -` will
# automatically update this ConfigMap
apiVersion: v1
data:
  benchmarks: cluster_boot fio
  pushgateway: pushgateway.address.test
kind: ConfigMap
metadata:
  name: kubemarks-conf

Experiments can be chosen amongst this list:

  • block_storage_workload
  • cluster_boot
  • fio
  • iperf
  • mesh_network
  • netperf

This ConfigMap will be automatically applied/updated by Kustomize.

Launching benchmarks

Benchmarks can be either launched singularly with (e.g., for iperf):

$ kubectl kustomize yaml/benchmarks/iperf | kubectl apply -f -

or programmatically, after having updated the list of benchmarks to run:

$ kubectl kustomize yaml/base | kubectl apply -f -

Developer guide

Take a look at the CONTRIBUTING.md file.

Conclusions

kubemarks is able to periodically run various kinds of benchmarks on a Kubernetes cluster. The current PerfKit Benchmarker fork (more details here) includes physical node identifiers into benchmark results and gradually exposes them to a Prometheus Pushgateway following the OpenMetrics format. The tool is configurable through a few handy configuration files.