Skip to content

Commit

Permalink
Update README with more detailed deployment details
Browse files Browse the repository at this point in the history
  • Loading branch information
pmundt committed Apr 10, 2020
1 parent e55a375 commit fafade5
Show file tree
Hide file tree
Showing 2 changed files with 86 additions and 8 deletions.
94 changes: 86 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,50 @@ It is inspired by and re-uses much of the Kubernetes controller plumbing from th
## Use Cases

`k8s-dt-node-labeller` was developed in order to facilitate targeted deployments into hybrid (e.g. armhf, arm64)
Kubernetes clusters, as well as for targeting heterogeneous accelerators in Edge deployments.
Kubernetes clusters, as well as for targeting heterogeneous accelerators in Edge deployments - specifically those that
show up as platform devices (FPGAs, embedded GPUs, etc.) as opposed to those that are dynamically discoverable at
run-time via bus enumeration (e.g. USB, PCI).

## Overview

`k8s-dt-node-labeller` is primarily concerned with top-level system characteristics, and therefore limits itself to
walking through the top-level `devicetree` node looking for `compatible` properties to expose as Kubernetes node labels.
As `k8s-dt-node-labeller` requires access to the target node's `devicetree` instance, the labeller must be run directly
on the target node that is to be labelled. An overview of the general labelling flow (in the context of a Kubernetes
DaemonSet) is provided below:

![Overview Diagram][overview]

It is expected that the node labeller is run when the node is registered with the cluster, such that pods relying on
specific labels are able to be scheduled and placed appropriately.

Labelling is possible in a number of different ways:
- Running Node-Local via the CLI App
- Running Node-Local via Docker
- Deploying as a Kubernetes DaemonSet

Regardless of the method of deployment, the labels will be generated and applied to the local node.

### Limitations

`k8s-dt-node-labeller` only generates labels from properties at the top-level of the `devicetree` and those specifically
defined by the user by design. It does not recurse down the tree, nor does it make any attempt to enumerate busses that
are better served by bus-specific device detection logic. While the labeller can attest to the existence of a node in
the `devicetree`, it offers no device-specific information or control - all of which would need to be implemented
through a device-specific device plugin.

## Installation

If planning to use the CLI App directly, this can be compiled and installed as other Go projects:

```
$ go get github.com/adaptant-labs/k8s-dt-node-labeller
```

For Docker containers and Kubernetes deployment instructions, see below.
## Usage

The node labeller is expected to be run node-local, and will need to be invoked on each individual node
requiring its own specific devicetree parsing and labelling.
General usage is as follows:

```
$ k8s-dt-node-labeller --help
Expand All @@ -24,10 +62,9 @@ Usage: k8s-dt-node-labeller [flags] [-n devicetree nodes...]
Paths to a kubeconfig. Only required if out-of-cluster.
-n string
Additional devicetree node names
```

By default, `compatible` strings from the top-level `/` node are discovered and converted to node labels.
A dry-run is possible by specifying the `-d` flag:

```
$ k8s-dt-node-labeller -d
Expand All @@ -37,7 +74,8 @@ beta.devicetree.org/nvidia-jetson-nano: 1
beta.devicetree.org/nvidia-tegra210: 1
```

additional node specifications are possible via the `-n` flag, as below:
By default, `compatible` strings from the top-level `/` node are discovered and converted to node labels. Additional
node specifications are possible via the `-n` flag, as below:

```
$ k8s-dt-node-labeller -d -n gpu pwm-fan
Expand All @@ -50,6 +88,11 @@ beta.devicetree.org/nvidia-gm20b: 1
beta.devicetree.org/pwm-fan: 1
```

In the case of Docker or Kubernetes Deployment, note that these arguments can also be passed on to the container
directly.

### Running Node-Local via the CLI App

In order to submit the labels to the cluster, ensure a valid kubeconfig can be found (e.g. by setting the `KUBECONFIG`
environment variable, or through specifying the location with the `-kubeconfig` parameter) and execute the node
labeller as-is:
Expand All @@ -67,7 +110,35 @@ $ k8s-dt-node-labeller
...
```

After which the node labels can be viewed from `kubectl`:
### Running Node-Local via Docker

Multi-arch Docker images are available on Docker Hub at [adaptant/k8s-dt-node-labeller]. These may be run as-is
in-cluster, or out of cluster with an appropriate `KUBECONFIG` passed through.

Note that as the labeller requires access to the host's `/sys/firmware` directory, this must either be passed through
explicitly, or the container run in privileged mode:

```
$ docker run --privileged adaptant/k8s-dt-node-labeller
...
```

### Deploying as a Kubernetes DaemonSet

An example deployment configuration for a DaemonSet is provided in `k8s-dt-labeller-ds.yaml`, which can be directly
applied to the running cluster:

```
$ kubectl apply -f k8s-dt-labeller-ds.yaml
```

This will create a special `dt-labeller` service account, cluster role, and binding with the permission to list and
reconcile nodes. Note that as the labeller requires access to an unmasked `/sys/firmware`, it must also be run in a
privileged securityContext.

### Verifying Labelling

After the labeller has run, the node labels can be viewed from `kubectl`:

```
$ kubectl describe node jetson-nano
Expand All @@ -87,6 +158,10 @@ Labels: beta.devicetree.org/nvidia-jetson-nano=1
...
```

## Features and bugs

Please file feature requests and bugs in the [issue tracker][tracker].

## Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant
Expand All @@ -97,6 +172,9 @@ agreement No 825480 ([SODALITE]).
`k8s-dt-node-labeller` is licensed under the terms of the Apache 2.0 license, the full
version of which can be found in the LICENSE file included in the distribution.

[tracker]: https://github.com/adaptant-labs/k8s-dt-node-labeller/issues
[devicetree]: https://www.devicetree.org
[SODALITE]: https://www.sodalite.eu
[amdgpu-node-labeller]: https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller
[overview]: https://raw.githubusercontent.com/adaptant-labs/k8s-dt-node-labeller/master/overview.png
[amdgpu-node-labeller]: https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller
[adaptant/k8s-dt-node-labeller]: https://hub.docker.com/repository/docker/adaptant/k8s-dt-node-labeller
Binary file added overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit fafade5

Please sign in to comment.