There is a future for private PaaS-solutions in a world where privacy and cost-control are primary concerns. We just needs to build it on the right foundation.
Create a private PaaS on Hetzner Cloud in minutes using nix-infra. Leverage NixOS and Nix Packages to build a reproducable and auditable private cloud for your projects.
Why did I build this? I wanted to test the limits of NixOS when it comes to maintainability and real world use.
- low and predictable cost -- runs on Hetzner Cloud
- reproducable and auditable -- 100% configuration in code
- privacy -- all data within your private walled garden
- easy to debug -- zero blackbox services
- extendable -- install anything than runs on NixOS or as an OCI-container
- customise in any way you like
You can easily share and extend modules to build the perfect private PaaS.
Low system requirements for each cluster node makes virtual machine isolation per service/application cost effective.
Features:
- runs NixOS as host OS
- uses Systemd credentials to secure secrets
- fault tolerant service mesh (with HAProxy + Etcd + Confd)
- private encrypted overlay network (with flanneld + wireguard)
- supports provisioning nodes in multiple data centers
- run apps in OCI-containers with podman
Limitations:
- NixOS doesn't support SELinux
- nix-infra currently only supports Hetzner Cloud
- the code is the primary documentation
Room for improvement:
- hardening with AppArmor [major]
- encryption of config folder at rest [medium]
- rotating secrets [medium]
- virtual tpm for systemd-credentials (not supported by Hetzner cloud)
- visualise cluster health [major]
Apple discusses privacy in a post about Private Cloud Compute.
-
Download and install the nix-infra binary
-
Install hcloud cli tool
Clone this repo and run nix-shell to ensure you get the right version of the hcloud tool.
- Try the nix-infra-test or nix-infra-ha-cluster cluster templates
You will find instructions in the each repo. Basically you will download and run a test script that automates the installation of the cluster. These scripts are also the documentation of the process.
To build without immediately tearing down the cluster:
$ ./test-nix-infra-with-apps.sh --env=.env --no-teardown
# -- or --
$ ./test-nix-infra-ha-base.sh --env=.env --no-teardownUseful commands to explore the running test cluster (check the bash script for more):
$ ./test-nix-infra-with-apps.sh etcd --env=.env "get /cluster --prefix"
$ ./test-nix-infra-with-apps.sh cmd --env=.env --target=ingress001 "uptime"
$ ./test-nix-infra-with-apps.sh ssh --env=.env ingress001
# -- or --
$ ./test-nix-infra-ha-base.sh etcd --env=.env "get /cluster --prefix"
$ ./test-nix-infra-ha-base.sh cmd --env=.env --target=ingress001 "uptime"
$ ./test-nix-infra-ha-base.sh ssh --env=.env ingress001
To tear down the cluster:
$ ./test-nix-infra-with-apps.sh --env=.env teardown
# -- or --
$ ./test-nix-infra-ha-base.sh --env=.env teardown- Install nix to build nix-infra (choose one)
- https://nixos.org/download/
- https://github.com/DeterminateSystems/nix-installer (supports uninstall)
- Clone the repo
$ git clone [email protected]:jhsware/nix-infra.git- Build nix-infra using the build script
$ cd nix-infra; ./build.sh
# ouput: bin/nix-infraConfiguration of your cluster using the Nix language.
Add remote actions written in Bash that can be run on cluster nodes.
- Clone a cluster template
- Run
nix-infra initto create the cluster configuration folder - Create a
.envfile - Add the created ssh-key to the ssh agent (probably:
ssh-add) - Provision nodes
nix-infra provision - Initialise control plane `nix-infra init-ctrl``
- Initialise cluster nodes
nix-infra init-node - Configure apps (apps consist of app_module and node specific configuration)
- Deploy apps
nix-infra deploy
To create similar clusters you create a cluster template and fork it for each cluster.
stateDiagram
direction LR
Cluster_1: Cluster 1
Cluster_2: Cluster 2
Template --> Cluster_1
Template --> Cluster_2
To share apps you copy them to your cluster repo.
stateDiagram-v2
direction LR
Cluster_1: Cluster 1
Cluster_2: Cluster 2
Template --> Cluster_1
Template --> Cluster_2
Cluster_1 --> Cluster_2 : copy<br>app_modules/monogdb.nix<br>app_modules/keydb.nix
To create an exact copy of cluster, use the same cluster repo but different .env-files.
stateDiagram-v2
direction LR
Cluster_1: Cluster 1
Template --> Cluster_1
note left of Cluster_1 : .env-a
note left of Cluster_1 : .env-b
The configuration files are related according to the diagram below. These are the files you would normally configure once you cluster is up and running:
nodes/[node_name].nix-- install and configure apps on each nodeapp_modules/[module_name].nix-- configure apps available on the cluster
When you add new files to app_modules you need to import them in app_modules/default.nix.
stateDiagram-v2
direction LR
flake: flake.nix
configuration: configuration.nix
hardwareConfiguration: hardware-configuration.nix
networking: networking.nix
nodeType: node_types/[node_type].nix
nodeConfig: nodes/[node_name].nix
modules: modules/[module_name].nix
appModules: app_modules/[module_name].nix
NixPkgs --> flake
Secrix --> flake
flake --> configuration : modules
configuration --> generated
configuration --> nodeType
configuration --> nodeConfig
configuration --> modules
configuration --> appModules
state generated {
direction LR
hardwareConfiguration
networking
}
Provisioning of nodes, deployment of configurations and container images is done through the nix-infra CLI. The overlay network and service mesh is configured via the etcd-database of the control plane (ctrl).
stateDiagram-v2
ctrl_node --> cluster_node : overlay network config<br>service mesh config
CLI --> ctrl_node : provision
CLI --> cluster_node : provision
CLI --> ctrl_node : deploy
CLI --> cluster_node : deploy
state ctrl_node {
etcd
systemd_ctrl
state "systemd" as systemd_ctrl
}
state cluster_node {
haproxy
confd
systemd
flanneld
}
The registry node contains a package cache and registry which allows you to provide private packages and caching.
The registry node also contains a container image registry where you push your private application images.
stateDiagram
registry --> service : img/pkg
registry --> worker : img/pkg
registry --> ingress : img/pkg
The cluster has a simple topology with three layers. Only the ingress layer is exposed to the outside world.
stateDiagram
service: Services
worker: Workers
ingress: Ingress
service --> worker
worker --> ingress
ingress
state service {
mongodb: MongoDB
keydb: KeyDB
elasticsearch: Elasticsearch
}
state worker {
API: API / Backend
APP: APP / Frontend
}
state ingress {
reverse_proxy: Nginx
}
Stateful services such as DBs run on the service nodes.
Worker nodes that run your stateless application containers.
The ingress node exposes the cluster to the internet via an Nginx reverse proxy.
Testing:
scripts/end-to-end-tests/test-nix-infra-ha-cluster.sh --env=./.env
scripts/end-to-end-tests/test-nix-infra-test.sh --env=./.env/cluster/frontends
[app_name]/
instances/
[node_name]={
"node": "[node_name]",
"ipv4": "123.23.23.0",
"port": 123
}
meta_data={
publish: { ”port”: 5001 } // Used by HA proxy to expose service on worker nodes
env_prefix: "[PREFIX]"
env: { "PROTOCOL", "HOST", "PORT", "PATH" }
}
/cluster/backends
[app_name]/
instances/
[node_name]={
"node": "[node_name]",
"ipv4": "123.23.23.0",
"port": 123
}
meta_data={
publish: { ”port”: 5001 } // Used by HA proxy to expose service on worker nodes
env_prefix: "[PREFIX]"
env: { "PROTOCOL", "HOST", "PORT", "PATH" }
}
/cluster/services
[app_name]/
instances/
[node_name]={
"node": "[node_name]",
"ipv4": "123.23.23.0",
"port": 123
}
meta_data={
publish: { ”port”: 5001 } // Used by HA proxy to expose service on worker nodes
env_prefix: "[PREFIX]"
env: { "PROTOCOL", "HOST", "PORT", "PATH" }
}
/cluster/nodes
[node_name]={
"name": "[node_name]",
"ipv4": "123.23.23.9",
"services": ["services", "frontends", "backends"] // Service types to access
}- Provision node
- Initialise node
- Resister node
- Unregister node
- Destroy node
CONSIDER: We might want to provide some kind of CI/CD-pipeline
- Register app
- Deploy app to node
- Register app instance
- Unregister app instance
- Remove app from node
- Unregister app
Secrets are created either:
- by storing the result of an action (i.e. when you create a user in a db), or
- by explicitly storing a provided secret (i.e. an external API-key)
nix-infra [...] action [...] --store-as-secret="[secret-name]"
nix-infra [...] store-secret [...] --secret="[your-secret]" --store-as-secret="[secret-name]"stateDiagram
direction LR
action : nix-infra action
secret : nix-infra secret
local_secrets_enc : secrets/
state cli_deploy {
local_secrets_dec : secrets/
ssh
systemd_credentials : systemd-credentials
}
cli_deploy : nix-infra deploy
action --> local_secrets_enc : encrypt
secret --> local_secrets_enc : encrypt
local_secrets_enc --> cli_deploy
local_secrets_dec --> ssh : decrypt
ssh --> systemd_credentials : encrypt
cli_deploy --> systemd
systemd --> application : decrypt
-
Update version in pubspec.yaml
-
Build macOS binary
./build.sh build-macos --env=.env-
Package and notarise macOS binary
-
Run build workflow to create draft release with Linux binary
-
Add macOS binary to release
-
Add release notes
-
Publish release
TODO: Investigate secret rotation
- https://partial.solutions/2024/understanding-systemd-credentials.html TODO: Automated builds
- https://blog.thestateofme.com/2023/05/17/multi-architecture-automated-builds-for-dart-binaries/
TODO: Font
- https://www.dafont.com/aristotelica.font?text=nix-infra
- https://fonts.google.com/specimen/Comfortaa?preview.text=nix-infra&categoryFilters=Sans+Serif:%2FSans%2FRounded
DONE: Investigate using systemd credentials
INVESTIGATE: Securing systemd services
INVESTIGATE: Tuning kernel and HAProxy
DONE: Investigate Nixos secrets management
- https://nixos.wiki/wiki/Comparison_of_secret_managing_schemes
- secrix appears to have systemd integration
DONE: Investigate using agenix for secrets