bbl&cfcr k8s walkthrough

Today we're going to provision a k8s using bosh. There is a TON of fragmentation in k8sland about what tools to use to initially provision VMs and install k8s components, and though cfcr+bosh aren't well-known there, they have the unique advantage of being familiar to us in CF. Last time we approached k8s from the perspective of an app dev trying to deploy a web app, this time we'll step into the shoes of bosh operators and platform engineers.

bbling

To start, we're each going to provision a fresh gcp bosh director with some modifications to make it more k8s friendly.

cd ~/workspace
git clone [email protected]:cloudfoundry/bosh-bootloader.git
cd ~/workspace/capi-ci-private
mkdir yourname-k8s
cd yourname-k8s

We're also going to cut corners on service keys and use the CI account key for Arya. If you were doing this from scratch, you'd need to make some new keys with appropriate permissions. Instead, we'll rip the key out of our CI credentials and point bbl at gcp:

export BBL_IAAS=gcp
export BBL_GCP_REGION=us-west2
export BBL_GCP_SERVICE_ACCOUNT_KEY="$(bosh int ${HOME}/workspace/capi-ci-private/ci/credentials.yml --path=/arya_gcp_json_key)"

and initialize your bbl plan:

bbl plan --name yourname-k8s

If you've never poked around in a bbl plan before, now is a good time to do git add . && git status to see a list of all the files that bbl packages that it will use to deploy a bosh director. There are a lot, including a full copy of bosh-deployment and jumpbox-deployment.

Next, we're gonna bring over the changes that help k8s run under bosh:

cp -r ${HOME}/workspace/bosh-bootloader/plan-patches/cfcr-gcp/. .
git status && bbl up -d

While bbl is working, go peruse the files you just copied, especially the terraform templates, to see what additional infrastructure k8s needs. At this point you're almost ready to deploy a k8s: you're nearly on step 2 of the plan-patch README. If that doesn't hold your attention, go poke around kubo-deployment and kubo-release.

When bbl is done (about 25 minutes), target your fresh bosh director:

eval "$(bbl print-env)"
bosh env

kuboing aka cfcring

At this point, you can keep following along with the plan-patch README. OR you can veer off where Connor's been and follow the kubo-release README.

Once you've got a k8s running, bosh ssh and monit summary onto each vm type to see what it looks like in comparison to a CC or other jobs you're familiar with. Maybe try a bosh instances --ps to get the lay of the land.

Then each README will have you set kubectl up to target your new cluster.

To take your k8s for a spin after you've got kubectl configured, try deploying the kubernetes-bootcamp app at the bottom of the patch README.

poking around and experimenting

What happens if you haven't configured all the capabilities? What happens if you remove the cloud-provider ops file that you might've copied from the plan-patch README? If you followed the kubo-release README, you probably didn't deploy it in the first place, but what capacities does it provide to a kubectl user?

bosh deploy -d cfcr ${KD}/manifests/cfcr.yml \
-o ${KD}/manifests/ops-files/use-vm-extensions.yml \
-o ${KD}/manifests/ops-files/add-hostname-to-master-certificate.yml \
-v deployment_name=cfcr \
-l <(bbl outputs)

What do the errands they package with kubo-release do?

bosh -d cfcr run-errand apply-specs
bosh -d cfcr run-errand smoke-tests

What about all the fun things you can install on k8s? Go try to install helm, and then find a helm chart and try using it to install something you're familiar with. How's that work? Does it?

What about service meshes? It's a lot easier to install a service mesh that's k8s native than it is to toy with service meshes on their own or in CF. Go check out LinkerD and try to install that. What if you wanted to try istio and envoy, like we use inside CF? How might you install those? How would you make them available to other folks trying to deploy onto your platform?