From ed164c3b64c75df517cc3449819a6d6c59f2eb05 Mon Sep 17 00:00:00 2001 From: GitHub Actions Date: Wed, 16 Oct 2024 21:01:51 +0000 Subject: [PATCH] site deploy Auto-generated via `{sandpaper}` Source : 3efe006c359ce1e8dcd7a2cdab30fca13b53dcbf Branch : md-outputs Author : GitHub Actions Time : 2024-10-16 21:01:40 +0000 Message : markdown source builds Auto-generated via `{sandpaper}` Source : c7c11828d38e4ce5d07c00709adbacdd3ab62132 Branch : main Author : Kati Lassila-Perini Time : 2024-10-16 20:48:14 +0000 Message : 05: add workflow --- 01-intro.html | 13 +- 02-storage.html | 25 +- 03-disk-image.html | 55 ++- 04-cluster.html | 117 +++++-- 05-workflow.html | 560 ++++++++++++++++++++++++++++++ 404.html | 13 +- CODE_OF_CONDUCT.html | 11 +- LICENSE.html | 11 +- aio.html | 509 ++++++++++++++++++++-------- config.yaml | 3 +- images.html | 17 +- index.html | 11 +- instructor-notes.html | 14 +- instructor/01-intro.html | 13 +- instructor/02-storage.html | 25 +- instructor/03-disk-image.html | 55 ++- instructor/04-cluster.html | 117 +++++-- instructor/05-workflow.html | 562 +++++++++++++++++++++++++++++++ instructor/404.html | 13 +- instructor/CODE_OF_CONDUCT.html | 11 +- instructor/LICENSE.html | 11 +- instructor/aio.html | 510 ++++++++++++++++++++-------- instructor/images.html | 17 +- instructor/index.html | 23 +- instructor/instructor-notes.html | 17 +- instructor/key-points.html | 33 +- instructor/profiles.html | 11 +- instructor/reference.html | 11 +- key-points.html | 33 +- md5sum.txt | 7 +- pkgdown.yml | 2 +- profiles.html | 11 +- reference.html | 11 +- sitemap.xml | 6 + 34 files changed, 2461 insertions(+), 397 deletions(-) create mode 100644 05-workflow.html create mode 100644 instructor/05-workflow.html diff --git a/01-intro.html b/01-intro.html index 95a3495..d938675 100644 --- a/01-intro.html +++ b/01-intro.html @@ -239,7 +239,7 @@

+ + + +
+
+ +
@@ -454,7 +463,7 @@

Too much to install? I just want a quick try…

-
+

If you don’t have access to a Linux terminal or prefer not to install tools locally, you can use Google Cloud Shell. You’ll need a Google diff --git a/02-storage.html b/02-storage.html index bab6554..f08dd7f 100644 --- a/02-storage.html +++ b/02-storage.html @@ -118,11 +118,11 @@ CMS Open Data processing in Google Cloud Platform (GCP)

-
-

In addition, on the storage side, the compute service account needs -write access to the bucket for the logs. Add it with

+
+
+

Application login

+

You need to create a credential file to run the script:

BASH

-
gcloud storage buckets add-iam-policy-binding gs://<BUCKET_FOR_LOGS>/ --project=<PROJECT_ID> --member=serviceAccount:<PROJECT_NR>-compute@developer.gserviceaccount.com --role=roles/storage.objectCreator
+
gcloud auth application-default login
+

and authenticate in the browser window that opens.

Get the code

Pull the code from https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/tools/gke-disk-image-builder

@@ -448,8 +460,35 @@

BASH 
go run ./cli --project-name=<PROJECT_ID> --image-name=pfnano-disk-image --zone=europe-west4-a --gcs-path=gs://<BUCKET_FOR_LOGS> --disk-size-gb=50 --container-image=ghcr.io/katilp/pfnano-image-build:main --timeout 100m
-

For the moment, for a new project created for testing these -instructions, this fails. Investigating…

+
+
+ +
+
+

Failure with errors of type

+
Code: QUOTA_EXCEEDED
+Message: Quota 'N2_CPUS' exceeded.
+

are due to requested machine type no being available in the requested +zone. Nothing to do with you quota.

+

Try in a different region or with a different machine type. You can +give them as parameters +e.g. --zone=europe-west4-a --machine-type=e2-standard-4. +Independent of the zone specified in parameters, the disk image will +have eu as the location, so any zone in europe +is OK (if you plan to create your cluster in a zone in +europe).

+

Note that the bucket for logs has to be in the same region so you +might need to create another one. Remove the old one with +gcloud storage rm -r gs://<BUCKET_FOR_LOGS>.

+
+
+
+

Costs

diff --git a/04-cluster.html b/04-cluster.html index fa0ce93..d478aae 100644 --- a/04-cluster.html +++ b/04-cluster.html @@ -118,11 +118,11 @@ CMS Open Data processing in Google Cloud Platform (GCP)
-
+ +
+
+ +
@@ -295,17 +306,17 @@

+
+

Bucket

+

If you worked through Section 02, you have now a storage bucket for +the output files.

+

List the buckets with

+
+

BASH +

+
gcloud storage ls
+
+
+
+

Secondary disk

+

If you worked through Section 03, you have a secondary boot disk +image available

Get the code

-

Create the cluster

-

Costs

+

The example Terraform scripts and Argo Workflow configuration are +in

+

Get them with

+
+

BASH +

+
git clone git@github.com:cms-dpoa/cloud-processing.git
+cd cloud-processing/standard-gke-cluster-gcs-imgdisk
+
+

Create the cluster

+

Set the variable in the terraform.tfvars files.

+

Run

+
+

BASH +

+
terraform apply
+
+

and confirm “yes”.

+

Connect to the cluster and inspect

+
+

BASH +

+
gcloud container clusters get-credentials cluster-2 --region europe-we
+st4-a --project hip-new-full-account
+
+
+

BASH +

+
kubectl get nodes
+
+
+

BASH +

+
kubectl get ns
+
+

Enable image streaming

+
+

BASH +

+
 gcloud container clusters update cluster-2 --zone europe-west4-a --ena
+ble-image-streaming
+
+

Costs

@@ -390,17 +459,17 @@

Key Points

-->