Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kicbase: update cri-o packaging sources #19504

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Kicbase: update cri-o packaging sources #19504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -733,7 +733,7 @@ build-and-upload-cri-dockerd-binaries:
.PHONY: local-kicbase
local-kicbase: ## Builds the kicbase image and tags it local/kicbase:latest and local/kicbase:$(KIC_VERSION)-$(COMMIT_SHORT)
touch deploy/kicbase/CHANGELOG
docker build -f ./deploy/kicbase/Dockerfile -t local/kicbase:$(KIC_VERSION) --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) --cache-from $(KICBASE_IMAGE_GCR) .
docker build -f ./deploy/kicbase/Dockerfile -t local/kicbase:$(KIC_VERSION) --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) --build-arg KUBERNETES_VERSION=$(KUBERNETES_VERSION) --cache-from $(KICBASE_IMAGE_GCR) .
docker tag local/kicbase:$(KIC_VERSION) local/kicbase:latest
docker tag local/kicbase:$(KIC_VERSION) local/kicbase:$(KIC_VERSION)-$(COMMIT_SHORT)

Expand All @@ -744,7 +744,7 @@ local-kicbase-debug: local-kicbase ## Builds a local kicbase image and switches

.PHONY: build-kic-base-image
build-kic-base-image: docker-multi-arch-build ## Build multi-arch local/kicbase:latest
docker buildx build -f ./deploy/kicbase/Dockerfile --platform $(KICBASE_ARCH) $(addprefix -t ,$(KICBASE_IMAGE_REGISTRIES)) --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) .
docker buildx build -f ./deploy/kicbase/Dockerfile --platform $(KICBASE_ARCH) $(addprefix -t ,$(KICBASE_IMAGE_REGISTRIES)) --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) --build-arg KUBERNETES_VERSION=$(KUBERNETES_VERSION) .

.PHONY: push-kic-base-image
push-kic-base-image: docker-multi-arch-build ## Push multi-arch local/kicbase:latest to all remote registries
Expand All @@ -759,7 +759,7 @@ ifndef CIBUILD
$(call user_confirm, 'Are you sure you want to push $(KICBASE_IMAGE_REGISTRIES) ?')
endif
./deploy/kicbase/build_auto_pause.sh $(KICBASE_ARCH)
docker buildx build -f ./deploy/kicbase/Dockerfile --platform $(KICBASE_ARCH) $(addprefix -t ,$(KICBASE_IMAGE_REGISTRIES)) --push --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) --build-arg PREBUILT_AUTO_PAUSE=true .
docker buildx build -f ./deploy/kicbase/Dockerfile --platform $(KICBASE_ARCH) $(addprefix -t ,$(KICBASE_IMAGE_REGISTRIES)) --push --build-arg VERSION_JSON=$(VERSION_JSON) --build-arg COMMIT_SHA=${VERSION}-$(COMMIT_NOQUOTES) --build-arg KUBERNETES_VERSION=$(KUBERNETES_VERSION) --build-arg PREBUILT_AUTO_PAUSE=true .

out/preload-tool:
go build -ldflags="$(MINIKUBE_LDFLAGS)" -o $@ ./hack/preload-images/*.go
Expand Down
29 changes: 12 additions & 17 deletions deploy/kicbase/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,10 @@ RUN if [ "$PREBUILT_AUTO_PAUSE" != "true" ]; then cd ./cmd/auto-pause/ && go bui
FROM ${UBUNTU_JAMMY_IMAGE} as kicbase

ARG BUILDKIT_VERSION="v0.15.2"
ARG CRIO_VERSION="1.24"
# cri-o depends on some kubernetes packages, the latest stable series
# is a reasonable choice.
ARG KUBERNETES_VERSION="v1.31.0"
ARG CRIO_VERSION="v1.30"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think we are ready to bump to the latest crio version yet (till we make a preload tarbal to include the critctl and other binaries)

ARG CRI_DOCKERD_VERSION="v0.3.15"
ARG CRI_DOCKERD_COMMIT="c1c566e0cc84abe6972f0bf857ecd8fe306258d9"
ARG CNI_PLUGINS_VERSION="v1.4.0"
Expand Down Expand Up @@ -187,30 +190,22 @@ RUN clean-install podman && \
echo "d /run/podman 0770 root podman" > /etc/tmpfiles.d/podman.conf && \
systemd-tmpfiles --create

# install cri-o dependencies:
RUN export ARCH=$(dpkg --print-architecture | sed 's/ppc64el/ppc64le/') && \
sh -c "echo 'deb https://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_22.04/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list" && \
curl -LO https://downloadcontent.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_22.04/Release.key && \
apt-key add - < Release.key && \
if [ "$ARCH" != "ppc64le" ]; then \
clean-install catatonit conmon cri-tools crun; \
else \
clean-install conmon crun; \
fi

# install containernetworking-plugins
RUN export ARCH=$(dpkg --print-architecture | sed 's/ppc64el/ppc64le/' | sed 's/armhf/arm/') && \
curl -LO "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-$ARCH-${CNI_PLUGINS_VERSION}.tgz" && \
mkdir -p /opt/cni/bin && \
tar -xf "cni-plugins-linux-$ARCH-${CNI_PLUGINS_VERSION}.tgz" -C /opt/cni/bin && \
rm "cni-plugins-linux-$ARCH-${CNI_PLUGINS_VERSION}.tgz"

# install cri-o based on https://github.com/cri-o/cri-o/blob/release-1.24/README.md#installing-cri-o
# install cri-o based on https://github.com/cri-o/packaging
RUN export ARCH=$(dpkg --print-architecture | sed 's/ppc64el/ppc64le/' | sed 's/armhf/arm-v7/') && \
if [ "$ARCH" != "ppc64le" ] && [ "$ARCH" != "arm-v7" ]; then sh -c "echo 'deb https://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/${CRIO_VERSION}/xUbuntu_22.04/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:${CRIO_VERSION}.list" && \
curl -LO https://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/${CRIO_VERSION}/xUbuntu_22.04/Release.key && \
apt-key add - < Release.key && \
clean-install cri-o cri-o-runc; fi
export KUBERNETES_RELEASE=$(echo $KUBERNETES_VERSION | grep -oE '[^.]+\.[^.]+') && \
if [ "$ARCH" != "ppc64le" ] && [ "$ARCH" != "arm-v7" ]; then \
sh -c "curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_RELEASE/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg" && \
sh -c "echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_RELEASE/deb/ /' > /etc/apt/sources.list.d/pkgs.k8s.io:core:stable.list" && \
sh -c "curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg" && \
sh -c "echo 'deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /' > /etc/apt/sources.list.d/pkgs.k8s.io:addons:cri-o:stable.list" && \
clean-install dbus cri-o cri-tools crun conmon catatonit; fi

# install NVIDIA container toolkit
RUN export ARCH=$(dpkg --print-architecture) && \
Expand Down
5 changes: 5 additions & 0 deletions hack/update/kubernetes_version/update_kubernetes_version.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,11 @@ var (
`kubernetesVersion:.*`: `kubernetesVersion: {{.LatestVersionP0}}`,
},
},
"deploy/kicbase/Dockerfile": {
Replace: map[string]string{
`ARG KUBERNETES_VERSION=.*`: `ARG KUBERNETES_VERSION="{{.StableVersion}}"`,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kicbase cannot have a single kubernetes version, if it is going to be reused for multiple versions.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it work for more cases if this kubernetes sources.list is then replaced by later tooling, when the specific version of k8s is installed?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The easiest would probably be to install the CRI adapter with the Kubernetes components (like cri-tools/cni-plugins/kubeadm/kubelet/kubectl) - possibly split it out into per container runtime, like the "preload" is now?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like CRI-O now uses an internal bundling of crun and conmon, so it doesn't use the system installation of those anymore,,, Otherwise such shared things could go with the OS installation still, if not k8s-version specific.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, the yak shaving continues (the path that led me here is quite convoluted already!), I'll check out the preload alternative.

Copy link
Collaborator

@afbjorklund afbjorklund Aug 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added CRI-O to the discuss items for today's meeting, since there was already some WIP PR for it

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Experimented with building static podman with Nix, similar to the cri-o packaging... (discussion)

},
},
}
)

Expand Down