Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry update with latest secret if plan still matches the applied plan #114

Merged

Conversation

Oats87
Copy link
Collaborator

@Oats87 Oats87 commented May 4, 2023

@Oats87
Copy link
Collaborator Author

Oats87 commented May 4, 2023

Tested through the integration framework for etcd restore on the etcd node:

May 04 19:36:47 test-node-zxb66 systemd[1]: Started Rancher System Agent.
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Rancher System Agent version 3460470 (3460470) is starting"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Starting remote watch of plans"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: E0504 19:36:47.644399     243 memcache.go:206] couldn't get resource list for management.cattle.io/v3:
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Starting /v1, Kind=Secret controller"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Detected first start, force-applying one-time instruction set"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.26.4-rke2r1 to directory /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 04 19:36:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:47Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.26.4-rke2r1"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/installer.sh"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/rke2.linux-amd64.tar.gz"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/sha256sum-amd64.txt"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/run.sh"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + mkdir -p /var/lib/rancher/rke2/system-agent-installer"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + check_target_mountpoint"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + mountpoint -q /usr/local"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + check_target_ro"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + touch /usr/local/.rke2-ro-test"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + test 0 -ne 0"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '' '!=' eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RESTART=true"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/sha256sum-amd64.txt"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/rke2.linux-amd64.tar.gz"
May 04 19:36:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:49Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  verifying tarball"
May 04 19:36:50 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:50Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ env"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ grep '^RKE2_'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ true"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_ENV="
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n '' ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ env"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ true"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + PROXY_ENV_INFO="
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n '' ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ awk '{print $1}'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -z '' ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + INSTALL_RKE2_TYPE=server"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + echo eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl daemon-reload"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '' = true ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' server = server ']'"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl is-active --quiet rke2-agent"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl enable rke2-server"
May 04 19:36:51 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:51Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service."
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '' = true ']'"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' true = true ']'"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl --no-block restart rke2-server"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-193647/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[Applyinator] Running command: sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs']"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:                                  Dload  Upload   Total   Spent    Left  Speed"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: \r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: curl: (7) Failed to connect to localhost port 9345 after 0 ms: Connection refused"
May 04 19:36:52 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:52Z" level=info msg="[Applyinator] Command sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs'] finished with err: <nil> and exit code: 1"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-193653/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_1"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] Running command: sh [-c cat /var/lib/rancher/rke2/server/db/etcd/name]"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_1:stdout]: test-node-zxb66-018c81b7"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] Command sh [-c cat /var/lib/rancher/rke2/server/db/etcd/name] finished with err: <nil> and exit code: 0"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-193653/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_2"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_2:stdout]: Name Location Size Created"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (4236 vs 4241)"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=error msg="error syncing 'test-ns-dv2rv/custom-bbc71888a47c-machine-plan': handler secret-watch: secret received was too old, requeuing"
May 04 19:36:53 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:53Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:36:58 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:36:58Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:37:03 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:03Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:37:08 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:08Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:37:13 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:13Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:37:18 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:18Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 04 19:37:18 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:18Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-193718/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0"
May 04 19:37:18 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:18Z" level=info msg="[Applyinator] Running command: sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs']"
May 04 19:37:18 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:18Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current"
May 04 19:37:18 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:18Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:                                  Dload  Upload   Total   Spent    Left  Speed"
May 04 19:37:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:20Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: \r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r100   130  100   130    0     0     64      0  0:00:02  0:00:02 --:--:--    64\r100   130  100   130    0     0     64      0  0:00:02  0:00:02 --:--:--    64"
May 04 19:37:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:20Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: {\"members\":[{\"name\":\"test-node-zxb66-018c81b7\",\"peerURLs\":[\"https://10.42.0.15:2380\"],\"clientURLs\":[\"https://10.42.0.15:2379\"]}]}"
May 04 19:37:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:20Z" level=info msg="[Applyinator] Command sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs'] finished with err: <nil> and exit code: 0"
May 04 19:37:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:20Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:37:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:37:25Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"


May 04 19:40:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:20Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29"
May 04 19:40:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:20Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.26.4-rke2r1 to directory /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0"
May 04 19:40:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:20Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 04 19:40:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:20Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.26.4-rke2r1"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/installer.sh"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/rke2.linux-amd64.tar.gz"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/sha256sum-amd64.txt"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/run.sh"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + check_target_mountpoint"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + mountpoint -q /usr/local"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + check_target_ro"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + touch /usr/local/.rke2-ro-test"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + test 0 -ne 0"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + PRIOR_RESTART_STAMP=eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -n '' ']'"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RESTART=false"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/sha256sum-amd64.txt"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0/rke2.linux-amd64.tar.gz"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stdout]: [INFO]  verifying tarball"
May 04 19:40:21 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:21Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ awk '{print $1}'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ grep '^RKE2_'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ true"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + RKE2_ENV="
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -n '' ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ true"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + PROXY_ENV_INFO="
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -n '' ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: ++ awk '{print $1}'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -z '' ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + INSTALL_RKE2_TYPE=server"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' -n '' ']'"
May 04 19:40:22 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:22Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + systemctl daemon-reload"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' '' = true ']'"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' server = server ']'"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + systemctl is-active --quiet rke2-agent"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + systemctl enable rke2-server"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + '[' true = true ']'"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_0:stderr]: + exit 0"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194020/9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[Applyinator] Running command: rke2 [etcd-snapshot save]"
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --agent-token found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --cluster-cidr found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --cni found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --disable-apiserver found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --disable-controller-manager found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --disable-scheduler found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --node-label found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --node-label found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --node-taint found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --private-registry found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --service-cidr found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=warning msg=\"Unknown flag --token found in config.yaml, skipping\\n\""
May 04 19:40:23 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:23Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:23Z\" level=info msg=\"Managed etcd cluster bootstrap already complete and initialized\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Applying CRD addons.k3s.cattle.io\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Applying CRD helmcharts.helm.cattle.io\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Applying CRD helmchartconfigs.helm.cattle.io\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-test-node-zxb66-1683229225\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.719Z\",\"caller\":\"snapshot/v3_snapshot.go:65\",\"msg\":\"created temporary db file\",\"path\":\"/var/lib/rancher/rke2/server/db/snapshots/on-demand-test-node-zxb66-1683229225.part\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.722Z\",\"logger\":\"client\",\"caller\":\"[email protected]/maintenance.go:212\",\"msg\":\"opened snapshot stream; downloading\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.723Z\",\"caller\":\"snapshot/v3_snapshot.go:73\",\"msg\":\"fetching snapshot\",\"endpoint\":\"https://127.0.0.1:2379\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.852Z\",\"logger\":\"client\",\"caller\":\"[email protected]/maintenance.go:220\",\"msg\":\"completed snapshot read; closing\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.911Z\",\"caller\":\"snapshot/v3_snapshot.go:88\",\"msg\":\"fetched snapshot\",\"endpoint\":\"https://127.0.0.1:2379\",\"size\":\"14 MB\",\"took\":\"now\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:40:25.912Z\",\"caller\":\"snapshot/v3_snapshot.go:97\",\"msg\":\"saved\",\"path\":\"/var/lib/rancher/rke2/server/db/snapshots/on-demand-test-node-zxb66-1683229225\"}"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[9c5fccafd1b123ff91e8a70fc6e891ee5d4934c0bff542f1c1edb1b616671f29_1:stderr]: time=\"2023-05-04T19:40:25Z\" level=info msg=\"Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete\""
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[Applyinator] Command rke2 [etcd-snapshot save] finished with err: <nil> and exit code: 0"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (5945 vs 6030)"
May 04 19:40:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:25Z" level=error msg="error syncing 'test-ns-dv2rv/custom-bbc71888a47c-machine-plan': handler secret-watch: secret received was too old, requeuing"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.26.4-rke2r1 to directory /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.26.4-rke2r1"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/installer.sh"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/rke2.linux-amd64.tar.gz"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/sha256sum-amd64.txt"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/run.sh"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + check_target_mountpoint"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + mountpoint -q /usr/local"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + check_target_ro"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + touch /usr/local/.rke2-ro-test"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + test 0 -ne 0"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + PRIOR_RESTART_STAMP=eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a '!=' eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RESTART=false"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/sha256sum-amd64.txt"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0/rke2.linux-amd64.tar.gz"
May 04 19:40:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:26Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  verifying tarball"
May 04 19:40:27 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:27Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ awk '{print $1}'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ grep '^RKE2_'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ true"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + RKE2_ENV="
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n '' ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ true"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + PROXY_ENV_INFO="
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n '' ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: ++ awk '{print $1}'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -z '' ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + INSTALL_RKE2_TYPE=server"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' -n eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + echo eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl daemon-reload"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '' = true ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' server = server ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl is-active --quiet rke2-agent"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + systemctl enable rke2-server"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' '' = true ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: + '[' false = true ']'"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Running command: sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs']"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]:                                  Dload  Upload   Total   Spent    Left  Speed"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stderr]: \r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   156  100   156    0     0  23944      0 --:--:-- --:--:-- --:--:-- 26000"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_0:stdout]: {\"members\":[{\"ID\":13677020063830845281,\"name\":\"test-node-zxb66-018c81b7\",\"peerURLs\":[\"https://10.42.0.15:2380\"],\"clientURLs\":[\"https://10.42.0.15:2379\"]}]}"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Command sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs'] finished with err: <nil> and exit code: 0"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_1"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Running command: sh [-c cat /var/lib/rancher/rke2/server/db/etcd/name]"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_1:stdout]: test-node-zxb66-018c81b7"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Command sh [-c cat /var/lib/rancher/rke2/server/db/etcd/name] finished with err: <nil> and exit code: 0"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194026/9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_2"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_2:stdout]: Name                                 Location                                                                              Size     Created"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[9a7d0963b287e58a6e80a9695361253f99cbc652bf68fb23efd42745e29c4f11_2:stdout]: on-demand-test-node-zxb66-1683229225 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-test-node-zxb66-1683229225 13537312 2023-05-04T19:40:25Z"
May 04 19:40:28 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:28Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
May 04 19:40:29 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:29Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194031/216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[Applyinator] Running command: /bin/sh [-c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi]"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + systemctl stop rke2-server.service"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + systemctl stop rke2-agent.service"
May 04 19:40:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:31Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + killtree 558 701 756 998 1144 1893 3062"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + kill -9 558 580 609 701 722 776 756 785 840 998 1019 1048 1144 1166 1389 1475 1482 1501 1509 1505 1487 1476 1483 1488 1477 1481 1500 1507 1503 1485 1478 1480 1504 1510 1508 1486 1479 1484 1499 1506 1502 1489 1893 1918 1973 3062"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + do_unmount_and_remove /run/k3s"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + umount /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f950367e4a60d411ab58d865398792a65c7c86ccbcc476b33cc3e8ca9baa254a/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cf0a70c50e4b903c27b72e7771e17bb4c29bf72632f524d2cd86ad051b59b54a/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9f2dd320ae13caae1de3ed1752c201870fba040050d61aa47e0e55eaf1c54213/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/979d93b7f9296a56c82e4bb447db9de3a5b469a95c3dd4392e9b231064927e80/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8b929e81e4922e515686bca43859a1f03e13793761d567272af2be3ad414dbf3/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/702e9b70e0e6ef77ec7fd3c3d7dcb8bbe43b9f949895cd40d11b880dff969aaf/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/60274a0a9a4be92a587286f5d0f7937551481ae61946decebd025b8b01e08dbb/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5c468ca6dff5e8efe5bd3f82022c4f316c74dfb9adbb5b384337d3f72e8b6be4/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/557d1a36c53f4bbdea498abc49305e7d7dfb7f0d61cacbe68d163dc0f0123b52/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/291552db4875879b3bdd7a1353efb4d218c76660b0537b85effd3c0d8c1abe84/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/04a6b2f142f9f4475b1f0a2759020d82143d89df14f90f2a5444da5d3c076eb9/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/036297385957591597f20a1e1e131c0c1415d49ffcb307837043642cb8676d99/rootfs /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/f950367e4a60d411ab58d865398792a65c7c86ccbcc476b33cc3e8ca9baa254a/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/cf0a70c50e4b903c27b72e7771e17bb4c29bf72632f524d2cd86ad051b59b54a/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9f2dd320ae13caae1de3ed1752c201870fba040050d61aa47e0e55eaf1c54213/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/5c468ca6dff5e8efe5bd3f82022c4f316c74dfb9adbb5b384337d3f72e8b6be4/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/291552db4875879b3bdd7a1353efb4d218c76660b0537b85effd3c0d8c1abe84/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/036297385957591597f20a1e1e131c0c1415d49ffcb307837043642cb8676d99/shm"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + rm -rf --one-file-system /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f950367e4a60d411ab58d865398792a65c7c86ccbcc476b33cc3e8ca9baa254a/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cf0a70c50e4b903c27b72e7771e17bb4c29bf72632f524d2cd86ad051b59b54a/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9f2dd320ae13caae1de3ed1752c201870fba040050d61aa47e0e55eaf1c54213/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/979d93b7f9296a56c82e4bb447db9de3a5b469a95c3dd4392e9b231064927e80/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8b929e81e4922e515686bca43859a1f03e13793761d567272af2be3ad414dbf3/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/702e9b70e0e6ef77ec7fd3c3d7dcb8bbe43b9f949895cd40d11b880dff969aaf/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/60274a0a9a4be92a587286f5d0f7937551481ae61946decebd025b8b01e08dbb/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5c468ca6dff5e8efe5bd3f82022c4f316c74dfb9adbb5b384337d3f72e8b6be4/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/557d1a36c53f4bbdea498abc49305e7d7dfb7f0d61cacbe68d163dc0f0123b52/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/291552db4875879b3bdd7a1353efb4d218c76660b0537b85effd3c0d8c1abe84/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/04a6b2f142f9f4475b1f0a2759020d82143d89df14f90f2a5444da5d3c076eb9/rootfs /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/036297385957591597f20a1e1e131c0c1415d49ffcb307837043642cb8676d99/rootfs /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/f950367e4a60d411ab58d865398792a65c7c86ccbcc476b33cc3e8ca9baa254a/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/cf0a70c50e4b903c27b72e7771e17bb4c29bf72632f524d2cd86ad051b59b54a/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9f2dd320ae13caae1de3ed1752c201870fba040050d61aa47e0e55eaf1c54213/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/5c468ca6dff5e8efe5bd3f82022c4f316c74dfb9adbb5b384337d3f72e8b6be4/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/291552db4875879b3bdd7a1353efb4d218c76660b0537b85effd3c0d8c1abe84/shm /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/036297385957591597f20a1e1e131c0c1415d49ffcb307837043642cb8676d99/shm"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + do_unmount_and_remove /var/lib/rancher/rke2"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + do_unmount_and_remove /var/lib/kubelet/pods"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + umount /var/lib/kubelet/pods/6f9dd7a9-53f6-4b15-867e-38d102a2b612/volumes/kubernetes.io~projected/kube-api-access-c6vrq /var/lib/kubelet/pods/49818c1f-5bed-4733-b068-de061a09daf6/volumes/kubernetes.io~secret/node-certs /var/lib/kubelet/pods/49818c1f-5bed-4733-b068-de061a09daf6/volumes/kubernetes.io~projected/kube-api-access-tx2gl /var/lib/kubelet/pods/3001dd10-736d-4a3e-a915-aae218c4a82b/volumes/kubernetes.io~projected/kube-api-access-t6cgv /var/lib/kubelet/pods/0eeab72b-7de7-49ee-a625-53ff8862bc5e/volumes/kubernetes.io~projected/kube-api-access-klxq5 /var/lib/kubelet/pods/0eeab72b-7de7-49ee-a625-53ff8862bc5e/volumes/kubernetes.io~downward-api/pod-info"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + rm -rf --one-file-system /var/lib/kubelet/pods/6f9dd7a9-53f6-4b15-867e-38d102a2b612/volumes/kubernetes.io~projected/kube-api-access-c6vrq /var/lib/kubelet/pods/49818c1f-5bed-4733-b068-de061a09daf6/volumes/kubernetes.io~secret/node-certs /var/lib/kubelet/pods/49818c1f-5bed-4733-b068-de061a09daf6/volumes/kubernetes.io~projected/kube-api-access-tx2gl /var/lib/kubelet/pods/3001dd10-736d-4a3e-a915-aae218c4a82b/volumes/kubernetes.io~projected/kube-api-access-t6cgv /var/lib/kubelet/pods/0eeab72b-7de7-49ee-a625-53ff8862bc5e/volumes/kubernetes.io~projected/kube-api-access-klxq5 /var/lib/kubelet/pods/0eeab72b-7de7-49ee-a625-53ff8862bc5e/volumes/kubernetes.io~downward-api/pod-info"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + do_unmount_and_remove /run/netns/cni-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + umount /run/netns/cni-baac84d0-fcef-f822-b592-5d8bd6243092"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + rm -rf --one-file-system /run/netns/cni-baac84d0-fcef-f822-b592-5d8bd6243092"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link show"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep 'master cni0'"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + read ignore iface ignore"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete cni0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"cni0\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete flannel.1"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"flannel.1\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete flannel-v6.1"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"flannel-v6.1\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete flannel-wg"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"flannel-wg\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete flannel-wg-v6"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"flannel-wg-v6\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete vxlan.calico"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete vxlan-v6.calico"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"vxlan-v6.calico\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete cilium_vxlan"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"cilium_vxlan\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete cilium_net"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"cilium_net\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip link delete kube-ipvs0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: Cannot find device \"kube-ipvs0\""
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + '[' -d /sys/class/net/nodelocaldns ']'"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + rm -rf /var/lib/cni/ /var/log/pods/ /var/log/containers"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + iptables-save"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v KUBE-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v CNI-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v cali-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v cali:"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v CILIUM_"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v flannel"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + iptables-restore"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip6tables-save"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v KUBE-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v CNI-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v cali-"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v cali:"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v CILIUM_"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + grep -v flannel"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + ip6tables-restore"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stderr]: + set +x"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stdout]: If this cluster was upgraded from an older release of the Canal CNI, you may need to manually remove some flannel iptables rules:"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stdout]: \texport cluster_cidr=YOUR-CLUSTER-CIDR"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stdout]: \tiptables -D POSTROUTING -s $cluster_cidr -j MASQUERADE --random-fully"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_0:stdout]: \tiptables -D POSTROUTING ! -s $cluster_cidr -d  -j MASQUERADE --random-fully"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Command /bin/sh [-c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi] finished with err: <nil> and exit code: 0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194031/216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_1"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Running command: rm [-rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml]"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Command rm [-rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml] finished with err: <nil> and exit code: 0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194031/216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_2"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Running command: touch [/var/lib/rancher/rke2/server/db/etcd/tombstone]"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Command touch [/var/lib/rancher/rke2/server/db/etcd/tombstone] finished with err: <nil> and exit code: 0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194031/216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_3"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Running command: rm [-rf /var/lib/rancher/rke2/server/tls]"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Command rm [-rf /var/lib/rancher/rke2/server/tls] finished with err: <nil> and exit code: 0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194031/216dff95266e6a46a82b7c4a9bc9c73948b6649b365874d68f5d7a0003cf12ea_4"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Running command: rm [-rf /var/lib/rancher/rke2/server/cred]"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[Applyinator] Command rm [-rf /var/lib/rancher/rke2/server/cred] finished with err: <nil> and exit code: 0"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (6139 vs 6188)"
May 04 19:40:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:35Z" level=error msg="error syncing 'test-ns-dv2rv/custom-bbc71888a47c-machine-plan': handler secret-watch: secret received was too old, requeuing"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Running command: /bin/sh [-c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi]"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + systemctl stop rke2-server.service"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + systemctl stop rke2-agent.service"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + killtree"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + kill -9"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + do_unmount_and_remove /run/k3s"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + do_unmount_and_remove /var/lib/rancher/rke2"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + do_unmount_and_remove /var/lib/kubelet/pods"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + do_unmount_and_remove /run/netns/cni-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link show"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep 'master cni0'"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + read ignore iface ignore"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete cni0"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"cni0\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete flannel.1"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"flannel.1\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete flannel-v6.1"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"flannel-v6.1\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete flannel-wg"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"flannel-wg\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete flannel-wg-v6"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"flannel-wg-v6\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete vxlan.calico"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"vxlan.calico\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete vxlan-v6.calico"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"vxlan-v6.calico\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete cilium_vxlan"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"cilium_vxlan\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete cilium_net"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"cilium_net\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip link delete kube-ipvs0"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: Cannot find device \"kube-ipvs0\""
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + '[' -d /sys/class/net/nodelocaldns ']'"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + rm -rf /var/lib/cni/ /var/log/pods/ /var/log/containers"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + iptables-save"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v KUBE-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v CNI-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v cali-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v cali:"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v CILIUM_"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v flannel"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + iptables-restore"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip6tables-save"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v KUBE-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v CNI-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v cali-"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v cali:"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v CILIUM_"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + grep -v flannel"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + ip6tables-restore"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stderr]: + set +x"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stdout]: If this cluster was upgraded from an older release of the Canal CNI, you may need to manually remove some flannel iptables rules:"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stdout]: \texport cluster_cidr=YOUR-CLUSTER-CIDR"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stdout]: \tiptables -D POSTROUTING -s $cluster_cidr -j MASQUERADE --random-fully"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_0:stdout]: \tiptables -D POSTROUTING ! -s $cluster_cidr -d  -j MASQUERADE --random-fully"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Command /bin/sh [-c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi] finished with err: <nil> and exit code: 0"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_1"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Running command: rm [-rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml]"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Command rm [-rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml] finished with err: <nil> and exit code: 0"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.26.4-rke2r1 to directory /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 04 19:40:45 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:45Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.26.4-rke2r1"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/installer.sh"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/rke2.linux-amd64.tar.gz"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/sha256sum-amd64.txt"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/run.sh"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + SA_INSTALL_PREFIX=/usr/local"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + mkdir -p /var/lib/rancher/rke2"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + check_target_mountpoint"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + mountpoint -q /usr/local"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + check_target_ro"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + touch /usr/local/.rke2-ro-test"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + rm -rf /usr/local/.rke2-ro-test"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + test 0 -ne 0"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' /usr/local = /opt/rke2 ']'"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + PRIOR_RESTART_STAMP=eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -n '' ']'"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RESTART=false"
May 04 19:40:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:46Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
May 04 19:40:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:47Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/sha256sum-amd64.txt"
May 04 19:40:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:47Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2/rke2.linux-amd64.tar.gz"
May 04 19:40:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:47Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stdout]: [INFO]  verifying tarball"
May 04 19:40:47 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:47Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stdout]: [INFO]  unpacking tarball file to /usr/local"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ awk '{print $1}'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ grep '^RKE2_'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ true"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + RKE2_ENV="
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -n '' ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ true"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + PROXY_ENV_INFO="
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -n '' ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: ++ awk '{print $1}'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -z '' ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + INSTALL_RKE2_TYPE=server"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' -n '' ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + systemctl daemon-reload"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' '' = true ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' server = server ']'"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + systemctl is-active --quiet rke2-agent"
May 04 19:40:48 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:48Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + systemctl enable rke2-server"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + '[' true = true ']'"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_2:stderr]: + exit 0"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_3"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] Running command: rm [-rf /var/lib/rancher/rke2/server/db/etcd]"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] Command rm [-rf /var/lib/rancher/rke2/server/db/etcd] finished with err: <nil> and exit code: 0"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194045/9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[Applyinator] Running command: rke2 [server --cluster-reset --etcd-arg=advertise-client-urls=https://127.0.0.1:2379 --cluster-reset-restore-path=db/snapshots/on-demand-test-node-zxb66-1683229225 --etcd-s3=false]"
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:49Z\" level=warning msg=\"not running in CIS mode\""
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:49Z\" level=info msg=\"Applying Pod Security Admission Configuration\""
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:49Z\" level=info msg=\"Static pod cleanup in progress\""
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:49Z\" level=info msg=\"Logging temporary containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log\""
May 04 19:40:49 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:49Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:49Z\" level=info msg=\"Running temporary containerd /var/lib/rancher/rke2/bin/containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd\""
May 04 19:40:59 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:40:59Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:40:59Z\" level=info msg=\"Removing pod etcd-test-node-zxb66\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Static pod cleanup completed successfully\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Starting rke2 v1.26.4+rke2r1 (e65f1c66df111d0d38b61da860653e761f6155b1)\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Managed etcd cluster initializing\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"generated self-signed CA certificate CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09.222119125 +0000 UTC notAfter=2033-05-01 19:41:09.222119125 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:admin,O=system:masters signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:kube-controller-manager signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:kube-scheduler signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:apiserver,O=system:masters signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:kube-proxy signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:rke2-controller signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=rke2-cloud-controller-manager signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"generated self-signed CA certificate CN=rke2-server-ca@1683229269: notBefore=2023-05-04 19:41:09.229604546 +0000 UTC notAfter=2033-05-01 19:41:09.229604546 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=kube-apiserver signed by CN=rke2-server-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"generated self-signed CA certificate CN=rke2-request-header-ca@1683229269: notBefore=2023-05-04 19:41:09.231550586 +0000 UTC notAfter=2033-05-01 19:41:09.231550586 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:auth-proxy signed by CN=rke2-request-header-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"generated self-signed CA certificate CN=etcd-server-ca@1683229269: notBefore=2023-05-04 19:41:09.233131409 +0000 UTC notAfter=2033-05-01 19:41:09.233131409 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=etcd-server signed by CN=etcd-server-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=etcd-client signed by CN=etcd-server-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"generated self-signed CA certificate CN=etcd-peer-ca@1683229269: notBefore=2023-05-04 19:41:09.235464946 +0000 UTC notAfter=2033-05-01 19:41:09.235464946 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=etcd-peer signed by CN=etcd-peer-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Pre-restore etcd database moved to /var/lib/rancher/rke2/server/db/etcd-old-1683229269\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:41:09.313Z\",\"caller\":\"snapshot/v3_snapshot.go:248\",\"msg\":\"restoring snapshot\",\"path\":\"db/snapshots/on-demand-test-node-zxb66-1683229225\",\"wal-dir\":\"/var/lib/rancher/rke2/server/db/etcd/member/wal\",\"data-dir\":\"/var/lib/rancher/rke2/server/db/etcd\",\"snap-dir\":\"/var/lib/rancher/rke2/server/db/etcd/member/snap\",\"stack\":\"go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\\n\\t/go/pkg/mod/github.com/k3s-io/etcd/etcdutl/[email protected]/snapshot/v3_snapshot.go:254\\ngithub.com/k3s-io/k3s/pkg/etcd.(*ETCD).Restore\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/etcd/etcd.go:2016\\ngithub.com/k3s-io/k3s/pkg/etcd.(*ETCD).Reset\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/etcd/etcd.go:365\\ngithub.com/k3s-io/k3s/pkg/cluster.(*Cluster).start\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cluster/managed.go:68\\ngithub.com/k3s-io/k3s/pkg/cluster.(*Cluster).Start\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cluster/cluster.go:75\\ngithub.com/k3s-io/k3s/pkg/daemons/control.prepare\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/daemons/control/server.go:283\\ngithub.com/k3s-io/k3s/pkg/daemons/control.Server\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/daemons/control/server.go:34\\ngithub.com/k3s-io/k3s/pkg/server.StartServer\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/server/server.go:59\\ngithub.com/k3s-io/k3s/pkg/cli/server.run\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cli/server/server.go:448\\ngithub.com/k3s-io/k3s/pkg/cli/server.RunWithControllers\\n\\t/go/pkg/mod/github.com/k3s-io/[email protected]/pkg/cli/server/server.go:46\\ngithub.com/rancher/rke2/pkg/rke2.Server\\n\\t/source/pkg/rke2/rke2.go:120\\ngithub.com/rancher/rke2/pkg/cli/cmds.ServerRun\\n\\t/source/pkg/cli/cmds/server.go:159\\ngithub.com/urfave/cli.HandleAction\\n\\t/go/pkg/mod/github.com/urfave/[email protected]/app.go:524\\ngithub.com/urfave/cli.Command.Run\\n\\t/go/pkg/mod/github.com/urfave/[email protected]/command.go:173\\ngithub.com/urfave/cli.(*App).Run\\n\\t/go/pkg/mod/github.com/urfave/[email protected]/app.go:277\\nmain.main\\n\\t/source/main.go:23\\nruntime.main\\n\\t/usr/local/go/src/runtime/proc.go:250\"}"
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=warning msg=\"dynamiclistener [::]:9345: no cached certificate available for preload - deferring certificate load until storage initialization or first client request\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Active TLS secret / (ver=) (count 10): map[listener.cattle.io/cn-10.42.0.15:10.42.0.15 listener.cattle.io/cn-10.45.0.1:10.45.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-test-node-zxb66:test-node-zxb66 listener.cattle.io/fingerprint:SHA1=B82D2EF002F0E7B30C3FA87EA1F5C61E97718A1F]\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:41:09.445Z\",\"caller\":\"membership/store.go:141\",\"msg\":\"Trimming membership information from the backend...\"}"
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:41:09.455Z\",\"caller\":\"membership/cluster.go:421\",\"msg\":\"added member\",\"cluster-id\":\"6a1531940fcac936\",\"local-member-id\":\"0\",\"added-peer-id\":\"86d1899113967526\",\"added-peer-peer-urls\":[\"https://10.42.0.15:2380\"]}"
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:41:09.477Z\",\"caller\":\"snapshot/v3_snapshot.go:269\",\"msg\":\"restored snapshot\",\"path\":\"db/snapshots/on-demand-test-node-zxb66-1683229225\",\"wal-dir\":\"/var/lib/rancher/rke2/server/db/etcd/member/wal\",\"data-dir\":\"/var/lib/rancher/rke2/server/db/etcd\",\"snap-dir\":\"/var/lib/rancher/rke2/server/db/etcd/member/snap\"}"
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Starting etcd for new cluster\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Tunnel server egress proxy mode: agent\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Tunnel server egress proxy waiting for runtime core to become available\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Server node token is available at /var/lib/rancher/rke2/server/token\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\\\"\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"To join server node to cluster: rke2 server -s https://10.42.0.15:9345 -t ${SERVER_NODE_TOKEN}\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Agent node token is available at /var/lib/rancher/rke2/server/agent-token\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"To join agent node to cluster: rke2 agent -s https://10.42.0.15:9345 -t ${AGENT_NODE_TOKEN}\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Wrote kubeconfig /etc/rancher/rke2/rke2.yaml\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Run: rke2 kubectl\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=warning msg=\"remove /var/lib/rancher/rke2/agent/etc/rke2-agent-load-balancer.json: no such file or directory\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Adding server to load balancer rke2-agent-load-balancer: 127.0.0.1:9345\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> [127.0.0.1:9345] [default: 127.0.0.1:9345]\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Running load balancer rke2-api-server-agent-load-balancer 127.0.0.1:6443 -> [] [default: ]\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=test-node-zxb66 signed by CN=rke2-server-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"certificate CN=system:node:test-node-zxb66,O=system:nodes signed by CN=rke2-client-ca@1683229269: notBefore=2023-05-04 19:41:09 +0000 UTC notAfter=2024-05-03 19:41:09 +0000 UTC\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Module overlay was already loaded\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Module nf_conntrack was already loaded\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Module br_netfilter was already loaded\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Module iptable_nat was already loaded\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Module iptable_filter was already loaded\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Set sysctl 'net/bridge/bridge-nf-call-iptables' to 1\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=error msg=\"Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Runtime image index.docker.io/rancher/rke2-runtime:v1.26.4-rke2r1 bin and charts directories already exist; skipping extract\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"No cluster configuration value changes necessary for manifest /var/lib/rancher/rke2/server/manifests/rancher/managed-chart-config.yaml\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Using private registry config file at /etc/rancher/rke2/registries.yaml\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log\""
May 04 19:41:09 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:09Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:09Z\" level=info msg=\"Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"containerd is now running\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 1.1336ms\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 603.135µs\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Pulling images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Imported images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt in 664.503µs\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=warning msg=\"Disabling CPU quotas due to missing cpu controller or cpu.cfs_period_us\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cluster-dns=10.45.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=test-node-zxb66 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/rke2 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=20483754-0f93-461d-b8fa-32f6fc728493 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --register-with-taints=node-role.kubernetes.io/etcd:NoExecute --resolv-conf=/etc/resolv.conf --runtime-cgroups=/rke2 --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key\""
May 04 19:41:10 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:10Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:10Z\" level=info msg=\"Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error\""
May 04 19:41:14 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:14Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:14Z\" level=info msg=\"Tunnel server egress proxy waiting for runtime core to become available\""
May 04 19:41:14 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:14Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"warn\",\"ts\":\"2023-05-04T19:41:14.488Z\",\"logger\":\"etcd-client\",\"caller\":\"[email protected]/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc001020380/127.0.0.1:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \\\"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\\\"\"}"
May 04 19:41:14 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:14Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:14Z\" level=warning msg=\"Failed to get apiserver address from etcd: context deadline exceeded\""
May 04 19:41:15 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:15Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:15Z\" level=info msg=\"Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error\""
May 04 19:41:19 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:19Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:19Z\" level=info msg=\"Tunnel server egress proxy waiting for runtime core to become available\""
May 04 19:41:19 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:19Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"warn\",\"ts\":\"2023-05-04T19:41:19.489Z\",\"logger\":\"etcd-client\",\"caller\":\"[email protected]/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc001020700/127.0.0.1:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \\\"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\\\"\"}"
May 04 19:41:19 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:19Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:19Z\" level=warning msg=\"Failed to get apiserver address from etcd: context deadline exceeded\""
May 04 19:41:20 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:20Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:20Z\" level=info msg=\"Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error\""
May 04 19:41:24 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:24Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"warn\",\"ts\":\"2023-05-04T19:41:24.312Z\",\"logger\":\"etcd-client\",\"caller\":\"[email protected]/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc0010201c0/127.0.0.1:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \\\"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\\\"\"}"
May 04 19:41:24 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:24Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"info\",\"ts\":\"2023-05-04T19:41:24.313Z\",\"logger\":\"etcd-client\",\"caller\":\"[email protected]/client.go:210\",\"msg\":\"Auto sync endpoints failed.\",\"error\":\"context deadline exceeded\"}"
May 04 19:41:24 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:24Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:24Z\" level=info msg=\"Tunnel server egress proxy waiting for runtime core to become available\""
May 04 19:41:24 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:24Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: {\"level\":\"warn\",\"ts\":\"2023-05-04T19:41:24.490Z\",\"logger\":\"etcd-client\",\"caller\":\"[email protected]/retry_interceptor.go:62\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"etcd-endpoints://0xc001020380/127.0.0.1:2379\",\"attempt\":0,\"error\":\"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \\\"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\\\"\"}"
May 04 19:41:24 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:24Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:24Z\" level=warning msg=\"Failed to get apiserver address from etcd: context deadline exceeded\""
May 04 19:41:25 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:25Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:25Z\" level=info msg=\"Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"Defragmenting etcd database\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"Reconciling bootstrap data between datastore and disk\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"Cluster reset: backing up certificates directory to /var/lib/rancher/rke2/server/tls-1683229286\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=warning msg=\"Updating bootstrap data on disk from datastore\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:admin,O=system:masters signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:kube-controller-manager signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:kube-scheduler signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:apiserver,O=system:masters signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:kube-proxy signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:rke2-controller signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=rke2-cloud-controller-manager signed by CN=rke2-client-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=kube-apiserver signed by CN=rke2-server-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=system:auth-proxy signed by CN=rke2-request-header-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=etcd-server signed by CN=etcd-server-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=etcd-client signed by CN=etcd-server-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:26 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:26Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:26Z\" level=info msg=\"certificate CN=etcd-peer signed by CN=etcd-peer-ca@1683229012: notBefore=2023-05-04 19:36:52 +0000 UTC notAfter=2024-05-03 19:41:26 +0000 UTC\""
May 04 19:41:29 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:29Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:29Z\" level=info msg=\"Tunnel server egress proxy waiting for runtime core to become available\""
May 04 19:41:29 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:29Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:29Z\" level=info msg=\"Container for etcd is running\""
May 04 19:41:30 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:30Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:30Z\" level=info msg=\"Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error\""
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="[9b9dcb3f303c28998797078875de56bea97fc2b11a2e1cd75c84a10a4ce76d63_4:stderr]: time=\"2023-05-04T19:41:31Z\" level=info msg=\"Managed etcd cluster membership has been reset, restart without --cluster-reset flag now. Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes\""
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="[Applyinator] Command rke2 [server --cluster-reset --etcd-arg=advertise-client-urls=https://127.0.0.1:2379 --cluster-reset-restore-path=db/snapshots/on-demand-test-node-zxb66-1683229225 --etcd-s3=false] finished with err: <nil> and exit code: 0"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (6261 vs 6528)"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=error msg="error syncing 'test-ns-dv2rv/custom-bbc71888a47c-machine-plan': handler secret-watch: secret received was too old, requeuing"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.26.4-rke2r1 to directory /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 04 19:41:31 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:31Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.26.4-rke2r1"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/installer.sh"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/rke2.linux-amd64.tar.gz"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/sha256sum-amd64.txt"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/run.sh"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + check_target_mountpoint"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + mountpoint -q /usr/local"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + check_target_ro"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + touch /usr/local/.rke2-ro-test"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + test 0 -ne 0"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + PRIOR_RESTART_STAMP=eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -n bec037c8ffad5d035a5c9e740f0ef961201c38ea74f4fb2f81aecaeace84ac76 ']'"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' eea6dcef3ecfaf8c520281da7705949f57c8bb6d995476469dfd7b2efab7254a '!=' bec037c8ffad5d035a5c9e740f0ef961201c38ea74f4fb2f81aecaeace84ac76 ']'"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RESTART=true"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/sha256sum-amd64.txt"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0/rke2.linux-amd64.tar.gz"
May 04 19:41:32 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:32Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stdout]: [INFO]  verifying tarball"
May 04 19:41:33 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:33Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ awk '{print $1}'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ grep '^RKE2_'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ true"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + RKE2_ENV="
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -n '' ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ true"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + PROXY_ENV_INFO="
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -n '' ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: ++ awk '{print $1}'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -z '' ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + INSTALL_RKE2_TYPE=server"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' -n bec037c8ffad5d035a5c9e740f0ef961201c38ea74f4fb2f81aecaeace84ac76 ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + echo bec037c8ffad5d035a5c9e740f0ef961201c38ea74f4fb2f81aecaeace84ac76"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + systemctl daemon-reload"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' '' = true ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' server = server ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + systemctl is-active --quiet rke2-agent"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + systemctl enable rke2-server"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' '' = true ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + '[' true = true ']'"
May 04 19:41:34 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:34Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: + systemctl --no-block restart rke2-server"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230504-194131/7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[Applyinator] Running command: sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs']"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]:                                  Dload  Upload   Total   Spent    Left  Speed"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: \r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[7530fe7cb25aec0b6b911d26292682916f435cd59a19e88433e608129509ee1b_0:stderr]: curl: (7) Failed to connect to localhost port 9345 after 0 ms: Connection refused"
May 04 19:41:35 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:35Z" level=info msg="[Applyinator] Command sh [-c curl -f --retry 100 --retry-delay 5 --cacert /var/lib/rancher/rke2/server/tls/server-ca.crt https://localhost:9345/db/info | grep 'clientURLs'] finished with err: <nil> and exit code: 1"
May 04 19:41:36 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:36Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:41:36 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:36Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:41:36 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:36Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (6591 vs 6595)"
May 04 19:41:36 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:36Z" level=error msg="error syncing 'test-ns-dv2rv/custom-bbc71888a47c-machine-plan': handler secret-watch: secret received was too old, requeuing"
May 04 19:41:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:46Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:41:46 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:41:46Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:42:01 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:42:01Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:42:16 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:42:16Z" level=info msg="[K8s] updated plan secret test-ns-dv2rv/custom-bbc71888a47c-machine-plan with feedback"
May 04 19:43:33 test-node-zxb66 rancher-system-agent[243]: time="2023-05-04T19:43:33Z" level=fatal msg="[K8s] received nil secret that was nil, stopping"
May 04 19:43:33 test-node-zxb66 systemd[1]: rancher-system-agent.service: Main process exited, code=exited, status=1/FAILURE
May 04 19:43:33 test-node-zxb66 systemd[1]: rancher-system-agent.service: Killing process 3896 (containerd-shim) with signal SIGKILL.
May 04 19:43:33 test-node-zxb66 systemd[1]: rancher-system-agent.service: Killing process 3896 (containerd-shim) with signal SIGKILL.
May 04 19:43:33 test-node-zxb66 systemd[1]: rancher-system-agent.service: Failed with result 'exit-code'.

}
// If we get a conflict, we can retrieve the latest secret and compare plan data to see if the plan changed.
latestSecret, getErr := core.Secret().Get(secret.Namespace, secret.Name, metav1.GetOptions{})
if getErr == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could avoid nesting if we invert the condition, but this is more of a nit as the current nesting doesn't significantly impact readability

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah... this was a little hasty in an attempt to fix this, I can come back and fix this later on.

@Oats87 Oats87 merged commit e696ff6 into rancher:main May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants