Proxy fails with error "Failed to connect to the host via scp: /bin/sh: 1: /usr/lib/sftp-server: not found\nscp: Connection closed\r\n" OpenSSH 9.0+ #100

invidian · 2022-05-30T16:34:51Z

Overview of the Issue

If one has OpenSSH version 9.0+ installed on their system, provisioning using this provisioner will fail with the following error:

learn-packer.docker.ubuntu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via scp: /bin/sh: 1: /usr/lib/sftp-server: not found\nscp: Connection closed\r\n", "unreachable": true}

The error occurs even though I have /usr/lib/sftp-server file on my machine:

$ ls -la /usr/lib/sftp-server
lrwxrwxrwx 1 root root 24 Apr 13 19:29 /usr/lib/sftp-server -> /usr/lib/ssh/sftp-server

This is because OpenSSH 9.0 has switched the default mode for scp from legacy mode to sftp mode and it seems SSH proxy set up by the plugin does not handle this correctly.

Reproduction Steps

Steps to reproduce this issue:

Prerequisites:

OpenSSH version 9.0+
Docker
Packer

Run the commands below:

cat <<EOF > docker-ubuntu.pkr.hcl
packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "ubuntu" {
  image  = "ubuntu:xenial"
  commit = true
}

build {
  name = "learn-packer"
  sources = [
    "source.docker.ubuntu"
  ]

  provisioner "ansible" {
    playbook_file = "./playbook.yml"
  }
}
EOF
cat <<EOF > playbook.yml
---
# playbook.yml
- name: 'Provision Image'
  hosts: default
  become: true

  tasks:
    - name: install Apache
      package:
        name: 'httpd'
        state: present
EOF
packer init .
ANSIBLE_SSH_ARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa" packer build docker-ubuntu.pkr.hcl

Plugin and Packer version

From packer version: Packer v1.8.1

Operating system and Environment details

OS: Arch Linux
OpenSSH version:

$ pacman -Qi openssh
Name            : openssh
Version         : 9.0p1-1

Log Fragments and crash.log files

https://gist.github.com/invidian/a9dbab69d1541b83f728f6009e27f593

CC @kopiczko

The text was updated successfully, but these errors were encountered:

scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used.

Since OpenSSH 9.0+ 'scp' uses SFTP protocol instead of legacy SCP protocol, which causes building errors like: bash: line 1: /usr/lib/sftp-server: No such file or directory\nscp: Connection closed\r\n"" However, -O option is not available in older OpenSSH version, so we cannot always set it as an option to use. To provide better out-of-the-box experience for users with newer versions of OpenSSH, we conditionally ensure -O is used when used OpenSSH version requires it. See kubernetes-sigs#859 and hashicorp/packer-plugin-ansible#100 for more details. Signed-off-by: Mateusz Gozdek <[email protected]>

Below commit messages from squashed commits: images/capi/packer: extract ansible common SSH args to a single place This is done to remove repetition of '-o IdentitiesOnly=yes' to make sure it is consistent across all platforms and to reduce amount of churn when adding new default arguments like we plan as part of mitigating issue with ssh-rsa keys (kubernetes-sigs#905). images/capi/packer: allow specifying extra scp arguments for Ansible This allows a workaround for issue kubernetes-sigs#859 when building host uses OpenSSH version 9.0+, which uses SFTP protocol for SCP instead of a legacy SCP protocol, which right now causes builds to fail with error message as below when Ansible is trying to copy files over to remote host. bash: line 1: /usr/lib/sftp-server: No such file or directory\nscp: Connection closed\r\n" This commit allows users with new OpenSSH version to specify ANSIBLE_SCP_EXTRA_ARGS="-O" to fix their builds. I plan to automate this in another commit, as it should be relatively simple and harmless. Refs kubernetes-sigs#859. images/capi/packer: allow using ssh-rsa keys with OpenSSH 8.8+ Since OpenSSH version 8.8+ ssh-rsa key algorithm is disabled by default, which right now causes builds to fail for builders which use OpenSSH version 8.8+. The problematic keys are generated by Ansible plugin for Packer and the problem is currently being discussed in issue hashicorp/packer-plugin-ansible#69. An alternative would be to consider using `use_proxy=false` option in plugin, however we are not sure what could be the implications of this. Given that building machine should be a rather short process, the workaround seem acceptable and actually allows being able to succesfully build images out of the box on more distributions. In implementation, 'PubkeyAcceptedKeyTypes' is used instead of 'PubkeyAcceptedAlgorithms', as it provides better backward compatibility, since 'PubkeyAcceptedAlgorithms' is only available since OpenSSH version 8.4. See issue kubernetes-sigs#905 for more details. Co-authored-by: Jeremi Piotrowski <[email protected]> images/capi/Makefile: set ANSIBLE_SCP_EXTRA_ARGS="-O" when needed Since OpenSSH 9.0+ 'scp' uses SFTP protocol instead of legacy SCP protocol, which causes building errors like: bash: line 1: /usr/lib/sftp-server: No such file or directory\nscp: Connection closed\r\n"" However, -O option is not available in older OpenSSH version, so we cannot always set it as an option to use. To provide better out-of-the-box experience for users with newer versions of OpenSSH, we conditionally ensure -O is used when used OpenSSH version requires it. See kubernetes-sigs#859 and hashicorp/packer-plugin-ansible#100 for more details. Signed-off-by: Mateusz Gozdek <[email protected]> Co-authored-by: Jeremi Piotrowski <[email protected]>

scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used.

* feat: Add Oracle Linux 9 build support to CAPI OCI provider * nutanix improvement * Unpin azure-cli to always install the latest * Update book Makefile `docs/book/build.sh` downloads `mdbook` to `/tmp/mdbook` serve should probably try to use the same location. * bump containerd to 1.16.15 * bump packer to 1.8.5 * capi/packer/qemu: Pass oem_id value to ansible PR 966 added support for setting OEM_ID in Flatcar QEMU images through the environment, but only defined the oem_id variable at the packer level and missed passing it through one of the `ansible_xxx_vars` variables to ansible. This results in the ansible step always being skipped. Extend ansible_extra_vars in qemu-flatcar.json to forward the oem_id to ansible so that the 'Set oem_id in grub' step can run correctly. Signed-off-by: Jeremi Piotrowski <[email protected]> * Storage Account should default to AZURE_LOCATION * Add `use_internal_ip` to gce's packer.json Allow Packer to use internal IP addr when building a GCE custom image. * remove 1.22 config and add 1.26 Signed-off-by: cpanato <[email protected]> * Sort elements in packer.json make json-sort * Add photon-4 OVA support * Update kubernetes-cni .deb to v1.2.0 * Add RHEL-8 AMI support * Support building RHEL-8 Azure VHD and SIG Images * Add support for Windows 2019/2022 to OCI capi provider image builder * Update to latest pause image from registry.k8s.io * Workaround scp behaviour in OpenSSH 9.0 scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used. * Initial support for building on OpenStack & Bump versions to build images for Kubernetes 1.24.2 * patch: changed deprecated floating_ip_pool to floating_ip_network * patch: Adding new variables and os.json files to allow building of ubuntu remotely in Openstack Added ability to override the image name to create unique images on subsequent runs. Added option to images as public via variables. * fix: corrected ubuntu20 variables in json file * docs: update to wording in docs Update docs/book/src/capi/providers/openstack-remote.md to changing the wording slightly around volume/image creation Co-authored-by: Mohammed Naser <[email protected]> * Include OpenStack builder task and ensure Hyper-V service is disabled There were no Hyper-V roles where this service is disabled being included, due to how they're being looked-up for inclusion i.e off the Packer builder name. This commit copies and trims the QEMU task and explicitly refers to it so that it's included in our builds on OpenStack. --------- Signed-off-by: Jeremi Piotrowski <[email protected]> Signed-off-by: cpanato <[email protected]> Co-authored-by: Joe Kratzat <[email protected]> Co-authored-by: Kubernetes Prow Robot <[email protected]> Co-authored-by: Christophe Jauffret <[email protected]> Co-authored-by: Kiran Keshavamurthy <[email protected]> Co-authored-by: Jeremi Piotrowski <[email protected]> Co-authored-by: Matt Boersma <[email protected]> Co-authored-by: RyuSA <[email protected]> Co-authored-by: cpanato <[email protected]> Co-authored-by: Hans Rakers <[email protected]> Co-authored-by: Nick Jones <[email protected]> Co-authored-by: Mohammed Naser <[email protected]>

scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used.

* feat: Add Oracle Linux 9 build support to CAPI OCI provider * nutanix improvement * Unpin azure-cli to always install the latest * Update book Makefile `docs/book/build.sh` downloads `mdbook` to `/tmp/mdbook` serve should probably try to use the same location. * bump containerd to 1.16.15 * bump packer to 1.8.5 * capi/packer/qemu: Pass oem_id value to ansible PR 966 added support for setting OEM_ID in Flatcar QEMU images through the environment, but only defined the oem_id variable at the packer level and missed passing it through one of the `ansible_xxx_vars` variables to ansible. This results in the ansible step always being skipped. Extend ansible_extra_vars in qemu-flatcar.json to forward the oem_id to ansible so that the 'Set oem_id in grub' step can run correctly. Signed-off-by: Jeremi Piotrowski <[email protected]> * Storage Account should default to AZURE_LOCATION * Add `use_internal_ip` to gce's packer.json Allow Packer to use internal IP addr when building a GCE custom image. * remove 1.22 config and add 1.26 Signed-off-by: cpanato <[email protected]> * Sort elements in packer.json make json-sort * Add photon-4 OVA support * Update kubernetes-cni .deb to v1.2.0 * Add RHEL-8 AMI support * Support building RHEL-8 Azure VHD and SIG Images * Add support for Windows 2019/2022 to OCI capi provider image builder * Update to latest pause image from registry.k8s.io * Workaround scp behaviour in OpenSSH 9.0 scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used. * Initial support for building on OpenStack & Bump versions to build images for Kubernetes 1.24.2 * patch: changed deprecated floating_ip_pool to floating_ip_network * patch: Adding new variables and os.json files to allow building of ubuntu remotely in Openstack Added ability to override the image name to create unique images on subsequent runs. Added option to images as public via variables. * fix: corrected ubuntu20 variables in json file * docs: update to wording in docs Update docs/book/src/capi/providers/openstack-remote.md to changing the wording slightly around volume/image creation Co-authored-by: Mohammed Naser <[email protected]> * Include OpenStack builder task and ensure Hyper-V service is disabled There were no Hyper-V roles where this service is disabled being included, due to how they're being looked-up for inclusion i.e off the Packer builder name. This commit copies and trims the QEMU task and explicitly refers to it so that it's included in our builds on OpenStack. --------- Signed-off-by: Jeremi Piotrowski <[email protected]> Signed-off-by: cpanato <[email protected]> Co-authored-by: Joe Kratzat <[email protected]> Co-authored-by: Kubernetes Prow Robot <[email protected]> Co-authored-by: Christophe Jauffret <[email protected]> Co-authored-by: Kiran Keshavamurthy <[email protected]> Co-authored-by: Jeremi Piotrowski <[email protected]> Co-authored-by: Matt Boersma <[email protected]> Co-authored-by: RyuSA <[email protected]> Co-authored-by: cpanato <[email protected]> Co-authored-by: Hans Rakers <[email protected]> Co-authored-by: Nick Jones <[email protected]> Co-authored-by: Mohammed Naser <[email protected]>

* Workaround scp behaviour in OpenSSH 9.0 scp in OpenSSH 9 now uses SFTP by default, which breaks the Packer / Ansible / SSH Proxy interaction See hashicorp/packer-plugin-ansible#100 And hashicorp/packer#11783 Adding the `-O` flag to the `scp` command forces the old protocol to be used. * Initial support for building on OpenStack & Bump versions to build images for Kubernetes 1.24.2 * patch: changed deprecated floating_ip_pool to floating_ip_network * patch: Adding new variables and os.json files to allow building of ubuntu remotely in Openstack Added ability to override the image name to create unique images on subsequent runs. Added option to images as public via variables. * fix: corrected ubuntu20 variables in json file * Support flatcar * Fix some vars * Enable blockstorage volume in default * Add more flatcar vars --------- Co-authored-by: Nick Jones <[email protected]> Co-authored-by: Drew <[email protected]>