Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incus storage volume attach prevents startup of VM for some (longer than 14 characters?) storage names with qemu 9.1.1 #1536

Open
christophhagemann opened this issue Dec 20, 2024 · 10 comments
Assignees
Labels
Bug Confirmed to be a bug Easy Good for new contributors
Milestone

Comments

@christophhagemann
Copy link

Required information

  • Distribution: debian
  • Distribution version: trixie
  • The output of "incus info"

config:
core.https_address: :8443
images.auto_update_interval: "0"
api_extensions:

  • storage_zfs_remove_snapshots
  • container_host_shutdown_timeout
  • container_stop_priority
  • container_syscall_filtering
  • auth_pki
  • container_last_used_at
  • etag
  • patch
  • usb_devices
  • https_allowed_credentials
  • image_compression_algorithm
  • directory_manipulation
  • container_cpu_time
  • storage_zfs_use_refquota
  • storage_lvm_mount_options
  • network
  • profile_usedby
  • container_push
  • container_exec_recording
  • certificate_update
  • container_exec_signal_handling
  • gpu_devices
  • container_image_properties
  • migration_progress
  • id_map
  • network_firewall_filtering
  • network_routes
  • storage
  • file_delete
  • file_append
  • network_dhcp_expiry
  • storage_lvm_vg_rename
  • storage_lvm_thinpool_rename
  • network_vlan
  • image_create_aliases
  • container_stateless_copy
  • container_only_migration
  • storage_zfs_clone_copy
  • unix_device_rename
  • storage_lvm_use_thinpool
  • storage_rsync_bwlimit
  • network_vxlan_interface
  • storage_btrfs_mount_options
  • entity_description
  • image_force_refresh
  • storage_lvm_lv_resizing
  • id_map_base
  • file_symlinks
  • container_push_target
  • network_vlan_physical
  • storage_images_delete
  • container_edit_metadata
  • container_snapshot_stateful_migration
  • storage_driver_ceph
  • storage_ceph_user_name
  • resource_limits
  • storage_volatile_initial_source
  • storage_ceph_force_osd_reuse
  • storage_block_filesystem_btrfs
  • resources
  • kernel_limits
  • storage_api_volume_rename
  • network_sriov
  • console
  • restrict_dev_incus
  • migration_pre_copy
  • infiniband
  • dev_incus_events
  • proxy
  • network_dhcp_gateway
  • file_get_symlink
  • network_leases
  • unix_device_hotplug
  • storage_api_local_volume_handling
  • operation_description
  • clustering
  • event_lifecycle
  • storage_api_remote_volume_handling
  • nvidia_runtime
  • container_mount_propagation
  • container_backup
  • dev_incus_images
  • container_local_cross_pool_handling
  • proxy_unix
  • proxy_udp
  • clustering_join
  • proxy_tcp_udp_multi_port_handling
  • network_state
  • proxy_unix_dac_properties
  • container_protection_delete
  • unix_priv_drop
  • pprof_http
  • proxy_haproxy_protocol
  • network_hwaddr
  • proxy_nat
  • network_nat_order
  • container_full
  • backup_compression
  • nvidia_runtime_config
  • storage_api_volume_snapshots
  • storage_unmapped
  • projects
  • network_vxlan_ttl
  • container_incremental_copy
  • usb_optional_vendorid
  • snapshot_scheduling
  • snapshot_schedule_aliases
  • container_copy_project
  • clustering_server_address
  • clustering_image_replication
  • container_protection_shift
  • snapshot_expiry
  • container_backup_override_pool
  • snapshot_expiry_creation
  • network_leases_location
  • resources_cpu_socket
  • resources_gpu
  • resources_numa
  • kernel_features
  • id_map_current
  • event_location
  • storage_api_remote_volume_snapshots
  • network_nat_address
  • container_nic_routes
  • cluster_internal_copy
  • seccomp_notify
  • lxc_features
  • container_nic_ipvlan
  • network_vlan_sriov
  • storage_cephfs
  • container_nic_ipfilter
  • resources_v2
  • container_exec_user_group_cwd
  • container_syscall_intercept
  • container_disk_shift
  • storage_shifted
  • resources_infiniband
  • daemon_storage
  • instances
  • image_types
  • resources_disk_sata
  • clustering_roles
  • images_expiry
  • resources_network_firmware
  • backup_compression_algorithm
  • ceph_data_pool_name
  • container_syscall_intercept_mount
  • compression_squashfs
  • container_raw_mount
  • container_nic_routed
  • container_syscall_intercept_mount_fuse
  • container_disk_ceph
  • virtual-machines
  • image_profiles
  • clustering_architecture
  • resources_disk_id
  • storage_lvm_stripes
  • vm_boot_priority
  • unix_hotplug_devices
  • api_filtering
  • instance_nic_network
  • clustering_sizing
  • firewall_driver
  • projects_limits
  • container_syscall_intercept_hugetlbfs
  • limits_hugepages
  • container_nic_routed_gateway
  • projects_restrictions
  • custom_volume_snapshot_expiry
  • volume_snapshot_scheduling
  • trust_ca_certificates
  • snapshot_disk_usage
  • clustering_edit_roles
  • container_nic_routed_host_address
  • container_nic_ipvlan_gateway
  • resources_usb_pci
  • resources_cpu_threads_numa
  • resources_cpu_core_die
  • api_os
  • container_nic_routed_host_table
  • container_nic_ipvlan_host_table
  • container_nic_ipvlan_mode
  • resources_system
  • images_push_relay
  • network_dns_search
  • container_nic_routed_limits
  • instance_nic_bridged_vlan
  • network_state_bond_bridge
  • usedby_consistency
  • custom_block_volumes
  • clustering_failure_domains
  • resources_gpu_mdev
  • console_vga_type
  • projects_limits_disk
  • network_type_macvlan
  • network_type_sriov
  • container_syscall_intercept_bpf_devices
  • network_type_ovn
  • projects_networks
  • projects_networks_restricted_uplinks
  • custom_volume_backup
  • backup_override_name
  • storage_rsync_compression
  • network_type_physical
  • network_ovn_external_subnets
  • network_ovn_nat
  • network_ovn_external_routes_remove
  • tpm_device_type
  • storage_zfs_clone_copy_rebase
  • gpu_mdev
  • resources_pci_iommu
  • resources_network_usb
  • resources_disk_address
  • network_physical_ovn_ingress_mode
  • network_ovn_dhcp
  • network_physical_routes_anycast
  • projects_limits_instances
  • network_state_vlan
  • instance_nic_bridged_port_isolation
  • instance_bulk_state_change
  • network_gvrp
  • instance_pool_move
  • gpu_sriov
  • pci_device_type
  • storage_volume_state
  • network_acl
  • migration_stateful
  • disk_state_quota
  • storage_ceph_features
  • projects_compression
  • projects_images_remote_cache_expiry
  • certificate_project
  • network_ovn_acl
  • projects_images_auto_update
  • projects_restricted_cluster_target
  • images_default_architecture
  • network_ovn_acl_defaults
  • gpu_mig
  • project_usage
  • network_bridge_acl
  • warnings
  • projects_restricted_backups_and_snapshots
  • clustering_join_token
  • clustering_description
  • server_trusted_proxy
  • clustering_update_cert
  • storage_api_project
  • server_instance_driver_operational
  • server_supported_storage_drivers
  • event_lifecycle_requestor_address
  • resources_gpu_usb
  • clustering_evacuation
  • network_ovn_nat_address
  • network_bgp
  • network_forward
  • custom_volume_refresh
  • network_counters_errors_dropped
  • metrics
  • image_source_project
  • clustering_config
  • network_peer
  • linux_sysctl
  • network_dns
  • ovn_nic_acceleration
  • certificate_self_renewal
  • instance_project_move
  • storage_volume_project_move
  • cloud_init
  • network_dns_nat
  • database_leader
  • instance_all_projects
  • clustering_groups
  • ceph_rbd_du
  • instance_get_full
  • qemu_metrics
  • gpu_mig_uuid
  • event_project
  • clustering_evacuation_live
  • instance_allow_inconsistent_copy
  • network_state_ovn
  • storage_volume_api_filtering
  • image_restrictions
  • storage_zfs_export
  • network_dns_records
  • storage_zfs_reserve_space
  • network_acl_log
  • storage_zfs_blocksize
  • metrics_cpu_seconds
  • instance_snapshot_never
  • certificate_token
  • instance_nic_routed_neighbor_probe
  • event_hub
  • agent_nic_config
  • projects_restricted_intercept
  • metrics_authentication
  • images_target_project
  • images_all_projects
  • cluster_migration_inconsistent_copy
  • cluster_ovn_chassis
  • container_syscall_intercept_sched_setscheduler
  • storage_lvm_thinpool_metadata_size
  • storage_volume_state_total
  • instance_file_head
  • instances_nic_host_name
  • image_copy_profile
  • container_syscall_intercept_sysinfo
  • clustering_evacuation_mode
  • resources_pci_vpd
  • qemu_raw_conf
  • storage_cephfs_fscache
  • network_load_balancer
  • vsock_api
  • instance_ready_state
  • network_bgp_holdtime
  • storage_volumes_all_projects
  • metrics_memory_oom_total
  • storage_buckets
  • storage_buckets_create_credentials
  • metrics_cpu_effective_total
  • projects_networks_restricted_access
  • storage_buckets_local
  • loki
  • acme
  • internal_metrics
  • cluster_join_token_expiry
  • remote_token_expiry
  • init_preseed
  • storage_volumes_created_at
  • cpu_hotplug
  • projects_networks_zones
  • network_txqueuelen
  • cluster_member_state
  • instances_placement_scriptlet
  • storage_pool_source_wipe
  • zfs_block_mode
  • instance_generation_id
  • disk_io_cache
  • amd_sev
  • storage_pool_loop_resize
  • migration_vm_live
  • ovn_nic_nesting
  • oidc
  • network_ovn_l3only
  • ovn_nic_acceleration_vdpa
  • cluster_healing
  • instances_state_total
  • auth_user
  • security_csm
  • instances_rebuild
  • numa_cpu_placement
  • custom_volume_iso
  • network_allocations
  • zfs_delegate
  • storage_api_remote_volume_snapshot_copy
  • operations_get_query_all_projects
  • metadata_configuration
  • syslog_socket
  • event_lifecycle_name_and_project
  • instances_nic_limits_priority
  • disk_initial_volume_configuration
  • operation_wait
  • image_restriction_privileged
  • cluster_internal_custom_volume_copy
  • disk_io_bus
  • storage_cephfs_create_missing
  • instance_move_config
  • ovn_ssl_config
  • certificate_description
  • disk_io_bus_virtio_blk
  • loki_config_instance
  • instance_create_start
  • clustering_evacuation_stop_options
  • boot_host_shutdown_action
  • agent_config_drive
  • network_state_ovn_lr
  • image_template_permissions
  • storage_bucket_backup
  • storage_lvm_cluster
  • shared_custom_block_volumes
  • auth_tls_jwt
  • oidc_claim
  • device_usb_serial
  • numa_cpu_balanced
  • image_restriction_nesting
  • network_integrations
  • instance_memory_swap_bytes
  • network_bridge_external_create
  • network_zones_all_projects
  • storage_zfs_vdev
  • container_migration_stateful
  • profiles_all_projects
  • instances_scriptlet_get_instances
  • instances_scriptlet_get_cluster_members
  • instances_scriptlet_get_project
  • network_acl_stateless
  • instance_state_started_at
  • networks_all_projects
  • network_acls_all_projects
  • storage_buckets_all_projects
  • resources_load
  • instance_access
  • project_access
  • projects_force_delete
  • resources_cpu_flags
  • disk_io_bus_cache_filesystem
  • instance_oci
  • clustering_groups_config
  • instances_lxcfs_per_instance
  • clustering_groups_vm_cpu_definition
  • disk_volume_subpath
  • projects_limits_disk_pool
  • network_ovn_isolated
  • qemu_raw_qmp
  • network_load_balancer_health_check
  • oidc_scopes
  • network_integrations_peer_name
  • qemu_scriptlet
  • instance_auto_restart
  • storage_lvm_metadatasize
  • ovn_nic_promiscuous
  • ovn_nic_ip_address_none
  • instances_state_os_info
  • network_load_balancer_state
  • instance_nic_macvlan_mode
  • storage_lvm_cluster_create
  • network_ovn_external_interfaces
  • instances_scriptlet_get_instances_count
  • cluster_rebalance
  • custom_volume_refresh_exclude_older_snapshots
  • storage_initial_owner
  • storage_live_migration
  • instance_console_screenshot
  • image_import_alias
  • authorization_scriptlet
  • console_force
    api_status: stable
    api_version: "1.0"
    auth: trusted
    public: false
    auth_methods:
  • tls
    auth_user_name: admin
    auth_user_method: unix
    environment:
    addresses:
    • 192.168.178.28:8443
    • '[fdbe:83d1:410d:0:6347:b141:8b86:e93d]:8443'
    • '[2a02:3100:258a:4f00:c8e4:fb47:fff7:e8ff]:8443'
    • 10.19.174.1:8443
    • '[fd42:d33f:57:5325::1]:8443'
      architectures:
    • x86_64
    • i686
      certificate: |
      -----BEGIN CERTIFICATE-----
      MIICEzCCAZmgAwIBAgIRAPdHUyKadiOihPOlPJZoPFwwCgYIKoZIzj0EAwMwODEZ
      MBcGA1UEChMQTGludXggQ29udGFpbmVyczEbMBkGA1UEAwwScm9vdEBwYy1oeXBl
      cnZpc29yMB4XDTI0MTExNjE5MDk0MloXDTM0MTExNDE5MDk0MlowODEZMBcGA1UE
      ChMQTGludXggQ29udGFpbmVyczEbMBkGA1UEAwwScm9vdEBwYy1oeXBlcnZpc29y
      MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEXrfQJsctleWwF03XLGnJGMVijQ0ZUnYD
      BK5bfmJexluhvFL2Pp9FtRaVSw/sBqR8AJ4oN5sqm3A9gB2obT7ZcfIOrWdNr1tY
      VI0IH4LveIPGkJCYoW4/seFBIePrRS1Mo2cwZTAOBgNVHQ8BAf8EBAMCBaAwEwYD
      VR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAwBgNVHREEKTAngg1wYy1o
      eXBlcnZpc29yhwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gA
      MGUCMEZ6Hs6TmNI+IeKWvxDoDmffkyBIjE8Ut8uFWxWCa85D6iwmlag7qa/DbH6F
      yK0UCwIxAOUmtzx/dhtaZXCbb74B0K/2TUUDKLPPHpZ7xvm6NtJKhHFCrDVBxlfn
      LKA0Mylg7Q==
      -----END CERTIFICATE-----
      certificate_fingerprint: 986d30c1248d8e6f2ba145232a39a7dfd1bb6bb11bcd8b91d684b64240005f82
      driver: lxc | qemu
      driver_version: 6.0.2 | 9.2.0
      firewall: nftables
      kernel: Linux
      kernel_architecture: x86_64
      kernel_features:
      idmapped_mounts: "true"
      netnsid_getifaddrs: "true"
      seccomp_listener: "true"
      seccomp_listener_continue: "true"
      uevent_injection: "true"
      unpriv_binfmt: "true"
      unpriv_fscaps: "true"
      kernel_version: 6.11.7-amd64
      lxc_features:
      cgroup2: "true"
      core_scheduling: "true"
      devpts_fd: "true"
      idmapped_mounts_v2: "true"
      mount_injection_file: "true"
      network_gateway_device_route: "true"
      network_ipvlan: "true"
      network_l2proxy: "true"
      network_phys_macvlan_mtu: "true"
      network_veth_router: "true"
      pidfd: "true"
      seccomp_allow_deny_syntax: "true"
      seccomp_notify: "true"
      seccomp_proxy_send_notify_fd: "true"
      os_name: Debian GNU/Linux
      os_version: ""
      project: default
      server: incus
      server_clustered: false
      server_event_mode: full-mesh
      server_name: pc-hypervisor
      server_pid: 778
      server_version: "6.8"
      storage: zfs
      storage_version: 2.2.6-2
      storage_supported_drivers:
    • name: zfs
      version: 2.2.6-2
      remote: false
    • name: btrfs
      version: 6.6.3
      remote: false
    • name: dir
      version: "1"
      remote: false

Issue description

Incus fails to start a VM when I attach specifically (too long?) named block devices with

Error: Failed setting up device via monitor: Failed adding block device for disk device “myblockdevice”: Failed adding device: The serial number can’t be longer than 20 characters when it is also used as the default for device_id
Try incus info --show-log win11vm for more info

Steps to reproduce

  1. create a VM: incus create images:debian/trixie/default/amd64 mytestvm --vm
  2. create a storage volume: incus storage volume create incus my_testing_blockdevice --type=block size=400GB
  3. attach it: incus storage volume attach incus my_testing_blockdevice mytestvm
  4. attempt to start the VM: incus start mytestvm
  5. Error

Error: Failed setting up device via monitor: Failed adding block device for disk device "my_testing_blockdevice": Failed adding device: The serial number can't be longer than 20 characters when it is also used as the default for device_id
Try incus info --show-log mytestvm for more info

Information to attach

The bug doesn't occur for device names shorter than 15 characters. my_testing_blo works, my_testing_bloc doesn't.

qemu.qmp.log is here: https://pastebin.com/i0tCXDGB

Thanks for this great piece of software and your hard work! Let me know if I can help debug things further.

Best regards,
Christoph Hagemann

@stgraber stgraber added Bug Confirmed to be a bug Easy Good for new contributors labels Dec 20, 2024
@stgraber stgraber added this to the incus-6.9 milestone Dec 20, 2024
@heldergomes
Copy link

Hi @stgraber
I am new contributing on incus, may I get this issue ?

@stgraber
Copy link
Member

Yep, I assigned it to you now!

@heldergomes
Copy link

Hi @christophhagemann, I just try to reproduce the same error, I coundn't get the same error though.

root@test-incus-debian:~# incus storage volume info incus my_testing_blockdevice
Name: my_testing_blockdevice
Type: custom
Content type: block
Usage: 12.00KiB
Created: 2024/12/21 18:45 UTC
root@test-incus-debian:~# incus storage volume show incus my_testing_blockdevice
config:
  size: 10GB
description: ""
name: my_testing_blockdevice
type: custom
used_by:
- /1.0/instances/mytestvm
location: none
content_type: block
project: default
created_at: 2024-12-21T18:45:16.851898181Z
root@test-incus-debian:~# incus list
+----------+---------+-------------------------+-----------------------------------------------+-----------------+-----------+
|   NAME   |  STATE  |          IPV4           |                     IPV6                      |      TYPE       | SNAPSHOTS |
+----------+---------+-------------------------+-----------------------------------------------+-----------------+-----------+
| mytestvm | RUNNING | 10.197.144.254 (enp5s0) | fd42:2bf:7f9a:13b:216:3eff:fed6:2452 (enp5s0) | VIRTUAL-MACHINE | 0         |
+----------+---------+-------------------------+-----------------------------------------------+-----------------+-----------+

@heldergomes
Copy link

heldergomes commented Dec 21, 2024

I setup a VM Debian 12 running incus 6.8 and ZFS as storage pool.

Maybe I didn't copy some configuration correctly, which's likely part of the problem 🤔

@christophhagemann
Copy link
Author

christophhagemann commented Dec 21, 2024 via email

@christophhagemann christophhagemann changed the title incus storage volume attach prevents startup of VM for some (longer than 14 characters?) storage names incus storage volume attach prevents startup of VM for some (longer than 14 characters?) storage names with qemu 9.1.1 Dec 21, 2024
@heldergomes
Copy link

heldergomes commented Dec 24, 2024

@christophhagemann I installed qemu 9.1.1 on debian 12 VM and incus 6.8.
Weirdly I didn't get the same error though, is there any clue how can I track the error ?

debian@test-debian-qemu-911:~$ sudo incus storage volume list incus
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
|      TYPE       |                               NAME                               | DESCRIPTION | CONTENT-TYPE | USED BY |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| custom          | my_testing_blockdevice                                           |             | block        | 1       |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| image           | 63a41233aed62002cad0bfb9da280ab5d569110cc96389d7cebd023a0a2e278f |             | block        | 1       |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
| virtual-machine | mytestvm                                                         |             | block        | 1       |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+
debian@test-debian-qemu-911:~$ sudo incus list
+----------+---------+------------------------+-----------------------------------------------+-----------------+-----------+
|   NAME   |  STATE  |          IPV4          |                     IPV6                      |      TYPE       | SNAPSHOTS |
+----------+---------+------------------------+-----------------------------------------------+-----------------+-----------+
| mytestvm | RUNNING | 10.31.185.123 (enp5s0) | fd42:bd4:c584:f89:216:3eff:fe8b:795c (enp5s0) | VIRTUAL-MACHINE | 0         |
+----------+---------+------------------------+-----------------------------------------------+-----------------+-----------+
debian@test-debian-qemu-911:~$ qemu-system-x86_64 --version
QEMU emulator version 9.1.1
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

@heldergomes
Copy link

I was thinking, maybe I need to try running incus on debian 13 🤔

@heldergomes
Copy link

Hi @stgraber, I had tried to reproduce the same error running incus on debian trixie but qemu 9.2.0 isn't supported yet.

Do you have any clue what could I do to reproduce the error, and also could we do something on incus side to handle this problem ?

@christophhagemann
Copy link
Author

christophhagemann commented Jan 10, 2025 via email

@pyfon
Copy link

pyfon commented Jan 11, 2025

I'm getting this issue on Arch Linux.

At first, I got the following error:

$ incus start trinity-poc --console
Error: Failed to add CPUs: Failed to add device: Invalid parameter type for 'core-id', expected: integer
Try `incus info --show-log trinity-poc` for more info

I saw this was a result of QEMU 9.2 breaking things, and that this was fixed in #1522 .
I couldn't see a pending update for Incus on Arch, so I downgraded QEMU to 9.1.2-1 instead.

Next, I received the following error when trying to start my VM:

$ incus start trinity-poc --console                                                                                                                                                                                             
Error: Failed setting up device via monitor: Failed adding block device for disk device "trinity_poc_hdd_0": Failed adding device: The serial number can't be longer than 20 characters when it is also used as the default for device_id                     
Try `incus info --show-log trinity-poc` for more info

Below are the steps I used to reproduce this error:

incus init trinity-poc \
	--empty \
	--vm \
	-c limits.cpu=4 \
	-c limits.memory=8GiB

incus storage volume create default trinity_poc_hdd_0 size=8GB --type=block
incus storage volume attach default trinity_poc_hdd_0 trinity-poc

incus storage volume import default \
	~/debian-12.8.0-amd64-netinst.iso \
	debian-iso-volume \
	--type=iso

incus config device add trinity-poc iso-volume disk \
	pool=default \
	source=debian-iso-volume \
	boot.priority=10

incus start trinity-poc --console

Crucially, this does not happen if the storage volumes attached to a VM have shorter names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Confirmed to be a bug Easy Good for new contributors
Development

No branches or pull requests

4 participants