Include replaces instead appends

[t-ru]

Hello,

I have a problem with includes.

keg version: 2.0.3
OS: OpenSUSE Tumbleweed @ WSL2 on Windows 11

I've a base image and a derived image.

images/sle-15-5.vmware/image_defaults.yaml

  preferences:
      - _include:
          - config/misc/packagemanager.zypper
          - config/misc/bootsplash-theme.sle
          - config/misc/bootloader-theme.sle
          - config/misc/rpm-check-signatures.false
          - config/misc/rpm-excludedocs.true
        version: "1.0.0"

images/sle-15-5.vmware/base-image/image.yaml

  preferences:
      - version: "2023.03"
      - _include:
          - config/disks/vmware.xfs
          - config/disks/sle-15-x.xfs
          - config/disks/boot.uefi.100mb
          - config/disks/vmware.kernelcmdline       
          - config/disks/bootloader.grub2
          - config/disks/lvm.root.30gb
          - config/disks/lvm.swap.20gb
        _attributes:
          profiles: [vmware-sles, vmware-sles4sap]

result

    <preferences profiles="vmware-sles,vmware-sles4sap">
        <type image="oem" filesystem="xfs" format="vmdk" devicepersistency="by-uuid" fscreateoptions="-i sparse=0" firmware="uefi" bootpartition="false" efipartsize="100" kernelcmdline="plymouth.enable=0 console=ttyS0,115200 console=tty0 mitigations=auto vmw_pvscsi.cmd_per_lun=1024 vmw_pvscsi.ring_pages=32 quiet">
            <bootloader name="grub2" timeout="10"/>
            <systemdisk name="system">
                <volume name="root" mountpoint="@root" size="30G"/>
            </systemdisk>
            <oemconfig>
                <oem-resize>false</oem-resize>
                <oem-swap>true</oem-swap>
                <oem-swapsize>20480</oem-swapsize>
                <oem-swapname>swap</oem-swapname>
            </oemconfig>
        </type>
    </preferences>

expected result

    <preferences>
        <version>2023.03</version>
        <packagemanager>zypper</packagemanager>
        <bootsplash-theme>SLE</bootsplash-theme>
        <bootloader-theme>SLE</bootloader-theme>
        <rpm-check-signatures>false</rpm-check-signatures>
        <rpm-excludedocs>true</rpm-excludedocs>
    </preferences>
    <preferences profiles="vmware-sles,vmware-sles4sap">
        <type image="oem" filesystem="xfs" format="vmdk" devicepersistency="by-uuid" fscreateoptions="-i sparse=0" firmware="uefi" bootpartition="false" efipartsize="100" kernelcmdline="plymouth.enable=0 console=ttyS0,115200 console=tty0 mitigations=auto vmw_pvscsi.cmd_per_lun=1024 vmw_pvscsi.ring_pages=32 quiet">
            <bootloader name="grub2" timeout="10"/>
            <systemdisk name="system">
                <volume name="root" mountpoint="@root" size="30G"/>
            </systemdisk>
            <oemconfig>
                <oem-resize>false</oem-resize>
                <oem-swap>true</oem-swap>
                <oem-swapsize>20480</oem-swapsize>
                <oem-swapname>swap</oem-swapname>
            </oemconfig>
        </type>
    </preferences>

I would like to append and not replace.
What do I need to change?

Best regards Thomas

[jgleissner]

Hi Thomas,

When merging, keys with the same name from upper layers get overwritten. That's on purpose so you can redefine them on lower layers. Even if keg could concatenate lists, you'd end up with two version entries in preferences.

The keg strategy of merging snippets together under the same key is using namespaces. Namespace keys are no different in terms of merging, but when generating the XML file, they do not produce an element (just a comment), so all elements in the namespace become a child element of the key they are under. We use this in keg-recipes for packages and other data, but this should work for preferences too, to some degree anyway. It's a little tricky in this case because preferences is a little different than e.g. packages, as it's a list directly under image, so you'd need to namespace it under image, like this:

image_defaults.yaml

image:
    _namespace_prefs:
        preferences:
            version: "1.0.0"
            _include:
                - config/misc/packagemanager.zypper
                - ...

base-image/image.yaml

image:
    _namespace_prefs:
        preferences:
            version: "2023.03"
    preferences:
        -  _include:
               - config/disk/...
           _attributes:
           profiles: [vmware-sles, vmware-sles4sap]

This looks a bit odd because there are two preferences keys with different type (the global one is a dict, allowing you to overwrite version, and the profile one is a list), but in the generated XML it's just <preferences> elements under <image> so it should work fine.

[t-ru]

Hello Thanks,

thanks for Support. Now I've implemented:

images-testing/sle-15-5.vmware/image_defaults.yaml

  _namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/includes:
    preferences:
      - _include:
          - config/misc/packagemanager.zypper
          - config/misc/bootsplash-theme.sle
          - config/misc/bootloader-theme.sle
          - config/misc/rpm-check-signatures.false
          - config/misc/rpm-excludedocs.true
  
  _namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/version:
    preferences:
      - version: "1.0.0"

images-testing/sle-15-5.vmware/base-image/image.yaml

  # overwrite 
  _namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/version:
      preferences:
        - version: "2023.03"
  
  preferences:
    - _include:
        - config/disks/vmware.xfs
        - config/disks/sle-15-x.xfs
        - config/disks/boot.uefi.100mb
        - config/disks/vmware.kernelcmdline       
        - config/disks/bootloader.grub2
        - config/disks/lvm.root.30gb
        - config/disks/lvm.swap.20gb
      _attributes:
        profiles: [vmware-sles, vmware-sles4sap]

result

<!-- begin namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/includes -->
    <preferences>
        <packagemanager>zypper</packagemanager>
        <bootsplash-theme>SLE</bootsplash-theme>
        <bootloader-theme>SLE</bootloader-theme>
        <rpm-check-signatures>false</rpm-check-signatures>
        <rpm-excludedocs>true</rpm-excludedocs>
    </preferences>
<!-- end namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/includes -->

<!-- begin namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/version -->
    <preferences>
        <version>2023.03</version>
    </preferences>
<!-- end namespace images/sle-15-5.vmware/image-defaults.yaml::preferences/version -->

<type image="oem" filesystem="xfs" format="vmdk" devicepersistency="by-uuid" fscreateoptions="-i sparse=0" firmware="uefi" bootpartition="false" efipartsize="100" kernelcmdline="plymouth.enable=0 console=ttyS0,115200 console=tty0 mitigations=auto vmw_pvscsi.cmd_per_lun=1024 vmw_pvscsi.ring_pages=32 quiet">
            <bootloader name="grub2" timeout="10"/>
            <systemdisk name="system">
                <volume name="root" mountpoint="@root" size="30G"/>
            </systemdisk>
            <oemconfig>
                <oem-resize>false</oem-resize>
                <oem-swap>true</oem-swap>
                <oem-swapsize>20480</oem-swapsize>
                <oem-swapname>swap</oem-swapname>
            </oemconfig>
        </type>
    </preferences>

Not nice. But working.

But if I implement this with packages I've a problem:

images-testing/sle-15-5.vmware/image_defaults.yaml

  _namespace images/sle-15-5.vmware/image-defaults.yaml::packages/includes:
    packages:

      - _include:
          - config/packages/sle-15-x/bootstrap
        _attributes:
            type: bootstrap
            patternType: onlyRequired

      - _include:
          - config/packages/sle-15-4/pattern-minimal-base.required
          - config/packages/sle-15-4/pattern-minimal-base.recommended
          - config/packages/sle-15-4/pattern-base.required
          - config/packages/sle-15-4/pattern-base.recommended
          - config/packages/sle-15-4/pattern-enhanced-base.required
          - config/packages/sle-15-4/pattern-enhanced-base.recommended
          - config/packages/sle-15-4/pattern-appamor.required
          - config/packages/sle-15-4/pattern-appamor.recommended
          - config/packages/sle-15-4/pattern-sw-management.required
          - config/packages/sle-15-4/pattern-sw-management.recommended

result

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/schema.py", line 400, in validate
    nvalue = Schema(svalue, error=e, ignore_extra_keys=i).validate(value, **kwargs)
  File "/usr/lib/python3.10/site-packages/schema.py", line 415, in validate
    raise SchemaMissingKeyError(message, e.format(data) if e else None)
schema.SchemaMissingKeyError: Missing key: 'packages'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/kiwi_keg/image_definition.py", line 133, in populate
    ImageSchema().validate(self._data)
  File "/usr/lib/python3.10/site-packages/kiwi_keg/image_schema.py", line 185, in validate
    self._schema.validate(data)
  File "/usr/lib/python3.10/site-packages/schema.py", line 404, in validate
    raise SchemaError([message] + x.autos, [e.format(data) if e else None] + x.errors)
schema.SchemaError: Key 'image' error:
Missing key: 'packages'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/kiwi_keg/keg.py", line 154, in main
    image_generator = KegGenerator(
  File "/usr/lib/python3.10/site-packages/kiwi_keg/generator.py", line 78, in __init__
    self.image_definition.populate()
  File "/usr/lib/python3.10/site-packages/kiwi_keg/image_definition.py", line 138, in populate
    raise KegDataError('Image definition malformed: {}'.format(err))
kiwi_keg.exceptions.KegDataError: Image definition malformed: Key 'image' error:
Missing key: 'packages'
[ ERROR   ]: 16:41:45 | KegDataError: Image definition malformed: Key 'image' error:
Missing key: 'packages'

I think the problem is that the packet definitions themselves already contain namespaces

How can i solve the issue?

best regards Thomas

[jgleissner]

Nested namespaces should be fine actually, albeit probably unnecessary. The image schema definition fails because it expects a packages key under images. TBH I didn't really consider namespacing inside the image definition as we don't actually use it the way we set up our image definitions. However, it's valid and there's no reason why it shouldn't work in theory. Namespacing makes the schema validation tricky though. I'll look into whether I can find a better way to handle namespaces during schema validation when I have a chance.

However, it should still work fine if you have one packages section that is not namespaced. I assume you'll have a packages section in images-testing/sle-15-5.vmware/base-image/image.yaml, so if you don't put that in a namespace (analog to the preferences section), validation should pass.