rendervulkan: Fix getting Vulkan DRM formats if backend doesn't use modifiers

[nullprop]

After commit 0110109 running gamescope nested SDL backend on top of x11 plasma causes assertion in wlroots. Related to issue #1218.

[gamescope] [Info]  console: gamescope version 3.15.9-10-g6497b0f (gcc 14.2.1)
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
[gamescope] [Info]  vulkan: selecting physical device 'AMD Radeon RX 6800 XT (RADV NAVI21)': queue family 1 (general queue family 0)
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
gamescope: types/wlr_linux_dmabuf_v1.c:532: feedback_compile: Assertion `table_len > 0' failed.
zsh: IOT instruction (core dumped)  ./build/src/gamescope -- glxgears

(gdb) bt
#0  0x00007ffff73b63f4 in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff735d120 in raise () from /usr/lib/libc.so.6
#2  0x00007ffff73444c3 in abort () from /usr/lib/libc.so.6
#3  0x00007ffff73443df in ?? () from /usr/lib/libc.so.6
#4  0x00007ffff7355177 in __assert_fail () from /usr/lib/libc.so.6
#5  0x00005555557f0472 in feedback_compile (feedback=0x7fffffffda20) at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:532
#6  0x00005555557f131d in set_default_feedback (linux_dmabuf=0x555555b19770, feedback=0x7fffffffda20)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:888
#7  0x00005555557f170e in wlr_linux_dmabuf_v1_create (display=0x7fffe44f2eb0, version=4, default_feedback=0x7fffffffda20)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:971
#8  0x00005555557f1806 in wlr_linux_dmabuf_v1_create_with_renderer (display=0x7fffe44f2eb0, version=4, renderer=0x555555b19970)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:1002
#9  0x00005555557c656a in wlr_renderer_init_wl_display (r=0x555555b19970, wl_display=0x7fffe44f2eb0) at ../subprojects/wlroots/render/wlr_renderer.c:87
#10 0x000055555563c225 in wlserver_init () at ../src/wlserver.cpp:1738
#11 0x00005555556344f5 in main (argc=1, argv=0x7fffffffdd28) at ../src/main.cpp:934

This PR returns the vulkan DRM format handling to what it was previously. i.e. they get added even if backend doesn't use modifiers.

[gamescope] [Info]  console: gamescope version 3.15.9-10-g6497b0f (gcc 14.2.1)
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
[gamescope] [Info]  vulkan: selecting physical device 'AMD Radeon RX 6800 XT (RADV NAVI21)': queue family 1 (general queue family 0)
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan:   AR24 (0x34325241)
[gamescope] [Info]  vulkan:   XR24 (0x34325258)
[gamescope] [Info]  vulkan:   AB24 (0x34324241)
[gamescope] [Info]  vulkan:   XB24 (0x34324258)
[gamescope] [Info]  vulkan:   RG16 (0x36314752)
[gamescope] [Info]  vulkan:   NV12 (0x3231564E)
[gamescope] [Info]  vulkan:   AB4H (0x48344241)
[gamescope] [Info]  vulkan:   XB4H (0x48344258)
[gamescope] [Info]  vulkan:   AB48 (0x38344241)
[gamescope] [Info]  vulkan:   XB48 (0x38344258)
[gamescope] [Info]  vulkan:   AB30 (0x30334241)
[gamescope] [Info]  vulkan:   XB30 (0x30334258)
[gamescope] [Info]  vulkan:   AR30 (0x30335241)
[gamescope] [Info]  vulkan:   XR30 (0x30335258)
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
[gamescope] [Info]  wlserver: Using explicit sync when available
[gamescope] [Info]  wlserver: Running compositor on wayland display 'gamescope-0'
[gamescope] [Info]  wlserver: [backend/headless/backend.c:17] Starting headless backend
[gamescope] [Info]  wlserver: Successfully initialized libei for input emulation!
[gamescope] [Error] wlserver: [xwayland/sockets.c:64] Failed to bind socket @/tmp/.X11-unix/X0: Address already in use
[gamescope] [Info]  wlserver: [xwayland/server.c:107] Starting Xwayland on :1
[gamescope] [Info]  pipewire: stream state changed: connecting
[gamescope] [Info]  pipewire: stream state changed: paused
[gamescope] [Info]  pipewire: stream available on node ID: 104
[gamescope] [Info]  vblank: Using timerfd.
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
[gamescope] [Warn]  xwm: got the same buffer committed twice, ignoring.
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning:          Unsupported maximum keycode 708, clipping.
>                   X11 cannot support keycodes above 255.
> Warning:          Could not resolve keysym XF86KbdInputAssistPrevgrou
> Warning:          Could not resolve keysym XF86KbdInputAssistNextgrou
Errors from xkbcomp are not fatal to the X server
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
828 frames in 5.0 seconds = 165.470 FPS

[HanabishiRecca]

This patch does not work for me on Polaris.

Instead, the one proposed in #1218 (comment)

--- a/src/rendervulkan.cpp
+++ b/src/rendervulkan.cpp
@@ -2799,7 +2799,7 @@ bool vulkan_init_format(VkFormat format, uint32_t drmFormat)
 	}
 	else
 	{
-		if ( !GetBackend()->SupportsFormat( drmFormat ) )
+		if ( GetBackend()->UsesModifiers() && !GetBackend()->SupportsFormat( drmFormat ) )
 			return false;
 
 		wlr_drm_format_set_add( &sampledDRMFormats, drmFormat, DRM_FORMAT_MOD_INVALID );

does work.

[nullprop]

@HanabishiRecca That path only gets hit if !g_device.supportsModifiers(). I didn't have an old enough GPU to test.

[HanabishiRecca]

Other reporters also confirm that: #1218 (comment).
So I don't see a problem combining both patches here.

[nullprop]

Updated to include that check too.

[HanabishiRecca]

Thank you.

[xylobol]

I'm pleased to report this is working well on an RX 580.

For fellow openSUSE users: while we wait for this to be merged, I've packaged this for Tumbleweed: https://build.opensuse.org/package/show/home:xylobol:playground/gamescope

[slynobody]

I'm pleased to report this is working well

For fellow openSUSE users: while we wait for this to be merged, I've packaged this for Tumbleweed: https://build.opensuse.org/package/show/home:xylobol:playground/gamescope

Debian / Ubuntu: did the same with a *.deb
https://github.com/slynobody/Debian-Packages/releases/tag/testing

[DocMAX]

pls merge asap!

[slynobody]

pls merge asap!

seconding this.

[jys1670]

3.15.15 worked fine for me; however, after updating to 3.16.1, an assertion happened:

[gamescope] [Info]  vulkan: selecting physical device 'AMD Radeon RX 6650M (RADV NAVI23)': queue family 1 (general queue family 0)
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
gamescope: types/wlr_linux_dmabuf_v1.c:532: feedback_compile: Assertion `table_len > 0' failed.

I can confirm that the patch resolves this issue on NixOS with X11 and Plasma.

[DocMAX]

thank god it's merged! now waiting for arch linux

[Yrds]

thank god it's merged! now waiting for arch linux

You will have to wait the next release and then after that for archlinux pkg. I believe this will take a while.

For now you can install gamescope-git from aur and edit the PKGBUILD to install a version from the master branch.

Edit: Removed instructions since the package already references master branch

[MorbidShell]

You will have to wait the next release and then after that for archlinux pkg. I believe this will take a while.

For now you can install gamescope-git from aur and edit the PKGBUILD to use the master branch.

To do that just replace it from PKGBUILD:

e.g. pkgver=3.15..... by pkgver=master

Off-topic, but that isn't necessary/doesn't change anything. The PKGBUILD already pulls from the master branch, pkgver isn't used for anything else in the PKGBUILD. The resulting package version is described further down in the PKGBUILD with pkgver().