Memory ballooning issue is back for the fedora-41 template #9663
Comments
lubellier
added
P: default
|
I think the issue is that |
andrewdavidwong
added
C: Fedora
needs diagnosis
|
@marmarek Both a macro and a script would work. An online relabel is racy and might or might not work, or it might work most of the time and fail occasionally. However, I expect that most policy upgrades do not require a relabel, and in any case whether a relabel should happen is up to Fedora, not Qubes OS, except for upgrades to our own policy. |
|
It seems that's not the correct diagnosis. After the update, |
marmarek
added
P: major
|
That's not it either... |
|
And at the same time, files in |
|
As for
I'm confused... |
|
While doing some tests, I noticed that reinstalling |
|
If I start with the Fedora 41 minimal template, install I think the best place to get help is from upstream Fedora. |
Which package was updated? |
|
@marmarek: can you check if a full relabel followed by two reboots solves the problem? |
If this happens while |
Why multiple relabels/reboots would change anything? We are talking about /run, which doesn't survive reboot... |
|
FWIW manually forcing full relabel does not help. interesting mode... |
|
Diff of the resulting @@ -5149,7 +5153,6 @@
/usr/bin/livecd-creator -- system_u:object_r:livecd_exec_t:s0
/usr/bin/lttng-sessiond -- system_u:object_r:lttng_sessiond_exec_t:s0
/usr/bin/mariadb-backup -- system_u:object_r:mysqld_exec_t:s0
-/usr/bin/meminfo-writer -- system_u:object_r:qubes_meminfo_writer_exec_t:s0
/usr/bin/modules-update -- system_u:object_r:kmod_exec_t:s0
/usr/bin/mount\.ecryptfs -- system_u:object_r:mount_ecryptfs_exec_t:s0
/usr/bin/neutron-server -- system_u:object_r:neutron_exec_t:s0
@@ -5160,7 +5163,6 @@
/usr/bin/partition_uuid -- system_u:object_r:fsadm_exec_t:s0
/usr/bin/qemu-pr-helper -- system_u:object_r:virtd_exec_t:s0
/usr/bin/quantum-server -- system_u:object_r:neutron_exec_t:s0
-/usr/bin/qubesdb-daemon -- system_u:object_r:qubes_qubesdb_daemon_exec_t:s0
/usr/bin/restart-dirsrv -- system_u:object_r:initrc_exec_t:s0
/usr/bin/roundup-server -- system_u:object_r:roundup_exec_t:s0
/usr/bin/samba-gpupdate -- system_u:object_r:samba_gpupdate_exec_t:s0This looks okay, as both files are in fact in /usr/sbin, not /usr/bin. And entries for /usr/sbin are still there. What I think is happening is that %post script of the policy package does So, I think fixfiles thinks In fact, there is surprisingly little entries for |
|
There is So, theoretically it should be called after the update, but I guess something went wrong here. And as can be seen above, there are specific packages that has it applied to via triggers. This I think the most reliable solution would be to add /usr/bin entries to our policy for all /usr/sbin binaries manually (have them more or less duplicated). This should fix the situation on Fedora (to be tested), regardless if |
/usr/sbin is now considered an alias for /usr/bin, so include rules for /usr/bin to ensure correct labelling. Do not remove the old rules to avoid regression on Fedora 40. Part-of: QubesOS/qubes-issues#9663
/usr/sbin is now considered an alias for /usr/bin, so include rules for /usr/bin to ensure correct labelling. Do not remove the old rules to avoid regression on Fedora 40. Suggested-by: Marek Marczykowski-Górecki <[email protected]> Part-of: QubesOS/qubes-issues#9663
/usr/sbin is now considered an alias for /usr/bin, so include rules for /usr/bin to ensure correct labelling. Do not remove the old rules to avoid regression on Fedora 40. Suggested-by: Marek Marczykowski-Górecki <[email protected]> Part-of: QubesOS/qubes-issues#9663
/usr/sbin is now considered an alias for /usr/bin, so include rules for /usr/bin to ensure correct labelling. Do not remove the old rules to avoid regression on Fedora 40. Suggested-by: Marek Marczykowski-Górecki <[email protected]> Part-of: QubesOS/qubes-issues#9663
|
The today stable fedora-41 template update fixed the issue. 👍 I noted some errors about the SELinux policy on package installation : |
|
The same updates prevented it for me in unaffected templates. I didn't notice such errors on my end, though. |
|
I am wondering if this issue is back with the latest Fedora 41 updates which I installed about an hour ago today. If I open a few programs my work qube becomes very laggy and in the log I see messages like this: In the update history I can see that selinux got some updates: Let me know if there is anything else I should check to find the root cause. Thanks. |
|
How much memory that qube has at the time? |
|
I set initial memory 1600 MB and max memory to 16000 MB. |
|
I can confirm the issue is back. My sys-firewall is set to 4000MB in its settings. After this last update which included selinux, it is down to a 500MB limit. Edit: This workaround still works #9663 (comment) |
|
I got this issue back too today after the latest |
|
Out of 4 fedora-41-xfce templates, I can reproduce the issue on only one. There, I have this: But I have |
Do you remember if you had any scriptlet errors while installing this update when it was pushed to the testing repository? |
|
Packages with SELinux policy changes cannot install successfully with only 400MiB RAM. The updater plugin needs to check that the packages have been installed properly and if they have not, reinstall them after increasing the amount of memory available. |
|
If you meant me, I wrote four zeros, as in four thousand, not four hundred. If you didn't mean me then never mind. Edit: As for the fedora template itself, its minimum is four hundred, 2000MB max. |
Can this be enough reason to rebuild the fedora-41 template? |
|
That won't help users with this issue. |
I didn't specify, sorry... My proposal is aimed at users that haven't installed the template yet so they don't face these problems like having to run the updater twice. Nevermind, I think I can handle it...
Reading this again makes me thing that if any SELinux policy changes needs to be added to fix qmemman again, any template or standalone that has the minimum amount of memory set to 400MiB will need to increase it permanently to be "future proof". |
When qmemman works, this isn't an issue in practice, and when it doesn't, rebuilding SELinux policy is just one of many issues... In fact, the thing that breaks with qmemman ist just qrexec signaling qmemman (or rather: meminfo-writer service) that it should start reporting memory usage (via sending USR1 signal), otherwise qmemman is functional. Maybe the updater should send the signal itself, just in case? As for changing initial memory, that isn't a good idea - settings on templates are used as default values for appvms based on it, so if you increase initial memory for the template, then all your vms will use more initially. |
I agree it should be done to avoid having to rerun the update. Problem is greater when installing a salt formula chained with highstate. As a state from a qube can't depend on another, if updating the fedora template fails, the next state such as cloning the fedora template will still occur having one more copy of a bad tree that will later be used to create other qubes based on this new template. Sure, I could do error checking with each qubesctl command using state.apply, to break the installation if any fails... which may be needed on other circumstances.
I agree, I don't want it also, just trying to see if it was needed at all and I was wrong. Thanks for the explanation as always. |
I just made a pull request based on that idea to replace the service restart which is unreliable. |
|
Why does qrexec need to tell qmemman to start reporting memory usage? That seems like a rather pointless requirement. Also, should the updater check if the policy is busted and forcibly reinstall the policy packages? |
* origin/pr/179: vmupdate: Send SIGUSR1 to meminfo-writer QubesOS/qubes-issues#9663
Qubes OS release
Qubes OS 4.2.3
Brief summary
The fedora-41 template based AppVMs stay to the minimal memory limit.
Other users got also this issue, see the related topic in the forum.
Steps to reproduce
Expected behavior
The XX current memory should grow over 400MB.
Actual behavior
Related to Fedora 41 template / selinux
Done checks
I did the
/.qubes-relabeledremoving /qrexec_timeoutprocedure in the fedora-41 template, the relabel job executed and re-created the/.qubes-relabeledfile as expected:fedora-41 logs:
On next boot:
Logs of the XX AppVM:
The text was updated successfully, but these errors were encountered: