Allow redeploying to interactive tests

[Radvendii]

Description of changes

• adds a rebuild() command to the NixOS testing driver, that will rebuild the test and push any changes to running VMs
• adds a --rebuild-cmd flag to the driver to specify what command rebuilds the test (can also be passed directly to rebuild()
• adds a --internal-print-update-driver-info-and-exit flag to the driver that will just print the information needed to update another driver.
• document the changes
• do more extensive stress-testing
  • adding machines
  • removing machines
  • multi-machine configs
    • adding one machine and removing another
    • removing a machine changes the other machines' configs accordingly
  • changing vlans
  • changing testScript

Bugs

• The vlans are getting updated on every rebuild() for some reason

This PR has been reworked since it's original formulation. See below for the original text.

Original PR description

• adds a command to the output of driverInteractive, as well as in its own output redeploy, which will deploy the current test machine configuration to an actively running driverInteractive
• Adds an extra machine to driverInteractive which serves as a jump host so that the user can ssh into machines in the test (including to do the redeploy).
• Adds an ssh config to the output of driverInteractive, as well as in its own output sshConfig to take advantage of that jump host. This can be used with ssh -F ./result/ssh_config
• document changes (haven't yet done. waiting for feedback)
• automatically start jump host. Currently this setup requires the manual step of running redeploy_jumphost.start() after starting the interactive driver (couldn't figure out how to do this. does anyone have ideas?)

Add a 👍 reaction to pull requests you find important.

[Radvendii]

I opened the issue (#281332) for discussion of alternate solutions and whether this is an issue worth solving at all.

[Radvendii]

Tagging some people who have touched the testing infrastructure recently, and therefore might be qualified to comment / advise

@roberth @K900 @tfc

[roberth]

Question: The nodes are observable by the rest of the test, which will probably lead to heisenbugs. Assuming there isn't a command that just forwards socket connections to an address and port on a VDE network, perhaps you could use a different option for this extra node config; not nodes.<something>?

[tfc]

It feels like this should neither increase the complexity of the python code nor of the standard test module code.
Can we maybe have this similar to profiles in the NixOS modules? So that it is not included by default and people who don't use it don't have to regard it?

The reason for my thinking is that this is quite an opinionated way to do it (i.e. not switching on a few flags but even adding another host that ought to be used as jumphost etc.) and might not automatically play well with any test.

[Radvendii]

Points well taken. To summarize

1. This is a disruptive way of doing this, such that it shouldn't be on by default.

2. Mostly this is because it adds an extra machine

I will work on finding a way around (2). But if there isn't a clean way to do it in the end, maybe the solution is more to put this in NUR (or a flake), and in Nixpkgs the only change necessary would be to expose the underlying module system structure of the tests, so that it can be extended outside of Nixpkgs.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-55/40996/1

[RaitoBezarius]

cc @nikstur as folks working on the NixOS tests ecosystem have not been paged.

[K900]

Hard disagree. If we have slow iteration times, we should be removing complexity, not adding more complexity to bypass original complexity.

[Radvendii]

It's not complexity that leads to the slow iteration time, it's the time to restart the VM and spin up all the services again. For some tests this is quick, but for others it can be slow and there's not much we can do to change that if the test needs services that are slow to start.

There's also the issue of: if I am running the interactive driver, and making stateful changes to test things out, and then I want to make a change to the configuration and test that out, I now have to redo all of the stateful changes I made. It's a question of workflow getting disrupted.

[Radvendii]

There's another approach demonstrated here: https://github.com/tweag/nix-hour/blob/master/templates/vm/configuration.nix#L164-L167

Which mounts the relevant directory into the VM so that you can run some version of nixos-rebuild from inside the VM and it has the latest version of the source code.

This only works though because the directory structure is highly constrained. NixOS tests, and really any NixOS VM are Nix derivations that can come from, in theory, any Nix code, so you can't know how to extract the derivation from the directory, or even which directory to mount. There could even be references to arbitrary other parts of the file system.

I don't see a way to solve that problem...

[Radvendii]

Okay, I've completley reworked how this PR functions. No jumphosts, no sshing, no modification of the machines in the test.

Instead, I've added a method to the driver, rebuild() which will run a rebuild command and then get information from the produced executable about how to update the driver.

It then does that update, by changing fields on the machines, adding new machines and removing old ones, and running nixos-rebuild on machines whose configurations have changed.

I've also added a flag to the driver for internal use which just produces that information needed to update.

There's a little bit of "magic" going on where we assume the path of the new driver executable will be the same as the one we're running now. This works well with nix-build and nix build which modify the result/ symlink, but for instance it will not work if you nix run .#nixosTests.foo.driverInteractive directly, since no result/ symlink is produced. I think this is an okay balance of convenience to magic, especially since you can override the exe path if needed.

[tfc]

If I understand correctly, with N machines running, we still need to stop all of them before executing rebuild(...). This approach doesn’t seem significantly more advantageous than stopping the entire driver and rerunning it with the --keep-vm-state parameter. This way, the interactive shell history and the VMs’ disks remain intact for the next run. The most important difference is that machines that are not stopped remain running with all their state in RAM if that is of value for some scenario, right?

Could you please elaborate on your specific use case and the exact benefits you see from this additional complexity? Understanding this will help evaluate if the added complexity is justified.

[Radvendii]

If I understand correctly, with N machines running, we still need to stop all of them before executing rebuild(...)

No! This is only if you delete a machine from the test, communicating that you no longer want it to be there at all.

If you change the config of a machine, it will do the equivalent of running nixos-rebuild switch on that machine. Namely, it will build the new configuration and run <new-configuration-store-path>/system/bin/switch-to-configuration test on the running VM

[Radvendii]

I expect deleting machines from the test in this context will be quite uncommon. This code is just handling an edge case gracefully. Most of the time you will be modifying existing machines, and that's where this PR shines, since it allows you to redeploy the VMs in-place rather than restart them.

[roberth]

The documentation team recommends one sentence per line, which can be a long line. It works better with GitHub suggestions, as it requires no reflowing of these ignored line breaks.
(Pre-existing problem and not urgent for these additions)

[Radvendii]

Yeah, I guess I was trying to match the style of the document. Should I reformat the page since I'm editing it anyways, or should that be its own PR?

[roberth]

Could we log info about the running instances?

[Radvendii]

I'm not sure how this would happen so I'm not sure what would be useful info. If it does happen it's certainly a bug in the testing framework and not a problem with the user's code. Maybe I should say that explicitly.

[roberth]

This only works if the host store is forwarded into the VM.
If the VM is completely image based, the new store paths will have to be copied in.
What if we always forward the host store to /root/host-store in the guests? No sane program will find it there, so it doesn't affect the purity or performance of the test, but it does allow us to do a nix copy just for the purpose of making this work.
This should error out early if

nixpkgs/nixos/modules/virtualisation/qemu-vm.nix

Line 771 in 883180e

virtualisation.useNixStoreImage =

[Radvendii]

We don't have access to the nixos configuration here, only the built output. We could check for the presence of the store path, and if it doesn't exist produce a helpful error message that points to this as a likely culprit. Were you imagining something else?

Always forwarding the host store to /root/host-store seems like a big change for only this purpose. It seems not very common to set useNixStoreImage in tests, and IIUC it's only for performance purposes, so it can be turned off while interactively developing the test.

That being said, my feelings against adding /root/host-store are not strong, so if you or other people feel like it is a good change, I'm happy to add it in.

[YorikSar]

Suggested change

	$ ./result/bin/nixos-test-driver --rebuild-cmd $build_cmd 2>machine_output
	$ ./result/bin/nixos-test-driver --rebuild-cmd "$build_cmd"

$build_cmd has spaces, so should be in quotes to avoid potential issues depending on shell configuration.

2>machine_output will consume all stderr output, not just output from the VM. Maybe remove it from the docs to avoid confusion?

[Radvendii]

Good call on the quotes.

What stderr output are you worried about? Python errors, for instance, are output to stdout.

I added it to the docs because in my experience the interactive driver is extremely irritating to use with the VM output clogging up the interactive terminal, and it took me a while to realise I could redirect it.

[Radvendii]

Oh indeed calls to logger.info() and the like get sent to stderr. I'll just remove this, since it's not actually related to the PR.

[YorikSar]

I think it would be better to write these arguments in JSON in the interactive driver derivation output instead. This way we don't have less stuff to run outside of the sandbox, and less guessing about the binary path and its arguments.

You could even add a separate derivation that only outputs this JSON to avoid having to fetch new versions of host packages (qemu) that we won't use here anyway.

[Radvendii]

The reason I did it the way I did is that things seem to be built so that the driver itself is a standalone executable. It gets wrapped in a wrapper that sets arguments. So I tried with this PR to maintain the separation between those layers. But they could also get set by command line arguments or environment variables.

What you're proposing puts us in a state where the executable depends on the wrapper being set up in a particular way, which feels like mixing layers to me.

Maybe I'm being overly concerned about this though, and the layers are already mixed up or we just don't care if we mix them up. If so, I think your solution is indeed better, but I didn't know enough to make that judgment call myself.

[Radvendii]

A couple of updates:

1. As has been noted in several issues Python: use environment variable for script name #24525 General problems with environment variable wrappers #60260 Finally handle wrappers correctly to make them invisible to applications #150841, argv0 doesn't get passed on properly to python scripts.

   Assuming the new driver executable is at argv0 does not work. It will have to be passed in explicitly, unless we want to assume result/bin/nixos-test-driver which feels like mixing layers of abstraction. Though admittedly, more convenient and would work in most cases.

2. Python globals() is... weird. Bottom line, it's not actually global variables. See stackoverflow explanation. I managed to hack together the behaviour I was going for (add/remove global variables) using inspect. It seems like a big kludge but I couldn't think of a better way to do it.

[Radvendii]

@roberth Wow, thanks for that test! At some point I tried to work out how to automate testing for this feature and kind of gave up as "tests within tests are broken". Good to see that it is possible.
I added my own stuff to the test for more edge-cases that I ran into. I think it should be good to go now. In theory JSON is better, but I think better to get this work in and do it as a follow-up if needed.

[Radvendii]

@roberth can we push this through?

[tfc]

Hey @Radvendii, i am rejoining this thread very late. Just tried it out on my machine and got an error where i am not sure if i did anything wrong or if it is a bug.

What i did so far was the following:

1. run nix-build . -A pkgs.nixosTests.bittorrent.driverInteractive (in the nixpkgs repo of course)
2. run ./result/bin/nixos-test-driver --rebuild-cmd "nix-build . -A pkgs.nixosTests.bittorrent.driverInteractive"
3. after start_all() and waiting a bit, i changed the configuration of one machine (client1 in that case) by adding some new package to its environment.systemPackages.
4. Run rebuild().
5. This leads to an error /nix/store/[...]-nixos-vm/system/bin/switch-to-configuration: No such file or directory

The store path exists, but there's no bin in the system folder. Not sure what the assumptions were that come with the new code.

Can you give me a pointer what i might have done wrong here?

[Radvendii]

You're holding it completely right. This is definitely a bug I introduced in later commits. Thanks for finding it! I know vaguely where it is but I haven't had the time and brainspace to fix it and work out why the test isn't failing. I'll ping you when I do.