Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the ability to devour only uncached derivations of a flake #10

Closed
wants to merge 9 commits into from
52 changes: 52 additions & 0 deletions devour/flake.nix
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't had the time to look at this (and the other) PR in detail yet, but could you aim to minimize the diff in your PRs? It would make the review easier for me. For eg., there is no need to move the code to a new devour/flake.nix.

Refactors are good, but should happen on a PR of their own.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For eg., there is no need to move the code to a new devour/flake.nix.

I wanted the actual flake itself to be usable without having to go through what's in nixpkgs. As it stands today the stubbed flake = {} input makes it so it cannot be evaluated. By moving it to a separate directory the scripts can still reference it while they themselves being exposed by the actual repo flake.

Refactors are good, but should happen on a PR of their own.

There wasn't a good way for me to introduce the new build script and show an end-to-end working concept. I tried to keep each of the commits small and incremental so they're easier to review that way. If you'd prefer me to split things out into multiple PRs I can do that, but the content will end up being the same

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you'd prefer me to split things out into multiple PRs I can do that, but the content will end up being the same

Yes. Every PR should do one thing, and one thing only. Even when you are updating flake.lock (0591b56), that should be a PR of its own.

Copy link
Contributor Author

@ipetkov ipetkov Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
systems.url = "github:nix-systems/default";
flake-parts.url = "github:hercules-ci/flake-parts";
flake = { };
};
outputs = inputs@{ flake-parts, systems, ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
systems = import systems;
perSystem = { self', pkgs, lib, system, ... }: {
packages.default =
let
# Given a flake output key, how to get the buildable derivation for
# any of its attr values?
flakeSchema = {
perSystem = {
lookupFlake = k: lib.attrByPath [ k system ] { };
getDrv = {
packages = _: x: [ x ];
checks = _: x: [ x ];
devShells = _: x: [ x ];
apps = _: app: [ app.program ];
legacyPackages = k: v:
if k == "homeConfigurations"
then
lib.mapAttrsToList (_: cfg: cfg.activationPackage) v
else [ ];
};
};
flake = {
lookupFlake = k: lib.attrByPath [ k ] { };
getDrv = {
nixosConfigurations = _: cfg:
lib.optional pkgs.stdenv.isLinux cfg.config.system.build.toplevel;
darwinConfigurations = _: cfg:
lib.optional pkgs.stdenv.isDarwin cfg.config.system.build.toplevel;
};
};
};
paths =
lib.flip lib.mapAttrsToList flakeSchema (lvl: lvlSchema:
lib.flip lib.mapAttrsToList lvlSchema.getDrv (kind: getDrv:
lib.mapAttrsToList
getDrv
(lvlSchema.lookupFlake kind inputs.flake))
);
in
pkgs.writeText "devour-output" (lib.strings.concatLines (lib.lists.flatten paths));
};
};
}
18 changes: 9 additions & 9 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

55 changes: 14 additions & 41 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -3,52 +3,25 @@
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
systems.url = "github:nix-systems/default";
flake-parts.url = "github:hercules-ci/flake-parts";
flake = { };
};
outputs = inputs@{ flake-parts, systems, ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
systems = import systems;
perSystem = { self', pkgs, lib, system, ... }: {
packages.default =
let
# Given a flake output key, how to get the buildable derivation for
# any of its attr values?
flakeSchema = {
perSystem = {
lookupFlake = k: lib.attrByPath [ k system ] { };
getDrv = {
packages = _: x: [ x ];
checks = _: x: [ x ];
devShells = _: x: [ x ];
apps = _: app: [ app.program ];
legacyPackages = k: v:
if k == "homeConfigurations"
then
lib.mapAttrsToList (_: cfg: cfg.activationPackage) v
else [ ];
};
};
flake = {
lookupFlake = k: lib.attrByPath [ k ] { };
getDrv = {
nixosConfigurations = _: cfg:
lib.optional pkgs.stdenv.isLinux cfg.config.system.build.toplevel;
darwinConfigurations = _: cfg:
lib.optional pkgs.stdenv.isDarwin cfg.config.system.build.toplevel;
};
};
};
paths =
lib.flip lib.mapAttrsToList flakeSchema (lvl: lvlSchema:
lib.flip lib.mapAttrsToList lvlSchema.getDrv (kind: getDrv:
lib.mapAttrsToList
getDrv
(lvlSchema.lookupFlake kind inputs.flake))
);
in
pkgs.runCommand "devour-output" { inherit paths; } ''
echo -n $paths > $out
'';
packages = rec {
default = pkgs.symlinkJoin {
name = "devour-flake";
paths = [
devour-flake
devour-flake-uncached
];
};

# Build all derivations in a flake (or download them from the cache)
devour-flake = pkgs.callPackage ./pkgs/devour-flake.nix { };
# Only build all uncached derivations in flake
devour-flake-uncached = pkgs.callPackage ./pkgs/devour-flake-uncached.nix { };
};
};
};
}
43 changes: 43 additions & 0 deletions pkgs/devour-flake-uncached.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
{ coreutils-full
, findutils
, jq
, nix
, nix-build-uncached
, writeShellApplication
}:

writeShellApplication {
name = "devour-flake-uncached";
runtimeInputs = [
coreutils-full
findutils
jq
nix
nix-build-uncached
];
text = ''
set -euo pipefail

FLAKE="$1"
shift 1 || true

outDir=$(mktemp -d devour-flake-uncached.XXXX)
cleanup() {
rm -rf "$outDir"
}
trap cleanup EXIT

nix derivation show \
${../devour}#default \
"$@" \
-L \
--reference-lock-file ${../flake.lock} \
--override-input flake "$FLAKE" \
| jq -r 'to_entries[].value.inputDrvs | to_entries[].key' \
| (
cd "$outDir"
1>&2 xargs nix-build-uncached
find . -type l -exec readlink '{}' \;
)
'';
}
3 changes: 2 additions & 1 deletion default.nix → pkgs/devour-flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ writeShellApplication {
FLAKE="$1"
shift 1 || true

nix build ${./.}#default \
nix build ${../devour}#default \
"$@" \
-L --no-link --print-out-paths \
--reference-lock-file ${../flake.lock} \
--override-input flake "$FLAKE" \
| xargs cat
'';
Expand Down