lib.packagesFromDirectoryRecursive: use explicit recursion, support nested scopes

[nbraud]

Follow-up on #359941

• Extend lib.filesystem.packagesFromDirectoryRecursive with support for explicit recursion.
• Add an example showing how to use recurseIntoDirectory.
• Update packagesFromDirectoryRecursive to create nested scopes for each (sub)directory (not just the top-level one) when newScope is given.
• Apply recurseIntoAttrs by default when recursing into a directory. (Deferred to follow-up PR)
• Add a test using scopes for packagesFromDirectoryRecursive, ensuring:
  • packages can depend on other packages in the same scope and in ancestor scopes (the scope's parent or grandparent) ;
  • packages can have mutual dependencies.
• Reject unknown parameters (breaking change!)

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested nix-instantiate --eval --strict lib/tests/misc.nix
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Note
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[nbraud]

Ran nixfmt over the new test fixtures

[nbraud]

Rebased following the parent PR being merged

[nbraud]

I'd prefer to land #359898 first, as there will be a merge conflict over the function's documentation, and having the improved documentation will make documenting this PR's changes simpler.

PS: Marked as draft again, since I'll wait for that to land before writing documentation in this PR.

[nbraud]

Rebased atop both blocking PRs, added documentation.

Still needs an example how to use recurseIntoDirectory

[nbraud]

This is only waiting for reviews, no functional change since the beginning of the month: only rebased to address the merge conflict and squash fixup commits, and moved the release notes entries. (One further change was tried and reverted, to be done in #369421)

@9999years if you re-review, I think we could get that merged soonish

[hsjobeki]

Nice changes. Thank you for the efforts that you already put into it.

[nbraud]

I'll hold off on rebasing (to address the merge conflict and squash fixup commits) until @hsjobeki can reply, to make the review work easier as it's possible to only display the new changes.

[infinisil]

Thanks for the PR! Me and @hsjobeki reviewed this PR more closely together in a call and agreed on this feedback:

• The new newScope attribute argument and the removal of ... from packagesFromDirectoryRecursive looks good to us.
• However, the recurseIntoDirectory concept feels a bit messy and hard to understand (we didn't quite get it in our ~1 hour of reviewing this PR together). In particular:
  • It's unclear what processDir does based on the documentation
  • recurseArgs has unclear motivation, is not tested and is missing example documentation.

In alignment with the one PR for each change lib PR guideline, we think this PR should defer the recurseIntoDirectory and recurseArgs introduction to a separate PR that clearly states motivation for the new interfaces and adds missing docs with examples.

Also note that there's now separate Nixpkgs release notes where you need to note the lib changes, which is the cause of the merge conflict.

[nbraud]

Thanks for the review (again!) @hsjobeki & @infinisil !
Sorry for the slow reply, I've been mainly stuck in bed by migraines 😬

[...] the recurseIntoDirectory concept feels a bit messy and hard to understand (we didn't quite get it in our ~1 hour of reviewing this PR together). In particular:

  * It's unclear what `processDir` does based on the documentation
  * `recurseArgs` has unclear motivation, is not tested and is missing example documentation.

Duly noted. FWIW, recurseArgs was introduced when starting to reject unknown args: without a dedicated argument, it becomes very inconvenient for a user-provided recurseIntoDirectory function to keep any data of its own. I just hadn't found a usecase compelling-enough to add an example for, where that's needed.

(I said “very inconvenient” because it's technically doable by changing recurseIntoDirectory between recursive calls; I'm not sure that would even occur as a possibility to most users.)

In alignment with the one PR for each change lib PR guideline, we think this PR should defer the recurseIntoDirectory and recurseArgs introduction to a separate PR that clearly states motivation for the new interfaces and adds missing docs with examples.

I'm somewhat confused there, since adding open recursion is part of how nested scopes are implemented (at least in this PR) but I can remove it from the public API and simplify as much as possible.

Also note that there's now separate Nixpkgs release notes where you need to note the lib changes, which is the cause of the merge conflict.

I'm aware. I simply haven't rebased yet, so it's easier to see which changes are introduced between reviews.

[infinisil]

I'm somewhat confused there, since adding open recursion is part of how nested scopes are implemented (at least in this PR) but I can remove it from the public API and simplify as much as possible.

The makeScope is already a useful change on its own. The fact that it's more limited allows guaranteeing more about the result in a more abstract sense. I like the saying "Constraints Liberate, Liberties Constrain" here. Of course, it's always possible to remove constraints later (by adding recurseIntoDirectory), but we can't go the other way and add more constraints (that would be a breaking change). So I think it's a good idea to start out with more constraints and loosen them as necessary :)

[nbraud]

The makeScope is already a useful change on its own. [...] So I think it's a good idea to start out with more constraints and loosen them as necessary :)

That's fair. In general I'm not keen on “hard-coding” behaviours like that without any way to pass a different one, but I guess it does cover the majority of usecases.
In any case, the fully-generic implementation will remain in the PR's history anyhow¹ if the need occurs.

Footnotes

1. Assuming GitHub never garbage-collects commits (from the actual git repos) which are linked-to by the PR UI... which I have no idea about 😅
   I guess I ought to tag that locally in case I need it. ↩

[infinisil]

The current code only applies recurseIntoAttrs if makeScope is passed. That seems like an oversight.

[infinisil]

Probably makes sense to move recurseIntoAttrs into processDir to apply it unconditionally.