dmd: bootstrap from source

[jtbx]

Description of changes

Bootstrap DMD from source (previously using GDC).

Supersedes #317845

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[jtbx]

Currently stuck on #324961

[jtbx]

Building the bootstrap package against 2.108.0 fails building the backend with the following error:

Command: /nix/store/cjx6hinsm7cjn4ps428q4v32lqbzmrn8-gdmd-0.1.0-unstable-2024-05-30/bin/gdmd -c -of/build/dmd/generated/linux/release/64/backend.o -w -de -fPIC -m64 -J/build/dmd/generated/linux/release/64 -I/build/dmd/compiler/src -O -inline -release src/dmd/backend/backend.d src/dmd/backend/bcomplex.d src/dmd/backend/evalu8.d src/dmd/backend/divcoeff.d src/dmd/backend/dvec.d src/dmd/backend/go.d src/dmd/backend/gsroa.d src/dmd/backend/glocal.d src/dmd/backend/gdag.d src/dmd/backend/gother.d src/dmd/backend/gflow.d src/dmd/backend/dout.d src/dmd/backend/inliner.d src/dmd/backend/gloop.d src/dmd/backend/compress.d src/dmd/backend/cgelem.d src/dmd/backend/cgcs.d src/dmd/backend/ee.d src/dmd/backend/cod4.d src/dmd/backend/cod5.d src/dmd/backend/eh.d src/dmd/backend/nteh.d src/dmd/backend/blockopt.d src/dmd/backend/mem.d src/dmd/backend/cg.d src/dmd/backend/cgreg.d src/dmd/backend/dtype.d src/dmd/backend/debugprint.d src/dmd/backend/fp.d src/dmd/backend/symbol.d src/dmd/backend/symtab.d src/dmd/backend/elem.d src/dmd/backend/dcode.d src/dmd/backend/cgsched.d src/dmd/backend/cg87.d src/dmd/backend/cgxmm.d src/dmd/backend/cgcod.d src/dmd/backend/cod1.d src/dmd/backend/cod2.d src/dmd/backend/cod3.d src/dmd/backend/cv8.d src/dmd/backend/dcgcv.d src/dmd/backend/pdata.d src/dmd/backend/util2.d src/dmd/backend/var.d src/dmd/backend/backconfig.d src/dmd/backend/drtlsym.d src/dmd/backend/dwarfeh.d src/dmd/backend/ptrntab.d src/dmd/backend/dvarstats.d src/dmd/backend/dwarfdbginf.d src/dmd/backend/cgen.d src/dmd/backend/goh.d src/dmd/backend/barray.d src/dmd/backend/cgcse.d src/dmd/backend/elpicpie.d src/dmd/backend/machobj.d src/dmd/backend/elfobj.d src/dmd/backend/mscoffobj.d src/dmd/backend/filespec.d src/dmd/backend/cgobj.d src/dmd/backend/aarray.d src/dmd/backend/disasm86.d

-----------------------------------------------------------
src/dmd/backend/machobj.d:2993:32: error: argument '&(*__withSym).major' for format specification '"%d"' must be 'int', not 'immutable(int)*'
 2993 |         str.sscanf("%d.%d.%d", &major, &minor, &build);
      |                                ^
src/dmd/backend/machobj.d:2993:40: error: argument '&(*__withSym).minor' for format specification '"%d"' must be 'int', not 'immutable(int)*'
 2993 |         str.sscanf("%d.%d.%d", &major, &minor, &build);
      |                                        ^
src/dmd/backend/machobj.d:2993:48: error: argument '&(*__withSym).build' for format specification '"%d"' must be 'int', not 'immutable(int)*'
 2993 |         str.sscanf("%d.%d.%d", &major, &minor, &build);
      |                                                ^

Building against 2.109.1 fails to link the dmd binary:

Command: /nix/store/cjx6hinsm7cjn4ps428q4v32lqbzmrn8-gdmd-0.1.0-unstable-2024-05-30/bin/gdmd -of/build/dmd/generated/linux/release/64/dmd -vtls -J/build/dmd/compiler/src/dmd/res -w -de -fPIC -m64 -J/build/dmd/generated/linux/release/64 -I/build/dmd/compiler/src -O -inline -release src/dmd/dinifile.d src/dmd/dmdparams.d src/dmd/gluelayer.d src/dmd/lib.d src/dmd/libelf.d src/dmd/libmach.d src/dmd/libmscoff.d src/dmd/libomf.d src/dmd/link.d src/dmd/mars.d src/dmd/main.d src/dmd/scanelf.d src/dmd/scanmach.d src/dmd/scanmscoff.d src/dmd/scanomf.d src/dmd/vsoptions.d src/dmd/access.d src/dmd/aggregate.d src/dmd/aliasthis.d src/dmd/argtypes_x86.d src/dmd/argtypes_sysv_x64.d src/dmd/argtypes_aarch64.d src/dmd/arrayop.d src/dmd/arraytypes.d src/dmd/astenums.d src/dmd/ast_node.d src/dmd/astcodegen.d src/dmd/asttypename.d src/dmd/attrib.d src/dmd/attribsem.d src/dmd/basicmangle.d src/dmd/blockexit.d src/dmd/builtin.d src/dmd/canthrow.d src/dmd/chkformat.d src/dmd/cli.d src/dmd/clone.d src/dmd/compiler.d src/dmd/cond.d src/dmd/constfold.d src/dmd/cppmangle.d src/dmd/cppmanglewin.d src/dmd/cpreprocess.d src/dmd/ctfeexpr.d src/dmd/ctorflow.d src/dmd/dcast.d src/dmd/dclass.d src/dmd/declaration.d src/dmd/delegatize.d src/dmd/denum.d src/dmd/dimport.d src/dmd/dinterpret.d src/dmd/dmacro.d src/dmd/dmangle.d src/dmd/dmodule.d src/dmd/doc.d src/dmd/dscope.d src/dmd/dstruct.d src/dmd/dsymbol.d src/dmd/dsymbolsem.d src/dmd/dtemplate.d src/dmd/dtoh.d src/dmd/dversion.d src/dmd/enumsem.d src/dmd/escape.d src/dmd/expression.d src/dmd/expressionsem.d src/dmd/func.d src/dmd/funcsem.d src/dmd/hdrgen.d src/dmd/iasm.d src/dmd/iasmgcc.d src/dmd/impcnvtab.d src/dmd/imphint.d src/dmd/importc.d src/dmd/init.d src/dmd/initsem.d src/dmd/inline.d src/dmd/inlinecost.d src/dmd/intrange.d src/dmd/json.d src/dmd/lambdacomp.d src/dmd/mtype.d src/dmd/mustuse.d src/dmd/nogc.d src/dmd/nspace.d src/dmd/ob.d src/dmd/objc.d src/dmd/opover.d src/dmd/optimize.d src/dmd/parse.d src/dmd/parsetimevisitor.d src/dmd/permissivevisitor.d src/dmd/postordervisitor.d src/dmd/pragmasem.d src/dmd/printast.d src/dmd/rootobject.d src/dmd/safe.d src/dmd/sapply.d src/dmd/semantic2.d src/dmd/semantic3.d src/dmd/sideeffect.d src/dmd/statement.d src/dmd/statement_rewrite_walker.d src/dmd/statementsem.d src/dmd/staticassert.d src/dmd/staticcond.d src/dmd/stmtstate.d src/dmd/target.d src/dmd/templatesem.d src/dmd/templateparamsem.d src/dmd/traits.d src/dmd/transitivevisitor.d src/dmd/typesem.d src/dmd/typinf.d src/dmd/utils.d src/dmd/visitor.d src/dmd/foreachvar.d src/dmd/cparse.d src/dmd/dmsc.d src/dmd/e2ir.d src/dmd/iasmdmd.d src/dmd/glue.d src/dmd/objc_glue.d src/dmd/s2ir.d src/dmd/tocsym.d src/dmd/toctype.d src/dmd/tocvdebug.d src/dmd/todt.d src/dmd/toir.d src/dmd/toobj.d src/dmd/backend/cc.d src/dmd/backend/cdef.d src/dmd/backend/cgcv.d src/dmd/backend/code.d src/dmd/backend/cv4.d src/dmd/backend/dt.d src/dmd/backend/el.d src/dmd/backend/global.d src/dmd/backend/obj.d src/dmd/backend/oper.d src/dmd/backend/rtlsym.d src/dmd/backend/code_x86.d src/dmd/backend/iasm.d src/dmd/backend/codebuilder.d src/dmd/backend/ty.d src/dmd/backend/type.d src/dmd/backend/mach.d src/dmd/backend/mscoff.d src/dmd/backend/dwarf.d src/dmd/backend/dwarf2.d src/dmd/backend/xmm.d src/dmd/backend/dlist.d src/dmd/backend/melf.d src/dmd/root/aav.d src/dmd/root/complex.d src/dmd/root/env.d src/dmd/root/longdouble.d src/dmd/root/man.d src/dmd/root/optional.d src/dmd/root/response.d src/dmd/root/speller.d src/dmd/root/string.d src/dmd/root/strtold.d ../generated/linux/release/64/lexer.o ../generated/linux/release/64/backend.o ../generated/linux/release/64/common.o

-----------------------------------------------------------
/nix/store/2p1sq1nr09xr3xb1a9lrjgdanvk1aakb-binutils-2.42/bin/ld: /nix/store/p93ircqs77faxik9b4766rgziyjv6269-gdc-11.4.0/lib/gcc/x86_64-unknown-linux-gnu/11.4.0/../../../../lib64/libgphobos.a(lifetime.o): in function `_d_allocmemory':
(.text._d_allocmemory+0x0): multiple definition of `_d_allocmemory'; ../generated/linux/release/64/lexer.o:console.d:(.text+0x10880): first defined here
/nix/store/2p1sq1nr09xr3xb1a9lrjgdanvk1aakb-binutils-2.42/bin/ld: /nix/store/p93ircqs77faxik9b4766rgziyjv6269-gdc-11.4.0/lib/gcc/x86_64-unknown-linux-gnu/11.4.0/../../../../lib64/libgphobos.a(lifetime.o): in function `_d_newclass':
(.text._d_newclass+0x0): multiple definition of `_d_newclass'; ../generated/linux/release/64/lexer.o:console.d:(.text+0x108e0): first defined here
/nix/store/2p1sq1nr09xr3xb1a9lrjgdanvk1aakb-binutils-2.42/bin/ld: /nix/store/p93ircqs77faxik9b4766rgziyjv6269-gdc-11.4.0/lib/gcc/x86_64-unknown-linux-gnu/11.4.0/../../../../lib64/libgphobos.a(lifetime.o): in function `_d_newitemT':
(.text._d_newitemT+0x0): multiple definition of `_d_newitemT'; ../generated/linux/release/64/lexer.o:console.d:(.text+0x10940): first defined here
/nix/store/2p1sq1nr09xr3xb1a9lrjgdanvk1aakb-binutils-2.42/bin/ld: /nix/store/p93ircqs77faxik9b4766rgziyjv6269-gdc-11.4.0/lib/gcc/x86_64-unknown-linux-gnu/11.4.0/../../../../lib64/libgphobos.a(lifetime.o): in function `_d_newitemiT':
(.text._d_newitemiT+0x0): multiple definition of `_d_newitemiT'; ../generated/linux/release/64/lexer.o:console.d:(.text+0x109a0): first defined here
collect2: error: ld returned 1 exit status

This could be caused by GDC apparently being broken when linking in C/C++ libraries:

https://github.com/NixOS/nixpkgs/blob/85a13515a5fd98ca1a4aeefcaedcb6417f841e13/pkgs/top-level/all-packages.nix#L15708-L15710

Another thing I found was this note in GCC's package.

nixpkgs/pkgs/development/compilers/gcc/default.nix

Lines 233 to 235 in 85a1351

	# TODO: fixup D bootstapping, probably by using gdc11 (and maybe other changes).
	# error: GDC is required to build d
	assert atLeast12 -> !langD;

These lines were added in commit cb4e7fd from #174125, which is related to #241341, which was fixed in #241628. #261947 is meant to fix a lot of these issues with the GDC package, but it's stalled so I think we've hit a dead end for now. I'll look into this a bit later.

[dukc]

It might be easier to go with LDC since it's already used at dlang.nix and is better up to date. It currently isn't bootstrapping either in Nixpkgs, but the compiler does support that.

And we'll get another perk switching compilers even before bootstrapping is supported: DMD gets optimised better and thus is faster for the user.

[jtbx]

Sounds good, I'll try that later

[jtbx]

Bootstrapping from LDC worked, thanks @dukc!

[CyberShadow]

But isn't LDC itself bootstrapped from an older LDC binary?

[jtbx]

Ah right, do you know if there's a tag of LDC without D we could build from? I think I remember one but I can't find it.

[CyberShadow]

Yes, dmd-rewrite-last-cdmd is the last commit with C++ source code.

Note that this was a while ago, so a few interim versions may be required to bootstrap all the way to the latest compiler versions.

[jtbx]

Awesome, thanks. I'll try building from 0.17; the wiki page for building LDC says that 0.17 can be used for bootstrapping, and by the looks dmd-rewrite-last-cdmd uses the same frontend as 0.17 anyway.

[jtbx]

The latest LLVM supported by LDC 0.17 is LLVM 6, which was removed a while ago. Trying to bootstrap from the dmd-cxx branch of DMD and then bootstrap LDC using that.

[jtbx]

I'm a bit stuck here, I've just realised that DMD can only generate code for x86 CPUs which won't work for all platforms. On the other hand LDC 0.17 won't compile because it's too old. The dmd-rewrite-last-cdmd branch seems to contain DMD code without an LLVM backend, essentially an older version of DMD's dmd-cxx. What would you suggest?

[dukc]

Honestly not sure, sorry. I opened an LDC issue for you but I can't tell you whether the maintainers will want to support this case. If not, I quess it's either back to gdc bootstrap (possibly still easier to do via intermediate LDC versions though) or readding support for older LLVM modules. I honestly don't know which is easier and I won't hold it on you if you decide it's not worth the effort.

Regardless of what happens, I suggest committing the LDC compilation for dmd, because it means DMD is faster.

[jtbx]

I think both GDC and LDC are not worth the hassle anymore. Starting from DMD is still an option though (about optimizing DMD using LDC, we can still build the final DMD using the final LDC). It sounds like a bit of a mess but I think it can be done as long as 095ba4f is reverted.