Installing MOOSE with CUDA

[rnpratoori]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

I am trying to install MOOSE with CUDA. For this, I followed the HPC installation with MPICH built from source. For the installation of PETSc, I modified the configure_petsc.sh with the following options

python3 ./configure --with-64-bit-indices CC=$CC CXX=$CXX FC=$FC F77=$F77 F90=$F90\
      --with-cxx-dialect=C++17 \
      --with-debugging=no \
      --with-fortran-bindings=0 \
      --with-mpi=1 \
      --with-openmp=1 \
      --with-strict-petscerrorcode=1 \
      --with-shared-libraries=1 \
      --with-sowing=0 \
      --with-cuda=1 \
      --download-fblaslapack=1 \
      --download-hpddm=1 \
      --download-hypre=1 \
      --download-metis=1 \
      --download-mumps=1 \
      --download-ptscotch=1 \
      --download-parmetis=1 \
      --download-scalapack=1 \
      --download-slate=1 \
      --download-slepc=1 \
      --download-strumpack=1 \
      --download-superlu_dist=1 \
      $HDF5_STR \
      $MUMPS_ARM_STR \
      $MAKE_NP_STR \
      "$@"

This runs fine with no errors.
But when I verify the PETSc installation using

make PETSC_DIR=/home/rnp/Software/MOOSE/projects/moose/scripts/../petsc PETSC_ARCH=arch-moose check

I get the following output

CH=arch-moose check
Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/rnp/Software/MOOSE/projects/moose/scripts/../petsc and PETSC_ARCH=arch-moose
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
3,5c3
<   1 SNES Function norm 4.12227e-06
<   2 SNES Function norm 6.098e-11
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/rnp/Software/MOOSE/projects/moose/petsc/src/snes/tutorials
Possible problem with ex19 running with HYPRE/CUDA, diffs above
=========================================
C/C++ example src/snes/tutorials/ex19 run successfully with CUDA
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Running SLEPc check examples to verify correct installation
Using SLEPC_DIR=/home/rnp/Software/MOOSE/projects/moose/petsc/arch-moose/externalpackages/git.slepc, PETSC_DIR=/home/rnp/Software/MOOSE/projects/moose/scripts/../petsc, and PETSC_ARCH=arch-moose
C/C++ example src/eps/tests/test10 run successfully with 1 MPI process
C/C++ example src/eps/tests/test10 run successfully with 2 MPI processes
C/C++ example src/eps/tests/test10 run successfully with CUDA
Completed SLEPc check examples
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

I also tried proceeding with the install with libmesh and wasp. Everything seems fine with no errors, but the make in moose/test is giving errors. Maybe, this is the cause for it?

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[GiudGiud]

Hello

A diff in make check is suspicious but GPU math isnt always reproducible for performance reasons, and the floating point ieee standards are not always obeyed.
I would say the installation went fine since libmesh installed fine.

What errors did you get from the make command in moose?

[milljm]

@milljm can you post your configure.log of the PETSc build on the PETSc issue thread?

Done!
I hope there isn't a thing as too much information, but I feel the more logs the merrier.

[rnpratoori]

@milljm @lindsayad I tried this again today and I still face make check error with PETSc. But the PETSC ERROR during tests are fewer, but there are other errors. Here is the output log in case someone is able to figure something out of it.
test_out.log

[lindsayad]

Can you remove the color from the log? It's promising to me that you getting "so few" failures. As to the occasional signal 11 crashes, I don't know what those are about. Unfortunately I probably won't be able to looking into this much until after mid-January

[rnpratoori]

Here is the log file without color
test_out_nocolor.log

Please take a look at this whenever you have time

[lindsayad]

I see a lot of the failures that you do. My list below. We will work to get these working with gpu builds over the coming weeks/months

outputs/iterative.exodus ................................................................ FAILED (EXODIFF)
multiapps/linearfv_nonlinearfv.linearfv/linearfv_mainapp ................................ FAILED (EXODIFF)
executioners/nullspace.test_singular .................................................... FAILED (CSVDIFF)
executioners/nl_divergence_tolerance.test ............................................... FAILED (EXODIFF)
outputs/iterative.iterative/exodus_steady ............................................... FAILED (EXODIFF)
executioners/nullspace.test_singular_contaminated ....................................... FAILED (CSVDIFF)
outputs/iterative.iterative/exodus_steady_sequence ...................................... FAILED (EXODIFF)
misc/signal_handler.test_signal_recover_exodiff ......................................... FAILED (EXODIFF)
outputs/iterative.start_stop/exodus_inline .............................................. FAILED (EXODIFF)
outputs/iterative.start_stop/exodus_start_time .......................................... FAILED (EXODIFF)
misc/signal_handler.test_signal_parallel_recover_exodiff ................................ FAILED (EXODIFF)
preconditioners/fsp.group/test ........................................................... FAILED (ERRMSG)
executioners/nl_forced_its.nl_forced_its ................................ FAILED (EXPECTED OUTPUT MISSING)
linearfvkernels/diffusion.mms-diffusion-1d-cd ............................................. FAILED (CRASH)
linearfvkernels/diffusion-reaction-advection.mms-adr-1d-dirichlet ......................... FAILED (CRASH)
preconditioners/vcp.condense_amg_test .................................................... FAILED (ERRMSG)
linearfvkernels/block-restriction.mms-diffusion ........................................... FAILED (CRASH)
linearfvkernels/anisotropic-diffusion.mms-anisotropic-diffusion-2d-orthogonal ............. FAILED (CRASH)
transfers/multiapp_copy_transfer/linear_sys_to_aux.test .................................. FAILED (ERRMSG)
multiapps/petsc_options.test ............................................ FAILED (EXPECTED OUTPUT MISSING)
preconditioners/fsp.group/vector ......................................................... FAILED (ERRMSG)
preconditioners/pbp.pbp/pbp_adapt_test ................................................... FAILED (ERRMSG)
linearfvkernels/diffusion.mms-diffusion-2d-orthogonal ..................................... FAILED (CRASH)
linearfvkernels/diffusion-reaction-advection.mms-adr-1d-outflow ........................... FAILED (CRASH)
linearfvkernels/block-restriction.mms-adr ................................................. FAILED (CRASH)
multiapps/linearfv_nonlinearfv.linearfv/nonlinearfv_mainapp .............................. FAILED (ERRMSG)
variables/linearfv.basic_pps ............................................................. FAILED (ERRMSG)
executioners/nl_divergence_tolerance.test_abs_divtol .................... FAILED (EXPECTED OUTPUT MISSING)
restart/restart_add_variable.add_variable_restart ........................................ FAILED (ERRMSG)
linearfvkernels/anisotropic-diffusion.mms-anisotropic-diffusion-2d-nonorthogonal .......... FAILED (CRASH)
fvkernels/two-var-flux-and-kernel.jac ........................................ FAILED (INCORRECT JACOBIAN)
preconditioners/fsp.group/array .......................................................... FAILED (ERRMSG)
preconditioners/vcp.condense_amg_test_dinv ............................................... FAILED (ERRMSG)
linearfvkernels/diffusion.mms-diffusion-2d-nonorthogonal .................................. FAILED (CRASH)
linearfvkernels/diffusion-reaction-advection.mms-adr-2d-dirichlet ......................... FAILED (CRASH)
linearfvkernels/block-restriction.diff-react-exact ....................................... FAILED (ERRMSG)
variables/linearfv.basic_aux ............................................................. FAILED (ERRMSG)
mortar/periodic_segmental_constraint.periodic_checker2d .................................. FAILED (ERRMSG)
linearfvkernels/diffusion-reaction-advection.mms-adr-2d-outflow ........................... FAILED (CRASH)
linearfvkernels/diffusion.mms-diffusion-2d-rz ............................................. FAILED (CRASH)
postprocessors/side_diffusive_flux_integral.linear_fv_var ................................ FAILED (ERRMSG)
mortar/continuity-2d-non-conforming.vcp .................................................. FAILED (ERRMSG)
mortar/continuity-2d-non-conforming.vcp-2nd-order ........................................ FAILED (ERRMSG)
preconditioners/hmg.hmg_strumpack .......................... [min_cpus=2] FAILED (EXPECTED OUTPUT MISSING)
kernels/array_kernels.test_array_custom_coupling ......................................... FAILED (ERRMSG)