HPC "Can't allocate memory" Error

[chunhuizhao478]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi all, I have started submitting jobs on INL sawtooth cluster using containerization and encounter the following error after the code executes for sometime (the job is not get killed but apparently it fails):

Warning, Exodus files cannot have titles longer than 80 characters.  Your title will be truncated.
Warning, Exodus files cannot have titles longer than 80 characters.  Your title will be truncated.
[r7i2n11:25   :0:25]      rcache.c:421  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
[r7i2n11:00025] *** Process received signal ***
[r7i2n11:00025] Signal: Aborted (6)
[r7i2n11:00025] Signal code:  (-6)
[r7i2n11:00025] [ 0] /lib64/libpthread.so.0(+0x12d20)[0x148708226d20]
[r7i2n11:00025] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x148706efb52f]
[r7i2n11:00025] [ 2] /lib64/libc.so.6(abort+0x127)[0x148706ecee65]
[r7i2n11:00025] [ 3] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucs.so.0(+0x699d7)[0x1487048579d7]
[r7i2n11:00025] [ 4] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucs.so.0(ucs_fatal_error_format+0xd1)[0x148704857ab1]
[r7i2n11:00025] [ 5] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucs.so.0(+0x746e8)[0x1487048626e8]
[r7i2n11:00025] [ 6] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucs.so.0(ucs_rcache_region_put+0x12c)[0x14870486294c]
[r7i2n11:00025] [ 7] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/ucx/libuct_xpmem.so.0(+0x251b)[0x14870071651b]
[r7i2n11:00025] [ 8] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucp.so.0(ucp_rkey_destroy+0x56)[0x148704e4a5a6]
[r7i2n11:00025] [ 9] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucp.so.0(ucp_ep_rkey_unpack_internal+0x4dd)[0x148704e4aecd]
[r7i2n11:00025] [10] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucp.so.0(ucp_proto_rndv_send_reply+0x1c7)[0x148704e88e37]
[r7i2n11:00025] [11] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucp.so.0(ucp_proto_rndv_receive_start+0x169)[0x148704e89459]
[r7i2n11:00025] [12] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libucp.so.0(ucp_tag_recv_nbx+0x15b4)[0x148704ed7b44]
[r7i2n11:00025] [13] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libmpi.so.40(mca_pml_ucx_recv+0xfb)[0x14870994b57b]
[r7i2n11:00025] [14] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libmpi.so.40(PMPI_Recv+0x18f)[0x1487097cc7af]
[r7i2n11:00025] [15] /apps/local/openmpi/5.0.5-gcc12.3.0/lib/libmpi_mpifh.so.40(pmpi_recv_+0x71)[0x148709cfbd41]
[r7i2n11:00025] [16] /opt/petsc/lib/libpetsc.so.3.21(dmumps_redistribution_+0x1355)[0x14870e5d21e2]
[r7i2n11:00025] [17] /opt/petsc/lib/libpetsc.so.3.21(dmumps_fac_driver_+0x4d08)[0x14870e59165e]
[r7i2n11:00025] [18] /opt/petsc/lib/libpetsc.so.3.21(dmumps_+0x2e37)[0x14870e51ea53]
[r7i2n11:00025] [19] /opt/petsc/lib/libpetsc.so.3.21(dmumps_f77_+0x173a)[0x14870e4b2cfc]
[r7i2n11:00025] [20] /opt/petsc/lib/libpetsc.so.3.21(dmumps_c+0x90f)[0x14870e4b0f47]
[r7i2n11:00025] [21] /opt/petsc/lib/libpetsc.so.3.21(+0x4c507c)[0x14870d77c07c]
[r7i2n11:00025] [22] /opt/petsc/lib/libpetsc.so.3.21(MatLUFactorNumeric+0x3ef)[0x14870da5bf65]
[r7i2n11:00025] [23] /opt/petsc/lib/libpetsc.so.3.21(+0xe982f4)[0x14870e14f2f4]
[r7i2n11:00025] [24] /opt/petsc/lib/libpetsc.so.3.21(PCSetUp+0x7f9)[0x14870e26a18d]
[r7i2n11:00025] [25] /opt/petsc/lib/libpetsc.so.3.21(KSPSetUp+0x1106)[0x14870e024fbc]
[r7i2n11:00025] [26] /opt/petsc/lib/libpetsc.so.3.21(+0xd6ff62)[0x14870e026f62]
[r7i2n11:00025] [27] /opt/petsc/lib/libpetsc.so.3.21(KSPSolve+0x13)[0x14870e02a56e]
[r7i2n11:00025] [28] /opt/petsc/lib/libpetsc.so.3.21(+0xfcec7a)[0x14870e285c7a]
[r7i2n11:00025] [29] /opt/petsc/lib/libpetsc.so.3.21(SNESSolve+0x11b9)[0x14870e2dc44d]
[r7i2n11:00025] *** End of error message ***
/hpc-common/moose/containers/moose-dev-openmpi/0ed51b1_0024d394e4653118fe3122a1de651cb59b7b9ae30e30809cf5c687d7b3f83b8b/bin/moose-dev-exec: line 177: 42710 Aborted                 (core dumped) apptainer exec instance://${INSTANCE_NAME} $@

I have started a ticket with HPC team, but they haven't found the possible issue, I do have tried reduce the number of cores per node to provide more memory, however the problem I run is fairly small. I wonder if someone has encountered similar issue before. Thanks!

I use the following command on sawtooth2:

module load use.moose moose-dev-openmpi/0ed51b1
moose-dev-shell

[loganharbour]

@chunhuizhao478 can you please share what you're running?

[chunhuizhao478]

dynamic_solve.txt
static_solve.txt

Hi @loganharbour, I basically run a large deformation dynamic solid mechanics problem, there are some modified objects, I wonder how easy you are able to get access to my directory on sawtooth2, I could direct you to the specific code I used. Thanks!

[chunhuizhao478]

As I tested, I was able to restart the problem from last checkpoints, so there seems to be a growing memory usage during the solve, which I'm not sure what could be the cause.