Running NekRS with 2 or more GPUs on cluster

[kit607]

Hi all,

I am trying to run a case on university supercomputer with 2 GPUs but I got the error Signal Aborted (6). The case run fine when I use 1 GPU. The details of the error is below.

[spartan-gpgpu120:76940:0:76940]      cma_ep.c:81   process_vm_readv(pid=76941 {0x148309938800,78194}-->{0x153ebb8c5a00,78194}) returned -1: Bad address
[spartan-gpgpu120:76941:0:76941]      cma_ep.c:81   process_vm_readv(pid=76940 {0x153ebb938800,78194}-->{0x1483098c5a00,78194}) returned -1: Bad address
==== backtrace (tid:  76941) ====
 0 0x000000000000370b uct_cma_ep_tx_error()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/cma/cma_ep.c:81
 1 0x0000000000003a36 uct_cma_ep_tx()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/cma/cma_ep.c:114
 2 0x000000000001c2f5 uct_scopy_ep_progress_tx()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/base/scopy_ep.c:151
 3 0x000000000001fcb2 ucs_arbiter_dispatch_nonempty()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/arbiter.c:321
 4 0x000000000001be39 ucs_arbiter_dispatch()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/arbiter.h:386
 5 0x0000000000020473 ucs_callbackq_slow_proxy()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/callbackq.c:404
 6 0x000000000003f11a ucs_callbackq_dispatch()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/callbackq.h:211
 7 0x000000000003f11a uct_worker_progress()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/api/uct.h:2638
 8 0x000000000003f11a ucp_worker_progress()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucp/core/ucp_worker.c:2782
 9 0x000000000002e20b opal_progress()  ???:0
10 0x0000000000052745 ompi_request_default_wait_all()  ???:0
11 0x000000000008f06f MPI_Waitall()  ???:0
12 0x000000000032cc57 pairwiseExchange()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:193
13 0x00000000003303ee oogs::finish()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:804
14 0x0000000000332093 oogs::setup()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:495
15 0x00000000002a9219 createMesh()  /home/lee16/nekRS/src/mesh/meshSetup.cpp:245
16 0x00000000000dda30 nrsSetup()  /home/lee16/nekRS/src/setup/setup.cpp:236
17 0x00000000000ad838 nekrs::setup()  /home/lee16/nekRS/src/lib/nekrs.cpp:193
18 0x000000000040e196 main()  /home/lee16/nekRS/src/main.cpp:469
19 0x000000000003feb0 __libc_start_call_main()  ???:0
20 0x000000000003ff60 __libc_start_main_alias_2()  :0
21 0x000000000040f955 _start()  ???:0
=================================
[spartan-gpgpu120:76941] *** Process received signal ***
[spartan-gpgpu120:76941] Signal: Aborted (6)
[spartan-gpgpu120:76941] Signal code:  (-6)
[spartan-gpgpu120:76941] [ 0] /lib64/libc.so.6(+0x54df0)[0x154438254df0]
[spartan-gpgpu120:76941] [ 1] /lib64/libc.so.6(+0xa154c)[0x1544382a154c]
[spartan-gpgpu120:76941] [ 2] /lib64/libc.so.6(raise+0x16)[0x154438254d46]
[spartan-gpgpu120:76941] [ 3] /lib64/libc.so.6(abort+0xd3)[0x1544382287f3]
[spartan-gpgpu120:76941] [ 4] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(+0x2662b)[0x15442ca2162b]
[spartan-gpgpu120:76941] [ 5] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_log_default_handler+0x753)[0x15442ca25513]
[spartan-gpgpu120:76941] [ 6] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_log_dispatch+0xd3)[0x15442ca25803]
[spartan-gpgpu120:76941] [ 7] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/ucx/libuct_cma.so.0(+0x370b)[0x15443400870b]
[spartan-gpgpu120:76941] [ 8] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/ucx/libuct_cma.so.0(uct_cma_ep_tx+0x186)[0x154434008a36]
[spartan-gpgpu120:76941] [ 9] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libuct.so.0(uct_scopy_ep_progress_tx+0x55)[0x1544340512f5]
[spartan-gpgpu120:76941] [10] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_arbiter_dispatch_nonempty+0xa2)[0x15442ca1acb2]
[spartan-gpgpu120:76941] [11] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libuct.so.0(uct_scopy_iface_progress+0x59)[0x154434050e39]
[spartan-gpgpu120:76941] [12] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(+0x20473)[0x15442ca1b473]
[spartan-gpgpu120:76941] [13] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucp.so.0(ucp_worker_progress+0x5a)[0x15442ca9311a]
[spartan-gpgpu120:76941] [14] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libopen-pal.so.40(opal_progress+0x2b)[0x154435f7e20b]
[spartan-gpgpu120:76941] [15] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libmpi.so.40(ompi_request_default_wait_all+0xf5)[0x15443881c745]
[spartan-gpgpu120:76941] [16] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libmpi.so.40(PMPI_Waitall+0x8f)[0x15443885906f]
[spartan-gpgpu120:76941] [17] /home/lee16/nekRS_mybuild/lib/libnekrs.so(+0x32cc57)[0x154438fe8c57]
[spartan-gpgpu120:76941] [18] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN4oogs6finishERN4occa6memoryEiiPKcS4_P6oogs_t+0xfce)[0x154438fec3ee]
[spartan-gpgpu120:76941] [19] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN4oogs5setupEP5ogs_tiiPKcSt8functionIFvvEE9oogs_mode+0x1583)[0x154438fee093]
[spartan-gpgpu120:76941] [20] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_Z10createMeshP19ompi_communicator_tiibRN4occa4jsonE+0xe99)[0x154438f65219]
[spartan-gpgpu120:76941] [21] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tR9setupAideP5nrs_t+0x1170)[0x154438d99a30]
[spartan-gpgpu120:76941] [22] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tS1_iiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_iii+0x1528)[0x154438d69838]
[spartan-gpgpu120:76941] [23] nekrs(main+0xbf6)[0x40e196]
[spartan-gpgpu120:76941] [24] /lib64/libc.so.6(+0x3feb0)[0x15443823feb0]
[spartan-gpgpu120:76941] [25] /lib64/libc.so.6(__libc_start_main+0x80)[0x15443823ff60]
[spartan-gpgpu120:76941] [26] nekrs(_start+0x25)[0x40f955]
[spartan-gpgpu120:76941] *** End of error message ***
==== backtrace (tid:  76940) ====
 0 0x000000000000370b uct_cma_ep_tx_error()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/cma/cma_ep.c:81
 1 0x0000000000003a36 uct_cma_ep_tx()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/cma/cma_ep.c:114
 2 0x000000000001c2f5 uct_scopy_ep_progress_tx()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/sm/scopy/base/scopy_ep.c:151
 3 0x000000000001fcb2 ucs_arbiter_dispatch_nonempty()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/arbiter.c:321
 4 0x000000000001be39 ucs_arbiter_dispatch()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/arbiter.h:386
 5 0x0000000000020473 ucs_callbackq_slow_proxy()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/callbackq.c:404
 6 0x000000000003f11a ucs_callbackq_dispatch()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucs/datastruct/callbackq.h:211
 7 0x000000000003f11a uct_worker_progress()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/uct/api/uct.h:2638
 8 0x000000000003f11a ucp_worker_progress()  /local_build/easybuild/build/UCX/1.13.1/GCCcore-11.3.0/ucx-1.13.1/src/ucp/core/ucp_worker.c:2782
 9 0x000000000002e20b opal_progress()  ???:0
10 0x0000000000052745 ompi_request_default_wait_all()  ???:0
11 0x000000000008f06f MPI_Waitall()  ???:0
12 0x000000000032cc57 pairwiseExchange()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:193
13 0x00000000003303ee oogs::finish()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:804
14 0x0000000000332093 oogs::setup()  /home/lee16/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:495
15 0x00000000002a9219 createMesh()  /home/lee16/nekRS/src/mesh/meshSetup.cpp:245
16 0x00000000000dda30 nrsSetup()  /home/lee16/nekRS/src/setup/setup.cpp:236
17 0x00000000000ad838 nekrs::setup()  /home/lee16/nekRS/src/lib/nekrs.cpp:193
18 0x000000000040e196 main()  /home/lee16/nekRS/src/main.cpp:469
19 0x000000000003feb0 __libc_start_call_main()  ???:0
20 0x000000000003ff60 __libc_start_main_alias_2()  :0
21 0x000000000040f955 _start()  ???:0
=================================
[spartan-gpgpu120:76940] *** Process received signal ***
[spartan-gpgpu120:76940] Signal: Aborted (6)
[spartan-gpgpu120:76940] Signal code:  (-6)
[spartan-gpgpu120:76940] [ 0] /lib64/libc.so.6(+0x54df0)[0x148892454df0]
[spartan-gpgpu120:76940] [ 1] /lib64/libc.so.6(+0xa154c)[0x1488924a154c]
[spartan-gpgpu120:76940] [ 2] /lib64/libc.so.6(raise+0x16)[0x148892454d46]
[spartan-gpgpu120:76940] [ 3] /lib64/libc.so.6(abort+0xd3)[0x1488924287f3]
[spartan-gpgpu120:76940] [ 4] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(+0x2662b)[0x14888c03d62b]
[spartan-gpgpu120:76940] [ 5] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_log_default_handler+0x753)[0x14888c041513]
[spartan-gpgpu120:76940] [ 6] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_log_dispatch+0xd3)[0x14888c041803]
[spartan-gpgpu120:76940] [ 7] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/ucx/libuct_cma.so.0(+0x370b)[0x14888c00570b]
[spartan-gpgpu120:76940] [ 8] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/ucx/libuct_cma.so.0(uct_cma_ep_tx+0x186)[0x14888c005a36]
[spartan-gpgpu120:76940] [ 9] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libuct.so.0(uct_scopy_ep_progress_tx+0x55)[0x14888c08c2f5]
[spartan-gpgpu120:76940] [10] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(ucs_arbiter_dispatch_nonempty+0xa2)[0x14888c036cb2]
[spartan-gpgpu120:76940] [11] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libuct.so.0(uct_scopy_iface_progress+0x59)[0x14888c08be39]
[spartan-gpgpu120:76940] [12] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucs.so.0(+0x20473)[0x14888c037473]
[spartan-gpgpu120:76940] [13] /apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1/lib/libucp.so.0(ucp_worker_progress+0x5a)[0x148886d6611a]
[spartan-gpgpu120:76940] [14] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libopen-pal.so.40(opal_progress+0x2b)[0x1488902c620b]
[spartan-gpgpu120:76940] [15] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libmpi.so.40(ompi_request_default_wait_all+0xf5)[0x148892b2b745]
[spartan-gpgpu120:76940] [16] /apps/easybuild-2022/easybuild/software/Compiler/GCC/11.3.0/OpenMPI/4.1.4/lib/libmpi.so.40(PMPI_Waitall+0x8f)[0x148892b6806f]
[spartan-gpgpu120:76940] [17] /home/lee16/nekRS_mybuild/lib/libnekrs.so(+0x32cc57)[0x1488932f7c57]
[spartan-gpgpu120:76940] [18] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN4oogs6finishERN4occa6memoryEiiPKcS4_P6oogs_t+0xfce)[0x1488932fb3ee]
[spartan-gpgpu120:76940] [19] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN4oogs5setupEP5ogs_tiiPKcSt8functionIFvvEE9oogs_mode+0x1583)[0x1488932fd093]
[spartan-gpgpu120:76940] [20] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_Z10createMeshP19ompi_communicator_tiibRN4occa4jsonE+0xe99)[0x148893274219]
[spartan-gpgpu120:76940] [21] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tR9setupAideP5nrs_t+0x1170)[0x1488930a8a30]
[spartan-gpgpu120:76940] [22] /home/lee16/nekRS_mybuild/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tS1_iiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_iii+0x1528)[0x148893078838]
[spartan-gpgpu120:76940] [23] nekrs(main+0xbf6)[0x40e196]
[spartan-gpgpu120:76940] [24] /lib64/libc.so.6(+0x3feb0)[0x14889243feb0]
[spartan-gpgpu120:76940] [25] /lib64/libc.so.6(__libc_start_main+0x80)[0x14889243ff60]
[spartan-gpgpu120:76940] [26] nekrs(_start+0x25)[0x40f955]
[spartan-gpgpu120:76940] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 76941 on node spartan-gpgpu120 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Appreciate for your help and time,
K3iTH

[dlwldud99]

Hi,

The issue was with UCX and we were able to solve this using the following modules:

foss/2022a
CMake/3.23.1
CUDA/12.2.0
UCX-CUDA/1.13.1-CUDA-12.2.0

Regards,
Joy