Code crashes in MPI_Win_create during setup

[tranm-ansto]

Hello,
I'm trying to run NekRS on GPU using a single node. The node has 128 CPU cores and 1 GPU.
When I execute the command

 ~/software/cardinal/install/bin/nrspre turbPipe 1

the code runs fine and exits with

Build successful.
finished with exit code 0

But when I run the command

~/software/cardinal/install/bin/nrsmpi turbPipe 1

I got

Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
UTC time: Tue Oct 24 02:48:32 2023

                 __    ____  _____
   ____   ___   / /__ / __ \/ ___/
  / __ \ / _ \ / //_// /_/ /\__ \ 
 / / / //  __// ,<  / _, _/___/ / 
/_/ /_/ \___//_/|_|/_/ |_|/____/  v23.0.7 (907edeac)

COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC

MPI tasks: 1

reading par file ...
Initializing device 
active occa mode: CUDA

using NEKRS_HOME: /data2/tranm/software/cardinal/install
using NEKRS_CACHE_DIR: /data2/tranm/examples/nek_standalone/turbPipe/.cache
using OCCA_CACHE_DIR: /data2/tranm/examples/nek_standalone/turbPipe/.cache/occa/

using existing SIZE file /data2/tranm/examples/nek_standalone/turbPipe/.cache/SIZE
skip building nekInterface (SIZE requires no update)
loading nek ... 
loading /data2/tranm/examples/nek_standalone/turbPipe/.cache/nek5000/libturbPipe.so
Loading uservp
Loading userf
Loading userq
Loading userbc
Loading useric
Loading userqtl
Loading usrsetvert
done
loading /data2/tranm/examples/nek_standalone/turbPipe/.cache/udf/libUDF.so

<<< [OCCA]
<<< backend = CUDA
<<< deviceNumber = LOCAL-RANK
<<< 
<<< [GENERAL]
<<< verbose = true
<<< polynomialOrder = 7
<<< #startFrom = "restart.fld"
<<< stopAt = endTime
<<< endTime = 0.06
<<< 
<<< dt = 6.0e-3
<<< #dt = targetCFL=4 + max=1e-2 + initial=6e-3
<<< 
<<< timeStepper = tombo2
<<< subCyclingSteps = 2
<<< 
<<< writeControl = simulationTime 
<<< writeInterval = 1
<<< 
<<< regularization = hpfrt + nModes=1 + scalingCoeff=10
<<< 
<<< [PRESSURE]
<<< residualTol = 1e-04
<<< #initialGuess = projectionAconj+nVector=10
<<< 
<<< [VELOCITY]
<<< #solver = pfcg+block
<<< boundaryTypeMap = inlet, outlet, wall
<<< density = 1.0
<<< viscosity = 1/19000
<<< residualTol = 1e-06
<<< 
<<< [CASEDATA]
<<< #/* diameters  */
<<< ZLENGTH = 20.0
<<< NSLABS  = 60
<<< NELSLAB = 108
<<< 


<<< //
<<< // nekRS User Defined File
<<< //
<<< #include <math.h>
<<< #include "udf.hpp"
<<< #include "plugins/velRecycling.hpp"
<<< #include "plugins/tavg.hpp"
<<< 
<<< static dfloat ZLENGTH;
<<< static int NSLABS;
<<< static int NELSLAB;
<<< 
<<< /* UDF Functions */                                                      
<<< 
<<< void UDF_LoadKernels(occa::properties& kernelInfo)
<<< {
<<<   velRecycling::buildKernel(kernelInfo);
<<<   tavg::buildKernel(kernelInfo);
<<< }
<<< 
<<< void UDF_Setup(nrs_t *nrs)
<<< {
<<<   // setup scratch space
<<<   platform_t* platform = platform_t::getInstance();
<<<   nrs->o_usrwrk = platform->device.malloc(nrs->NVfields*nrs->fieldOffset,  sizeof(dfloat));
<<< 
<<<   // recycling
<<<   const dfloat wbar  = 1.0;
<<<   const int bID      = 1; 
<<<   dfloat zRecycLayer = 0.25*ZLENGTH;
<<<   const hlong offset = NELSLAB * round(NSLABS * zRecycLayer/ZLENGTH); 
<<<   velRecycling::setup(nrs, nrs->o_usrwrk, offset, bID, wbar);
<<< 
<<<   tavg::setup(nrs);
<<< }
<<< void UDF_Setup0(MPI_Comm comm, setupAide &options)
<<< {
<<<   platform->par->extract("casedata", "zlength", ZLENGTH);
<<<   platform->par->extract("casedata", "nslabs", NSLABS);
<<<   platform->par->extract("casedata", "nelslab", NELSLAB);
<<<   if (platform->options.compareArgs("BUILD ONLY", "FALSE")) {
<<<     double* const nek_cb_scnrs = (double*) nek::ptr("cb_scnrs");
<<<     nek_cb_scnrs[0] = ZLENGTH;
<<<     nek_cb_scnrs[1] = NSLABS;
<<<     nek_cb_scnrs[2] = NELSLAB;
<<<   }
<<< }
<<< 
<<< void UDF_ExecuteStep(nrs_t *nrs, dfloat time, int tstep)
<<< {
<<<   mesh_t *mesh = nrs->meshV;
<<< 
<<<   velRecycling::copy();
<<<   tavg::run(time);
<<< 
<<<   if (nrs->isOutputStep) {
<<<     occa::memory o_UZ = nrs->o_U + 2*nrs->fieldOffset * sizeof(dfloat);
<<<     const dfloat ubar = platform->linAlg->innerProd(mesh->Nlocal, o_UZ, mesh->o_LMM, platform->comm.mpiComm)/mesh->volume;
<<<     if (platform->comm.mpiRank == 0) printf(" uBulk: %g\n", ubar);
<<< 
<<<     tavg::outfld();
<<<   }
<<< }
<<< 

<<< // Boundary conditions
<<< void velocityDirichletConditions(bcData *bc)
<<< {                                                                        
<<<   bc->u = bc->usrwrk[bc->idM + 0*bc->fieldOffset];
<<<   bc->v = bc->usrwrk[bc->idM + 1*bc->fieldOffset];
<<<   bc->w = bc->usrwrk[bc->idM + 2*bc->fieldOffset];
<<< }
<<< 
<<< // Stabilized outflow (Dong et al)
<<< void pressureDirichletConditions(bcData *bc)
<<< {
<<<   const dfloat iU0delta = 20.0;
<<<   const dfloat un = bc->u*bc->nx + bc->v*bc->ny + bc->w*bc->nz;
<<<   const dfloat s0 = 0.5 * (1.0 - tanh(un*iU0delta)); 
<<<   bc->p = -0.5 * (bc->u*bc->u + bc->v*bc->v + bc->w*bc->w) * s0;
<<< }
<<< #ifdef __okl__
<<< void meshVelocityDirichletConditions(bcData *bc){}
<<< void velocityNeumannConditions(bcData *bc){}
<<< void scalarNeumannConditions(bcData *bc){}
<<< void scalarDirichletConditions(bcData *bc){}
<<< #endif
<<< 

loading udf kernels ... 
done (0.00616698s)
benchmarking hot kernels ...
advSub: MPItasks=1 OMPthreads=1 NRepetitions=74 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.00227846 wordSize=64 GDOF/s=2.9265 GB/s=263.244 GFLOPS/s=1627.94 kernelVer=0
advSub: MPItasks=1 OMPthreads=1 NRepetitions=70 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.0028487 wordSize=64 GDOF/s=2.34069 GB/s=210.549 GFLOPS/s=1302.06 kernelVer=6
advSub: MPItasks=1 OMPthreads=1 NRepetitions=122 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.00167763 wordSize=64 GDOF/s=3.97462 GB/s=357.524 GFLOPS/s=2210.98 kernelVer=7
advSub: MPItasks=1 OMPthreads=1 NRepetitions=120 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.0016899 wordSize=64 GDOF/s=3.94575 GB/s=354.927 GFLOPS/s=2194.92 kernelVer=8
advSub: MPItasks=1 OMPthreads=1 NRepetitions=121 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.00168888 wordSize=64 GDOF/s=3.94812 GB/s=355.14 GFLOPS/s=2196.24 kernelVer=9
advSub: MPItasks=1 OMPthreads=1 NRepetitions=180 N=7 cubN=10 nEXT=2 Nfields=3 Nelements=6480 elapsed time=0.00116527 wordSize=64 GDOF/s=5.72223 GB/s=514.724 GFLOPS/s=3183.13 kernelVer=16
Ax: MPItasks=1 OMPthreads=1 NRepetitions=663 N=7 Nelements=6480 elapsed time=0.00029448 wordSize=64 GDOF/s=7.54768 GB/s=721.057 GFLOPS/s=1261.85 constCoeff=1 poisson=1 kernelVer=0
Ax: MPItasks=1 OMPthreads=1 NRepetitions=629 N=7 Nelements=6480 elapsed time=0.000297806 wordSize=64 GDOF/s=7.46338 GB/s=713.003 GFLOPS/s=1247.75 constCoeff=1 poisson=1 kernelVer=1
Ax: MPItasks=1 OMPthreads=1 NRepetitions=638 N=7 Nelements=6480 elapsed time=0.000293954 wordSize=64 GDOF/s=7.56118 GB/s=722.346 GFLOPS/s=1264.11 constCoeff=1 poisson=1 kernelVer=2
Ax: MPItasks=1 OMPthreads=1 NRepetitions=565 N=7 Nelements=6480 elapsed time=0.000328154 wordSize=64 GDOF/s=6.77317 GB/s=647.065 GFLOPS/s=1132.36 constCoeff=1 poisson=1 kernelVer=4
Ax: MPItasks=1 OMPthreads=1 NRepetitions=580 N=7 Nelements=6480 elapsed time=0.000327244 wordSize=64 GDOF/s=6.79201 GB/s=648.864 GFLOPS/s=1135.51 constCoeff=1 poisson=1 kernelVer=5
Ax: MPItasks=1 OMPthreads=1 NRepetitions=644 N=7 Nelements=6480 elapsed time=0.000297982 wordSize=64 GDOF/s=7.45898 GB/s=712.583 GFLOPS/s=1247.02 constCoeff=1 poisson=1 kernelVer=6
Ax: MPItasks=1 OMPthreads=1 NRepetitions=720 N=7 Nelements=6480 elapsed time=0.000271993 wordSize=64 GDOF/s=8.17169 GB/s=780.671 GFLOPS/s=1366.17 constCoeff=1 poisson=1 kernelVer=7
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1234 N=7 Nelements=6480 elapsed time=0.000150417 wordSize=32 GDOF/s=14.7765 GB/s=705.826 GFLOPS/s=2470.39 constCoeff=1 poisson=1 kernelVer=0
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1220 N=7 Nelements=6480 elapsed time=0.000149094 wordSize=32 GDOF/s=14.9077 GB/s=712.091 GFLOPS/s=2492.32 constCoeff=1 poisson=1 kernelVer=1
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1003 N=7 Nelements=6480 elapsed time=0.000179541 wordSize=32 GDOF/s=12.3796 GB/s=591.333 GFLOPS/s=2069.67 constCoeff=1 poisson=1 kernelVer=2
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1286 N=7 Nelements=6480 elapsed time=0.000144425 wordSize=32 GDOF/s=15.3896 GB/s=735.111 GFLOPS/s=2572.89 constCoeff=1 poisson=1 kernelVer=4
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1269 N=7 Nelements=6480 elapsed time=0.000145586 wordSize=32 GDOF/s=15.2669 GB/s=729.249 GFLOPS/s=2552.37 constCoeff=1 poisson=1 kernelVer=5
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1196 N=7 Nelements=6480 elapsed time=0.000149414 wordSize=32 GDOF/s=14.8757 GB/s=710.566 GFLOPS/s=2486.98 constCoeff=1 poisson=1 kernelVer=6
Ax: MPItasks=1 OMPthreads=1 NRepetitions=1221 N=7 Nelements=6480 elapsed time=0.000149891 wordSize=32 GDOF/s=14.8284 GB/s=708.305 GFLOPS/s=2479.07 constCoeff=1 poisson=1 kernelVer=7
fdm:MPItasks=1 OMPthreads=1 NRepetitions=708 N=9 Nelements=6480 elapsed time=0.000264751 wordSize=32 GDOF/s=17.8429 GB/s=323.081 GFLOPS/s=2961.58 kernelVer=0
fdm:MPItasks=1 OMPthreads=1 NRepetitions=935 N=9 Nelements=6480 elapsed time=0.000200184 wordSize=32 GDOF/s=23.5979 GB/s=427.287 GFLOPS/s=3916.8 kernelVer=1
fdm:MPItasks=1 OMPthreads=1 NRepetitions=573 N=9 Nelements=6480 elapsed time=0.000322545 wordSize=32 GDOF/s=14.6458 GB/s=265.191 GFLOPS/s=2430.92 kernelVer=2
fdm:MPItasks=1 OMPthreads=1 NRepetitions=639 N=9 Nelements=6480 elapsed time=0.000290146 wordSize=32 GDOF/s=16.2812 GB/s=294.803 GFLOPS/s=2702.36 kernelVer=3
fdm:MPItasks=1 OMPthreads=1 NRepetitions=945 N=9 Nelements=6480 elapsed time=0.000194058 wordSize=32 GDOF/s=24.3428 GB/s=440.774 GFLOPS/s=4040.43 kernelVer=4
Ax: MPItasks=1 OMPthreads=1 NRepetitions=5304 N=3 Nelements=6480 elapsed time=2.48787e-05 wordSize=32 GDOF/s=7.03252 GB/s=533.43 GFLOPS/s=1066.86 constCoeff=1 poisson=1 kernelVer=0
Ax: MPItasks=1 OMPthreads=1 NRepetitions=6365 N=3 Nelements=6480 elapsed time=2.03444e-05 wordSize=32 GDOF/s=8.59992 GB/s=652.32 GFLOPS/s=1304.64 constCoeff=1 poisson=1 kernelVer=1
Ax: MPItasks=1 OMPthreads=1 NRepetitions=7766 N=3 Nelements=6480 elapsed time=1.38302e-05 wordSize=32 GDOF/s=12.6505 GB/s=959.566 GFLOPS/s=1919.13 constCoeff=1 poisson=1 kernelVer=2
Ax: MPItasks=1 OMPthreads=1 NRepetitions=7112 N=3 Nelements=6480 elapsed time=1.59815e-05 wordSize=32 GDOF/s=10.9476 GB/s=830.398 GFLOPS/s=1660.8 constCoeff=1 poisson=1 kernelVer=4
Ax: MPItasks=1 OMPthreads=1 NRepetitions=7097 N=3 Nelements=6480 elapsed time=1.59844e-05 wordSize=32 GDOF/s=10.9456 GB/s=830.248 GFLOPS/s=1660.5 constCoeff=1 poisson=1 kernelVer=5
Ax: MPItasks=1 OMPthreads=1 NRepetitions=6397 N=3 Nelements=6480 elapsed time=2.03435e-05 wordSize=32 GDOF/s=8.60028 GB/s=652.347 GFLOPS/s=1304.69 constCoeff=1 poisson=1 kernelVer=6
Ax: MPItasks=1 OMPthreads=1 NRepetitions=7501 N=3 Nelements=6480 elapsed time=1.55875e-05 wordSize=32 GDOF/s=11.2244 GB/s=851.392 GFLOPS/s=1702.78 constCoeff=1 poisson=1 kernelVer=7
fdm:MPItasks=1 OMPthreads=1 NRepetitions=2413 N=5 Nelements=6480 elapsed time=7.09208e-05 wordSize=32 GDOF/s=11.4212 GB/s=276.302 GFLOPS/s=1440.72 kernelVer=0
fdm:MPItasks=1 OMPthreads=1 NRepetitions=3720 N=5 Nelements=6480 elapsed time=4.19984e-05 wordSize=32 GDOF/s=19.2864 GB/s=466.578 GFLOPS/s=2432.87 kernelVer=1
fdm:MPItasks=1 OMPthreads=1 NRepetitions=3045 N=5 Nelements=6480 elapsed time=5.38848e-05 wordSize=32 GDOF/s=15.0321 GB/s=363.655 GFLOPS/s=1896.2 kernelVer=2
fdm:MPItasks=1 OMPthreads=1 NRepetitions=3182 N=5 Nelements=6480 elapsed time=5.14823e-05 wordSize=32 GDOF/s=15.7336 GB/s=380.626 GFLOPS/s=1984.69 kernelVer=3
fdm:MPItasks=1 OMPthreads=1 NRepetitions=3379 N=5 Nelements=6480 elapsed time=4.56375e-05 wordSize=32 GDOF/s=17.7486 GB/s=429.374 GFLOPS/s=2238.88 kernelVer=4
Ax: MPItasks=1 OMPthreads=1 NRepetitions=270 Ndim=3 N=7 Nelements=6480 elapsed time=0.000704985 wordSize=64 GDOF/s=9.45824 GB/s=564.737 GFLOPS/s=1651.86 constCoeff=0 poisson=0 kernelVer=0
Ax: MPItasks=1 OMPthreads=1 NRepetitions=342 Ndim=3 N=7 Nelements=6480 elapsed time=0.000553523 wordSize=64 GDOF/s=12.0463 GB/s=719.267 GFLOPS/s=2103.86 constCoeff=0 poisson=0 kernelVer=1
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (11.1713s)

initializing linAlg ...
wdotp: 1.275e-04s  

ENABLE GS COMM OVERLAP disabled

 reading turbPipe.re2                                                                                                                        
 hdr:#v002     6480  3     6480 hdr                                                  
[moose106:63553] *** An error occurred in MPI_Win_create
[moose106:63553] *** reported by process [2961440769,0]
[moose106:63553] *** on communicator MPI COMMUNICATOR 5 DUP FROM 3
[moose106:63553] *** MPI_ERR_WIN: invalid window
[moose106:63553] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[moose106:63553] ***    and potentially your MPI job)

I'm new to NekRS. I'm not sure how to go about solving this issue. Any help would be much appreciated.

Many thanks,
Minh

[stgeke]

Not sure why MPI_Win_create fails. I have not seen this before nor can I reproduce this on my end. What system are you using?

[yslan]

I made this mistake many times and it fails at this same location and same error message.
Most likely, your MPI executable mpiexec (or other wrapper) doesn't match your MPI compiler.

[tranm-ansto]

Thank you for your responses.

@stgeke, I'm using Ubuntu 20.04.3 LTS and mpiexec (OpenRTE) 4.1.2. I don't know if this is just because of system I'm using.

@yslan, could you please elaborate a bit more? How do I go about checking my MPI executable and my MPI compiler version? I'm using standalone NekRS executable from ANL Cardinal.

Thank you for your help.

[yslan]

Try adding this into your mpiexec --mca osc ucx

For example, in nrsbmpi

nohup mpirun --mca osc ucx -np $2 $NEKRS_HOME/bin/nekrs --setup $1 ${@:3} >$1.log.$2 </dev/null 2>&1 &

[tranm-ansto]

@yslan, thank you so very much. This works like a charm. I'm so happy. After days tinkering with this, it finally works.

[yslan]

Forgot to mention the source: open-mpi/ompi#10726

[tranm-ansto]

@yslan,
I have a follow up question, which has been asked in the Cardinal discussion page. I'm not sure if this is the right discussion page for it.

When I tried running coupled NekRS & MOOSE, e.g., sfr_7pin, GPU works on NekRS part, but when Cardinal calls MOOSE, it seems MOOSE is only running on 1 CPU core. If I specify more cores -np 8, then the run crashes. I guess Cardinal is trying to run on 8 GPUs, but I only have 1.

Is there a way to have NekRS run on GPU and MOOSE run on CPU?

Many thanks for your help.

[yslan]

I'm sorry. I have zero experience in MOOSE.

You can also run NekRS on CPU.
I guess the alternative way is to run everything on CPU.