Periodic Pipe: Error after loading mesh from nek

[tonyzahtila]

Hi,

I am able to run the turbPipePeriodic tutorial. However, I am trying to modify it to run many more grid points.

In order to achieve this, I create a new mesh with gmsh, following the style of the KTH pipe meshes.

I change the usrdat2 routine in turbPipe.usr

      !Set all non-periodic BCs here. This is required due to generating mesh by gmsh and converting it by gmsh2nek
      do iel=1,nelv
         do ifc=1,2*ndim
            id_face = bc(5,ifc,iel,1)
            if (id_face.eq.3) then    ! wall
               cbc(ifc,iel,1) = 'W  '
            endif
         enddo
      enddo

      do iel=1,nelt
      do ifc=1,2*ndim
         if (cbc(ifc,iel,1) .eq. 'W  ') boundaryID(ifc,iel) = 1
      enddo
      enddo

I then run nrspre turbPipe 8. Compilation is successful.

The error I am getting occurs,

 call usrdat3
 done :: usrdat3

gridpoints unique/tot:      15846516     23526912
dofs vel/pr:                15676584     15846516
 nek setup done in    4.8325E+00 s

 set initial conditions
 nekuic (1) for ifld            1
 call nekuic for vel
 xyz min   -0.49998     -0.49998       0.0000
 uvwpt min  0.45434E-10 -0.99997E-01 -0.20998       0.0000       0.0000
 PS min      0.0000       0.0000       0.0000      0.99000E+22
 xyz max    0.49998      0.49998       6.2832
 uvwpt max  0.49999E-01  0.99997E-01   1.2100       0.0000       0.0000
 PS max      0.0000       0.0000       0.0000     -0.99000E+22
 done :: set initial conditions

calling nek_userchk ...
loading mesh from nek ... NboundaryIDs: 2, NboundaryFaces: 5270 done (0.000930932s)
[spartan-gpgpu033:09444] *** An error occurred in MPI_Waitall
[spartan-gpgpu033:09444] *** reported by process [3431923712,2]
[spartan-gpgpu033:09444] *** on communicator MPI_COMM_WORLD
[spartan-gpgpu033:09444] *** MPI_ERR_REQUEST: invalid request
[spartan-gpgpu033:09444] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[spartan-gpgpu033:09444] ***    and potentially your MPI job)
In: PMI_Abort(7, N/A)
slurmstepd: error: *** STEP 24824975.0 ON spartan-gpgpu033 CANCELLED AT 2021-03-30T15:05:52 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: spartan-gpgpu042: tasks 8-11: Killed
srun: error: spartan-gpgpu033: tasks 0-1,3: Killed
srun: error: spartan-gpgpu033: task 2: Exited with exit code 7
srun: error: spartan-gpgpu043: tasks 12-15: Killed
srun: error: spartan-gpgpu041: tasks 4-7: Killed

Note, my 'goal mesh' contains about 250M grid points.

I have previously modified the tutorial to an 8M grid point pipe that I generated in gmsh. It is quite strange behaviour as sometimes it runs smoothly, but most of the time, it shows the error above.

When I run the 26M grid point pipe that I have generated, I always get the above error.

Tony

[stgeke]

Can you reproduce this with a much smaller (<10'000 element) mesh?

Please try:

      do iel=1,nelv
      do ifc=1,2*ndim
        if (boundaryID(ifc,iel) .eq. 3) cbc(ifc,iel,1) = 'W  '
      enddo
      enddo

      do iel=1,nelt
      do ifc=1,2*ndim
         boundaryID(ifc,iel) = 0
         if (cbc(ifc,iel,1) .eq. 'W  ') boundaryID(ifc,iel) = 1
      enddo
      enddo

[tonyzahtila]

If I add in the line 'boundaryID(ifc,iel) = 0', the case runs.

Would you like me to try reproduce the error with <10,000 elements tomorrow, without including the above line? I didn't have one handy, the smallest case I run with my own .re2 file was 16,000 elements.

[stgeke]

The number of BCs (in par) has max(boundaryID). Note, boundaryID is 1-based and needs to be consecutive.