Skip to content

MPI Wrappers

MeisterEule edited this page Jul 29, 2021 · 6 revisions

Functionality

Vftrace has wrappers to MPI function calls. These wrappers are instrumented and call a vftr_MPI version of the function. The wrappers for C and Fortran both call the same vftr_MPI routine to reduce code duplication and enable easier maintenance.

int MPI_Send(const void *buf, int count, MPI_Datatype datatype,
             int dest, int tag, MPI_Comm comm) {
   return vftr_MPI_Send(buf, count, datatype, dest, tag, comm);
}

In these vftr_MPI-functions the communication is executed by calling the corresponding PMPI function:

int vftr_MPI_Send(const void *buf, int count, MPI_Datatype datatype,
                  int dest, int tag, MPI_Comm comm) {

   // disable profiling based on the Pcontrol level
   if (vftrace_Pcontrol_level == 0) {
      return PMPI_Send(buf, count, datatype, dest, tag, comm);
   } else {
      long long tstart = vftr_get_runtime_usec();
      int retVal = PMPI_Send(buf, count, datatype, dest, tag, comm);
      long long tend = vftr_get_runtime_usec();

      vftr_store_sync_message_info(send, count, datatype, dest, tag, comm, tstart, tend);

      return retVal;
   }
}

The PMPI_ symbols do the same as their MPI_ counterpart. This way, the MPI functions as used by the application are instrumented. The functionality inside the wrapper enables in-depth MPI sampling. The vftrace MPI-wrapper record which ranks are communicating (who sends, who receives), the message size, message type, and communication time. Non blocking communication is sampled by registering the non blocking call's request and checking for completion from time to time in the background. Persistent requests are registered and handled like normal non blocking calls upon calling MPI_Start or MPI_Startall. For collective communications inter- and intra-communicators are distinguished and handled to reflect their unique communication patterns. Special buffers like MPI_IN_PLACE are taken care of and handled accordingly.

Supported Functions

Function Type Blocking? vftr-handling C-wrapper C-Regtest vftr-c2F F-wrapper F-Regtest F08 wrapper F08-Regtest
94,94% 94,94% 94,94% 94,94% 94,94% 94,94% 35,44% 30,38%
MPI_Send P2P Blocking Yes Yes Yes Yes Yes Yes Yes No
MPI_Bsend P2P Blocking Yes Yes Yes Yes Yes Yes Yes No
MPI_Ssend P2P Blocking Yes Yes Yes Yes Yes Yes Yes No
MPI_Rsend P2P Blocking Yes Yes Yes Yes Yes Yes Yes No
MPI_Recv P2P Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Mrecv P2P Blocking No No No No No No No No
MPI_Isend P2P Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Ibsend P2P Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Issend P2P Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Irsend P2P Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Irecv P2P Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Imrecv P2P Non-Blocking No No No No No No No No
MPI_Sendrecv P2P Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Sendrecv_replace P2P Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Probe Probing Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Iprobe Probing Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Mprobe Probing Blocking No No No No No No No No
MPI_Improbe Probing Non-Blocking No No No No No No No No
MPI_Send_init Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Bsend_init Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Ssend_init Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Rsend_init Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Recv_init Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Start Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Startall Persistent Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Test Test/Wait Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Testany Test/Wait Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Testsome Test/Wait Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Testall Test/Wait Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Wait Test/Wait Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Waitany Test/Wait Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Waitsome Test/Wait Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Waitall Test/Wait Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Request_free Test/Wait Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Barrier Collective Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Bcast Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ibcast Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Gather Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Igather Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Gatherv Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Igatherv Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Scatter Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iscatter Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Scatterv Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iscatterv Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Allgather Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iallgather Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Allgatherv Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iallgatherv Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Alltoall Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ialltoall Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Alltoallv Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ialltoallv Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Alltoallw Collective Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ialltoallw Collective Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Allreduce GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iallreduce GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Reduce GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ireduce GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Reduce_scatter_block GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ireduce_scatter_block GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Reduce_scatter GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Ireduce_scatter GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Scan GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iscan GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Exscan GlobalReduction Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Iexscan GlobalReduction Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Put Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Rput Onesided Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Get Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Rget Onesided Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Accumulate Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Raccumulate Onesided Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Get_accumulate Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Rget_accumulate Onesided Non-Blocking Yes Yes Yes Yes Yes Yes No No
MPI_Fetch_and_op Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Compare_and_swap Onesided Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Init Environment Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Finalize Environment Blocking Yes Yes Yes Yes Yes Yes Yes Yes
MPI_Pcontrol Environment Blocking Yes Yes Yes Yes Yes Yes Yes No
Clone this wiki locally