How to copy into scratch

[aprilnovak]

What is the correct syntax to copy data of length N starting from position M in a host-side array, into position M in nrs->o_usrwrk? I tried the following,

nrs->o_usrwrk.copyFrom(nrs->usrwrk, 
  N /* length to copy */,
  M /* where to put it */,
  M /* where to source it */);

but I get a compile error:

/home/anovak/cardinal/src/base/NekInterface.C:651:100: error: call of overloaded ‘copyFrom(dfloat*&, long unsigned int, long unsigned int, long unsigned int)’ is ambiguous
  651 |       first_slot_to_copy * n /* where to put it */, first_slot_to_copy * n /* where to source it */);
      |                                                                                                    ^
In file included from /home/anovak/cardinal/install/include/occa/core/device.hpp:8,
                 from /home/anovak/cardinal/install/include/occa/core/base.hpp:16,
                 from /home/anovak/cardinal/install/include/occa/core.hpp:4,
                 from /home/anovak/cardinal/install/include/occa.hpp:4,
                 from /home/anovak/cardinal/contrib/nekRS/src/core/nrssys.hpp:68,
                 from /home/anovak/cardinal/contrib/nekRS/src/nrs.hpp:13,
                 from /home/anovak/cardinal/contrib/nekRS/src/nekInterface/nekInterfaceAdapter.hpp:10,
                 from /home/anovak/cardinal/contrib/nekRS/src/bdry/bcMap.hpp:6,
                 from /home/anovak/cardinal/include/base/NekInterface.h:26,
                 from /home/anovak/cardinal/src/base/NekInterface.C:21:
/home/anovak/cardinal/install/include/occa/core/memory.hpp:345:10: note: candidate: ‘void occa::memory::copyFrom(const void*, occa::dim_t, occa::dim_t, const occa::json&)’
  345 |     void copyFrom(const void *src,
      |          ^~~~~~~~
/home/anovak/cardinal/install/include/occa/core/memory.hpp:371:10: note: candidate: ‘void occa::memory::copyFrom(occa::memory, occa::dim_t, occa::dim_t, occa::dim_t, const occa::json&)’
  371 |     void copyFrom(const memory src,

[yslan]

This is OCCA function btw.

There are many different function interface for copyFrom.

• The simplest one is

  nrs->o_usrwrk.copyFrom(nrs->usrwrk, bytes)
  

  so you only need to specify the size of the array. For example, bytes = mesh->Nlocal * sizeof(dfloat)

• The one with full capability is

  nrs->o_usrwrk.copyFrom(nrs->usrwrk, bytes, destOffset, srcOffset)
  

  The starting location of source and destination will be "offset (in bytes)" by

  nrs->o_usrwrk + destOffset = nrs->usrwrk + srcOffset
  

Here is another example from (navierStokes/constantFlowRate.cpp) that we copies o_LMM into three components of the o_BF

  o_BF.copyFrom(mesh->o_LMM,
      mesh->Nlocal * sizeof(dfloat),
      0 * nrs->fieldOffset * sizeof(dfloat),
      0);
  o_BF.copyFrom(mesh->o_LMM,
      mesh->Nlocal * sizeof(dfloat),
      1 * nrs->fieldOffset * sizeof(dfloat),
      0);
  o_BF.copyFrom(mesh->o_LMM,
      mesh->Nlocal * sizeof(dfloat),
      2 * nrs->fieldOffset * sizeof(dfloat),
      0);

If we write it into loops, it will be

for (i=0;i<mesh->Nlocal,;i++) {
  o_BF[ 0 * nrs->fieldOffset  + i] = mesh->o_LMM[0+i];
}
for (i=0;i<mesh->Nlocal,;i++) {
  o_BF[ 1 * nrs->fieldOffset  + i] = mesh->o_LMM[0+i];
}
for (i=0;i<mesh->Nlocal,;i++) {
  o_BF[ 2 * nrs->fieldOffset  + i] = mesh->o_LMM[0+i];
}

[aprilnovak]

I did try that, but no success :/

Can you reproduce what I am seeing? Just to be sure I'm not missing something obvious?

[yslan]

Yes, I can reproduce this on my laptop using v22.
I think you are right. The host to device one might not support 4 argument copyFrom

[aprilnovak]

@stgeke do you know how to circumvent this? Unfortunately we cannot use v23 yet, we're still waiting on some MOOSE stuff in order to update fully in Cardinal.

fyi @yukkiezy this is the last thing I need in order to allow us to shift the scratch space in Cardinal for your use case.

[stgeke]

You want to use

nrs->o_usrwrk.copyFrom(nrs->usrwrk + srcOffSet, sizeBytes, destOffsetBytes);

[aprilnovak]

Thanks, that worked!

[yukkiezy]

Thanks April for initiating the discussion here. Is there any smart way we can shift the velocity component location in nrs->o_usrwrk
velRecycling::setup(nrs, nrs->o_usrwrk, offset, bID, wbar);

[aprilnovak]

No worries @yukkiezy, I've got a PR in Cardinal which has the fixes you need. Should be ready for you in about an hour after the tests pass.