Realm: barrier assertion

[syamajala]

I'm seeing the following assertion when running S3D on Frontier:

s3d.x: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_tdb/legion/runtime/realm/barrier_impl.cc:1259: static void Realm:
:BarrierTriggerMessage::handle_message(Realm::NodeID, const Realm::BarrierTriggerMessage&, const void*, size_t, Realm::
TimeLimit): Assertion `datalen == (impl->redop->sizeof_lhs * (trigger_args.internal.trigger_gen - trigger_args.internal
.previous_gen))' failed.

It seems to happen when I turn on profiling as my run without profiling succeeded.

[syamajala]

Also I am running on the onepool branch.

[lightsighter]

The critical path profiling requires the use of Realm barrier reductions which are not used when you are not profiling. What are the values of datalen, impl->redop->sizeof_lhs, trigger_args.internal.trigger_gen, trigger_args.internal.previous_gen, and trigger_args.internal.first_gen?

[lightsighter]

If you want to avoid this error for now and you don't mind not having critical path profiling then you can run with -lg:prof_no_critical_paths.

I'll also note that this doesn't have anything to do with onepool and will happen in the master branch as well.

[apryakhin]

Yep that is certainly a bug due to my recent merge of the notification broadcast. That will be fixed at the earliest convenience.

[apryakhin]

That will be the patch:

• https://gitlab.com/StanfordLegion/legion/-/merge_requests/1613

That needs a test to be added which I will do shortly and then it's ready

[apryakhin]

@syamajala The assert you are hitting is only on the onepool branch? I wonder if we can actually test the path with S3D

[lightsighter]

The onepool branch is merged into the master branch now, so you can upstream those changes to whatever feature branch you are testing.

[apryakhin]

PR1613 should already include changes from one pool branch then. @syamajala just to follow up again on testing the fix

[syamajala]

Will test the pull request.

[syamajala]

Frontier is down today.

[syamajala]

Job is in the queue, but it looks like the compute nodes are still down on Frontier.

[syamajala]

I tested on sapling and it seems to be working.

[syamajala]

oops. meant to click comment not close. Please close after you merge to master.