Sensei2 ghost libsim #3

BradWhitlock · 2018-03-26T18:24:39Z

Some changes to pjacobi to expose ghost nodes via SENSEI 2 API.

…ome changes so we don't have empty vtkMultiBlockDataSet blocks.

burlen

number of blocks tells the total across all ranks, then non-local blocks are nullptrs. that's the convention in vtk.

 -    this->Mesh->SetNumberOfBlocks(n_ranks);
 -    this->Mesh->SetBlock(rank, block);
 +    this->Mesh->SetNumberOfBlocks(1);
 +    this->Mesh->SetBlock(0, block);

this change is no good, might work with liibsiom but breaks the rules, probably cause paraview to crash or produce incorrect output.

burlen · 2018-03-26T22:38:50Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    y->Delete();
+    z->Delete();
+    this->Mesh = vtkSmartPointer<vtkMultiBlockDataSet>::New();
+    this->Mesh->SetNumberOfBlocks(1);


this should be n_ranks, because the way pjacobi decomposes the domain each rank has 1 block

burlen · 2018-03-26T22:39:46Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    z->Delete();
+    this->Mesh = vtkSmartPointer<vtkMultiBlockDataSet>::New();
+    this->Mesh->SetNumberOfBlocks(1);
+    this->Mesh->SetBlock(0, block);


should be SetBlock(rank, block)

burlen · 2018-03-26T22:41:47Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    y->Delete();
+    z->Delete();
+    this->Mesh = vtkSmartPointer<vtkMultiBlockDataSet>::New();
+    this->Mesh->SetNumberOfBlocks(1);


should be
this->Mesh->SetNumberOfBlocks(n_ranks)
this->Mesh->SetBlock(rank, block)

burlen · 2018-03-26T23:00:46Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+  gn->SetNumberOfTuples(nx*ny);
+  gn->SetName(GHOST_NODE_ARRAY_NAME().c_str());
+  unsigned char *gptr = (unsigned char *)gn->GetVoidPointer(0);
+  memset(gn->GetVoidPointer(0), 0, nx*nx*sizeof(unsigned char));


should be nx*ny

burlen · 2018-03-26T23:08:38Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    for( ; i < nx; ++i)
+        gptr[j*nx+i] = ghost;
+    }
+  block->GetPointData()->SetScalars(gn);


use AddArray here, SetScalars should be used for the array you want to visualize

burlen · 2018-03-26T23:12:54Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+  memset(gn->GetVoidPointer(0), 0, nx*nx*sizeof(unsigned char));
+  unsigned char ghost = 1;
+  // Left column
+  if(this->sim->rankx > 0)


all the patches have full set of ghost nodes, so I think these tests on rank is not needed

burlen · 2018-03-27T17:30:58Z

also want to mention that using rectilnear mesh add substantial performance hit in ParaView for some operations such as stream line tracing. I have profiled this before and image data is much much faster than the same mesh represented in rectilinear form. It has to do with cell locators and having to do explicit searches through coordinate axes in rectilinear meshes. I realize that rectilinear is currently the only option in visit, but we want to hold this up as an example of best practices we might want to leave it as image data? ON the other hand we currently do not have a test case for rectilinear meshes, so changing it would be beneficial in that regard. I don't have strong feeling either way. @jfavre may have some opinion about this as he is using this in tutorials at cscs.

as a related question, isn't about time VisIt supports image data? I have thought "yes" for a long time, and while it would be tedious to make that change, I think it is likely straight forward one?

…es the same mesh as the rectilinear version. Changed vtkMultiBlockDataSet code so we set the rank dataset and have many empties. Fixed nx*nx typo.

BradWhitlock · 2018-03-28T00:39:30Z

I fixed the vtkMultiBlockDataset issues. I was not aware of having to make the dataset mostly empty except for the local data. I guess when Catalyst runs on 1M ranks, it makes 1M-1 empty pointers then. Bleh.

I added vtkImageData code back in and adjusted the extents, etc so it makes the same picture as the vtkRectilinearMesh which was using the sim's actual coordinate arrays. I had changed it to vtkRectilinearGrid because I did not believe the vtkImageData's coordinates when I looked at things in VisIt.

I made the recommended changes to the ghost nodes array. You're right. I did a temporary test where the temperature array returns the ghost nodes and they now appear around the edges of all domains.

burlen

only thing I see now is extents/coordinates not putting origin of the valid mesh at 0,0

burlen · 2018-03-28T17:08:02Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+#ifdef USE_IMAGE_DATA
+    int Extent[6];
+    double Origin[3], Spacing[3];
+    Extent[0] = this->sim->rankx*this->sim->bx;


I'd like to keep the origin of the valid cells in the mesh at (0,0), which with grid spacing of 1 puts the origin of the mesh with ghost zones at (-1,-1). I think the rectilinear mesh should do the same.

burlen · 2018-03-28T17:22:12Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    Spacing[1] = this->sim->cy[1] - this->sim->cy[0];
+    Spacing[2] = 0.;
+
+    // Origin on rank 0 using cx[0], cy[0] is zero.


the origin of valid part of the mesh should be located at 0,0, with ghost zones origin will be negative (-dx*ng, -dy*ng)

I was just exposing the coordinates as computed in the simulation's AllocateGridMemory routine. The minimum values are zero, not negative. If that's not what it ought to be then the simulation should be different - not the adaptor.

Rank 0: cx={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195}
Rank 0: cy={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195}
Rank 1: cx={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195}
Rank 1: cy={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000}
Rank 2: cx={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000}
Rank 2: cy={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195}
Rank 3: cx={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000}
Rank 3: cy={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000}

@BradWhitlock see comment below

burlen · 2018-03-28T17:27:16Z

Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx

+    Spacing[1] = this->sim->cy[1] - this->sim->cy[0];
+    Spacing[2] = 0.;
+
+    // Origin on rank 0 using cx[0], cy[0] is zero.


the origin of valid part of the mesh should be located at 0,0, with ghost zones origin will be negative (-dx*ng, -dy*ng)

@burlen @BradWhitlock

Sorry guys, I have been in backlog mode ever since I finished the tutorial with Burlen, teaching a 3rd day event, then attending a class myself. I cannot find the time to properly interact with you until after April 9 (will be on vacation for the Easter week). Cheers. Jean

burlen · 2018-03-30T18:20:12Z

Hi Brad, I don't disagree with your assessment of where the bug is. My issue with your change is that you introduced a new bug into the uniform mesh case. Take a look at how the extents were set before your change. It took me some effort to get the extents correct, and hence I do not want to see that effort undone. Could you fix the simulation so that the coordinate axes are generated to reflect the ghost nodes correctly? Burlen

…

On 03/30/2018 10:50 AM, Brad Whitlock wrote: ***@***.**** commented on this pull request. ------------------------------------------------------------------------ In Jacobi/Sensei/C/solution/JacobiDataAdaptor.cxx <#3 (comment)>: > - block->SetSpacing(this->Spacing); +#ifdef USE_IMAGE_DATA + int Extent[6]; + double Origin[3], Spacing[3]; + Extent[0] = this->sim->rankx*this->sim->bx; + Extent[1] = Extent[0] + this->sim->bx + 2 - 1; + Extent[2] = this->sim->ranky*this->sim->by; + Extent[3] = Extent[2] + this->sim->by + 2 - 1; + Extent[4] = 0; + Extent[5] = 0; + + Spacing[0] = this->sim->cx[1] - this->sim->cx[0]; + Spacing[1] = this->sim->cy[1] - this->sim->cy[0]; + Spacing[2] = 0.; + + // Origin on rank 0 using cx[0], cy[0] is zero. I was just exposing the coordinates as computed in the simulation's AllocateGridMemory routine. The minimum values are zero, not negative. If that's not what it ought to be then the simulation should be different - not the adaptor. Rank 0: cx={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195} Rank 0: cy={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195} Rank 1: cx={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195} Rank 1: cy={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000} Rank 2: cx={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000} Rank 2: cy={, 0.000000, 0.024390, 0.048780, 0.073171, 0.097561, 0.121951, 0.146341, 0.170732, 0.195122, 0.219512, 0.243902, 0.268293, 0.292683, 0.317073, 0.341463, 0.365854, 0.390244, 0.414634, 0.439024, 0.463415, 0.487805, 0.512195} Rank 3: cx={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000} Rank 3: cy={, 0.487805, 0.512195, 0.536585, 0.560976, 0.585366, 0.609756, 0.634146, 0.658537, 0.682927, 0.707317, 0.731707, 0.756098, 0.780488, 0.804878, 0.829268, 0.853658, 0.878049, 0.902439, 0.926829, 0.951219, 0.975610, 1.000000} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHBbSDe6scHudZfI3mtJuCzBxTuyY29ks5tjnBNgaJpZM4S7rX2>.

Brad Whitlock added 4 commits March 21, 2018 10:03

Use rectilinear grid that uses simulation's coordinate arrays. Make s…

68ae584

…ome changes so we don't have empty vtkMultiBlockDataSet blocks.

Pass simulation_data structure to the data adaptor.

a133e65

Use simulation_data in the adaptor. Ghost node support.

d26e55f

Ghost node support.

69dc6fb

burlen requested changes Mar 26, 2018

View reviewed changes

burlen reviewed Mar 26, 2018

View reviewed changes

Changed back to vtkImageData by default and changed extents so it mak…

ae5e05f

…es the same mesh as the rectilinear version. Changed vtkMultiBlockDataSet code so we set the rank dataset and have many empties. Fixed nx*nx typo.

burlen reviewed Mar 28, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sensei2 ghost libsim #3

Sensei2 ghost libsim #3

BradWhitlock commented Mar 26, 2018

burlen left a comment

burlen Mar 26, 2018

burlen Mar 26, 2018

burlen Mar 26, 2018

burlen Mar 26, 2018

burlen Mar 26, 2018

burlen Mar 26, 2018

burlen commented Mar 27, 2018

BradWhitlock commented Mar 28, 2018

burlen left a comment

burlen Mar 28, 2018

burlen Mar 28, 2018

BradWhitlock Mar 30, 2018

burlen Mar 30, 2018

burlen Mar 28, 2018

jfavre Mar 30, 2018

burlen commented Mar 30, 2018 via email

Sensei2 ghost libsim #3

Are you sure you want to change the base?

Sensei2 ghost libsim #3

Conversation

BradWhitlock commented Mar 26, 2018

burlen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burlen commented Mar 27, 2018

BradWhitlock commented Mar 28, 2018

burlen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burlen commented Mar 30, 2018 via email