Optimizing input file for faster 3d D phase-field simulation

[PengWei97]

Dear Moose Community,

I am currently trying to reproduce the simulation results from the paper "Verma, M., & Mukherjee, R. (2021). Grain growth stagnation in solid state thin films: A phase-field study. Journal of Applied Physics, 130(2)." In this study, I am modeling a 3D bicrystal simulation (
surf_circular_matrix_gg_3d_split.txt), as shown below.

However, I am experiencing significantly slower computation times for the 3D simulation compared to the 2D case. Specifically, when running the simulation for a time step t=1, it takes around 4021.53s on my server for the 3D case. On the other hand, when I reduce the problem to 2D, the same simulation step only takes 27.78s.

Given this, I would like to know what aspects of my input file I should focus on optimizing to improve the computational efficiency of the 3D simulation. Any suggestions on specific parameters, solvers, or techniques that could help speed up the calculation would be greatly appreciated.

Thank you in advance for your insights!

Best regards,
Wei

[GiudGiud]

Hello

Mesh and time discretizations are big factors on the computing time. Making sure they are optimal helps.

A concern often when using adaptivity is that the initial partitioning is not great once the mesh is refined. Re partitioning is expensive so it can be a good idea to use a prior simulation mesh (using the checkpoint system, not using exodus) as the starting point.

Here you only have two crystals right? So not very many order parameters. That's usually impactful but not here.

Solver wise PJFNK with block jacobi is probably a weak option. Try to look into Newton if you have enough memory and a perfect Jacobian, and try to use a different preconditioner

Why are you using vinewton for the snes type? Do you have constraints?

[PengWei97]

Hi @GiudGiud, based on your suggestions, I have made the necessary modifications and attached the results. I look forward to your next advice.

A concern often when using adaptivity is that the initial partitioning is not great once the mesh is refined.

[Mesh]
  type = GeneratedMesh
  dim = 3
  nx = 50
  ny = 50
  nz = 20
  xmin = 0
  xmax = 100
  ymin = 0
  ymax = 100
  zmin = 0
  zmax = 40
  elem_type = HEX8

  parallel_type = distributed
[]

Not using mesh adaptivity indeed accelerated the computation time, but it is still slow. (Even when calculating up to t=0.01, it takes 115.82s, and the output to the terminal is： 02_3d_split_noAdaptivity.txt
. Moreover, using mesh adaptivity, the time consumption is 426.74s.

Re partitioning is expensive so it can be a good idea to use a prior simulation mesh (using the checkpoint system, not using exodus) as the starting point.

Sorry, I don’t quite understand using the checkpoint system, not using exodus. I didn’t output the checkpoint file for recovery in [Outputs], but only the Nemesis-type file.

Here you only have two crystals right? So not very many order parameters. That's usually impactful but not here.

Yes, there are only two order parameters here, as well as two conservative field variables cv (to distinguish vapor and bulk) and w. This case is a very simple bicrystal model, used to verify the reliability and efficiency of the model.

Solver wise PJFNK with block jacobi is probably a weak option.

Why is using the PJFNK solver with block Jacobi considered a weak option? Apologies for my lack of experience and theoretical knowledge regarding the [Executioner] module settings. If there are good practical summaries or learning materials on this, I would appreciate it if you could share them with me.

Try to look into Newton if you have enough memory and a perfect Jacobian, and try to use a different preconditioner.

Got it. Based on your suggestion, I set the solve type to Newton, and indeed, this method is faster, taking only 86.6s. Additionally, I noticed that there are several options in the Preconditioning System such as Split, FDP, PSP, PBP, SMP, and VCP. I will try these further tomorrow. Thanks for your advice on this.

Why are you using vinewton for the snes type? Do you have constraints?

Actually, I just copied an example from Moose, although I don’t fully understand what it means. Maybe I should add some knowledge about PETCs. Do you have any other advice on this?

Finally, I greatly appreciate your valuable suggestions. I will further understand and try the adjustments based on your responses, and I hope you can provide further guidance.

Wei

Let me know if you'd like to adjust anything!

[GiudGiud]

Nemesis and exodus files don't keep track of the levels of refinement for each element, which means we cannot use them for restarting simulations after APR has been used.

On the other hand, checkpoints and their associated mesh files keep the refinement information.

[GiudGiud]

The petsc manual will describe this preconditioning better than I can, but block Jacobi, from what I recall, is essentially just scaling rows of the system matrix (or its action in PJFNK) with a processor-dependent factor (from the diagonal? I would need to check the manual)

As such it's not effective for all problems. As the calculation gets larger you'll want to look for better scaling approaches such as algebraic multi-grid.

[GiudGiud]

this is the petsc manual page for block jacobi
https://petsc.org/release/manualpages/PC/PCBJACOBI/