GPU test fails restart comparison

[nusbaume]

What happened?

It was found that the changes brought in by PR #1175 caused the aux_cam GPU test:

ERS_Ln9.ne30pg3_ne30pg3_mg17.F2000dev.derecho_nvhpc.cam-outfrq9s_gpu_default

To fail the restart comparison (i.e. restarting the model changes the answers versus a no-restart run). All restart tests on CPUs pass as expected.

What are the steps to reproduce the bug?

Run the CAM regression tests with the nvhpc compiler option on Derecho with CAM tag cam6_4_052 or later.

What CAM tag were you using?

cam6_4_052

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

NVHPC

Path to a case directory, if applicable

No response

Will you be addressing this bug yourself?

Yes, but I will need some help

Extra info

@huebleruwm @sjsprecious I will likely need your help on this, especially given that it only occurs for the GPU test.

[huebleruwm]

I took a look at the code and found these variables "pdf_zm_w_1, pdf_zm_w_2, pdf_zm_varnce_w_1, pdf_zm_varnce_w_2, pdf_zm_mixt_frac" that are allocated on the GPU with a create statement, but they should be copied in and out using a copy statement.

I thought moving line 2896, in clubb_intr.F90, to the copy section in the same clause around line 2870 would fix the issue, but I tried this and the test still failed. I don't think there's any other code specific to restarts in clubb_intr, so I'm not immediately sure what else to try, I'll have more time to look at this after next week.