Gsl ufs dev #1
Closed
Gsl ufs dev #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In PR NOAA-EMC#1601, the removal of the old `applications.py` was inadvertently excluded.
Update logic for interpolated wave gridded post Fixes NOAA-EMC#1658
Switches the runtime pyenv from ufswm to gfs_workflow on Hera and Orion. gfs_workflow is the new one created just for workflow and is the target going forward. CI modules have already been using it.
…sistent with updated GSI-fix (NOAA-EMC#1650) (NOAA-EMC#1666)
Add JEDI-based LETKFOI update for snow depth analysis in the global-workflow. These jobs are run in a single landanl task. Additional filter to_f90bool is added to jinja template processing.
The global-workflow runs offline UPP to generate master (and optionally flux) grib2 files for: - analysis - GOES simulated satellite imagery - WAFS products WAFS has been removed from the global-workflow in a previous PR. The offline UPP and product generation in the current workflow is done in a single job. This PR aims to partition the offline UPP from the product generation (for atmos grib2 files). The changes in this PR are not yet activated in the workflow. The offline post is now completely yaml driven for all the variants of the upp run conditions. In a future PR, product generation jobs will be added and subsequently current post-processing jobs will be retired.
This feature changes the CI system to use the test cases from the PR in question. This allows the PR to add or modify tests and have them applied to itself to ensure they work. Fixes NOAA-EMC#1662
Updates the state of GitHub badges based on the results of CI testing of the last PR to easily see the state and stability of the develop branch. GitHub's Workflow Actions are used to generate the badges in the README.md file for the development branch. The GitHub Actions will update the badges only when a PR is merged to the **develop** branch and only update them according to the state of the Labels in the PR itself. This Action must be able to use the final state of the CI labels of the PR. The labels must therefore be dynamically "pulled" from the PR just before it is merged. Here is the map between the Label states and the Badges (for Orion and Hera respectively): ``` Pass -> Pass Fail -> Fail None -> Pending ``` For example, if the Badges are currently _Pass_ and a PR is merged without CI test Pass Label the corresponding Badge will be updated to _Pending_. Fixes NOAA-EMC#1643
Updates the UFS hash to bf60924. This required a change in the compile signature due to changes in UFS to how the compiler is defined.
A recent fix of another issue meant that wave config settings were set incorrectly for glo_025 resolution. This removes pass through options which caused the issue since a few variables were intentionally set to nothing but then were set incorrectly to another default.
When setup_expt.py or setup_xml.py fails in CI, the error logs are now reported back to the user via the message stream in the PR. Fixes NOAA-EMC#1672
…A-EMC#1667)" (NOAA-EMC#1707) This reverts commit 38cd0bc.
This PR provides support for running UFS weather model forecasts, using the global-workflow, on the AWS Parallel Works (PW) platform. This PR accomplishes the following: - Building the UFS weather model within the global-workflow build system; - Running a UFS weather model atmosphere-only deterministic forecast using the global-workflow infrastructure. Follow-up PRs will include the following: - Adding documentation for building and using the AWS PW UFSWM spack-stack environment; this is heavily borrowed from Sadegh Sadeghi Tabas's presentation 03 May 2023; - Addressing remaining `TODO` issues within this PR; several notes are made throughout as to how to better streamline the connections between the UFSWM and the global-workflow for the respective cloud platforms; - Adding data-transfer capabilities within the global-workflow for cloud deployments; - Testing additional UFSWM capabilities (e.g., coupling, etc.,) and resolutions (this PR has been tested at C48 and has not been configured for other cubed-sphere resolutions). No dependencies other than the loading of the appropriate AWS PW modules are required. These are included in this PR.
By default, the early-cycle EnKF is designed to write analysis files. The switch controlling this was using `$CDUMP` which was changed by PR NOAA-EMC#1421. This current PR updates this switch to use `$RUN` in accordance with issue NOAA-EMC#1299, and restores the default analysis file write capability.
Fix Ocean post treating zero-padded forecast hours as octals. Fixes NOAA-EMC#1710 Refs NOAA-EMC#1195
…AA-EMC#1700) user will provide a single integer restart_interval (hours) via config.base and config.fcst, config.efcs. scripts will convert this as necessary for model_configure and nems.configure. Logic for determining RERUN revisited based on files available rather than restart list. Fixed a bunch of shell norms --------- Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
* Create gw_setup script and extend to other machines Add a new script to handle setting up the lmod environment needed to run all the setup for checkout and experiment generation. C `checkout.sh` now uses this script as more recent versions of `git` are needed on some machines. Documentation has been updated to recommend this method on all machines rather than the previous instructions. Updates documentation to add the checkout with UFSDA option. Closes NOAA-EMC#1677
`BASE_CPLIC` depends on the machine. It is now defined in `hosts/machine.yaml` together with other machine specific paths.
During the COM refactor, a location for the ocn/ice daily files was defined within the products directory. However, the daily files are output directly by the model and land in the history directory. The archive job was looking in the otherwise unused daily product directory for the files and not finding them. The 2D and 3D ocean files had the opposite problem: archive was looking in the history directory, but these files have their own directories in products. This update now: - Removes the COM variable for the daily files - Updates the archive job to: - pull the daily ocean files from history - pull 2D ocean files from the 2D product directory - pull 3D ocean files from the 3D product directory Fixes NOAA-EMC#1709
* Add S2SA nems.configure template Most of the configuration was setup to allow S2SA, but there was no appropriate nems.configure template. A new one was created by starting with the S2SWA template and removing the wave portions. Also added two UFSDA scripts that were missing from the git ignore list. Fixes NOAA-EMC#1592 * Update documentation for cycled mode setup The experiment setup documentation for cycled mode had not been updated as coupled capability was added. Also added notes to both the cycled and forecast-only section that ATMW is currently not working. * Update setup_expt to recognize more S2S cycled exps setup_expt would only detect two of the possible S2S options when staging cycled ICs. Refs: NOAA-EMC#1592 * Change path to volcanic aerosol data The volcanic aerosol data was moved to a new location within the GOCART emissions directory. Refs: NOAA-EMC#1592 * Disable enkf versions of coupled settings The enkf versions of coupled settings (DO_OCN, etc.) do not work properly because some settings have already been determined by that point based on the non-enkf versions of the settings. The enkf versions are now disabled until there is a solution (see NOAA-EMC#1692). Refs: NOAA-EMC#1593 Refs: NOAA-EMC#1692 * Fix ocnpost detection for gfs free-forecast The task list for the free-forecast was checking against a list of apps that did not include S2SA to determine whether ocnpost is included. That check is now changed to check on do_ocean rather than looking at the app directly. Refs: NOAA-EMC#1593
…OAA-EMC#1718) This PR: - allows self cycling of the ocean and ice components in a cycled atmosphere with 3DEnVar (without IAU). - allows staging of initial conditions for coupled ensemble when setting up cycled ensemble. This PR also includes: - numerous improvements involving appropriate locations for defining relevant variables for components. This PR does not: - address archiving of ensemble data when cycling - probably many other issues that are necessary e.g. IAU Partially addresses NOAA-EMC#613
Using numpy to cast type has been deprecated in favor of using the native python functions (i.e. `int()`, `float()`, etc.). Resolves NOAA-EMC#1725
Partition name and node size for AWS were missing from `config.resources`. This information has now been added. Resolves NOAA-EMC#1727
`pygw` from the global-workflow has been moved to its own repository at [wxflow](https://github.com/NOAA-EMC/wxflow). This PR: - removes `pygw` and replaces it with a checkout of `wxflow` in `checkout.sh` and `Externals.cfg` Going forward, until `wxflow` becomes a fixture as a module, `global-workflow` will be required to be checked out and is done as part of `checkout.sh` and `Externals.cfg`
* Add GEFS fcst job support - Makes the minor changes necessary to run GEFS fcst job only from pre-staged ICs. Also many unnecessary DA settings are removed from the GEFS version of `config.base`. - Stop using DA diag table from GEFS, remove unneeded settings Also removed more unneeded DA settings from the GEFS side. - Use RUN instead of CDUMP to determine output frequencies in forecast Refs: NOAA-EMC#826
- this is a placeholder with changes from last year's HFIP experiment; will likely need to be updated in the future
Update GSL template for hera Create EMC and GSL versions of files under workflow/rocoto - tasks.py, workflow_tasks.py
Add new GSL archive script for archiving grib2 and ICs, but no netcdf files
Add GSL archive script and correct typos/fixed paths
Squashed commit of the following: commit c590476 Author: Rahul Mahajan <[email protected]> Date: Fri Jan 12 19:21:02 2024 -0500 Update hash to ufs-weather-model. The noahmptable.tbl was reorganized, so update link_workflow.sh to use the same one used in UFSWM RT (NOAA-EMC#2219) commit c041968 Author: Kate Friedman <[email protected]> Date: Fri Jan 12 12:09:35 2024 -0500 Add ocean resolution to setup_expt invocation and retire/reduce COMROT/ROTDIR usage (NOAA-EMC#2214) Two series of updates: 1) Update setup scripts to now allow users to provide ocean resolution 2) Housekeeping to retire the `COMROT` variable, replacing it with other variables as needed, and reduce the `ROTDIR` variable usage. Both updates change options for the workflow setup API. Refs NOAA-EMC#2061 commit 997f978 Author: AndrewEichmann-NOAA <[email protected]> Date: Fri Jan 12 10:46:36 2024 -0500 Allow use of ocean obs prep in WCDA cycling and remove R2D2 (NOAA-EMC#2215) Enables use of ocean obs prep task in WCDA cycling and removes R2D2 from same. Runs task gdasprepoceanobs before gdasocnanalprep -- obtains ocean data nc4 files from DMPDIR, processes them into IODA format and copies them to COM_OBS. Replaces the current R2D2 processing. commit 12a5bb1 Merge: 4cb5802 6492c2d Author: David Huber <[email protected]> Date: Fri Jan 12 09:26:27 2024 -0500 Merge pull request NOAA-EMC#2217 from DavidHuber-NOAA/update/versions Update and clean up version and module files commit 6492c2d Author: David Huber <[email protected]> Date: Thu Jan 11 11:49:08 2024 -0600 Update orion module/version files for met/metplus NOAA-EMC#2123 commit 94c9937 Author: DavidHuber <[email protected]> Date: Thu Jan 11 17:42:53 2024 +0000 Comment met/metplus out from Hera modulefile. NOAA-EMC#2123 commit 8c32f8b Merge: a65e4c6 4cb5802 Author: David Huber <[email protected]> Date: Thu Jan 11 11:34:44 2024 -0600 Merge branch 'develop' of github.com:noaa-emc/global-workflow into develop commit a65e4c6 Author: David Huber <[email protected]> Date: Thu Jan 11 11:34:19 2024 -0600 Initial update of version/module files NOAA-EMC#2123 commit 4cb5802 Author: souopgui <[email protected]> Date: Wed Jan 10 08:30:22 2024 -0600 Fix OpenMP over-allocation of resources in exglobal_atmos_products.sh when running MPMD tasks (NOAA-EMC#2212) Fix OpenMP over-allocation of resources running MPMD tasks Co-authored-by: Innocent Souopgui <[email protected]> commit b056b53 Author: TerrenceMcGuinness-NOAA <[email protected]> Date: Mon Jan 8 17:28:05 2024 -0500 Add Hercules as valid machine in CI scripts (NOAA-EMC#2207) Few updates to CI scripts to include names for hercules that where missed the first time. commit 6574d29 Author: Rahul Mahajan <[email protected]> Date: Mon Jan 8 17:25:47 2024 -0500 Fix invalid GH action and restart file name (NOAA-EMC#2210) Resolves a typo that leads to an invalid workflow yaml and fixes the restart filename in restart detection. Resolves NOAA-EMC#2205 commit 69605ea Author: Rahul Mahajan <[email protected]> Date: Mon Jan 8 17:00:28 2024 -0500 Stop attempting to comment link to RTD for non-PRs (NOAA-EMC#2209) Adds a check so comments with a link to documentation are only generated for PRs. commit 4e160a8 Author: Rahul Mahajan <[email protected]> Date: Mon Jan 8 13:10:15 2024 -0500 Enable UPP for GOES processing (NOAA-EMC#2203) Wnables the creation of special master grib2 files from UPP for GOES processing commit c15875b Author: David Huber <[email protected]> Date: Mon Jan 8 09:56:06 2024 -0500 Port cycling to Hercules (NOAA-EMC#2196) Adds cycled support for Hercules (excluding gsi-monitor). Partially resolves NOAA-EMC#1588 GSI monitoring is disabled on Hercules due to missing Perl modules. That will be enabled in a later PR. commit ef6827d Author: Walter Kolczynski - NOAA <[email protected]> Date: Mon Jan 8 09:43:12 2024 -0500 Refactor rocoto task XML creation (NOAA-EMC#2189) Refactors the rocoto task generation to be recursive. This will allow nested metatasks to loop over multiple variables, which is needed for GEFS product generation. As part of this refactor, there is no longer separate arguments to designate metatasks. Instead, task dicts can include a nested 'task_dict' as well as a 'var_dict' containing the variables to loop over. The nested task dict can then either have another layer, or be the innermost task. To accommodate the new recursive nature, some defaults that were previously defined in create_wf_task() had to be pushed down into the function that creates the innermost task. Also, former keywords have been absorbed by the task dict. Refs NOAA-EMC#823 Refs NOAA-EMC#827 commit 2b81cfa Author: Walter Kolczynski - NOAA <[email protected]> Date: Mon Jan 8 09:41:03 2024 -0500 Update fix versions (NOAA-EMC#2198) Updates fix versions for a few components: - Update cice and mom6 versions to support C96/1p00 marine - Update wave to change betamax setting for glo_025 waves Resolves NOAA-EMC#2004 Resolves NOAA-EMC#2107
Remove 'rrtmg_lw_pre' from C3 and MYNN SDFs.
Add c3 SDF in the build.
Add c3 and mynn template
Merge to 2024 Jan 12 NOAA-EMC/develop and update to 2024 Jan 29 UFS
|
Link to ReadTheDocs sample build for this PR can be found at: |
|
Link to ReadTheDocs sample build for this PR can be found at: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Type of change
Change characteristics
How has this been tested?
Checklist