The WRF Physics Review Process¶
2015 WRF Users’ Workshop discussions confirmed strong community support for a physics review panel, composed of community experts, to evaluate and recommend new WRF physics packages. This was approved by the WRF Research Applications Board (RAB), who nominated community scientists with extensive WRF physics experience to determine review procedures, as well as the WRF Physics Review Panel (PRP; wprp at ucar dot edu). These are described below.
New WRF physics packages must offer improvements or new features, and must have broad user interest. Developers must provide sufficient testing and documentation detailing the scheme’s science, applications, proper use, and suggested configuration to demonstrate its benefits. Details are provided below.
The Panel and Review Process¶
The WRF PRP includes three scientists who are historically involved with WRF physics, and who value advancing WRF for the community. The panel
reviews developer-proposed physics packages;
corresponds with prospective contributors;
enlists relevant scientific community experts to anonymously review proposals;
consideres the reviewer input to determine whether a recommendation for acceptance will be forwarded to the WRF Developers’ Committee (DC).
If the DC approves the addition, the submission must then meet further testing and prerequisites for repository commits. Note that the PRP only considers new physics package submissions. They are not utilized for any other updates, including bug fixes, non-fundamental modifications to existing physics schemes, developments/modifications to any other part of the code.
The panel’s recommendations are not mandates, and the DC has the final say on repository integration of submitted code. Thus, there may be situations in which additions are denied despite a recommendation. This is because, ultimately, the NCAR/MMM is responsible for the maintenance of the WRF repository and has constrained resources for supporting for the WRF system. For code that is adopted, the developer will work with a designated point of contact on technical aspects of implementation of the code into the designated repository branch.
The PRP selects reviewers based on their recognized scientific expertise in areas relevant to the submitted physics. Reviewers agree to review the submission materials in a timely manner (e.g., within three weeks). The process aims to avoid unduly burdening the reviewers or hampering the addition of worthwhile physics to WRF.
The reviews of the materials are akin to reviews of proposals, which result in a recommendation on the merits. They are not, however, like reviews of journal articles, where iterations on questions and revisions routinely occur. Any lack of consensus among the reviewers is resolved by the PRP.
Review Process and Materials Required¶
The review process for new physics packages consists of two stages:
Letter of intent
Review of materials
The PRP then provides its recommendation to the DC on whether the new package merits inclusion in the WRF repository.
Letter of Intent¶
Developers intending to submit physics packages must initiate the process with a letter of intent to the PRP, who will (1) determine whether the submission proceeds and, (2) tailor it for testing. The letter can be short (e.g., a half-page), but must provide statements on:
Proposed changes and innovations
Testing procedures
Merit criteria at this stage are:
Scientific soundness
Novelty compared to WRF’s current capabilities
Potential for interest from the community
After review, the panel informs the developer on whether to proceed with the submission, and if so, suggests testing procedures. The developer can then prepare and submit the required testing and documentation materials.
Testing¶
Developers must conduct tests, and provide results to the PRP, demonstrating the scientific advantages of the package. They must also address any interactions or dependencies of the proposed package with other WRF schemes/code. Developers should outline specific testing plans in their letter of intent and the plan should be agreed upon by all parties before testing begins.
Note
Testing addressing other issues (e.g., coding standards, computational issues) are handled later during the code request and commit process.
Testing must employ at least one model configuration that incorporates recommended or commonly-used WRF packages. For this, developers are provided designated subsets of 2-3 acceptable namelist options for each WRF physical process area (e.g., microphysics, PBL, etc.). Testing must also include comparisons with one or more (at the panel’s discretion) additional recommended WRF options related to the targeted physical process. For e.g., if a new pBL scheme is proposed, testing should compare the proposed PBL scheme with at least one existing/recommended WRF PBL scheme.
The PRP communicates the required testing, which may include simulations described in Information for Code Contributors, or other applications the PRP deems appropriate. For testing analysis, verification against observations and statistical evaluation of the results are highly desired. At the discretion of the PRP, published results relevant to the proposed technique(s) may also be considered in meeting the testing requirements.
The testing should demonstrate benefits from the new package. The PRP standard expects that the proposed physics package or code should be comparable to a publishable development, meaning it:
is sufficiently novel (not just incremental), and
demonstrates potential advancement, compared to existing WRF options
Documentation¶
Reviewers, panelists, and future users rely on clear documentation to understand and evaluate the proposed code. It should detail the code’s function, advancement, and intended application areas. Additionally, documentation within the code must be sufficient so that it is understandable by users and WRF support.
Documentation must sufficiently describe:
The scheme, its motivation, and the principles/assumptions of its development;
Why the approach is an improvement over existing schemes/code in WRF;
How the scheme works and whether/how it is connected to other physics packages; and
All of its adjustable parameters
Email¶
All correspondence for the WRF Physics Review Panel may be directed to: wprp at ucar dot ed