The WRF Physics
Review Process
Background
In discussions at the 2015 WRF Users'
Workshop, the community strongly supported the formation of a physics review
panel that, with community representation and expertise, would screen and make recommendations
on the addition to WRF of proposed new physics packages. Subsequently the WRF
Research Applications Board (RAB) approved the idea and suggested scientists
from the community with extensive experience with WRF physics to initially
oversee the process. Through further discussions, the procedures for the
review and approval of new physics packages for WRF were refined, and the WRF
Physics Review Panel (PRP) was created (wprp@ucar.edu). The WRF physics review
process and the role of the PRP are described below.
For new physics packages submitted for
inclusion in WRF, developers are required to demonstrate that the package
provides an improvement or feature not available in other WRF physics and is of
broad potential interest to users. In addition, sufficient testing of the
package must be conducted and described to demonstrate its beneficial features,
along with documentation that explains the scientific basis for the scheme, its
intended areas of application, and its proper use and suggested option configuration.
The Panel and Review Process
The WRF PRP is a board of three scientists
with a history of involvement with WRF and its physics and who understand the
issues of maintaining and advancing and the system for the community. The
panel oversees the review of new physics packages proposed for WRF by
developers. Similar to a journal editorial committee, the panel communicates
with prospective contributors and enlists relevant experts from the scientific
community to review the submissions anonymously. The panel receives the
reviewers' input and makes a recommendation on whether the package should be
accepted for inclusion in the WRF repository. The PRP then communicates this
decision to the WRF Developers' Committee (DC). If the DC confirms a PRP
recommendation of addition, the submission must then meet the further testing
and other prerequisites for a commit to the repository trunk. Note that the
PRP is only engaged for submissions of new physics packages, not for
updates such as bugfixes, non-fundamental modifications to existing physics
schemes, or developments/modifications of the WRF dynamics, numerics, or
software framework. Neither does the panel review proposed additions to WRFDA
or WRF-Chem.
The review process begins with the
developer submitting to the panel a letter of intent (LOI) to propose a new
physics package for WRF. The LOI can be submitted prior to running any cases
necessary for WRF testing purposes, as the cases appropriate for testing and
evaluation may vary according to the physics being proposed. The LOI is thus a
way for developers to avoid full testing of physics prior to an initial
expression of interest from the panel. If there is interest, the panel will
then request the necessary review materials (described below) and engage
appropriate reviewers, communicating with the developer as needed. Based on
the reviews, the panel then makes a recommendation on whether the code should
be added to the WRF repository.
The panel's recommendations are not
mandates, and the DC has the final say on repository integration of submitted
code. Thus, there may be situations in which additions are denied despite a
recommendation. This is because, ultimately, the NCAR/MMM is responsible for
the maintenance of the WRF repository and has constrained resources for
supporting for the WRF system. For code that is adopted, the developer will
work with a designated point of contact on technical aspects of implementation
of the code into the designated repository branch.
The PRP selects reviewers based on their
recognized scientific expertise in areas relevant to the submitted physics.
Reviewers agree to review the submission materials in a timely manner (e.g.,
within three weeks). The process aims to avoid unduly burdening the reviewers
or hampering the addition of worthwhile physics to WRF.
The reviews of the materials are akin to
reviews of proposals, which result in a recommendation on the merits. They are
not, however, like reviews of journal articles, where iterations on questions
and revisions routinely occur. Any lack of consensus among the reviewers is
resolved by the PRP.
Review Process and Materials Required
The process for new physics review has two
stages: (1) letter of intent and (2) review of materials. After these, the PRP
provides its recommendation to the Developers' Committee on whether the new
package merits inclusion in the WRF repository. The PRP may be contacted at wprp@ucar.edu . The following
are the procedures and requirements for the submission and evaluation of new
physics packages for WRF.
Requirements
Letter of Intent
Developers seeking to submit physics
packages must first submit a letter of intent to the PRP. This allows the
panel, first, to determine whether the submission should proceed and, second,
to tailor for relevance its testing. The letter can be short (e.g., a
half-page), but must provide statements on: what is being proposed and what is
new about it, and the testing performed or to be performed. Merit criteria at
this stage are: (i) scientific soundness, (ii) novelty compared to WRF’s
current capabilities, (iii) potential for interest from the community. Based
on the letter, the panel will direct the developer on whether to proceed with
the submission and suggest testing if so. The developer can then prepare and
submit the required materials for testing and documentation.
Testing
The developer must conduct tests with WRF
simulations to demonstrate clearly the scientific improvement from, or
advantages of, the package, and the developer must provide the results of these
tests to the PRP. At this stage, however, testing is not required to address
other issues that are important (e.g., coding standards, computational issues),
but that will eventually be handled to any commit. The developer must also
address any interactions or dependencies of the proposed package with other WRF
schemes/code.
To ensure the relevance of the results and
to assist in their review and interpretation, the testing must be done with at
least one model configuration using recommended/commonly-used WRF packages. To
this end, there are designated subsets of 2–3 acceptable namelist options for
each of the main WRF physical process areas (e.g., microphysics, PBL, etc.) for
use in the test simulations. In addition, for the submission's targeted
physical process, the testing must include comparisons with one or more (at the
discretion of the panel) of the other recommended WRF options for that
process. Thus, if the developer is proposing the addition of a new PBL scheme,
the testing should compare the proposed PBL scheme with at least one of the
currently-available WRF PBL schemes that are among those recommended.
Prospective contributors should outline their specific testing plans in their
letter of intent to reach agreement with the PRP before testing begins.
The PRP will communicate to the developer
the testing that will be required, which may involve the simulation of cases as
described in the WRF "Information for Code Contributors" document or
other applications that the PRP deems appropriate. For testing analysis,
verification against observations and statistical evaluation of the results are
highly desired. At the discretion of the PRP, published results relevant to
the proposed technique(s) may also be provided and considered in meeting the
testing requirements.
The testing should demonstrate improvement
or benefits from the new package. The standard for assessment by the reviewers
and the PRP is that the proposed physics package or code should be comparable
to a publishable development. Here, this means that it: (a) is sufficiently
novel (not just incremental) and (b) demonstrates potential for advancement
over what exists in WRF.
Documentation
Documentation is essential to the
understanding and evaluation of the proposed code by the reviewers, the panel,
and subsequent users. It should explain what the code is and does, what the
advancement is, and what the intended areas of application are. In addition,
the documentation within the code itself must be sufficient so that future
users and WRF support can understand it.
The documentation provided with
submissions of new physics must be sufficient to explain:
(i) The scheme, its motivation, and the
principles/assumptions of its development;
(ii) Why the approach is an improvement
over existing schemes/code in WRF;
(iii) How the scheme works and whether/how
it is connected to other physics packages; and
(iv) All of its adjustable parameters.
E-mail: All communications for the WRF
Physics Review Panel, including letters of intent and queries, may be directed
to: wprp@ucar.edu .
5/25
