WRF Feature, Capabilities, Infrastructure Testing

For the release, a number of existing capability options in the WRF code need to 
be explicitly tested to insure the continued correct working of the model for users.  
Among these tested options are:
       adaptive time step
       analysis nudging
       observational nudging
       digital filtering initialization
       diffusion options
       program NDOWN
       program OBSGRID
       sst update
       global domain
       vortex following (moving nest)

While the testing of some of the new physics options is done for 28 48-h time periods, 
all of the new physics options are run through a couple of canned cases, and verified 
with subjective analysis.  The level of success is easier to achieve, generally producing 
a complete forecast that gives reasonable results.  The specific new additions to the 
model are scrutinized much more closely.  An important consideration for this level of 
testing is the interworking of schemes, from source-code incompatibility to the 
consequences of unphysical results.  Testing for these new features includes:
       aerosols (Goddard, Thompson, RRTMG)
       CLM
       Grell Freitas cumulus
       Grenier Bretherton PBL
       GRIMS shallow cumulus scheme
       KF-cup cumulus scheme
       lightning options 1, 2, 3, 11
       Morrison Gettleman microphysics
       New Tiedke cumulus scheme
       NoahMP land surface
       Noah subgrid tiling 
       NSSL microphysics options 17, 19, 21
       NWP diagnostics
       ozone options 0, 2
       sea ice
       scale aware cumulus scheme
       SKEBS (backscatter scheme)
       TEMF PBL Scheme 
       topo_wind option 2
       trajectories
       urban physics 1, 2, 3
       P3 microphysics options 50, 51
       
For each of the above tests (types of land data, various physics options, feature 
upgrades, previous capability re-assurances), a fixed set of permutations were conducted:
       single domain case, concurrent nesting, restart, nested restart
The restart runs compared output from an original end-to-end run to the restarted simulation 
to ensure the two data sets were bit-wise identical.

Additionally, several of the above combinations of namelist settings were tested as follows:
       ndown, one-way-nest, nest-starts-later


As part of the assurance that the WRF code is not suffering from accidental side-effects, 
all of the idealized cases have been run, comparing the previous release and the current 
WRF 3.9 release.  The cases all build, run, and produce similar results.  As some of the 
cases use physics schemes that have been upgraded, it is not really possible to get 
identical results in most of these tests. This portion of the testing handles all of the 
lateral boundary condition options in WRF, and exercises many of the dynamics switches. 


The feature testing was conducted on a Linux platform. All tests were run with MPI (with
the exception of the idealized cases, which were run serially) with GNU V4.8.2. 


The following are known to NOT give bit for bit identical answers with a restart:
       Global Run
       adaptive timestep (with a nest)
       ncd_nofill = .true.
       SKEBS
       urban physics 1, 2, 3

NSSL option 17 needs a small time-step (4xDX).