WRF Feature, Capabilities, Infrastructure Testing

For the release, a number of existing capability options in the WRF code need to 
be explicitly tested to ensure the continued correct working of the model for users.  
Among these tested options are:
       adaptive time step
       analysis nudging
       observational nudging
       digital filtering initialization
       diffusion options
	   nest starts later than parent
	   one-way nest
       program NDOWN
	   restart
       sst update
       global domain
	   vertical nesting
       vortex following (moving nest)
	   All idealized cases (3D cases tested in serial/parallel, 2D/1D cases only tested in serial)

While the testing of some of the new physics options is done for 28 48-h time periods, 
all of the new physics options are run through a couple of canned cases, and verified 
with subjective analysis.  The level of success is easier to achieve, generally producing 
a complete forecast that gives reasonable results.  The specific new additions to the 
model are scrutinized much more closely.  An important consideration for this level of 
testing is the interworking of schemes, from source-code incompatibility to the 
consequences of unphysical results.  Testing for these new features includes:
       aerosols (Goddard, Thompson, RRTMG)
	   CAM microphysics
       CLM
       Grell Freitas cumulus
       Grenier Bretherton PBL
       GRIMS shallow cumulus scheme
	   HUJI SBM, regular and fast options
	   icloud option 3
       KF-cup cumulus scheme
	   Lake model
       lightning options 1, 2, 3, 11
	   Milbrandt microphysics
	   Modified YSU PBL
       Morrison Gettleman microphysics
       New Tiedke cumulus scheme
       NoahMP land surface
       Noah subgrid tiling 
       NSSL microphysics options 17, 19, 21
       NWP diagnostics
       ozone options 0, 2
       sea ice
       scale aware cumulus scheme
       SKEBS (backscatter scheme)
	   smooth options 1, 2
       TEMF PBL Scheme 
       topo_wind option 2
       trajectories
       urban physics 1, 2, 3
       P3 microphysics options 50, 51
       
For each of the above tests (types of land data, various physics options, feature 
upgrades, previous capability re-assurances), a fixed set of permutations were conducted:
       single domain case, concurrent nesting, restart, nested restart
The restart runs compared output from an original end-to-end run to the restarted simulation 
to ensure the two data sets were bit-wise identical.

As part of the assurance that the WRF code is not suffering from accidental side-effects, 
all of the idealized cases have been run, comparing the previous release and the current 
WRF 3.9 release.  The cases all build, run, and produce similar results.  As some of the 
cases use physics schemes that have been upgraded, it is not really possible to get 
identical results in most of these tests. This portion of the testing handles all of the 
lateral boundary condition options in WRF, and exercises many of the dynamics switches. 


The feature testing was conducted on a Linux platform. All tests were run with MPI (with
the exception of the idealized cases, which were run serially) with Intel V17.0.1.132, and a 
few tests were run with GNU V6.3.0 and PGI V17.9-0. 


The following are known to NOT give bit for bit identical answers with a restart:
       Global Run
       SKEBS
       

NSSL option 17 needs a small time-step (4xDX).