WRF Feature, Capabilities, Infrastructure Testing

For the release, a number of existing capability options in the WRF code need to 
be explicitly tested to ensure the continued correct working of the model for users.  
Among these tested options are:
       adaptive time step
       analysis nudging
       observational nudging
       digital filtering initialization
       diffusion options
       nest starts later than parent
       one-way nest
       program NDOWN
       restart
       sst update
       global domain
       vertical nesting
       vortex following (moving nest)
       All idealized cases (3D cases tested in serial/parallel, 2D/1D cases only tested in serial)

While the testing of some of the new physics options is done for 28 48-h time periods, 
all of the physics options are run through a couple of canned cases, and verified 
with subjective analysis.  The level of success is easier to achieve, generally producing 
a complete forecast that gives reasonable results.  The specific new additions to the 
model are scrutinized much more closely.  An important consideration for this level of 
testing is the interworking of schemes, from source-code incompatibility to the 
consequences of unphysical results.  Testing for these features includes:

Microphysics
Kessler (option 1)
Lin (option 2)
WSM3/5/6/7 (options 3, 4, 6, 24)
Eta/Ferrier (option 5)
Goddard 4-ice (option 7)
Milbrandt (option 9)
Morrison (option 10)
CAM (option 11)
SBU Ylin (option 13)
WDM5/6/7 (options 14, 16, 26)
NSSL (options 17, 18, 19, 21, 22)
Thompson (options 8, 28)
SBM Fast/Full (options 30, 32)
Morrison/CESM aerosol (option 40)
P3 (options 50, 51, 52)
Jensen (option 55)

PBL   
YSU (option 1)
MYJ (option 2)
QNSE (option 4)
MYNN2/3 (options 5, 6)
BouLac (option 8)
UW (option 9)
TEMF (option 10)
Shin Hong (option 11)
Grenier Bretherton (option 12)

Cumulus
Kain-Fritsch/CUP/scale-aware (options 1, 10, 11)
BMJ (option 2)
Grell/Freitas/3/Devenyi (options 3, 5, 93)
Old SAS (option 4)
Tiedtke/new (option 6, 16)
Zhang McFarlane (option 7)
Kiaps SAS (option 14)
Shallow Cumulus Deng/Grims 

Surface Layer
Revised MM5 (option 1)
Eta/MYJ (option 2)
QNSE (option 4)
MYNN (option 5)
Pleim-Xu (option 7)
TEMF (option 10)

LSM
Slab (option 1)
Noah/MP (options 2, 4)
RUC (option 3)
CLM4 (option 5)
SSIB (option 8)

Radiation
RRTM/Dudhia (option 1)
CAM (option 3)
RRTMG/K/Fast (options 4, 14, 24)
Goddard (option 5)
FLG (option 7)
GFDL (option 99)
Goddard Aerosol (aer_opt)

Urban
UCM (option 1)
BEP (option 2)
BEM (option 3)

Other Options
icloud option 3
Lake model
Lightning options 1, 2, 3, 11
NWP diagnostics
Ozone options 0, 2
sea ice
SKEBS (backscatter scheme)
smooth options 1, 2
topo_wind option 2
trajectories
Noah Subgrid Tiling

       
For each of the above tests (types of land data, various physics options, feature 
upgrades, previous capability re-assurances), a fixed set of permutations were conducted:
       single domain case, concurrent nesting, restart, nested restart
The restart runs compared output from an original end-to-end run to the restarted simulation 
to ensure the two data sets were bit-wise identical.

As part of the assurance that the WRF code is not suffering from accidental side-effects, 
all of the idealized cases have been run, comparing the previous release and the current 
WRF 3.9 release.  The cases all build, run, and produce similar results.  As some of the 
cases use physics schemes that have been upgraded, it is not really possible to get 
identical results in most of these tests. This portion of the testing handles all of the 
lateral boundary condition options in WRF, and exercises many of the dynamics switches. 

The feature testing was conducted on a Linux platform. All tests were run with MPI (with
the exception of the 2d idealized cases, which were run serially) with GNU V8.3.0, and some 
tests were run with Intel V18.0.5.

The following are known to NOT give bit for bit identical answers with a restart:
        SKEBS
	adaptive time-stepping
	Eta-Ferrier microphysics
	SBM Full microphysics
	GFDL Radiation
	SSiB LSM
	Lightning Options 1, 2, 11 will only give bit-for-bit results with a restart if cudt=0
	
       
The following are known to be problematic and may not run to completion:
	Lightning Option 3
	 
	
NSSL option 17 needs a small time-step (4xDX).