WRF Software Testing

The testing conducted on the WRF code to insure bit-for-bit behavior on differing 
processor counts runs through hundreds of short forecasts in about 30 minutes. These 
simulations are very short (about 10 time steps).  The purpose is to activate as many 
possible physics options.  If single processor vs multiple processor results differ, 
then there is a strong likelihood that improper initialization of variables or 
missing communications or race conditions exist.  While tracking down the root cause 
of the problem is extremely time consuming, physics options that exhibit clean 
bit-wise reproducible results are more likely to be robust.  This testing is handled 
entirely with a newly developed mechanism designed to run on small desktops, or with
additional scripting on distributed cluster systems.

See https://github.com/davegill/wrf-coop/blob/master/README_user.md


 
===========================
       WRF Test Framework
===========================


1.   Overview 

The WRF Testing Framework is designed to build, test, and analyze test results for 
one or more versions of the WRF model. This is a docker containerized solution that
is utilized for interactive testing and is also part of the WRF automated jenkins
testing.

     
   COMPILERS:    
             GNU fortran versions 8 and 9.

    PARALLEL BUILD CONFIGURATIONS:
             Serial      (single-processor) build
             OpenMP      (multithreaded, shared memory) build
             MPI         (multiprocessor, distributed memory) build

     WRF COMPILE-TIME VARIATIONS
	     ARW 
	     ARW real*8
	     ARW Nested
	     ARW Moving Nest

             NMM HWRF
             CHEM

             Idealized Fire
             Idealized Super Cell
             Idealized Super Cell real*8
             Idealized Baroclinc Wave
             Idealized 2D Hill
             Idealized Single Column Model

             

      ARW em_real RUN-TIME VARIATIONS
             Adaptive Time Stepping
             Digital Filtering
             FDDA: Spectral nudging, obs nudging, surface and upper air nudging
             Stochastic Forcing

             Nesting
             Vertical Nesting
             Hybrid and Terrain Following Vertical Coordinates
             Dry and Moist Potential Temperature


2.   Physics Options Applied in WRF Tests


The following Table and associated table Key summarizes the combinations of physics 
options that are tested for WRF.   It is important to note that while the choice of 
one physics option should not influence the choice of another physics option (i.e., 
the choice of a microphysics scheme should be independent of the cumulus scheme choice), 
in practice certain options are developed and tested for a small subset of other physics 
option combinations.   Therefore, the following table is useful as a guide for 
combinations of WRF physics options that are known to provide bit-for-bit results 
between serial and MPI versions of WRF.   Each row in the table represents a specific 
test, and each column a specific physics option.   Each of the physics combinations 
listed in the tables can be considered a "safe" combination that will provide successful 
short-term forecasts with bit-for-bit results when comparing single-processor output 
against multi-processor output.  



            				KEY 1:   Column Labels      (Tables 1-7)
            				----------------------------------------
            				NL   => Test Namelist Identifier
            				PBL  => Planetary Boundary Layer Scheme
            				CU   => Cumulus Scheme
            				MP   => Microphysics Scheme
            				LW   => Longwave Radiation Scheme
            				SW   => Shortwave Radiation Scheme
            				SFC  => Surface Physics Scheme
            				LAND => Land Surface Scheme
            				URB  => Urban Physics Scheme
            				SHCU => Shallow Cumulus Scheme
            				TOPO => Topography-Following Wind Scheme
            				----------------------------------------



                 			KEY 2:  Test Namelist Codes (Tables 1-7)
                 			----------------------------------------
                 			AD => Adaptive Time Stepping

                 			DF => Digital Filtering
                 			FD => FDDA
                 			NE => Basic Nesting
                 			VN => Vertical Nesting
                 			----------------------------------------








                                 TABLE 1: 
          WRF ARW Tests, Providing Successful 10 Time Step Forecasts, 
                       and Bit-for-Bit Results for
              Serial vs. MPI Runs and Serial vs. OpenMP Runs


NL		MP	CU	LW	SW	PBL	SFC	LSM	URB	
3dtke		D	D	D	D	D	1	D	0	
conus		D	D	D	D	D	D	D	0	
rap		28	3	4	4	5	5	3	0	
tropical	D	D	D	D	D	D	D	0	
03		3	3	24	24	4	4	1	0	
03DF		3	3	4	4	4	4	1	0	
03FD		3	3	4	4	4	4	1	0	
06		6	6	24	24	8	2	1	0	
07NE		8	14	5	5	8	1	2	2	
10		10	2	1	2	4	4	7	0	
11		10	2	1	2	4	4	7	0	
14		3	6	3	3	4	4	3	0	
16		8	14	5	5	9	2	7	0	
16DF		8	14	5	5	9	2	7	0	
17		4	2	3	3	2	2	2	0	
17AD		4	2	3	3	2	2	2	0	
18		8	6	5	5	10	10	7	0	
20		4	1	1	2	12	1	2	0	
20NE		4	1	1	2	12	1	2	0	
38		2	14	4	4	2	2	7	0	
48		3	3	24	24	4	4	1	0	
49		3	1	24	24	1	91	2	0	
50		3	1	24	24	1	91	4	0	
51		3	1	24	24	1	91	4	0	
52		17	3	24	24	4	4	1	0	
52DF		17	3	4	4	4	4	1	0	
52FD		17	3	4	4	4	4	1	0	
60		6	11	24	24	1	1	4	0	
60NE		6	11	4	4	1	1	4	0	
65DF		28	7	4	4	9	2	3	0	
66FD		3	1	4	4	4	4	1	0	
71		8	1	4	4	1	1	2	0	
78		52	1	4	4	1	1	2	0	
79		2	14	4	4	5	2	7	0	
cmt		6	11	4	4	1	1	2	0	
kiaps1NE	16	14	14	14	11	1	4	0	
kiaps2		16	14	14	14	1	91	4	1	
solaraNE	8	1	4	4	5	5	2	3	
NL		MP	CU	LW	SW	PBL	SFC	LSM	URB	





                                 TABLE 2: 
          WRF ARW Tests, Providing Successful 10 Time Step Forecasts, 
                       and Bit-for-Bit Results for
               Serial vs. MPI Runs and Serial vs. OpenMP Runs
                             REAL*8

NL	PBL	CU	MP	LW	SW	SFC	LAND	URB	SHCU	TOPO
14	 4	 6	 3	 3	 3	 4	 3	 0	 0	 0
17AD	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
NL	PBL	CU	MP	LW	SW	SFC	LAND	URB	SHCU	TOPO






                                 TABLE 3: 
       WRF Idealized Supercell Tests, Providing Successful 10 Time Step Forecasts, 
                       and Bit-for-Bit Results for
               Serial vs. MPI Runs and Serial vs. OpenMP Runs

NL	PBL	CU	MP	LW	SW	SFC
02NE	 0	 0	 1	 0	 0	 1
03	 0	 0	 1	 0	 0	 1
03NE	 0	 0	 1	 0	 0	 1
04	 0	 0	 2	 0	 0	 1
NL	PBL	CU	MP	LW	SW	SFC






                                 TABLE 4: 
        WRF Idealized B-Wave Tests, Providing Successful 10 Time Step Forecasts, 
                       and Bit-for-Bit Results for
               Serial vs. MPI Runs and Serial vs. OpenMP Runs

NL	PBL	CU	MP	LW	SW	SFC
1NE	 0	 0	 1	 0	 0	 0
2	 0	 0	 1	 0	 0	 0
2NE	 0	 0	 1	 0	 0	 0
3	 0	 0	 2	 0	 0	 0
NL	PBL	CU	MP	LW	SW	SFC