WRF Software Testing

The testing conducted on the WRF code to insure bit-for-bit behavior on differing 
processor counts runs through hundreds of short forecasts in a couple hours.  These 
simulatins are very short (about 10 time steps).  The purpose is to activate as many 
possible physics options.  If single processor vs multiple processor results differ, 
then there is a strong likelihood that improper initialization of variables or 
missing communications or race conditions exist.  While tracking down the root cause 
of the problem is extremely time consuming, physics options that exhibit clean 
bit-wise reproducible results are more likely to be robust.  This testing is handled 
entirely with a newly developed mechanism designed to run on small desktops and on 
batch systems.
 
===========================
       WRF Test Framework
===========================


1.   Overview 

The WRF Testing Framework is designed to build, test, and analyze test results for 
one or more versions of the WRF model.    With the advent of NCAR's flagship mainframe, 
Yellowstone, in early 2013,  the testing framework for WRF was rewritten to accommodate 
Yellowstone's additional flexibility.    For example, a large number of Fortran compilers 
are available now, instead of just a single Fortran compiler that was available on NCAR's 
"bluefire" machine.   Yellowstone's greater speed and greater job throughput has also 
allowed the number of tests performed for WRF to expand.   Despite all of the additional 
variations available on Yellowstone, a complete WRF test run (compiling WRF, running 
tests, and analyzing results) takes about two hours on Yellowstone.   WRF is now tested 
regularly for the following compilers, parallel processing configurations, compile-time 
variations, and run-time variations of the WRF software:

     
   COMPILERS:    
             GNU fortran version 4.9.2
             PGI  fortran version 15.1
             Intel fortran version 15.0.1

    PARALLEL BUILD CONFIGURATIONS:
             Serial      (single-processor) build
             OpenMP (multithreaded, shared memory) build
             MPI         (multiprocessor, distributed memory) build

     WRF COMPILE-TIME VARIATIONS
	     ARW 
             NMM Nested
             NMM HWRF
             CHEM
             CHEM with KPP
             Idealized Super Cell
             Idealized Baroclinc Wave
             Idealized 2D Hill
             3D Var
             4D Var
             WRFPlus 
             

      ARW em_real RUN-TIME VARIATIONS
             Adaptive Time Stepping
             Digital Filtering
             FDDA
             Grib 1 WRF Output
             Binary WRF Output
             Nesting
             Quilting
             Global Domain
             Vertical Nesting


2.   Physics Options Applied in WRF Tests


The following Table and associated table Key summarizes the combinations of physics 
options that are tested for WRF.   It is important to note that while the choice of 
one physics option should not influence the choice of another physics option (i.e., 
the choice of a microphysics scheme should be independent of the cumulus scheme choice), 
in practice certain options are developed and tested for a small subset of other physics 
option combinations.   Therefore, the following table is useful as a guide for 
combinations of WRF physics options that are known to provide bit-for-bit results 
between serial and MPI versions of WRF.   Each row in the table represents a specific 
test, and each column a specific physics option.   All of the following tests are 
exercised using all three compilers on Yellowstone.   Each of the physics combinations 
listed in the tables can be considered "safe" combinations that will provide successful 
short-term forecasts with bit-for-bit results when comparing single-processor output 
against multi-processor output.  





                                 TABLE 1: 
          WRF ARW Tests, Providing Successful 30-Minute Forecasts, 
              and Bit-for-Bit Results, on Serial vs. MPI Runs


NL	PBL	CU	MP	LW	SW	SFC	LAND	URB	SHCU	TOPO
01	 1	 1	 1	 1	 1	 1	 1	 0	 0	 0
01ST	 1	 1	 1	 1	 1	 1	 1	 0	 0	 0
02	 2	 2	 4	 3	 3	 2	 4	 0	 0	 0
02GR	 2	 2	 4	 3	 3	 2	 4	 0	 0	 0
02ST	 2	 2	 4	 3	 3	 2	 4	 0	 0	 0
03	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
03DF	 4	 3	 3	 4	 4	 4	 1	 0	 0	 0
03FD	 4	 3	 3	 4	 4	 4	 1	 0	 0	 0
03ST	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
03VN	 4	 3	 3	 1	 1	 4	 1	 0	 0	 0
04FD	 4	 3	 3	 4	 4	 4	 1	 0	 0	 0
05	 1	 1	 3	 1	 1	 1	 2	 0	 0	 0
05AD	 1	 1	 3	 1	 1	 1	 2	 0	 0	 0
05ST	 1	 1	 3	 1	 1	 1	 2	 0	 0	 0
06	 8	 6	 6	 24	 24	 2	 1	 0	 0	 0
06BN	 8	 6	 6	 4	 4	 2	 1	 0	 0	 0
06VN	 8	 6	 6	 1	 1	 2	 1	 0	 0	 0
07	8	14	 7	 7	 7	1	 2	 2	 0	 0
07NE	8	14	 7	 7	 7	1	 2	 2	 0	 0
07VN	8	14	 7	 1	 1	1	 2	 2	 0	 0
08	 9	 7	 8	 5	 5	 2	 3	 0	 0	 0
09	 5	 0	 9	 1	 2	 5	 3	 0	 0	 0
09QT	 5	 0	 9	 1	 2	 5	 3	 0	 0	 0
10	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
10VN	 4	 2	 10	 1	 1	 4	 7	 0	 0	 0
11	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
12	 8	 3	 16	 24	 24	 1	 2	 3	 0	 0
12GR	 8	 3	 16	 4	 4	 1	 2	 3	 0	 0
13	 9	 7	13	 1	 1	 2	 3	 0	 2	 0
14	 4	 6	 3	 3	 3	 4	 3	 0	 0	 0
14VN	 4	 6	 3	 1	 1	 4	 3	 0	 0	 0
15	5	14	 2	 5	 5	1	 7	 0	 0	 0
15AD	5	14	 2	 5	 5	1	 7	 0	 0	 0
16	9	14	 4	 5	 5	2	 7	 0	 0	 0
16BN	9	14	 4	 5	 5	2	 7	 0	 0	 0
16DF	9	14	 4	 5	 5	2	 7	 0	 0	 0
16VN	2	14	 4	 1	 1	2	 7	 0	 0	 0
17	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
17AD	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
17VN	 2	 2	 4	 1	 1	 2	 2	 0	 0	 0
18	10	6	 4	 5	 5	10	 7	 0	 0	 0
18BN	10	6	 4	 5	 5	10	 7	 0	 0	 0
18VN	10	6	 4	 1	 1	10	 7	 0	 0	 0
19	 1	 1	 4	 1	 2	 1	 5	 0	 0	 0
20	 12	 1	 4	 1	 2	 1	 2	 0	 0	 0
20NE	 12	 1	 4	 1	 2	 1	 2	 0	 0	 0
20VN	 1	 1	 3	 1	 1	 1	 2	 0	 0	 0
21	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
25	 1	 1	 1	 1	 1	 91	 1	 0	 0	 0
26	 2	 1	 1	 1	 1	 3	 1	 0	 0	 0
29	 9	 3	 4	 1	 2	 1	 5	 0	 2	 0
29QT	 9	 3	 4	 1	 2	 1	 5	 0	 2	 0
30	 2	 3	 4	 1	 1	 2	 1	 0	 2	 0
31	 7	 2	 14	 3	 3	 7	 1	 0	 0	 0
31AD	 7	 2	 14	 3	 3	 7	 1	 0	 0	 0
31VN	 7	 2	 14	 1	 1	 7	 1	 0	 0	 0
32	 9	 7	 11	 3	 3	 1	 5	 0	 2	 0
33	 9	 7	 11	 24	 24	 1	 5	 0	 2	 0
34	 9	 7	 11	 24	 24	 1	 2	 0	 2	 0
35	 9	 7	 11	 3	 3	 1	 2	 0	 2	 0
36	 8	 3	 16	 24	 24	 1	 2	 0	 0	 0
36GR	 8	 3	 16	 4	 4	 1	 2	 0	 0	 0
37	 9	 7	 11	 24	 24	 2	 2	 0	 2	 0
38	5	14	 2	 5	 5	2	 7	 0	 0	 0
38AD	5	14	 2	 5	 5	2	 7	 0	 0	 0
38VN	5	14	 2	 1	 1	2	 7	 0	 0	 0
39	5	14	 2	 5	 5	5	 7	 0	 0	 0
39AD	5	14	 2	 5	 5	5	 7	 0	 0	 0
40	 7	 2	 14	 3	 3	 7	 1	 0	 0	 0
41	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
42	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
42VN	 4	 2	 10	 1	 1	 4	 7	 0	 0	 0
43	 2	 93	 4	 1	 1	 2	 1	 0	 0	 0
46	 1	 1	 1	 1	 1	 1	 1	 0	 0	 0
47	 9	 7	 28	 5	 5	 2	 3	 0	 0	 0
48	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
48VN	 4	 3	 3	 1	 1	 4	 1	 0	 0	 0
49	 1	 1	 3	 24	 24	 91	 2	 0	 0	 0
49VN	 1	 1	 3	 1	 1	 91	 2	 0	 0	 0
50	 1	 1	 3	 24	 24	 91	 4	 0	 0	 0
50VN	 1	 1	 3	 1	 1	 91	 4	 0	 0	 0
51	 1	 1	 3	 24	 24	 91	 4	 0	 0	 0
52	 4	 3	 17	 24	 24	 4	 1	 0	 0	 0
52DF	 4	 3	 17	 4	 4	 4	 1	 0	 0	 0
52FD	 4	 3	 17	 4	 4	 4	 1	 0	 0	 0
52VN	 4	 3	 17	 1	 1	 4	 1	 0	 0	 0
54	 1	 1	 32	 24	 24	 1	 2	 0	 0	 0
55FD	 4	 3	 17	 4	 4	 4	 1	 0	 0	 0
56	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
56NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
56VN	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
57	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
57NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
58	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
58NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
60	 1	 11	 6	 24	 24	 1	 4	 0	 0	 0
60NE	 1	 11	 6	 4	 4	 1	 4	 0	 0	 0
61	 1	 16	 4	 24	 24	 1	 2	 0	 0	 0
61NE	 1	 16	 4	 4	 4	 1	 2	 0	 0	 0
62	 11	 0	 6	 1	 24	 1	 4	 0	 0	 0
63	 2	 3	 10	 1	 2	 2	 2	 0	 0	 0
64	 4	 10	 17	 3	 3	 4	 1	 0	 0	 0
64FD	 4	 10	 17	 3	 3	 4	 1	 0	 0	 0
64VN	 4	 10	 17	 1	 1	 4	 1	 0	 0	 0
65	 9	 7	 28	 4	 4	 2	 3	 0	 0	 0
65DF	 9	 7	 28	 4	 4	 2	 3	 0	 0	 0
65NE	 9	 7	 28	 4	 4	 2	 3	 0	 0	 0
66FD	 4	 1	 3	 4	 4	 4	 1	 0	 0	 0
67	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
67NE	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
67VN	 1	 1	 8	 1	 1	 1	 2	 1	 0	 0
68	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
68NE	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
68VN	 1	 1	 8	 1	 1	 1	 2	 1	 0	 0
global	 1	 1	 3	 1	 1	 1	 1	 0	 0	 0



            				KEY 1:   Column Labels      (Tables 1-4)
            				----------------------------------------
            				NL   => Test Namelist Identifier
            				PBL  => Planetary Boundary Layer Scheme
            				CU   => Cumulus Scheme
            				MP   => Microphysics Scheme
            				LW   => Longwave Radiation Scheme
            				SW   => Shortwave Radiation Scheme
            				SFC  => Surface Physics Scheme
            				LAND => Land Surface Scheme
            				URB  => Urban Physics Scheme
            				SHCU => Shallow Cumulus Scheme
            				TOPO => Topography-Following Wind Scheme
            				----------------------------------------



                 			KEY 2:  Test Namelist Codes (Tables 1-2)
                 			----------------------------------------
                 			AD => Adaptive Time Stepping
                 			BN => Binary WRF Output
                 			DF => Digital Filtering
                 			FD => FDDA
                 			GR => Grib 1 WRF Output
                 			NE => Basic Nesting
                 			QT => Quilting
                 			VN => Vertical Nesting
                 			----------------------------------------




                                 TABLE 2: 
          WRF ARW Tests, Providing Successful 30-Minute Forecasts, 
             and Bit-for-Bit Results, on Serial vs. OpenMP Runs

NL	PBL	CU	MP	LW	SW	SFC	LAND	URB	SHCU	TOPO
03	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
03DF	 4	 3	 3	 4	 4	 4	 1	 0	 0	 0
03FD	 4	 3	 3	 4	 4	 4	 1	 0	 0	 0
03VN	 4	 3	 3	 1	 1	 4	 1	 0	 0	 0
06	 8	 6	 6	 24	 24	 2	 1	 0	 0	 0
06BN	 8	 6	 6	 4	 4	 2	 1	 0	 0	 0
06VN	 8	 6	 6	 1	 1	 2	 1	 0	 0	 0
07	8	14	 7	 7	 7	1	 2	 2	 0	 0
07NE	8	14	 7	 7	 7	1	 2	 2	 0	 0
07VN	8	14	 7	 1	 1	1	 2	 2	 0	 0
09	 5	 0	 9	 1	 2	 5	 3	 0	 0	 0
09QT	 5	 0	 9	 1	 2	 5	 3	 0	 0	 0
10	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
10VN	 4	 2	 10	 1	 1	 4	 7	 0	 0	 0
11	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
14	 4	 6	 3	 3	 3	 4	 3	 0	 0	 0
16	9	14	 4	 5	 5	2	 7	 0	 0	 0
16BN	9	14	 4	 5	 5	2	 7	 0	 0	 0
16DF	9	14	 4	 5	 5	2	 7	 0	 0	 0
16VN	2	14	 4	 1	 1	2	 7	 0	 0	 0
17	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
17AD	 2	 2	 4	 3	 3	 2	 2	 0	 0	 0
17VN	 2	 2	 4	 1	 1	 2	 2	 0	 0	 0
18	10	6	 4	 5	 5	10	 7	 0	 0	 0
18BN	10	6	 4	 5	 5	10	 7	 0	 0	 0
18VN	10	6	 4	 1	 1	10	 7	 0	 0	 0
20	 12	 1	 4	 1	 2	 1	 2	 0	 0	 0
20NE	 12	 1	 4	 1	 2	 1	 2	 0	 0	 0
20VN	 1	 1	 3	 1	 1	 1	 2	 0	 0	 0
21	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
31	 7	 2	 14	 3	 3	 7	 1	 0	 0	 0
31AD	 7	 2	 14	 3	 3	 7	 1	 0	 0	 0
31VN	 7	 2	 14	 1	 1	 7	 1	 0	 0	 0
38	5	14	 2	 5	 5	2	 7	 0	 0	 0
38VN	5	14	 2	 1	 1	2	 7	 0	 0	 0
42	 4	 2	 10	 1	 2	 4	 7	 0	 0	 0
42VN	 4	 2	 10	 1	 1	 4	 7	 0	 0	 0
48	 4	 3	 3	 24	 24	 4	 1	 0	 0	 0
48VN	 4	 3	 3	 1	 1	 4	 1	 0	 0	 0
49	 1	 1	 3	 24	 24	 91	 2	 0	 0	 0
49VN	 1	 1	 3	 1	 1	 91	 2	 0	 0	 0
50	 1	 1	 3	 24	 24	 91	 4	 0	 0	 0
50VN	 1	 1	 3	 1	 1	 91	 4	 0	 0	 0
51	 1	 1	 3	 24	 24	 91	 4	 0	 0	 0
52	 4	 3	 17	 24	 24	 4	 1	 0	 0	 0
52DF	 4	 3	 17	 4	 4	 4	 1	 0	 0	 0
52FD	 4	 3	 17	 4	 4	 4	 1	 0	 0	 0
52VN	 4	 3	 17	 1	 1	 4	 1	 0	 0	 0
56	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
56NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
56VN	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
57	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
57NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
58	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
58NE	 1	 1	 3	 1	 1	 1	 4	 0	 0	 0
60	 1	 11	 6	 24	 24	 1	 4	 0	 0	 0
60NE	 1	 11	 6	 4	 4	 1	 4	 0	 0	 0
62	 11	 0	 6	 1	 24	 1	 4	 0	 0	 0
65DF	 9	 7	 28	 4	 4	 2	 3	 0	 0	 0
66FD	 4	 1	 3	 4	 4	 4	 1	 0	 0	 0
67	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
67NE	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
67VN	 1	 1	 8	 1	 1	 1	 2	 1	 0	 0
68	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
68NE	 1	 1	 8	 4	 4	 1	 2	 1	 0	 0
68VN	 1	 1	 8	 1	 1	 1	 2	 1	 0	 0
global	 1	 1	 3	 1	 1	 1	 1	 0	 0	 0






                                 TABLE 3: 
       WRF Idealized Supercell Tests, Providing Successful 30-Minute Forecasts, 
       and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI)

NL      PBL     CU      MP      LW      SW      SFC
02       0       0       1       0       0       1
02NE     0       0       1       0       0       1
03       0       0       1       0       0       1
03NE     0       0       1       0       0       1
04       0       0       2       0       0       1
04NE     0       0       2       0       0       1
05       0       0       2       0       0       1
05NE     0       0       2       0       0       1
06       0       0       18      0       0       1
06NE     0       0       18      0       0       1
08       0       0       18      0       0       1
09       0       0       19      0       0       1
10       0       0       21      0       0       1
11NE     0       0       1       0       0       0
12NE     0       0       1       0       0       0
13NE     0       0       1       0       0       0
14NE     0       0       1       0       0       0






                                 TABLE 4: 
        WRF Idealized B-Wave Tests, Providing Successful 30-Minute Forecasts, 
         and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI)

NL      PBL     CU      MP      LW      SW      SFC
1        0       0       1       0       0       0
1NE      0       0       1       0       0       0
2        0       0       1       0       0       0
2NE      0       0       1       0       0       0
3        0       0       2       0       0       0
3NE      0       0       2       0       0       0
4        0       0       2       0       0       0
4NE      0       0       2       0       0       0
5        0       0       0       0       0       0
5NE      0       0       0       0       0       0