WRF Software Testing
The testing conducted on the WRF code to insure bit-for-bit behavior on differing
processor counts runs through hundreds of short forecasts in a couple hours. These
simulations are very short (about 10 time steps). The purpose is to activate as many
possible physics options. If single processor vs multiple processor results differ,
then there is a strong likelihood that improper initialization of variables or
missing communications or race conditions exist. While tracking down the root cause
of the problem is extremely time consuming, physics options that exhibit clean
bit-wise reproducible results are more likely to be robust. This testing is handled
entirely with a newly developed mechanism designed to run on small desktops and on
batch systems.
===========================
WRF Test Framework
===========================
1. Overview
The WRF Testing Framework is designed to build, test, and analyze test results for
one or more versions of the WRF model. With the advent of NCAR's flagship mainframe,
cheyenne, in 2017, the testing framework for WRF was rewritten to accommodate
cheyenne's additional flexibility. For example, a large number of Fortran compilers
are available now, instead of just a single Fortran compiler that was available on NCAR's
"bluefire" machine. cheyenne's greater speed and greater job throughput has also
allowed the number of tests performed for WRF to expand. Despite all of the additional
variations available on cheyenne, a complete WRF test run (compiling WRF, running
tests, and analyzing results) takes about three hours on cheyenne. WRF is now tested
regularly for the following compilers, parallel processing configurations, compile-time
variations, and run-time variations of the WRF software:
COMPILERS:
GNU fortran version 6.2.0, 6.3.0, 7.2.0
PGI fortran version 17.9
Intel fortran version 17.0.1
PARALLEL BUILD CONFIGURATIONS:
Serial (single-processor) build
OpenMP (multithreaded, shared memory) build
MPI (multiprocessor, distributed memory) build
WRF COMPILE-TIME VARIATIONS
ARW
ARW real*8
ARW Nested
ARW Moving Nest
NMM Nested
NMM HWRF
CHEM
Idealized Fire
Idealized Super Cell
Idealized Super Cell real*8
Idealized Baroclinc Wave
Idealized 2D Hill
Idealized Single Column Model
ARW em_real RUN-TIME VARIATIONS
Adaptive Time Stepping
Digital Filtering
FDDA: Spectral nudging, obs nudging, surface and upper air nudging
Stochastic Forcing
Nesting
Quilting
Global Domain
Vertical Nesting
Hybrid and Terrain Following Vertical Coordinates
Dry and Moist Potential Temperature
2. Physics Options Applied in WRF Tests
The following Table and associated table Key summarizes the combinations of physics
options that are tested for WRF. It is important to note that while the choice of
one physics option should not influence the choice of another physics option (i.e.,
the choice of a microphysics scheme should be independent of the cumulus scheme choice),
in practice certain options are developed and tested for a small subset of other physics
option combinations. Therefore, the following table is useful as a guide for
combinations of WRF physics options that are known to provide bit-for-bit results
between serial and MPI versions of WRF. Each row in the table represents a specific
test, and each column a specific physics option. All of the following tests are
exercised using all three compilers on cheyenne. Each of the physics combinations
listed in the tables can be considered "safe" combinations that will provide successful
short-term forecasts with bit-for-bit results when comparing single-processor output
against multi-processor output.
KEY 1: Column Labels (Tables 1-7)
----------------------------------------
NL => Test Namelist Identifier
PBL => Planetary Boundary Layer Scheme
CU => Cumulus Scheme
MP => Microphysics Scheme
LW => Longwave Radiation Scheme
SW => Shortwave Radiation Scheme
SFC => Surface Physics Scheme
LAND => Land Surface Scheme
URB => Urban Physics Scheme
SHCU => Shallow Cumulus Scheme
TOPO => Topography-Following Wind Scheme
----------------------------------------
KEY 2: Test Namelist Codes (Tables 1-7)
----------------------------------------
AD => Adaptive Time Stepping
BN => Binary WRF Output
DF => Digital Filtering
FD => FDDA
GR => Grib 1 WRF Output
NE => Basic Nesting
QT => Quilting
VN => Vertical Nesting
----------------------------------------
TABLE 1:
WRF ARW Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. MPI Runs
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
01 1 1 1 1 1 1 1 0 0 0
01ST 1 1 1 1 1 1 1 0 0 0
02 2 2 4 3 3 2 4 0 0 0
02GR 2 2 4 3 3 2 4 0 0 0
02ST 2 2 4 3 3 2 4 0 0 0
03 4 3 3 24 24 4 1 0 0 0
03DF 4 3 3 4 4 4 1 0 0 0
03FD 4 3 3 4 4 4 1 0 0 0
03ST 4 3 3 24 24 4 1 0 0 0
03VN 4 3 3 1 1 4 1 0 0 0
04FD 4 3 3 4 4 4 1 0 0 0
05 1 1 3 1 1 1 2 0 0 0
05AD 1 1 3 1 1 1 2 0 0 0
05ST 1 1 3 1 1 1 2 0 0 0
06 8 6 6 24 24 2 1 0 0 0
06BN 8 6 6 4 4 2 1 0 0 0
06VN 8 6 6 1 1 2 1 0 0 0
07 8 14 7 7 7 1 2 2 0 0
07NE 8 14 7 7 7 1 2 2 0 0
07VN 8 14 7 1 1 1 2 2 0 0
08 9 7 8 5 5 2 3 0 0 0
09 5 0 9 1 2 5 3 0 0 0
09QT 5 0 9 1 2 5 3 0 0 0
10 4 2 10 1 2 4 7 0 0 0
10VN 4 2 10 1 1 4 7 0 0 0
11 4 2 10 1 2 4 7 0 0 0
12 8 1 16 24 24 1 2 3 0 0
12GR 8 1 16 4 4 1 2 3 0 0
13 9 7 13 1 1 2 3 0 2 0
14 4 6 3 3 3 4 3 0 0 0
14VN 4 6 3 1 1 4 3 0 0 0
15 5 14 2 5 5 1 7 0 0 0
15AD 5 14 2 5 5 1 7 0 0 0
16 9 14 4 5 5 2 7 0 0 0
16BN 9 14 4 5 5 2 7 0 0 0
16DF 9 14 4 5 5 2 7 0 0 0
16VN 2 14 4 1 1 2 7 0 0 0
17 2 2 4 3 3 2 2 0 0 0
17AD 2 2 4 3 3 2 2 0 0 0
17VN 2 2 4 1 1 2 2 0 0 0
18 10 6 4 5 5 10 7 0 0 0
18BN 10 6 4 5 5 10 7 0 0 0
18VN 10 6 4 1 1 10 7 0 0 0
19 1 1 4 1 2 1 5 0 0 0
20 12 1 4 1 2 1 2 0 0 0
20NE 12 1 4 1 2 1 2 0 0 0
20VN 1 1 3 1 1 1 2 0 0 0
21 4 3 3 24 24 4 1 0 0 0
25 1 1 1 1 1 91 1 0 0 0
26 2 1 1 1 1 3 1 0 0 0
29 9 3 4 1 2 1 5 0 2 0
29QT 9 3 4 1 2 1 5 0 2 0
30 2 3 4 1 1 2 1 0 2 0
31 7 2 14 3 3 7 1 0 0 0
31AD 7 2 14 3 3 7 1 0 0 0
31VN 7 2 14 1 1 7 1 0 0 0
32 9 7 11 3 3 1 5 0 2 0
33 9 7 11 24 24 1 5 0 2 0
34 9 7 11 24 24 1 2 0 2 0
35 9 7 11 3 3 1 2 0 2 0
37 9 7 11 24 24 2 2 0 2 0
38 5 14 2 5 5 2 7 0 0 0
38AD 5 14 2 5 5 2 7 0 0 0
38VN 5 14 2 1 1 2 7 0 0 0
39 5 14 2 5 5 5 7 0 0 0
39AD 5 14 2 5 5 5 7 0 0 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
40 7 2 14 3 3 7 1 0 0 0
41 2 2 4 3 3 2 2 0 0 0
42 4 2 10 1 2 4 7 0 0 0
42VN 4 2 10 1 1 4 7 0 0 0
43 2 93 4 1 1 2 1 0 0 0
48 4 3 3 24 24 4 1 0 0 0
48VN 4 3 3 1 1 4 1 0 0 0
49 1 1 3 24 24 91 2 0 0 0
49VN 1 1 3 1 1 91 2 0 0 0
50 1 1 3 24 24 91 4 0 0 0
50VN 1 1 3 1 1 91 4 0 0 0
51 1 1 3 24 24 91 4 0 0 0
52 4 3 17 24 24 4 1 0 0 0
52DF 4 3 17 4 4 4 1 0 0 0
52FD 4 3 17 4 4 4 1 0 0 0
52VN 4 3 17 1 1 4 1 0 0 0
54 1 1 32 24 24 1 2 0 0 0
55FD 4 3 17 4 4 4 1 0 0 0
56 1 1 3 1 1 1 4 0 0 0
56NE 1 1 3 1 1 1 4 0 0 0
56VN 1 1 3 1 1 1 4 0 0 0
57 1 1 95 1 1 1 4 0 0 0
57NE 1 1 95 1 1 1 4 0 0 0
58 1 1 3 1 1 1 4 0 0 0
58NE 1 1 3 1 1 1 4 0 0 0
60 1 11 6 24 24 1 4 0 0 0
60NE 1 11 6 4 4 1 4 0 0 0
61 1 16 4 24 24 1 2 0 0 0
61NE 1 16 4 4 4 1 2 0 0 0
62 11 0 6 1 24 1 4 0 0 0
63 2 3 10 1 2 2 2 0 0 0
64 4 10 17 3 3 4 1 0 0 0
64FD 4 10 17 3 3 4 1 0 0 0
64VN 4 10 17 1 1 4 1 0 0 0
65DF 9 7 28 4 4 2 3 0 0 0
66FD 4 1 3 4 4 4 1 0 0 0
67 1 1 8 4 4 1 2 1 0 0
67NE 1 1 8 4 4 1 2 1 0 0
67VN 1 1 8 1 1 1 2 1 0 0
68 1 1 8 4 4 1 2 1 0 0
68NE 1 1 8 4 4 1 2 1 0 0
68VN 1 1 8 1 1 1 2 1 0 0
69 1 10 4 4 4 1 2 0 0 0
70 1 10 4 4 4 1 2 0 0 0
71 1 1 8 4 4 1 2 0
72 5 1 4 4 4 5 2 0
73 11 0 5 1 24 1 4 0
global 1 1 3 1 1 1 1 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
TABLE 2:
WRF ARW Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. OpenMP Runs
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
03 4 3 3 24 24 4 1 0 0 0
03DF 4 3 3 4 4 4 1 0 0 0
03FD 4 3 3 4 4 4 1 0 0 0
03VN 4 3 3 1 1 4 1 0 0 0
06 8 6 6 24 24 2 1 0 0 0
06BN 8 6 6 4 4 2 1 0 0 0
06VN 8 6 6 1 1 2 1 0 0 0
07 8 14 7 7 7 1 2 2 0 0
07NE 8 14 7 7 7 1 2 2 0 0
07VN 8 14 7 1 1 1 2 2 0 0
09 5 0 9 1 2 5 3 0 0 0
09QT 5 0 9 1 2 5 3 0 0 0
10 4 2 10 1 2 4 7 0 0 0
10VN 4 2 10 1 1 4 7 0 0 0
11 4 2 10 1 2 4 7 0 0 0
14 4 6 3 3 3 4 3 0 0 0
16 9 14 4 5 5 2 7 0 0 0
16BN 9 14 4 5 5 2 7 0 0 0
16DF 9 14 4 5 5 2 7 0 0 0
16VN 2 14 4 1 1 2 7 0 0 0
17 2 2 4 3 3 2 2 0 0 0
17AD 2 2 4 3 3 2 2 0 0 0
17VN 2 2 4 1 1 2 2 0 0 0
18 10 6 4 5 5 10 7 0 0 0
18BN 10 6 4 5 5 10 7 0 0 0
18VN 10 6 4 1 1 10 7 0 0 0
20 12 1 4 1 2 1 2 0 0 0
20NE 12 1 4 1 2 1 2 0 0 0
20VN 1 1 3 1 1 1 2 0 0 0
21 4 3 3 24 24 4 1 0 0 0
31 7 2 14 3 3 7 1 0 0 0
31AD 7 2 14 3 3 7 1 0 0 0
31VN 7 2 14 1 1 7 1 0 0 0
38 5 14 2 5 5 2 7 0 0 0
38VN 5 14 2 1 1 2 7 0 0 0
42 4 2 10 1 2 4 7 0 0 0
42VN 4 2 10 1 1 4 7 0 0 0
48 4 3 3 24 24 4 1 0 0 0
48VN 4 3 3 1 1 4 1 0 0 0
49 1 1 3 24 24 91 2 0 0 0
49VN 1 1 3 1 1 91 2 0 0 0
50 1 1 3 24 24 91 4 0 0 0
50VN 1 1 3 1 1 91 4 0 0 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
51 1 1 3 24 24 91 4 0 0 0
52 4 3 17 24 24 4 1 0 0 0
52DF 4 3 17 4 4 4 1 0 0 0
52FD 4 3 17 4 4 4 1 0 0 0
52VN 4 3 17 1 1 4 1 0 0 0
56 1 1 3 1 1 1 4 0 0 0
56NE 1 1 3 1 1 1 4 0 0 0
56VN 1 1 3 1 1 1 4 0 0 0
57 1 1 95 1 1 1 4 0 0 0
57NE 1 1 95 1 1 1 4 0 0 0
58 1 1 3 1 1 1 4 0 0 0
58NE 1 1 3 1 1 1 4 0 0 0
60 1 11 6 24 24 1 4 0 0 0
60NE 1 11 6 4 4 1 4 0 0 0
62 11 0 6 1 24 1 4 0 0 0
65DF 9 7 28 4 4 2 3 0 0 0
66FD 4 1 3 4 4 4 1 0 0 0
67 1 1 8 4 4 1 2 1 0 0
67NE 1 1 8 4 4 1 2 1 0 0
67VN 1 1 8 1 1 1 2 1 0 0
68 1 1 8 4 4 1 2 1 0 0
68NE 1 1 8 4 4 1 2 1 0 0
68VN 1 1 8 1 1 1 2 1 0 0
71 1 1 8 4 4 1 2 0 0 0
72 5 1 4 4 4 5 2 0 0 0
73 11 0 5 1 24 1 4 0
global 1 1 3 1 1 1 1 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
TABLE 3:
WRF ARW Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. MPI Runs
REAL*8
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
07 8 14 7 7 7 1 2 2 0 0
14 4 6 3 3 3 4 3 0 0 0
16 9 14 4 5 5 2 7 0 0 0
17 2 2 4 3 3 2 2 0 0 0
17AD 2 2 4 3 3 2 2 0 0 0
18 10 6 4 5 5 10 7 0 0 0
31 7 2 14 3 3 7 1 0 0 0
31AD 7 2 14 3 3 7 1 0 0 0
38 5 14 2 5 5 2 7 0 0 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
TABLE 4:
WRF ARW Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. OpenMP Runs
REAL*8
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
14 4 6 3 3 3 4 3 0 0 0
16 9 14 4 5 5 2 7 0 0 0
17 2 2 4 3 3 2 2 0 0 0
17AD 2 2 4 3 3 2 2 0 0 0
18 10 6 4 5 5 10 7 0 0 0
31 7 2 14 3 3 7 1 0 0 0
31AD 7 2 14 3 3 7 1 0 0 0
38 5 14 2 5 5 2 7 0 0 0
NL PBL CU MP LW SW SFC LAND URB SHCU TOPO
TABLE 5:
WRF Idealized Supercell Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI)
NL PBL CU MP LW SW SFC
02 0 0 1 0 0 1
02NE 0 0 1 0 0 1
03 0 0 1 0 0 1
03NE 0 0 1 0 0 1
04 0 0 2 0 0 1
04NE 0 0 2 0 0 1
05 0 0 2 0 0 1
05NE 0 0 2 0 0 1
06 0 0 18 0 0 1
06NE 0 0 18 0 0 1
08 0 0 18 0 0 1
09 0 0 19 0 0 1
10 0 0 21 0 0 1
11NE 0 0 1 0 0 0
12NE 0 0 1 0 0 0
13NE 0 0 1 0 0 0
14NE 0 0 1 0 0 0
NL PBL CU MP LW SW SFC
TABLE 6:
WRF Idealized Supercell Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI)
REAL*8
NL PBL CU MP LW SW SFC
02 0 0 1 0 0 1
03 0 0 1 0 0 1
04 0 0 2 0 0 1
05 0 0 2 0 0 1
06 0 0 18 0 0 1
08 0 0 18 0 0 1
09 0 0 19 0 0 1
10 0 0 21 0 0 1
NL PBL CU MP LW SW SFC
TABLE 7:
WRF Idealized B-Wave Tests, Providing Successful 10 Time Step Forecasts,
and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI)
NL PBL CU MP LW SW SFC
1 0 0 1 0 0 0
1NE 0 0 1 0 0 0
2 0 0 1 0 0 0
2NE 0 0 1 0 0 0
3 0 0 2 0 0 0
3NE 0 0 2 0 0 0
4 0 0 2 0 0 0
4NE 0 0 2 0 0 0
5 0 0 0 0 0 0
5NE 0 0 0 0 0 0
NL PBL CU MP LW SW SFC