|
WRF Software Testing
The testing conducted on the WRF code to ensure bit-for-bit behavior on differing processor counts runs through hundreds of short forecasts (only about 10 time-steps) in about an hour. The purpose is to activate as many possible physics options. If single processor vs. multiple processor results differ, then there is a strong likelihood that improper initialization of variables, missing communications, or race conditions exist. While tracking down the root cause of the problem is extremely time-consuming, physics options that exhibit clean bit-wise reproducible results are more likely to be robust. This testing is handled entirely with a newly-developed mechanism designed to run on small desktops, as well as on batch systems.
WRF Test Framework
1. Overview
The WRF Testing Framework is designed to build, test, and analyze test results for one or more versions of the WRF model. With the advent of NCAR's flagship mainframe, Yellowstone, in early 2013, the testing framework for WRF was rewritten to accommodate Yellowstone's additional flexibility. For example, a large number of Fortran compilers are available now, instead of just a single compiler that was available on NCAR's Bluefire machine. Yellowstone's faster speed and greater job throughput has also allowed the number of tests performed for WRF to expand. Despite all of the additional variations available on Yellowstone, a complete WRF test run (compiling WRF, running tests, and analyzing results) often takes less than one hour on Yellowstone. WRF is now tested regularly for the following compilers, parallel processing configurations, compile-time variations, and run-time variations of the WRF software:
COMPILERS
GNU Fortran version 4.7.2
PGI Fortran version 12.5
Intel Fortran version 12.1.5
PARALLEL BUILD CONFIGURATIONS
Serial (single-processor) build
OpenMP (multithreaded, shared memory) build
MPI (multiprocessor, distributed memory) build
WRF COMPILE-TIME VARIATIONS
ARW
NMM
NMM Nested
CHEM
CHEM with KPP
Idealized Super Cell
Idealized Baroclinic Wave
ARW em_real RUN-TIME VARIATIONS
Adaptive Time-Stepping
Digital Filtering
FDDA
Grib 1 WRF Output
Binary WRF Output
Nesting
Quilting
Global Domain
2. Physics Options Applied in WRF Tests
The following table and associated table key summarize the combinations of physics options that are tested for WRF. It is important to note that while the choice of one physics option should not influence the choice of another physics option (i.e., the choice of a microphysics scheme should be independent of the cumulus scheme), in practice, certain options are developed and tested for a small subset of other physics option combinations; therefore, the following table is useful as a guide for combinations of WRF physics options that are known to provide bit-for-bit results between serial and MPI compilations of WRF. Each row in the table represents a specific test, and each column represents a specific physics option. All of the following tests are exercised using all three compilers on Yellowstone. Each of the physics combinations listed in the tables can be considered "safe" combinations that will provide successful short-term forecasts with bit-for-bit results when comparing single-processor output against multi-processor output.
KEY 1: Column Labels (Tables 1-4) |
KEY 2: Test Namelist Codes (Tables 1-4) |
NL => Test Namelist Identifier
PBL => Planetary Boundary Layer Scheme
CU => Cumulus Scheme
MP => Microphysics Scheme
LW => Longwave Radiation Scheme
SW => Shortwave Radiation Scheme
SFC => Surface Physics Scheme
LAND => Land Surface Scheme
URB => Urban Physics Scheme
SHCU => Shallow Cumulus Scheme
TOPO => Topography-Following Wind Scheme
|
AD => Adaptive Time Stepping
BN => Binary WRF Output
DF => Digital Filtering
FD => FDDA
GR => Grib 1 WRF Output
NE => Basic Nesting
QT => Quilting
|
TABLE 1:
WRF ARW Tests, Providing Successful 30-minute Forecasts,
and Bit-for-Bit Results, on Serial vs. MPI runs
NL |
PBL |
CU |
MP |
LW |
SW |
SFC |
LAND |
URB |
SHCU |
TOPO |
global |
1 |
1 |
3 |
1 |
1 |
1 |
1 |
0 |
0 |
0 |
01 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
0 |
0 |
0 |
02 |
1 |
2 |
4 |
3 |
3 |
1 |
4 |
0 |
0 |
0 |
02GR |
1 |
2 |
4 |
3 |
3 |
1 |
4 |
0 |
0 |
0 |
03 |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
03DF |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
03FD |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
05 |
7 |
5 |
5 |
5 |
5 |
7 |
7 |
0 |
0 |
0 |
05AD |
7 |
5 |
5 |
5 |
5 |
7 |
7 |
0 |
0 |
0 |
05FD |
7 |
5 |
5 |
5 |
5 |
7 |
7 |
0 |
0 |
0 |
06 |
8 |
6 |
6 |
4 |
4 |
2 |
1 |
0 |
0 |
0 |
06BN |
8 |
6 |
6 |
4 |
4 |
2 |
1 |
0 |
0 |
0 |
07 |
8 |
14 |
7 |
7 |
7 |
1 |
2 |
2 |
0 |
0 |
07NE |
8 |
14 |
7 |
7 |
7 |
1 |
2 |
2 |
0 |
0 |
08 |
9 |
7 |
8 |
5 |
5 |
2 |
3 |
0 |
0 |
0 |
09 |
6 |
1 |
9 |
3 |
3 |
5 |
3 |
0 |
0 |
0 |
09QT |
6 |
1 |
9 |
3 |
3 |
5 |
3 |
0 |
0 |
0 |
10 |
4 |
2 |
10 |
1 |
2 |
4 |
7 |
0 |
0 |
0 |
12 |
8 |
3 |
16 |
4 |
4 |
1 |
2 |
3 |
0 |
0 |
12GR |
8 |
3 |
16 |
4 |
4 |
1 |
2 |
3 |
0 |
0 |
13 |
9 |
7 |
13 |
1 |
1 |
2 |
3 |
0 |
2 |
0 |
14 |
4 |
6 |
3 |
3 |
3 |
4 |
3 |
0 |
0 |
0 |
15 |
5 |
14 |
2 |
5 |
5 |
1 |
7 |
0 |
0 |
0 |
15AD |
5 |
14 |
2 |
5 |
5 |
1 |
7 |
0 |
0 |
0 |
16 |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
16BN |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
16DF |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
17 |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
17AD |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
19 |
1 |
1 |
4 |
1 |
2 |
1 |
5 |
0 |
0 |
0 |
20 |
12 |
1 |
4 |
1 |
2 |
1 |
2 |
0 |
0 |
0 |
20NE |
12 |
1 |
4 |
1 |
2 |
1 |
2 |
0 |
0 |
0 |
25 |
1 |
1 |
1 |
1 |
1 |
11 |
1 |
0 |
0 |
0 |
26 |
2 |
1 |
1 |
1 |
1 |
3 |
1 |
0 |
0 |
0 |
29 |
9 |
3 |
4 |
1 |
2 |
1 |
5 |
0 |
2 |
0 |
29QT |
9 |
3 |
4 |
1 |
2 |
1 |
5 |
0 |
2 |
0 |
30 |
2 |
93 |
4 |
1 |
1 |
2 |
1 |
0 |
2 |
0 |
31 |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
31AD |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
32 |
9 |
7 |
11 |
3 |
3 |
1 |
5 |
0 |
2 |
0 |
33 |
9 |
7 |
11 |
4 |
4 |
1 |
5 |
0 |
2 |
0 |
34 |
9 |
7 |
11 |
4 |
4 |
1 |
2 |
0 |
2 |
0 |
35 |
9 |
7 |
11 |
3 |
3 |
1 |
2 |
0 |
2 |
0 |
37 |
9 |
7 |
11 |
4 |
4 |
2 |
2 |
0 |
2 |
0 |
38 |
5 |
14 |
2 |
5 |
5 |
2 |
7 |
0 |
0 |
0 |
38AD |
5 |
14 |
2 |
5 |
5 |
2 |
7 |
0 |
0 |
0 |
39 |
5 |
14 |
2 |
5 |
5 |
5 |
7 |
0 |
0 |
0 |
39AD |
5 |
14 |
2 |
5 |
5 |
5 |
7 |
0 |
0 |
0 |
40 |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
41 |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
42 |
4 |
2 |
10 |
1 |
2 |
4 |
7 |
0 |
0 |
0 |
TABLE 2:
WRF ARW Tests, Providing Successful 30-minute Forecasts,
and Bit-for-Bit Results, on Serial vs. OpenMP runs
NL |
PBL |
CU |
MP |
LW |
SW |
SFC |
LAND |
URB |
SHCU |
TOPO |
global |
1 |
1 |
3 |
1 |
1 |
1 |
1 |
0 |
0 |
0 |
03 |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
03DF |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
03FD |
4 |
3 |
3 |
4 |
4 |
4 |
1 |
0 |
0 |
0 |
06 |
8 |
6 |
6 |
4 |
4 |
2 |
1 |
0 |
0 |
0 |
06BN |
8 |
6 |
6 |
4 |
4 |
2 |
1 |
0 |
0 |
0 |
07 |
8 |
14 |
7 |
7 |
7 |
1 |
2 |
2 |
0 |
0 |
07NE |
8 |
14 |
7 |
7 |
7 |
1 |
2 |
2 |
0 |
0 |
08 |
9 |
7 |
8 |
5 |
5 |
2 |
3 |
0 |
0 |
0 |
10 |
4 |
2 |
10 |
1 |
2 |
4 |
7 |
0 |
0 |
0 |
14 |
4 |
6 |
3 |
3 |
3 |
4 |
3 |
0 |
0 |
0 |
16 |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
16BN |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
16DF |
10 |
14 |
4 |
5 |
5 |
10 |
7 |
0 |
0 |
0 |
17 |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
17AD |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
20 |
12 |
1 |
4 |
1 |
2 |
1 |
2 |
0 |
0 |
0 |
20NE |
12 |
1 |
4 |
1 |
2 |
1 |
2 |
0 |
0 |
0 |
31 |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
31AD |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
38 |
5 |
14 |
2 |
5 |
5 |
2 |
7 |
0 |
0 |
0 |
40 |
7 |
2 |
14 |
3 |
3 |
7 |
1 |
0 |
0 |
0 |
41 |
2 |
2 |
4 |
3 |
3 |
2 |
2 |
0 |
0 |
0 |
42 |
4 |
2 |
10 |
1 |
2 |
4 |
7 |
0 |
0 |
0 |
TABLE 3:
WRF Idealized Supercell Tests, Providing Successful 30-minute Forecasts,
and Bit-for-Bit Results, on Serial vs. Non-serial runs (OpenMP and MPI)
NL |
PBL |
CU |
MP |
LW |
SW |
SFC |
LAND |
URB |
SHCU |
TOPO |
01 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
01NE |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
02 |
0 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
02NE |
0 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
03 |
0 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
03NE |
0 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
04 |
0 |
0 |
2 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
04NE |
0 |
0 |
2 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
05 |
0 |
0 |
2 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
05NE |
0 |
0 |
2 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
06 |
0 |
0 |
18 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
06NE |
0 |
0 |
18 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
07 |
0 |
0 |
17 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
08 |
0 |
0 |
18 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
09 |
0 |
0 |
19 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
10 |
0 |
0 |
21 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
TABLE 4:
WRF Idealized B-wave Tests, Providing Successful 30-minute Forecasts,
and Bit-for-Bit Results, on Serial vs. Non-serial runs (OpenMP and MPI)
NL |
PBL |
CU |
MP |
LW |
SW |
SFC |
LAND |
URB |
SHCU |
TOPO |
1 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
1NE |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
2 |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
2NE |
0 |
0 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
3 |
0 |
0 |
2 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
3NE |
0 |
0 |
2 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
4 |
0 |
0 |
2 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
4NE |
0 |
0 |
2 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
5 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
5NE |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|