WRF Software Testing The testing conducted on the WRF code to insure bit-for-bit behavior on differing processor counts runs through hundreds of short forecasts in a couple hours. These simulatins are very short (about 10 time steps). The purpose is to activate as many possible physics options. If single processor vs multiple processor results differ, then there is a strong likelihood that improper initialization of variables or missing communications or race conditions exist. While tracking down the root cause of the problem is extremely time consuming, physics options that exhibit clean bit-wise reproducible results are more likely to be robust. This testing is handled entirely with a newly developed mechanism designed to run on small desktops and on batch systems. =========================== WRF Test Framework =========================== 1. Overview The WRF Testing Framework is designed to build, test, and analyze test results for one or more versions of the WRF model. With the advent of NCAR's flagship mainframe, Yellowstone, in early 2013, the testing framework for WRF was rewritten to accommodate Yellowstone's additional flexibility. For example, a large number of Fortran compilers are available now, instead of just a single Fortran compiler that was available on NCAR's "bluefire" machine. Yellowstone's greater speed and greater job throughput has also allowed the number of tests performed for WRF to expand. Despite all of the additional variations available on Yellowstone, a complete WRF test run (compiling WRF, running tests, and analyzing results) takes about two hours on Yellowstone. WRF is now tested regularly for the following compilers, parallel processing configurations, compile-time variations, and run-time variations of the WRF software: COMPILERS: GNU fortran version 4.9.2 PGI fortran version 15.1 Intel fortran version 15.0.1 PARALLEL BUILD CONFIGURATIONS: Serial (single-processor) build OpenMP (multithreaded, shared memory) build MPI (multiprocessor, distributed memory) build WRF COMPILE-TIME VARIATIONS ARW NMM Nested NMM HWRF CHEM CHEM with KPP Idealized Super Cell Idealized Baroclinc Wave Idealized 2D Hill 3D Var 4D Var WRFPlus ARW em_real RUN-TIME VARIATIONS Adaptive Time Stepping Digital Filtering FDDA Grib 1 WRF Output Binary WRF Output Nesting Quilting Global Domain Vertical Nesting 2. Physics Options Applied in WRF Tests The following Table and associated table Key summarizes the combinations of physics options that are tested for WRF. It is important to note that while the choice of one physics option should not influence the choice of another physics option (i.e., the choice of a microphysics scheme should be independent of the cumulus scheme choice), in practice certain options are developed and tested for a small subset of other physics option combinations. Therefore, the following table is useful as a guide for combinations of WRF physics options that are known to provide bit-for-bit results between serial and MPI versions of WRF. Each row in the table represents a specific test, and each column a specific physics option. All of the following tests are exercised using all three compilers on Yellowstone. Each of the physics combinations listed in the tables can be considered "safe" combinations that will provide successful short-term forecasts with bit-for-bit results when comparing single-processor output against multi-processor output. TABLE 1: WRF ARW Tests, Providing Successful 30-Minute Forecasts, and Bit-for-Bit Results, on Serial vs. MPI Runs NL PBL CU MP LW SW SFC LAND URB SHCU TOPO 01 1 1 1 1 1 1 1 0 0 0 01ST 1 1 1 1 1 1 1 0 0 0 02 2 2 4 3 3 2 4 0 0 0 02GR 2 2 4 3 3 2 4 0 0 0 02ST 2 2 4 3 3 2 4 0 0 0 03 4 3 3 24 24 4 1 0 0 0 03DF 4 3 3 4 4 4 1 0 0 0 03FD 4 3 3 4 4 4 1 0 0 0 03ST 4 3 3 24 24 4 1 0 0 0 03VN 4 3 3 1 1 4 1 0 0 0 04FD 4 3 3 4 4 4 1 0 0 0 05 1 1 3 1 1 1 2 0 0 0 05AD 1 1 3 1 1 1 2 0 0 0 05ST 1 1 3 1 1 1 2 0 0 0 06 8 6 6 24 24 2 1 0 0 0 06BN 8 6 6 4 4 2 1 0 0 0 06VN 8 6 6 1 1 2 1 0 0 0 07 8 14 7 7 7 1 2 2 0 0 07NE 8 14 7 7 7 1 2 2 0 0 07VN 8 14 7 1 1 1 2 2 0 0 08 9 7 8 5 5 2 3 0 0 0 09 5 0 9 1 2 5 3 0 0 0 09QT 5 0 9 1 2 5 3 0 0 0 10 4 2 10 1 2 4 7 0 0 0 10VN 4 2 10 1 1 4 7 0 0 0 11 4 2 10 1 2 4 7 0 0 0 12 8 3 16 24 24 1 2 3 0 0 12GR 8 3 16 4 4 1 2 3 0 0 13 9 7 13 1 1 2 3 0 2 0 14 4 6 3 3 3 4 3 0 0 0 14VN 4 6 3 1 1 4 3 0 0 0 15 5 14 2 5 5 1 7 0 0 0 15AD 5 14 2 5 5 1 7 0 0 0 16 9 14 4 5 5 2 7 0 0 0 16BN 9 14 4 5 5 2 7 0 0 0 16DF 9 14 4 5 5 2 7 0 0 0 16VN 2 14 4 1 1 2 7 0 0 0 17 2 2 4 3 3 2 2 0 0 0 17AD 2 2 4 3 3 2 2 0 0 0 17VN 2 2 4 1 1 2 2 0 0 0 18 10 6 4 5 5 10 7 0 0 0 18BN 10 6 4 5 5 10 7 0 0 0 18VN 10 6 4 1 1 10 7 0 0 0 19 1 1 4 1 2 1 5 0 0 0 20 12 1 4 1 2 1 2 0 0 0 20NE 12 1 4 1 2 1 2 0 0 0 20VN 1 1 3 1 1 1 2 0 0 0 21 4 3 3 24 24 4 1 0 0 0 25 1 1 1 1 1 91 1 0 0 0 26 2 1 1 1 1 3 1 0 0 0 29 9 3 4 1 2 1 5 0 2 0 29QT 9 3 4 1 2 1 5 0 2 0 30 2 3 4 1 1 2 1 0 2 0 31 7 2 14 3 3 7 1 0 0 0 31AD 7 2 14 3 3 7 1 0 0 0 31VN 7 2 14 1 1 7 1 0 0 0 32 9 7 11 3 3 1 5 0 2 0 33 9 7 11 24 24 1 5 0 2 0 34 9 7 11 24 24 1 2 0 2 0 35 9 7 11 3 3 1 2 0 2 0 36 8 3 16 24 24 1 2 0 0 0 36GR 8 3 16 4 4 1 2 0 0 0 37 9 7 11 24 24 2 2 0 2 0 38 5 14 2 5 5 2 7 0 0 0 38AD 5 14 2 5 5 2 7 0 0 0 38VN 5 14 2 1 1 2 7 0 0 0 39 5 14 2 5 5 5 7 0 0 0 39AD 5 14 2 5 5 5 7 0 0 0 40 7 2 14 3 3 7 1 0 0 0 41 2 2 4 3 3 2 2 0 0 0 42 4 2 10 1 2 4 7 0 0 0 42VN 4 2 10 1 1 4 7 0 0 0 43 2 93 4 1 1 2 1 0 0 0 46 1 1 1 1 1 1 1 0 0 0 47 9 7 28 5 5 2 3 0 0 0 48 4 3 3 24 24 4 1 0 0 0 48VN 4 3 3 1 1 4 1 0 0 0 49 1 1 3 24 24 91 2 0 0 0 49VN 1 1 3 1 1 91 2 0 0 0 50 1 1 3 24 24 91 4 0 0 0 50VN 1 1 3 1 1 91 4 0 0 0 51 1 1 3 24 24 91 4 0 0 0 52 4 3 17 24 24 4 1 0 0 0 52DF 4 3 17 4 4 4 1 0 0 0 52FD 4 3 17 4 4 4 1 0 0 0 52VN 4 3 17 1 1 4 1 0 0 0 54 1 1 32 24 24 1 2 0 0 0 55FD 4 3 17 4 4 4 1 0 0 0 56 1 1 3 1 1 1 4 0 0 0 56NE 1 1 3 1 1 1 4 0 0 0 56VN 1 1 3 1 1 1 4 0 0 0 57 1 1 3 1 1 1 4 0 0 0 57NE 1 1 3 1 1 1 4 0 0 0 58 1 1 3 1 1 1 4 0 0 0 58NE 1 1 3 1 1 1 4 0 0 0 60 1 11 6 24 24 1 4 0 0 0 60NE 1 11 6 4 4 1 4 0 0 0 61 1 16 4 24 24 1 2 0 0 0 61NE 1 16 4 4 4 1 2 0 0 0 62 11 0 6 1 24 1 4 0 0 0 63 2 3 10 1 2 2 2 0 0 0 64 4 10 17 3 3 4 1 0 0 0 64FD 4 10 17 3 3 4 1 0 0 0 64VN 4 10 17 1 1 4 1 0 0 0 65 9 7 28 4 4 2 3 0 0 0 65DF 9 7 28 4 4 2 3 0 0 0 65NE 9 7 28 4 4 2 3 0 0 0 66FD 4 1 3 4 4 4 1 0 0 0 67 1 1 8 4 4 1 2 1 0 0 67NE 1 1 8 4 4 1 2 1 0 0 67VN 1 1 8 1 1 1 2 1 0 0 68 1 1 8 4 4 1 2 1 0 0 68NE 1 1 8 4 4 1 2 1 0 0 68VN 1 1 8 1 1 1 2 1 0 0 global 1 1 3 1 1 1 1 0 0 0 KEY 1: Column Labels (Tables 1-4) ---------------------------------------- NL => Test Namelist Identifier PBL => Planetary Boundary Layer Scheme CU => Cumulus Scheme MP => Microphysics Scheme LW => Longwave Radiation Scheme SW => Shortwave Radiation Scheme SFC => Surface Physics Scheme LAND => Land Surface Scheme URB => Urban Physics Scheme SHCU => Shallow Cumulus Scheme TOPO => Topography-Following Wind Scheme ---------------------------------------- KEY 2: Test Namelist Codes (Tables 1-2) ---------------------------------------- AD => Adaptive Time Stepping BN => Binary WRF Output DF => Digital Filtering FD => FDDA GR => Grib 1 WRF Output NE => Basic Nesting QT => Quilting VN => Vertical Nesting ---------------------------------------- TABLE 2: WRF ARW Tests, Providing Successful 30-Minute Forecasts, and Bit-for-Bit Results, on Serial vs. OpenMP Runs NL PBL CU MP LW SW SFC LAND URB SHCU TOPO 03 4 3 3 24 24 4 1 0 0 0 03DF 4 3 3 4 4 4 1 0 0 0 03FD 4 3 3 4 4 4 1 0 0 0 03VN 4 3 3 1 1 4 1 0 0 0 06 8 6 6 24 24 2 1 0 0 0 06BN 8 6 6 4 4 2 1 0 0 0 06VN 8 6 6 1 1 2 1 0 0 0 07 8 14 7 7 7 1 2 2 0 0 07NE 8 14 7 7 7 1 2 2 0 0 07VN 8 14 7 1 1 1 2 2 0 0 09 5 0 9 1 2 5 3 0 0 0 09QT 5 0 9 1 2 5 3 0 0 0 10 4 2 10 1 2 4 7 0 0 0 10VN 4 2 10 1 1 4 7 0 0 0 11 4 2 10 1 2 4 7 0 0 0 14 4 6 3 3 3 4 3 0 0 0 16 9 14 4 5 5 2 7 0 0 0 16BN 9 14 4 5 5 2 7 0 0 0 16DF 9 14 4 5 5 2 7 0 0 0 16VN 2 14 4 1 1 2 7 0 0 0 17 2 2 4 3 3 2 2 0 0 0 17AD 2 2 4 3 3 2 2 0 0 0 17VN 2 2 4 1 1 2 2 0 0 0 18 10 6 4 5 5 10 7 0 0 0 18BN 10 6 4 5 5 10 7 0 0 0 18VN 10 6 4 1 1 10 7 0 0 0 20 12 1 4 1 2 1 2 0 0 0 20NE 12 1 4 1 2 1 2 0 0 0 20VN 1 1 3 1 1 1 2 0 0 0 21 4 3 3 24 24 4 1 0 0 0 31 7 2 14 3 3 7 1 0 0 0 31AD 7 2 14 3 3 7 1 0 0 0 31VN 7 2 14 1 1 7 1 0 0 0 38 5 14 2 5 5 2 7 0 0 0 38VN 5 14 2 1 1 2 7 0 0 0 42 4 2 10 1 2 4 7 0 0 0 42VN 4 2 10 1 1 4 7 0 0 0 48 4 3 3 24 24 4 1 0 0 0 48VN 4 3 3 1 1 4 1 0 0 0 49 1 1 3 24 24 91 2 0 0 0 49VN 1 1 3 1 1 91 2 0 0 0 50 1 1 3 24 24 91 4 0 0 0 50VN 1 1 3 1 1 91 4 0 0 0 51 1 1 3 24 24 91 4 0 0 0 52 4 3 17 24 24 4 1 0 0 0 52DF 4 3 17 4 4 4 1 0 0 0 52FD 4 3 17 4 4 4 1 0 0 0 52VN 4 3 17 1 1 4 1 0 0 0 56 1 1 3 1 1 1 4 0 0 0 56NE 1 1 3 1 1 1 4 0 0 0 56VN 1 1 3 1 1 1 4 0 0 0 57 1 1 3 1 1 1 4 0 0 0 57NE 1 1 3 1 1 1 4 0 0 0 58 1 1 3 1 1 1 4 0 0 0 58NE 1 1 3 1 1 1 4 0 0 0 60 1 11 6 24 24 1 4 0 0 0 60NE 1 11 6 4 4 1 4 0 0 0 62 11 0 6 1 24 1 4 0 0 0 65DF 9 7 28 4 4 2 3 0 0 0 66FD 4 1 3 4 4 4 1 0 0 0 67 1 1 8 4 4 1 2 1 0 0 67NE 1 1 8 4 4 1 2 1 0 0 67VN 1 1 8 1 1 1 2 1 0 0 68 1 1 8 4 4 1 2 1 0 0 68NE 1 1 8 4 4 1 2 1 0 0 68VN 1 1 8 1 1 1 2 1 0 0 global 1 1 3 1 1 1 1 0 0 0 TABLE 3: WRF Idealized Supercell Tests, Providing Successful 30-Minute Forecasts, and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI) NL PBL CU MP LW SW SFC 02 0 0 1 0 0 1 02NE 0 0 1 0 0 1 03 0 0 1 0 0 1 03NE 0 0 1 0 0 1 04 0 0 2 0 0 1 04NE 0 0 2 0 0 1 05 0 0 2 0 0 1 05NE 0 0 2 0 0 1 06 0 0 18 0 0 1 06NE 0 0 18 0 0 1 08 0 0 18 0 0 1 09 0 0 19 0 0 1 10 0 0 21 0 0 1 11NE 0 0 1 0 0 0 12NE 0 0 1 0 0 0 13NE 0 0 1 0 0 0 14NE 0 0 1 0 0 0 TABLE 4: WRF Idealized B-Wave Tests, Providing Successful 30-Minute Forecasts, and Bit-for-Bit Results, on Serial vs. Non-Serial Runs (OpenMP and MPI) NL PBL CU MP LW SW SFC 1 0 0 1 0 0 0 1NE 0 0 1 0 0 0 2 0 0 1 0 0 0 2NE 0 0 1 0 0 0 3 0 0 2 0 0 0 3NE 0 0 2 0 0 0 4 0 0 2 0 0 0 4NE 0 0 2 0 0 0 5 0 0 0 0 0 0 5NE 0 0 0 0 0 0