======================
WRF & WPS Code Testing
======================

Prior to a WRF code release, the code is extensively tested to ensure new and modified code does not negatively impact building or running the code. The following describes the three areas on which testing is focused.

|

.. note::
   It is impossible to test all combinations of options. The possible combinations of physics options, alone, totals more than two million, and that number increases with the addition of dynamics options, urban effects, shallow convection, and impacts of climatologically-sensitive constituents. 

|

|

|

|

|

Forecast Verification
=====================

To verify that new and/or modified code produces reasonable and expected results. Simulations are run using the following criteria:

*   28 48-hour forecasts
*   Conducted over a month-long winter period and a month-long late spring period
*   CONUS domain
*   15 km resolution

Forecasts include each of a selected combination of physics and dynamics options, and the results are compared to observations and analyses (NCEP GFS surface and upper-air), subjectively and objectively. 

.. rst-class:: horizbuttons-next-m 

* `See details on forecast verification tests <./testing_forecast_verification.html>`_

|

|

|

|

|

WRF Capabilities & Features
===========================

To verify that new and/or modified code does not break any exisiting WRF capabilities, features, or infrastructure, tests are performed, using short simulations with numerous options. Additionally, a restart simulation is conducted with each option, and output from the restart is compared to that of a standard simulation at corresponding times. It is expected that restarts provide bit-for-bit results as if the same simulation was run from beginning to end, without restarting at any point during the simulation. 

.. rst-class:: horizbuttons-next-m 

* `See details on testing WRF capabilities and features <./testing_capabilities_features.html>`_ 

|

|

|

|

|

Software Verification
=====================

It is assumed that WRF code that runs on a single processor is correct, and that the WRF parallel infrastructure is able to repeat those values using differing numbers of processors. These software-oriented tests are conducted using as many physics options as is possible, and are therefore primarily automated since the solution "correctness" is determined by reproducing bit-for-bit results with different core counts. This testing occurs for each proposed source code modification, prior to its acceptance into the model. 

.. rst-class:: horizbuttons-next-m 

* `See details on software verification tests <./testing_software_verification.html>`_

|

|

|

|

|

Testing Architecture
====================

All above tests are primarily conducted on NSF NCAR's Linux-based HPC, which supports GNU and Intel compilers. The below table shows the compilers and versions used for each WRF code release version. 

|

.. csv-table::
   :escape: \
   :header: WRF Version, Intel Version, gfortran Version, Compiler for Automated Testing

   v4.5, 19.1.1, 10.1.0, gfortran 9.3.1
   v4.4, 19.1.1, 10.1.0, gfortran 9.3.1
   v4.3, 19.0.5, 9.1.0, gfortran 9.3.1
   v4.2, 18.0.5, 8.3.0, gfortran 9.1.0
   v4.1, 17.0.1, 8.1.0, ""
   v4.0, 17.0.1, 7.2.0, ""
   v3.9, 15.0.1 |br| 16.0.2, 4.9.2
   v3.8, 15.0.1 |br| 16.0.2, 4.9.2
   v3.7, 15.0.1, 4.9.2
   v3.6, 12.1.5 |br| 14.0.2, 4.7.2 |br| 4.8.2
   v3.5, 12.1.5, 4.7.2
   v3.4, 10.1 |br| 11.1 |br| ifort 9 does not work, 4.3.2
   v3.3, 10.1 |br| 11.1 |br| ifort 9 does not work, 4.3.2
   v3.2, 10.1 |br| ifort 9 does not work, 4.1.2 |br| 4.2.4 |br| 4.3.2
   v3.1, 10.1, 4.1.2 |br| 4.2.4 |br| 4.3.2

|

|

|

|

|

.. toctree::
   :maxdepth: 3
   :hidden:

   Forecast Verification <testing_forecast_verification.rst>
   Capabilities & Features <testing_capabilities_features.rst>  
   Software Verification <testing_software_verification.rst>