WRF Software & Computation¶

WRF Build Mechanism¶

The WRF build mechanism provides a uniform apparatus for configuring and compiling the WRF model, WRFDA, WRF-Chem, and WRF-Hydro systems, as well as the WRF Preprocessing System (WPS) over a range of platforms, with a variety of options. This section describes the components and functioning of the build mechanism. For instructions on building the WRF modeling system code, see the Compiling chapter of this Users’ Guide.

Build Components¶

The primary components of the build mechanism are included in the WRF source code, and consist of the following:

Scripts¶

The top-level WRF directory contains three user-executable scripts: configure (which relies on the Perl script in arch/Config_new.pl), compile, and clean

Programs¶

A significant number of WRF lines of code are automatically generated at compile time by the tools/registry file.

Makefiles¶

The primary Makefile (input to the UNIX “make” utility) is in the top-level WRF directory. There are also makefiles in most subdirectories, as well. “make” is called recursively over the directory structure when the compile script is run by the user.

Configuration Files¶

The user-invoked configure script generates a file called configure.wrf. This file contains a compiler, linker, and other build settings, as well as rules and macro definitions used by the “make” utility. It is included by the Makefiles in most of the WRF source distribution (Makefiles in tools and external directories do not include configure.wrf), and is deleted when the code system is cleaned by the user-issued clean script. Thus, configure.wrf can be modified for temporary changes, such as optimization levels and compiling with debugging. Permanent changes should be made in arch/configure_new.defaults. The configure.wrf file is composed by the following three files:

arch/configure_new.defaults : contains lists of compiler options for all supported platforms and configurations. Changes made to this file are permanent.
arch/preamble_new : constitutes the generic parts (non-architecture-specific) of the configure.wrf file generated by the configure script. See the file for details.*
arch/postamble_new : similar to arch/preamble_new. See the file for details.

Registry¶

The Registry directory contains files that control many compile-time aspects of the WRF code. The files are named Registry.X, where X corresponds to the intended application specified during configuration (for e.g., if a WRF-Chem compile is chosen, the Registry.EM_CHEM file will be used, where X corresponds to the intended application, specified during configuration). The configure script copies the appropriate Registry.X file to a generic file named Registry, which is the file that tools/registry uses as input. Any changes to the the Registry file will be lost; permanent changes should be made to Registry.X. For a standard WRF-ARW build, the file is typically Registry.EM_COMMON. The command “include” is used by the registry program, allowing the file Registry.EM_COMMON to be included during a build. This file holds all shared entries for all applications, which helps to reduce the amount of replicated registry information. See the full Registry section for additional details.

Environment Variables¶

The following aspects of the configuration and build are controlled by environment variables:

The non-standard locations of netCDF libraries or the Perl command
Which dynamic core to compile
Machine-specific features
Optional build libraries (such as Grib Edition 2, HDF, and parallel netCDF)

In addition to WRF-related environment settings, there may also be settings specific to particular compilers or libraries. For e.g., local installations may require setting a variable like MPICH_F90 to ensure the correct instance of the Fortran 90 compiler is used by the mpif90 command.

How the WRF Build Works¶

The two steps in the WRF build process are configuration and compilation.

Configuration

The configure script configures the model to determine information about the user’s computing environment prior to compilation.

Configuration only takes a few seconds to run through the following processes:

Attempts to locate mandatory and optional libraries (e.g., netCDF or HDF) and tools, such as Perl; it checks for these in standard paths, or uses settings from the user’s shell environment
Calls the UNIX “uname” command to determine the user’s platform
Calls the Perl script arch/Config_new.pl, which traverses the list of known machine configurations and displays a list of available options to the user
The selected set of options is then used to create the configure.wrf file in the top-level directory.

Note

The configure.wrf file may be edited, but changes could be temporary if the file is later deleted by the user-invoked clean script, or overwritten by the next invocation of the configure script.

Compilation

The compile script is used to compile the WRF code after it has been configured.

The compile cshell script performs a number of checks, constructs an argument list, copies the appropriate Registry.X file to Registry/Registry, and then invokes the UNIX “make” command in the top-level directory. The Makefile in the top-level directory directs the rest of the build, accomplished as a set of recursive invocations of “make” in the WRF subdirectories. Most makefiles include configure.wrf from the top-level directory. The order of a complete build is as follows:

make in the external directory
- make in external/io_{grib1,grib_share,int,netcdf} for Grib Edition 1, binary, and netCDF implementations of I/O API
- make in external/RSL_LITE to build the communications layer (DM_PARALLEL only)
- make in external/esmf_time_f90 to build the ESMF time manager library
- make in external/fftpack to build the FFT library for the global filters
- make in other external directories, as specified by the “external:” target in the configure.wrf file
make in the tools directory to build the program that reads the Registry/Registry file and auto-generates files in the inc directory
make in the frame directory to build the WRF framework specific modules
make in the share directory to build the mediation layer routines, including WRF I/O modules that call the I/O API
make in the phys directory to build the WRF model layer routines for physics
make in the dyn_em directory for mediation-layer and model-layer subroutines
make in the main directory to build the main programs for WRF, with a symbolic link to create executable files (location depending on the build case that was selected as the argument to the compile script; for e.g., for a real-data case, the executables are linked to test/em_real)
- Source files (.F files, and in some of the external directories, .F90 files) are preprocessed to produce .f90 files, which are input to the compiler. As part of the preprocessing, Registry-generated files from the inc directory may be included.
- .f90 files are compiled, resulting in the creation of .o (object) files that are added to the library main/libwrflib.a. Most external directories generate their own library file.
- The linking step produces the wrf.exe executable and other executables, depending on the case argument to the “compile” command: real.exe (a preprocessor for real-data cases) or ideal.exe (a preprocessor for idealized cases), and the ndown.exe program, for one-way nesting of real-data cases.

The .o files and .f90 files created during the compile are retained until the next invocation of the clean script. The .f90 files provide the true reference for tracking down run time errors that refer to line numbers or for sessions using interactive debugging tools such as “dbx” or “gdb.”

Registry¶

Much of the WRF code is automatically generated from a user-edited text-based table called the “Registry.” The Registry provides a high-level, single-point of control over the fundamental structure of the model data. It contains lists describing state data fields and their attributes:

dimensionality
binding to particular solvers
association with WRF I/O streams
communication operations
run time configuration options (namelist elements and their bindings to model control structures).

Adding or modifying a state variable to WRF involves modifying a single line of a single file, which is then automatically propagated to scores of locations in the source code the next time the code is compiled.

The WRF Registry has two components: the Registry file (which the user may edit), and the Registry program. The Registry file is located in the Registry directory and contains entries that direct auto-generation of WRF code by the Registry program. There are multiple Registry files in this directory, with the filename syntax Registry.X, where X specifies the type of build selected during configuration.

File Name	Type of WRF Build
Registry.EM_COMMON	Standard/Basic WRF
Registry.EM_COMMON.var	Minimal subset of Registry.EM_COMMON required for a WRFDA compile
Registry.EM_CHEM”	WRF Chemistry (WRF-Chem)
Registry.wrfvar”	WRF Data Assimilation (WRFDA)
Registry.wrfchemvar”	WRF-Chem + WRFDA
Registry.tladj”	WRFPLUS

Note

The Registry.EM_COMMON file contains variables that are common to all applications. The above Registry.X files make use of this file, reducing replicated registry information. Additionally, files in the Registry directory named registry.x (where x represents a specific application) are also included in this manner.

The WRF build mechanism copies the appropriate Registry.X file to Registry/Registry and this file is used to direct the Registry program, which is included in the WRF source code in the tools directory. It is built automatically when WRF is compiled from clean code. The executable file is tools/registry, which reads the contents of the Registry/Registry file and generates files in the inc directory. These “include” files are inserted (with cpp “#include” commands) into WRF Fortran source files prior to compilation. The Registry program, itself, is written in C. The source files and makefile are in the tools directory. The syntax and semantics for entries in the Registry are described in detail in WRF Tiger Team Documentation: The Registry.

In addition to the WRF model itself, Registry/Registry is used to build the accompanying preprocessors, such as real.exe (for real data) or ideal.exe (for ideal simulations), and the ndown.exe program (used for one-way, off-line nesting).

"An image that shows when the user compiles WRF, the Registry Program reads Registry/Registry, producing auto-generated sections of code that are stored in files in the inc directory. These are included into WRF using the CPP preprocessor and the Fortran compiler."

When WRF is compiled, the Registry program reads Registry/Registry, producing auto-generated sections of code that are stored in files in the inc directory. These are included in WRF, using the CPP preprocessor and the Fortran compiler.

The Registry describes every variable that is an input or an output field, or is required for parallel communication, specifically associated with a physics package, or needs to provide a tendency to multiple physics or dynamics routines. The following is defined for each variables:

Index ordering
Horizontal and vertical staggering
Feedback and nesting interpolation requirements
The associated IO

Adding a variable into the model typically only requires the addition of a single line to the Registry.X file, regardless of dimensionality. The same is true when defining a new run-time option (i.e., a new namelist entry). As with the model state arrays and variables, the entire model configuration is described in the Registry. Since the Registry modifies code for compile-time options, any change to the Registry (i.e., any file in the Registry directory) requires the code to be returned to the original unbuilt status with the clean -a command, and then reconfigured and recompiled.

The Registry file also provides input to generate all communications for distributed memory processing (halo interchanges between patches, support for periodic lateral boundaries, and array transposes for FFTs to be run in the X, Y, or Z directions). The Registry associates various fields with particular physics packages so the memory footprint reflects the actual selection of the options, not a maximal value.

Registry Syntax¶

Each entry in the Registry is specific to a single variable, regardless of whether it applies to a model dimension, a new field, new namelist value, or communication. A single entry may be spread across several lines with a backslach () at the end of a line to denote the entry is continuing. When adding a new line to the Registry, keep in mind the following:

It is recommended to copy an entry that is similar to the new entry, and then modify the new entry.
The Registry is not sensitive to spatial formatting.
White space separates identifiers in each entry.
After any change to a registry file, the code must be cleaned, reconfigured, and recompiled.
If a column has no entry, do not leave it blank. Use the dash character ( - ).

Registry Entries¶

The following entry types are used in the WRF Registry (not case-dependent).

Dimspec	Describes dimensions used to define arrays in the model
State	Describes state variables and arrays in the domain structure
I1	Describes local variables and arrays in ‘solve’
Typedef	Describes derived types that are subtypes of the domain structure
Rconfig	Describes a configuration (e.g. namelist) variable or array
Package	Describes attributes of a package (e.g. physics)
Halo	Describes halo update interprocessor communications
Period	Describes communications for periodic boundary updates
Xpose	Describes communications for parallel matrix transposes
include	Similar to a CPP #include file

These keywords appear as the first word in each line of the Registry file to define which type of information is being provided. Following are details and examples of the more-likely Registry types.

Dimspec¶

The first set of entries in the Registry are specifications of the dimensions for the fields to be defined. To keep the WRF system consistent between build types, a unified registry.dimspec file is used (located in the Registry directory), and is included in each Registry.* file, using the keyword “include.” In the example below, three dimensions are defined: i, j, and k. Issuing the command ncdump -h on a WRF file, provides three directional dimensions as west_east, south_north, and bottom_top. That information is contained in this example.

#<Table>  <Dim>  <Order> <How defined>  <Coord-axis>  <Dimname in Data sets>
dimspec    i      1     standard_domain     x          west_east
dimspec    j      3     standard_domain     y          south_north
dimspec    k      2     standard_domain     z          bottom_top

Because WRF uses horizontal and vertical staggering, dimension names are extended with a _stag suffix, representing staggered sizes. The list of names in the <Dim> column may either be a single, unique character, or a string with no embedded spaces (e.g., my_dim). When this dimension is used later to dimension-ize a state or i1 variable, it must be surrounded by curly braces (e.g., {my_dim}). This <Dim> variable is not case specific.

State and I1¶

A WRF state variable is a field that is eligible for input/output (IO) and communications, and exists for the duration of the model forecast. I1 variables (intermediate level one) are typically considered tendency terms, computed during a single model time-step, and then discarded prior to the next time-step. The space allocation and de-allocation for these I1 variables is automatic (on the stack for the model solver).

Below are the columns used for state and I1 variables (printed in order from left to right in the Registry file).

Column Heading	Description	Notes
Table	either ‘state’ or ‘I1’	string
Type	type of variable or array	real, double, integer, logical, character, or derived
Sym	symbolic name inside WRF	string; not case-sensitive
Dims	dimensionality of the array	string; use hyphen (-) when dimensionless
Use	denotes association with the solver (dyn_em), or 4D scalar array	string; if not specific, declared ‘misc’
NumTLev	number of time levels	integer for arrays; use hyphen (-) for variables
Stagger	indicates staggered dimensions	string; X,Y,Z, or hyphen (-) if not staggered
IO	whether and how it is subject to I/O and nesting	string; use hyphen (-) when no specification
Dname	metadata name	string; in quotes
Descrip	metadata description	string; in quotes
Units	metadata units	string; in quotes

The following is a state variable that is a Fortran type “real.” The name of the field inside the WRF model is u_gc. It is a three-dimensional array (igj), has a single time level, and is staggered in the X and Z directions. This field is input only to the real program (i1). On output, the netCDF name is UU, with the accompanying description and units provided(the example is broken across two lines for this guide, but represents a single line in the Registry file).

#<Table> <Type> <Sym> <Dims>  <Use>  <NumTLev>
state    real   u_gc   igj     dyn_em     1

<Stagger> <IO> <DNAME>  <DESCRIP>             <UNITS>
XZ       i1   "UU"    "x-wind component"    "m s-1"

Note that instead of leaving a blank column, a hypen/dash (-) is used for some fields listed in the table above. The variable description and units columns are used for post-processing purposes only; this information is not directly utilized by the model, but is useful for users to understand them better.

The above example can be used as a template for adding a new variable. Use information provided in the table above, as well as some additional details below.

New variables added to the Registry file must use unique variable names, and should not use imbedded spaces. It is not required that <Sym> and <DNAME> use the same character string, but it is highly recommended.
If the new variable is not specific to any build type (e.g., basic WRF, WRF-Chem), the <Use> column entry should be ‘misc’ (for miscellaneous). The misc entry is typical for fields used in physics packages.
Only dynamics variables have more than a single time level (<NumTLev>). Note: this introductory guide is not suitable for describing the impact of multiple time periods on the registry program.
For <Stagger>, select any subset from {X, Y, Z} or {-}, where the dash character signifies no staggering. For example, the x-direction wind component (u) is staggered in the X direction, and the y-direction wind component (v) is staggered in the Y direction.
<DESCRIP> and <UNITS> are optional, but are highly encouraged for users to understand them better. Since the <DESCRIP> value is used in the automatic code generation, restrict the variable description to 40 characters or less.
The <IO> column handles file input and output, output streams, and the nesting specification for the field.
- Input & Output : There are three options for I/O: i (input), r (restart), and h (history). For e.g., if the field should be in the input file to the model, the restart file from the model, and the history file from the model, the entry would be ‘irh’ (in any order).
- Streams : To allow more flexibility, the input and history fields are associated with streams. A digit can be specified after the i or the h token, stating that this variable is associated with a specified stream (1 through 9) instead of the default (0, which means ‘history’). A single variable may be associated with multiple streams. Once any digit is used with the i or h tokens, the default 0 stream must be explicitly stated. For example, i and i0 are the same. However, h1 outputs the field to the first auxiliary stream, but does not output the field to the default history stream. h01 outputs the field to both the default history stream and the first auxiliary stream. For streams larger than a single digit, such as stream number thirteen, the multi-digit numerical value is enclosed inside braces (e.g., i{13} ). The maximum stream value is 24 for both input and history.
- Nesting Specification : The letter values parsed for nesting are: u (up, as in feedback up), d (down, as in downscale from coarse to fine grid), f (forcing, how the lateral boundaries are processed), and s (smoothing). Users must determine whether it is reasonable to smooth the field in the area of the coarse grid, where the fine-grid feeds back to the coarse grid. Variables that are defined over land and water, non-masked, are usually smoothed. Lateral boundary forcing is primarily for dynamics variables (not discussed in this guide). For non-masked fields (e.g., wind, temperature, pressure), downward interpolation (controlled by d) and feedback (controlled by u) use default routines. Variables that are land fields (e.g., soil temperature - TSLB) or water fields (e.g., sea ice - XICE) have special interpolators, as shown in the examples below (again, interleaved for readability):
```
#<Table> <Type> <Sym> <Dims>  <Use>  <NumTLev> <Stagger>
state    real   TSLB   ilj    misc       1          Z
state    real   XICE   ij     misc       1          -

<IO>
i02rhd=(interp_mask_land_field:lu_index)u=(copy_fcnm)
i0124rhd=(interp_mask_water_field:lu_index)u=(copy_fcnm)

<DNAME>  <DESCRIP>           <UNITS>
"TSLB"   "SOIL TEMPERATURE"   "K"
"SEAICE" "SEA ICE FLAG"       ""
```
  Note
  
  The d and u entries are followed by an “=” then a parenthesis-enclosed subroutine, and a colon-separated list of additional variables to pass to the routine. It is recommended to follow the pattern: du for non-masked variables, and the above syntax for the existing interpolators for masked variables.

Rconfig¶

Run-time configuration options, rconfig, (namelist.input options) are also defined in the Registry file, where each namelist option is described in its own line. Default values for each namelist variable are assigned in the Registry. The standard form for the entry for two namelist variables is given in the example below.

<Table>  <Type>    <Sym>       <How set>              <Nentries>   <Default>
rconfig  integer   run_days    namelist,time_control        1           0
rconfig  integer   start_year  namelist,time_control   max_domains    1993

Below are the columns used for rconfig variables (printed in order from left to right in the Registry file).

Column Heading	Description	Notes
Table	entry type	‘rconfig’
Type	type of variable or array	real, integer, logical
Sym	symbolic name inside WRF	string; not case-sensitive
How set	namelist,<fortran record>	string
Nentries	dimensionality (# of entries)	either ‘1’ (single entry valid for all domains) or ‘max_domains’ (value specified for each domain)
Default	default setting for the variable	if entry is missing from namelist, this value is used

Note

Each namelist variable is a member of one of the specific namelist records. The previous example shows that run_days and start_year are both members of the time_control record.

The registry program constructs two subroutines for each namelist variable: one to retrieve the value of the namelist variable, and the other to set the value. For an integer variable named my_nml_var, the following code snippet provides an example of the easy access to the namelist variables:

INTEGER :: my_nml_var, dom_id
CALL nl_get_my_nml_var ( dom_id , my_nml_var )

The subroutine takes two arguments. The first is the input integer domain identifier, dom_id, (for example, 1 for the most coarse grid, 2 for the second domain), and the second argument is the returned value of the namelist variable. The associated subroutine to set the namelist variable, with the same argument list, is nl_set_my_nml_var. For namelist variables that are scalars, the grid identifier should be set to 1.

The rconfig line may also be used to define variables that are convenient to pass around in the model, usually part of a derived configuration (such as the number of microphysics species associated with a physics package). In this case, the <How set> column entry is derived. This variable does not appear in the namelist, but is accessible with the same generated nl_set and nl_get subroutines.

Halo¶

Distributed memory, inter-processor communications are fully described in the Registry file. An entry in the Registry constructs a code segment which is included (with cpp) in the source code. Following is an example of a halo communication:

#<Table>  <CommName>   <Core>   <Stencil:varlist>
halo      HALO_EM_D2_3 dyn_em    24:u_2,v_2,w_2,t_2,ph_2;24:moist,chem,scalar;4:mu_2,al

Below are the columns used for halos (printed in order from left to right in the Registry file).

Column Heading	Description	Notes
Table	entry type	‘halo’
CommName	communication	case-sensitive, starts with ‘HALO_EM’
Core	dynamical core	all are set to ‘dyn_em;’ no longer relevant since dyn_nmm is removed from WRF
Stencil:varlist	stencil size and variables communicated with that stencil size	portion prior to colon (:) is size; comma-separated list after defines variables;

Note

Different stencil sizes are available, and are separated in the same <Stencil:varlist> column by a semi-colon (;). Stencil sizes 8, 24, and 48 all refer to a square with an odd number of grid cells on a side, with the center grid cell removed (8 = 3x3-1, 24 = 5x5-1, 48 = 7x7-1). The special small stencil 4 is just a simple north, south, east, west communication pattern.

The WRF model provides a communication immediately after a variable has been updated. The communications are restricted to the mediation layer (an intermediate layer of the software placed between the framework and model levels). The model level is where developers spend most of their time. The majority of users will insert communications into the dyn_em/solve_em.F subroutine. The HALO_EM_D2_3 communication shown in the above example is activated by inserting a small section of code that includes an automatically-generated code segment into the solve routine, via standard cpp directives:

#ifdef DM_PARALLEL
#    include "HALO_EM_D2_3.inc"
#endif

Parallel communications are only required when the code is built for distributed-memory parallel processing, which accounts for the surrounding #ifdef.

Period¶

Period communications are required when periodic lateral boundary conditions are selected. The Registry syntax is very similar for period and halo communications, but here stencil size refers to how many grid cells to communicate, in a direction that is normal to the periodic boundary:

#<Table>     <CommName>        <Core>   <Stencil:varlist>
period   PERIOD_EM_COUPLE_A    dyn_em    2:mub,mu_1,mu_2

Xpose¶

The xpose (a data transpose) entry is used when decomposed data is to be re-decomposed. This is required when doing FFTs in the x-direction for polar filtering, for example. No stencil size is necessary:

#<Table>     <CommName>        <Core>      <Varlist>
xpose    XPOSE_POLAR_FILTER_T  dyn_em   t_2,t_xxx,dum_yyy

It is likely additions will be added to the the parallel communications portion of the Registry file (halo and period), but unlikely that additions will be added to xpose fields.

Package¶

The package option in the Registry file associates fields with particular physics packages. It is mandatory that all 4-D arrays be assigned. Any 4-D array not associated with the selected physics option at run-time is neither allocated, used for IO, nor communicated. All other 2-D and 3-D arrays are eligible for use with a package assignment, but that is not required. The package option’s purpose is to allow users to reduce the memory used by the model, since only necessary fields are processed. An example for a microphysics scheme is given below:

#<Table>  <PackageName>  <NMLAssociated>    <Variables>
package   kesslerscheme   mp_physics==1   - moist:qv,qc,qr

Column Heading	Description	Notes
Table	entry type	‘package’
PackageName	name of associated options for code IF/CASE statements	string
NMLAssociated	package is associated with these namelist options	namelist setting, using ‘==’ syntax
Variables	all variables included in the package	syntax is: dash (-), space, 4-D array name, colon (:), then comma-separated list of 3-D arrays constituting 4-D amalgamation

In the example above, the 4-D array is moist, and the selected 3-D arrays are qv, qc, and qr. If more than one 4-D array is required, a semi-colon (;) separates those sections from each other in the <Variables> column.

The package entry is also used to associate generic state variables, as shown in the example following. If the namelist variable use_wps_input is set to 1, then the variables u_gc and v_gc are available to be processed:

#<Table>  <PackageName> <NMLAssociated>     <Variables>
package     realonly    use_wps_input==1  - state:u_gc,v_gc

WRF Architecture¶

The WRF model architecture is divided into the following different layers:

Driver Layer

The WRF Driver is responsible for handling domains and their time loop. It allocates, stores, decomposes, and represents abstractly as single data objects. To handle time looping, there are algorithms for integration over nest hierarchy (see Timekeeping).

Mediation Layer

The mediation layer contains the Solve routine, which uses a sequence of calls to take a domain object and advance it one time step. It also controls nest forcing, interpolation, and feedback routines. It dereferences fields in calls to the physics dirvers and dynamics code. Calls to message-passing are also contained in the Solve routine.

Model Layer

The model layer contains the actual WRF model routines for physics and dynamics, which are written to perform computation over arbitrarily-sized/shaped, 3D, and rectangular subdomains.

The Call structure in the code is as follows (this example shows a call for a specific cumulus parameterization scheme):

main/wrf.F -> frame/module_integrate.F -> dyn_em/solve_em.F -> dyn_em/module_first_rk_step_part1.F -> phys/module_cumulus_driver.F -> phys/module_cu_g3.F

The main WRF program calls the integration, which calls the solve interface, which calls the mediation solver, which calls the specific physics driver, which calls the specific physics routine.

I/O Applications Program Interface (I/O API)¶

The software that implements WRF I/O, like the software that implements the model in general, is organized hierarchically, as a “software stack.” From top (closest to the model code itself) to bottom (closest to the external package implementing the I/O), the I/O stack looks like this:

Domain I/O (operations on an entire domain)
Field I/O (operations on individual fields)
Package-neutral I/O API
Package-dependent I/O API (external package)

The lower-levels of the stack, associated with the interface between the model and the external packages, are described in the I/O and Model Coupling API specification document.

Domain I/O¶

This layer implements the routines that perform I/O on a domain object to a particular data stream. These are defined in share/module_io_domain.F. Routines include the following:

input_initial
output_initial
input_restart
output_restart
input_history
output_history
input_boundary
output_boundary

There are also a series of routines that are used to input and output the five auxiliary input and five auxiliary history data streams if these are opened. In the the below set, n is may be the number of the input or history auxiliary stream, 1 through 5:

input_aux_model_inputn
output_aux_model_inputn
input_aux_histn
output_aux_histn

The high-level routines to open and close datasets for writing and reading, open_w_dataset and open_r_dataset, are also provided at this layer. They are considered high-level because in addition to opening and closing the datasets, they also contain logic to initialize datasets for formats that require it (e.g. netCDF).

Field I/O¶

This layer contains the routines that take a domain object as an argument and then generate the series of calls to input or output all the individual fields and metadata elements that constitute a dataset. The Registry auto-generates much of this code, which appears in these routines through include statements to registry-generated files in the inc directory:

wrf_initialin.inc
wrf_initialout.inc
wrf_restartin.inc
wrf_restartout.inc
wrf_histin.inc
wrf_histout.inc
wrf_bdyin.inc
wrf_bdyout.inc

The routines of this layer are defined in share/module_io_wrf.F.

Package-independent I/O API¶

This layer contains definitions of the subroutines specified in the WRF I/O API. Some of these are:

wrf_open_for_write_begin
wrf_open_for_write_commit
wrf_open_for_read
wrf_write_field
wrf_read_field
wrf_get_dom_ti_real

There are quite a few of these routines, most of which are for getting or putting metadata in different forms. These routines, defined in frame/module_io.F, fre considered part of the WRF framework and have the wrf_ prefix in their names to distinguish them from package-dependent implementations of each routine (next layer down). The principal function of this layer is to select the package-dependent routine from the appropriate linked I/O package to call, based on the definition of the io_form_initial, io_form_restart, io_form_history, and io_form_boundary namelist variables. The association of io_form values with particular packages is specified in the Registry using “rconfig” and package entries.

Package-specific I/O API¶

This layer contains definitions of the subroutines specified in the WRF I/O API for a particular package. There may be a number of packages linked into the program. External packages are defined in the external/packagename subdirectory of the WRF distribution. The names are the same as above, except their names have a package specific prefix. External packages have the prefix ext_pkg where pkg specifies the particular package. For example, the netCDF implementation of the I/O API uses the prefix ext_ncd. HDF uses ext_hdf. At present there are three placeholder package names, xxx, yyy, and zzz, which are included in the code but inactive (#ifdef’d out using CPP directives). There is also an internal-I/O package, which may be selected to input/output data in a fast, non-self-describing, machine specific intermediate format. The routine names in this package are prefixed with the string int.

Timekeeping¶

Starting times, stopping times, and time intervals in WRF are stored and manipulated as Earth System Modeling Framework (ESMF) time manager objects. This allows exact representation of time instants and intervals as integer numbers of years, months, hours, days, minutes, seconds, and fractions of a second (numerator and denominator are specified separately as integers). All time computations involving these objects are performed exactly by using integer arithmetic, with the result of no accumulated time step drift or rounding, even for fractions of a second.

The WRF implementation of the ESMF Time Manger is distributed with WRF in the external/esmf_time_f90 directory. This implementation is entirely Fortran90 (as opposed to the ESMF implementation in C++) and it is conformant to the version of the ESMF Time Manager API that was available in 2009.

WRF source modules and subroutines that use the ESMF routines do so by use-association of the top-level ESMF Time Manager module, esmf_mod:

USE esmf_mod

The code is linked to the library file libesmf_time.a in the external/esmf_time_f90 directory.

ESMF timekeeping is set up on a domain-by-domain basis in the routine setup_timekeeping (in share/set_timekeeping.F). Each domain keeps track of its own clocks and alarms. Since the time arithmetic is exact, clocks on separate domains will not become unsynchronized.

Computation¶

The WRF model can be run serially or as a parallel job, depending on the available resources in the computing environment and the option chosen during the WRF compile. When the model is configured with the serial option, only a single processor is used to run a simulation. This is typically only useful when using a small domain (no greater than 100x100 grid cells), usually just for instructional purposes. The model can be configured to use parallel processes with the distributed-memory (dmpar) option, shared memory (smpar) option, or a combination of both (dm+sm).

Parallel processing allows processes to be carried out simultaneously by independent processing units. Most often, simulations require parallel processing. Besides teaching applications, it is not recommended to use domains any smaller than 100x100 grid spaces, meaning most domains will be larger. Parallel computing uses domain decomposition to divide the total amount of work over parallel processes. Since less work is performed by each individual processor, the elapsed time to complete the computational task for each processor is reduced, reducing overall simulation time.

Distributed-memory Communications
Distributed-memory parallelism takes place over patches, which are sections of the model domain allocated to a single distributed memory node. These patches send and receive information from each other (message passing). WRF uses HALO regions to assist with this. The image below shows an imaginary domain that is using nine processors. Each of the nine squares represents a single patch and is computed by a single processor. The yellow (or lighter) surrounding area in each square is the halo region. Halos are able to communicate with halos in neighboring, tangential patches, sending and receiving information.

"An image that shows a domain divided into nine grid spaces (3x3). Each grid represents a patch that is processed by a single processing unit. Each patch has a thin outlined area that represents the halo region."

Shared-memory Communications
Shared-memory parallelism uses OpenMP and takes place over tiles within patches. Tiles are sections of a patch allocated to a shared-memory processor within a single node. With shared memory processing, the domain is split among the run-time available OpenMP threads, and this is not sufficient for larger jobs (use dmpar instead for those). OpenMP is always within a single shared-memory processing unit.

Quilting¶

Note

This option is unstable and may not work properly.
This option is only used for wrf.exe. It does not work for real or ndown.

This option allows reserving a few processors to manage output only, which can be useful and performance-friendly if the domain size is large, and/or the time taken to write an output time is significant when compared to the time taken to integrate the model in between output times. There are two variables for setting the options:

nio_tasks_per_group: Number of processors to use per IO group for IO quilting (1 or 2 is typically sufficient)
nio_groups: How many IO groups for IO (default is 1).

Software Documentation¶

Detailed and comprehensive documentation aimed at WRF software is available in WRF v2 Software Tools and Documentation. Though the document was written for WRFv2, the information is still relevant for current versions.

Performance¶

Benchmark information for some WRF model versions is available from the WRF Benchmarks page.