&&

Beginning with Version 2.8 of MM5 (March, 1998) users may download additional components that support execution of the model on distributed memory (DM) parallel machines. The DM option to MM5 has been implemented so that users who do not wish to run on these types of machines may simply download and build the model as before without downloading the additional components for the DM option. Further, all additional components for the DM option reside in a new directory, MPP, within the top-level MM5 source directory and this MPP directory need exist only if the DM extension is to be employed. Therefore, the first step to obtaining and building the model for a distributed-memory parallel machine is to obtain and unarchive the main model distribution file (a compressed UNIX tar archive file) into a directory. Then, a secondary DM-specific distribution file (also a compressed UNIX tar archive file) is unarchived, creating the MPP directory. Subsequent to downloading, uncompressing, and unarchiving the main model and the DM-specific components, the model may be compiled and run. The first section of this page provides details and step by step instructions for downloading and unarchiving the main model and DM-specific components. The next section describes configuration and compilation of the model. This is followed by a section with information on running the model on a distributed memory parallel machine.

All the standard job decks and source code tar files needed to run the MM5 model and its pre- and post- processors are available via anonymous ftp to ftp://ftp.ucar.edu/mesouser/MM5V2/MM5. The model and the additional code for the DM-parallel option are distributed in two archive files:

If you do not intend to use the DM-parallel option to run on a distributed memory parallel machine, you do not need the second file, nor do you need to read further in this documentation. Please refer to the standard MM5 documentation for instructions on running MM5 without the DM-parallel option.

If you intend to run the model on a distributed-memory parallel computer using the DM-option, download both these files onto your local machine using the hyperlink listed above, or download them from ftp.ucar.edu using the anonymous ftp. If you are using ftp directly, be sure to type "bin" at the ftp prompt before downloading, to make certain that the files are transferred in binary format. Otherwise, you may be unable to uncompress and unarchive them.

Once you have downloaded the compressed files onto your local machine, they should be uncompressed and unarchived. The following sequence of UNIX shell will create the MM5 source directory directly below the one in which the mm5v2.tar.Z and MPP.TAR.gz archive files are stored. The commands will then fill the directory with the necessary files and directories to compile the model with the DM-parallel option. Upon completion of the commands, the current working directory will be the just-created MM5 top-level source directory. The percent characters are not to be typed; they represent the command line prompt, which may be different on your system.

The last command, ls, should output the following:

The -F option to the /bin/ls command causes directory names to end with a trailing slash (so you can see which are directories arand which are normal files. An asterisk is appended to some of the regular file names; this is not a concern.

At this point, the MM5 code and the DM-option components have been installed in your MM5 directory. The next section describes how to configure and compile the model.

Configuring, compiling, and running the model with the distributed-memory parallel option involves the following steps:

1. Modify the configure.user file to reflect the machine-specific and scenario-specific settings for your run.
2. Compile the model to use the distributed memory parallel option.
3. Create the job deck for your run (optional).
4. Submit the job (platform specific). 

1. Configuring the model

The configure.user file in the top level of the MM5 directory comprises seven sections of machine- and scenario-specific settings, some of which may need to be modified. The sections are as follows:

1. System variables: not affected
2. User variables: RUNTIME_SYSTEM
3. Fortran Options: not affected (these are ignored when compiling with "make mpp")
4. General commands: not affected
5. Model configuration: NHYDRO, FDDAGD, FDDAOBS, MAXNES, MIX, MJX, MKX
6. Physics options: IMPHYS, MPHYSTBL, ICUPA, IBLTYP, FRAD, ISOIL, ISHALLO
7. MPP options: this section is analogous to Section 3 of the configure.user file. The set of compiler, linker, and other options for the desired platform should be uncommented (remove the leading # character) and edited as necessary.

Please refer to the sample configure.user file for details.

2. Making the Executable

Issue the command "make mpp". Assuming a successful compile, the resulting executable file will be named mm5.mpp and it will appear in the directory named Run in the main MM5 directory. Note that the first time you execute this command after downloading and unarchiving the file in the MM5 directory, some additional installation will occur; specifically, software in the MPP directory will be compiled and some additional directories and links between files will be set up. There is additional discussion of compilation issues for the DM-parallel version of the code at the end of this page.

3. Making the mm5.deck (optional)

Issue the command "make mm5.deck".  This will produce a file named mm5.deck in your MM5 directory. The mm5.deck is a script that sets up and runs the model for a given scenario. To use the script for the SESAME case, make the following changes to the mm5.deck:

To use the script for the largedomain data set make the following changes to the mm5.deck:

You may use the mm5.deck file to create the mmlif namelist file in the Run directory and to set up the run. The mm5.deck file may also be used to execute the job. However, because file systems, run commands, and queuing mechanisms vary from installation to installation, it may be easier to execute the model directly from the command line after consulting with the documentation and/or administration personnel for your system. The discussion which follows assumes access to some form of the mpirun command on your system.

4. Submit the job

In the Run directory, or in some other run directory, the following files or symbolic links should exist:

mm5.mpp	Executable file
fort.8	Transmissivities file: not needed for MPP code
fort.9	Copy of or symbolic link to lateral boundary conditions file
fort.10	Copy of or symbolic link to mmlif namelist file
fort.11	Initial conditions file for coarse domain
fort.12, fort.13, …	Initial conditions (IOVER=1) or Terrain file (IOVER=2) for nests
fort.31, fort.32, …	Used only for FDDA
fort.71, fort.72, …	Used only for FDDA
restrts	Directory for restart files

These may have been set up using the mm5.deck script, or they may be set up manually. Run the parallel model using the command for your system. Many parallel machines will use some version of the mpirun command. Here are some examples:

Each of these commands runs the model on 16 processors of a particular machine. Check the manual pages for the mpirun command on your system. You may also need to interact with a queuing system or batch scheduler. When the job runs, it will create files in the run directory:

fort.41	Output files from coarse domains
fort.42, fort.43, …	Output files from nests
rsl.out.0000	Standard output from processor zero
rsl.out.0001, rsl.out.0002, …	Standard output from other processors
rsl.error.0000	Standard error from processor zero
rsl.error.0001	Standard error from other processors
restrts/…	Restart files in restrt directory

As with the non-distributed memory versions of the model, the distributed-memory version is compiled using the UNIX make utility and controlled by settings within the configure.user file in the main MM5 directory. This section provides some additional detail that may be helpful in working with the DM-parallel model.

There are several DM-specific options to the "make" command. The command "make mpp" is used to build the model and this command is analogous to just typing "make" to build the non-DM version. The command "make mpclean" is used to put the code back to a non-configured state by deleting object (.o) files from the source tree; it is analogous to the "make clean" command for the non-DM version. The command "make uninstall" is unique to the DM version of the model, and it is used to return the code to a pristine, more or less as-downloaded state.

Command: make mpp

The "make mpp" command builds (compiles) the model based on the settings in the configure.user file and places the resulting executable, mm5.mpp, in the directory named Run in the top-level MM5 directory. The first time after unarchiving the mm5v2.tar and MPP.TAR files into the MM5 directory, the "make mpp" command installs software needed by the DM version in the directory named MPP in the MM5 directory prior to building the model. It also sets up the MPP/build directory and creates symbolic links to MM5 source files in the other directories of the MM5 source tree. The DM code is compiled only in the MPP/build directory, not in the various subdirectories of the MM5 source tree, where the non-DM version is compiled. This, and the fact that the executable file has a different name from the non-DM version, allows both the DM and non-DM versions of the model to be compiled and coexist within the same MM5 directory. Subsequent invocations of "make mpp" will only recompile the model and not reinstall the MPP software.

Prior to compilation, the MPP/build directory will contain a Makefile.RSL and a large number of files, actually symbolic links, with names that end in .F and a few that end in .c .

Command: make mpclean

The "make mpclean" command is used to restore the model to a clean, un-compiled state. The "make mpclean" command causes all of the files ending with .o and any other intermediate files in the MPP/build directory to be removed. This should be done whenever the model is reconfigured in the configure.user file, prior to rebuilding the code with the "make mpp" command. Usually, when making code changes to MM5 source files, it is not necessary to "make mpclean" between each compile. However, any time to code describing a COMMON block is altered, the model should be completely rebuilt using "make mpclean" first.

Command: make uninstall

The "make uninstall" command undoes what was done to install the DM option when the "make mpp" command was executed the first time. It uninstalls software needed by the DM option and it removes the MPP/build directory, including all symbolic links. The effect is to restore the code to an "as downloaded" state with respect to the DM option (it won't undo changes you've made to the configure.user file or to model source files). Because the installation of the DM option sets up some non-relative directory path information within the MPP directory. The "make uninstall" command should be used whenever the code is moved to a different directory. Otherwise, there are very few instances for which the "make uninstall" command is necessary; "make mpclean" will usually suffice.

Copyright © UCAR 1997 - Disclaimer - mmminfo@ncar.ucar.edu