NCAR IBM Help Guide
New users, please note:
- Build and run MM5 from /ptmp rather than $HOME
There is simple not enough space in your $HOME directory to keep
the executables and output files.
- Don't use /tmp on the NCAR IBM's - always use /ptmp.
- Define PROCMIN_EW and PROCMIN_NS to suite the domain size you
are running, and the number of processors you plan on using.
If you plan on running on 16 processors, set PROCMIN_EW and PROCMIN_NS
to 4,4 rather than 1,1. Otherwise, the job will likely incur the
dreaded "memory swapping" problem, which will result
in the user's account locked until the user contacts
Consulting (consult1@ucar.edu).
Two problems with programs TERRAIN and REGRID have been
resolved for use on NCAR's IBM computers. Please use the updated
tar files.
This Web page has been set up to help with the transition from
NCAR Cray's to the NCAR IBM's. If you require more information,
or have additional information you would like to share with other
NCAR IBM users, please send mail to mesouser at mesouser@ucar.edu.
General Questions
To get a logon on SCD's IBM computers babyblue and blackforest,
please call SCD Database Services at 303 497-1225,
08:00 - 17:00 MDT Monday to Friday and request one. For
security reasons, this MUST be done by phone, and not via email
or FAX. For general information on applying for computer resources
at NCAR, check out the SCD
Web Page.
For introductory information about IBM computer blackforest,
please see http://www.scd.ucar.edu/computers/blackforest
By default, your logon shell on the IBM's is the Korn shell (ksh).
To change it to sh or csh or tcsh, please follow the instructions
in
http://www.scd.ucar.edu/docs/blackforest/env.html#spd
Message from SCD Consulting Office to MM5 users switching over
to IBM computers. "Do not hesitate to call the SCD
Consulting Office 303 497-1302, 08:00 - 17:00 MDT.
Monday to Friday, or email consult1@ucar.edu for any help
you may need. You will undoubtedly have questions.
SCD is anxious to help you run successfully on blackforest."
NO, Cray format file cannot be read directly on the IBM's
You can convert the files from Cray to IBM format on the IBM.
This process is not as straight forward as on the Cray. The method
involves writing a program to read the Cray records and convert
them with calls to the ncaru library. A program has been
developed to convert
MM5V3 data files on the IBM. This program should run
on many other Unix platforms (like SGI and SUN), since the ncaru
software is freely available. For more information on the use
of the ncaru library, see
http://www.scd.ucar.edu/docs/conversion.tools/ncaru.html
Have Cray format intermediate files, you would like to read on
the IBM? Use the cray2ibm_intermediate.f
utility.
To submit jobs on the IBM's to run as a batch job, make use of
the LoadLeveler.
A typical batch job will look as follows:
#!/bin/csh |
|
# @ job_type = parallel |
! When you run a parallel job, always set
"job_type = parallel" because the default is serial. In
a parallel job, you must also specify tasks per node and
the number of nodes. |
# @ environment =
COPY_ALL;MP_EUILIB=us |
! Specifies your initial environment variables
when your job step starts. |
# @ job_name = my_job |
! Specifies the name of the job. |
# @ output = my_job.out |
! Specifies a file to
which standard output is written. |
# @ error
= my_job.err |
! Specifies the file to which standard-error
output is written. |
# @ node
= 1,1 |
! Number of nodes required for your job.
|
# @ network.MPI = css0,shared,us |
! Use "shared" in all cases. |
# @ tasks_per_node = 4 |
! Tasks per node. |
# @ node_usage = not_shared |
! Whether to share nodes acquired for your
tasks |
# @ checkpoint = no |
! Specifies whether you want to checkpoint
your program. |
# @ wall_clock_limit = 1800 |
! Specifies the maximum wall-clock time
that LoadLeveler will allow the job to run. Will prevent
runaway jobs from consuming excessive resources. This can
help your job run sooner when blackforest queues are full. |
# @ class
= com_reg |
! Batch queue name. Currently, blackforest
has these queues
for users. |
# @ queue |
! Causes LoadLeveler to submit the preceding
macros to the queue |
|
|
set MP_SHARED_MEMORY = yes |
|
|
cd /home/username/MM5
pwd
ls -l
make mpp
cd Run
timex poe ./mm5.mpp |
|
|
|
exit |
|
Submit this job to the queue, by making use of llsubmit.
To check the status of your job, use llq.
To cancel a jon that is in the queue, use llcancel.
Yes. Read more about these systems at: IRJE
& MIGS
-
Will I be able to use the new IBM scripts on other IBM computers?
-
The basics of the scripts will work on other IBM's.
-
The LoadLeveler commands, especially the batch classes,
could be different.
-
Most of the scripts have been developed around the NCAR
MSS facilities, and those are only available on NCAR computers.
-
Where can I find the IBM scripts?
-
How do I get the data back to my local computer?
Since version 3.6 all the batch job scripts has been made more
user friendly. All "rcp" commands has therefore been removed
from the scripts to prevent the script from stopping due to
firewall problems. If you need the data on your local machine,
please ftp it over manually.
Thanks to Dick Valent from NCAR/SCD that helped
with answers to these questions
TERRAIN
- How does one run TERRAIN on the IBM?
- Copy the script terrain.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the terrain.deck
- Parameters in green
is worth looking at
- [comment] is used
for extra information
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./terrain.deck
- Do I have to use this script to run TERRAIN on NCAR
IBM computers?
- No
- you can download the source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- run code manually
- I get the followining message whne running TERRAIN:
"Cannot load program ./terrain.exe because of the following
error
System error: There is not enough memory available
now"
- Edit the TERRAIN Makefile and under the IBM options, change
the following line from:
"LDOPTIONS = "
to:
"LDOPTIONS = -bmaxdata:2000000000"
Then recompile the code, before running the script again.
REGRID
- How does one run REGRID on the IBM?
- Copy the script regrid.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the regrid.deck
- Parameters in green
is worth looking at
- [comment] is used
for extra information
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./regrid.deck
- This script only works for input data that is available on
NCAR mass storage (NCEP GDAS data in ON84 or GRIB format, NCEP/NCAR
Reanalysis data, ECMWF Reanalysis data, FNL Data, Eta AWIP DATA
and TOGA data).
- The script expects to find a TERRAIN_DOMAINn file on mass
storage for input to the "regridder" part of the run.
- All output files ("pregrid" & "regridder" output files)
will be written to mass storage.
- Do not run script as a batch job
- The new NISE data is not yet available via
the regrid.deck script.
- Can I download data without running the regrid.deck script?
- Yes, all the "get" routines are available on the
IBM (for instance, get_nnrp.csh, to download NNRP data).
- You can copy the script from ~mesouser/MM5V3/Util
- Copy script to your working directory to run.
- To run you have 3 options:
- Edit startdate ; enddate and itimint inside script,
run script by typing (eg):
get_nnrp.csh
[Data will be downloaded in
/ptmp/$USER]
- OR type (eg - for Mar 13-14, 1993, evry 12 hours):
get_nnrp.csh
1994-03-13_00 1994-03-14_00 12
[Data will be downloaded in
/ptmp/$USER]
- OR type (eg - for Mar 13-14, 1993, evry 12 hours):
get_nnrp.csh
1994-03-13_00 1994-03-14_00 12 /ptmp/$USER/myDATA
[Data will be downloaded in
/ptmp/$USER/myData]
- Since version 3.6, the scripts no longer provide the option
to automatically copy the files to your local computer. This
creates to many problems with user computer firewalls. If
you need the data on your local computer, please ftp the files
over manually.
- Do I have to use this script to run REGRID on NCAR IBM
computers?
- No
- you can download teh source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- get data using "get" script above (or use
your own data)
- run code manually
RAWINS [using
LITTLE_R is recommended]
- How does one run RAWINS on the IBM?
- Copy the script rawins.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the rawins.deck
- Parameters in green
is worth looking at
- [comment] is used
for extra information
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./rawins.deck
- The script will download the necessary observational files
from the mass store and run RAWINS
- The script expects to find the input files on mass store
- Can I download data without running the rawins.deck script?
- Yes, a "fetch" routine is available on the IBM
- You can copy the script from ~mesouser/MM5V3/Util/fetch-rawins-data.csh
- Copy script to your working directory to run.
- Edit startdate and enddate, run script by typing:
fetch-rawins-data.csh
[Data will be downloaded in
/ptmp/$USER]
- Do I have to use this script to run RAWINS on NCAR IBM
computers?
- No
- you can download teh source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- get data using "fetch" script above (or use
your own data)
- run code manually
LITTLE_R
- How does one run LITTLE_R on the IBM?
- Copy the script little_r.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the little_r.deck
- Parameters in green
is worth looking at
- [comment] is used
for extra information
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./little_r.deck
- The script will download the necessary observational files
from the mass store and run LITTLE_R
- The script expects to find the input files on mass store
- Can I download data without running the little_r.deck script?
- Yes, a "fetch" routine is available on the IBM
- You can copy the script from ~mesouser/MM5V3/Util/fetch-little_r-data.csh
- Copy script to your working directory to run.
- Edit startdate and enddate, run script by typing:
fetch-little_r-data.csh
[Data will be downloaded in
/ptmp/$USER]
- Three types of data will will be generated,
surface_obs files
upper-air_obs files
obs files
- The surface obs files have the format "surface_obs_r:DATE".
They are necessary if f4d are set to TRUE. If FDDA is not
done, these files are not needed for input to little_r as
this data will also be included in the obs files. At the
end of the run, these files will be tarred, zipped and saved
in mass storage under $ExpName/SURFACE_OBS_R.TAR.gz
- The upper_air obs files have the format "upper-air_obs_r:DATE".
They are not needed as direct input to little_r, but needed
to generate the obs files. For completeness these are tarred,
zipped and save in mass storage under $ExpName/UPPER-AIR_OBS_R.TAR.gz
- The obs files have the format "obs:DATE".
They contain both surface and upper-air data. These files
are always necessary as input to little_r. At the end of
the run, they are tarred, zipped and save in mass storage
under $ExpName/OBS.TAR.gz
- Do I have to use this script to run RAWINS on NCAR IBM
computers?
- No
- you can download teh source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- get data using "fetch" script above (or use
your own data)
- run code manually
INTERP
-
INTERPF
- How does one run INTERPF on the IBM?
- Copy the script interpf.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the interpf.deck
- Parameters in green
is worth looking at
- [comment] is
used for extra information
- DO NOT ADD SPACES between the
new sigma level values - this will prevent the namelist
from bring created correctly
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./interpf.deck
- Do I have to use this script to run INTERPF on NCAR
IBM computers?
- No
- you can download the source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- run code manually
-
INTERPB
- How does one run INTERPB on the IBM?
- Copy the script interpb.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the interpb.deck
- Parameters in green
is worth looking at
- [comment] is
used for extra information
- DO NOT ADD SPACES between the
pressure level values - this will prevent the namelist
from bring created correctly
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./interpb.deck
- Once the run is complete the
output, MMOUTP_DOMAINn, REGRID_DOMAINn and the intermediate
Files (FILE_MMOUTP:*), will be written to mass storage.
Take care that the new REGRID_DOMAINn file does
not overwrite one on the mass store that was created
by REGRID.
- Do I have to use this script to run INTERPB on NCAR
IBM computers?
- No
- you can download the source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- run code manually
NESTDOWN
- How does one run NESTDOWN on the IBM?
- Copy the script nestdown.deck
from ~mesouser/MM5V3/IBM to your working directory.
- Add your changes to the script
- Pay special attention to parameters maked in red
in the nestdown.deck
- Parameters in green
is worth looking at
- [comment] is used
for extra information
- If you do not want to make use
of a LOWBDY_DOMAINn file, change
"set input_lowbdy_file = ( LOWBDY_DOMAINn )" to "set
input_lowbdy_file = ( )", and
set use_mm5_lowbdy = .FALSE.
- DO NOT ADD SPACES between the
new sigma level values - this will prevent the namelist
from bring created correctly
- Run script:
- Run script interactively (this prevent problems with
ftp and mass store access)
- TYPE: ./nestdown.deck
- Once the run is complete the output,
MMINPUT_DOMAINn, LOWBDY_DOMAINn and BDYOUT_DOMAINn, will
be written to mass storage. Take care that the new
files do not overwrite the ones on the mass store that
was created by INTERPF.
- Do I have to use this script to run NESTDOWN on NCAR
IBM computers?
- No
- you can download the source code from ~mesouser/MM5V3
- unzip and untar the file in /ptmp/$USER
- run code manually
GRAPH
- Can GRAPH be compiled and run on an IBM?
- Yes, GRAPH has been updated to compile and run on am IBM.
(Thanks to Dave Gill for the changes).
- I get a namelist read error, what when wrong?
- When running on an IBM SP2, change the last line in "g_defaults.nml",
from "&END" to "/"
RIP
- Anything special to look out for the running RIP on an IBM?
- When running on an IBM SP2, if the namelist in rip.in is
in the default format of beginning and ending with a "&",
then an environment variable must be set to recognize this.
setenv
XLFRTEOPTS "namelist=old"
- If one desires to run on the IBM without setting this variable,
the namelist delimiters must be modified to begin with "&"
and end with "/".
- When compiling on an IBM SP, replace calls to "flush" in
driver.f and pltitle.f to calls to "fflush".
MM5
- MM5 Serial code
- It is recommended that you use the MPP code on the NCAR
IBM's. Therefore no specific NCAR script is generated when
choosing the "IBM" RUNTIME_SYSTEM option.
- If you would like to do this for testing purposes an example
script "mm5.deck.ibm-omp", is available under the directory
MM5/Templates.
- This script works similar to the MM5 MPP script, but can
be submitted to batch or run interactively.
- MM5 MPP code
- Since the NCAR IBM computers are multi processor machines,
this is recommended.
- The easiest way to run the code is by making use of the provided
script (mm5-mpp.deck).
This script is available from ~mesouser/MM5V3/IBM
- Copy mm5-mpp.deck
to your working directory, and edit the script
- Make sure your "wall_clock_limit" is set large enough
to complete you job (the SOC case run in under 1500).
- This script has been set up to run on 4 processors (on
blackforest, the setup will be different for bluesky as it
has 8 and 32 processor nodes, whereas blackforest has 4 processor
nodes). If you want to run the job on 16 processors,
set "node=4" and "task_per_node=4"
- ALWAYS make sure that
PROCMIN_NS and PROCMIN_EW match the number of processors
you are running on. For instance if you want to run on
64 processors, set (for
blackforest which has 4 processor nodes)
node=16, tasks_per_node=4 AND set PROCMIN_NS=8 ; PROCMIN_EW=8
- The script expects to find all the input files on mass
storage.
- To run TYPE:
- ./mm5-mpp.deck
- This will envoke 3 jobs:
- 1) an interactive job to get the source code, compile
the code and get the input data from mass store;
- 2) a multiple processor job will be submitted to the
loadleveler to run the MM5 job; and
- 3) an interactive job will be submitted to the loadleveler
to write the output to mass store. This job will automatically
run once the MM5 job has been completed
- This 3 job approach ensure better resource utilization
for IO processors.
Suggestion for changes to any of the above scripts are welcome.
If you have any suggestions/comments/problems, please send mail
to mesouser at mesouser@ucar.edu.
|