site map - email us - search  
 

General Questions

 NCAR IBM Related Questions

 Terrain

 Regrid

Datagrid

Little_R

 Rawins

 MM5

Graph

 Conventions used in MM5

 Trouble shooting and clues on various computers

 

NCAR IBM Help Guide

New users, please note:

  1. Build and run MM5 from /ptmp rather than $HOME
    There is simple not enough space in your $HOME directory to keep the executables and output files.
  2. Don't use /tmp on the NCAR IBM's - always use /ptmp.
  3. Define PROCMIN_EW and PROCMIN_NS to suite the domain size you are running, and the number of processors you plan on using.
    If you plan on running on 16 processors, set PROCMIN_EW and PROCMIN_NS to 4,4 rather than 1,1. Otherwise, the job will likely incur the dreaded "memory swapping" problem, which will result in the user's account locked until the user contacts
    Consulting (consult1@ucar.edu).

Two problems with programs TERRAIN and REGRID have been resolved for use on NCAR's IBM computers. Please use the updated tar files.

This Web page has been set up to help with the transition from NCAR Cray's to the NCAR IBM's. If you require more information, or have additional information you would like to share with other NCAR IBM users, please send mail to mesouser at mesouser@ucar.edu.

General Questions

  • How do I obtain an account on NCAR's IBM machines?

To get a logon on SCD's IBM computers babyblue and blackforest, please call SCD Database Services at 303  497-1225,  08:00 - 17:00 MDT Monday to Friday and request one.  For security reasons, this MUST be done by phone, and not via email or FAX. For general information on applying for computer resources at NCAR, check out the SCD Web Page.

For introductory information about IBM computer blackforest, please see http://www.scd.ucar.edu/computers/blackforest

By default, your logon shell on the IBM's is the Korn shell (ksh).  To change it to sh or csh or tcsh, please follow the instructions in

            http://www.scd.ucar.edu/docs/blackforest/env.html#spd

Message from SCD Consulting Office to MM5 users switching over to IBM computers.   "Do not hesitate to call the SCD Consulting Office 303  497-1302, 08:00 - 17:00 MDT.  Monday to Friday,  or email consult1@ucar.edu for any help you may need.  You will undoubtedly have questions.  SCD is anxious to help you run successfully on blackforest."

 

  • Will I be able to read my old Cray files on the IBM's without modifications?

NO, Cray format file cannot be read directly on the IBM's

You can convert the files from Cray to IBM format on the IBM. This process is not as straight forward as on the Cray. The method involves writing a program to read the Cray records and convert them with calls to the ncaru library. A program has been developed to convert MM5V3 data files on the IBM.  This program should run on many other Unix platforms (like SGI and SUN), since the ncaru software is freely available. For more information on the use of the ncaru library, see

       http://www.scd.ucar.edu/docs/conversion.tools/ncaru.html

Have Cray format intermediate files, you would like to read on the IBM? Use the cray2ibm_intermediate.f utility.

  • How do I submit jobs on the IBM?

To submit jobs on the IBM's to run as a batch job, make use of the LoadLeveler.  A typical batch job will look as follows:

#!/bin/csh

 

# @ job_type   = parallel

! When you run a parallel job, always set
"job_type = parallel" because the default is serial. In a parallel job, you must also specify tasks per node and the number of nodes.

# @ environment =
       COPY_ALL;MP_EUILIB=us

! Specifies your initial environment variables when your job step starts.

# @ job_name   = my_job

! Specifies the name of the job.

# @ output     = my_job.out 

!    Specifies a file to which standard output is written.

# @ error      = my_job.err

! Specifies the file to which standard-error output is written.

# @ node       = 1,1

! Number of nodes required for your job.

# @ network.MPI    = css0,shared,us

! Use "shared" in all cases.

# @ tasks_per_node = 4

! Tasks per node.

# @ node_usage = not_shared 

! Whether to share nodes acquired for your tasks

# @ checkpoint = no

! Specifies whether you want to checkpoint your program.

# @ wall_clock_limit = 1800

! Specifies the maximum wall-clock time that LoadLeveler will allow the job to run. Will prevent runaway jobs from consuming excessive resources. This can help your job run sooner when blackforest queues are full.

# @ class      = com_reg

! Batch queue name.  Currently, blackforest has these queues for users.

# @ queue 

! Causes LoadLeveler to submit the preceding macros to the queue

 

 

set MP_SHARED_MEMORY = yes

 

 

cd /home/username/MM5
pwd
ls -l
make mpp
cd Run
timex poe ./mm5.mpp

 

 

 

exit

 

            Submit this job to the queue, by making use of llsubmit.
            To check the status of your job, use llq.
            To cancel a jon that is in the queue, use llcancel.
 
 

  • Will I still be able to use IRJE / MIGS to submit my jobs?

Yes.  Read more about these systems at: IRJE & MIGS

  • I have limited space on babyblue/blackforest, will I be able to run large jobs?

    • All the new IBM scripts need to run on the  "/ptmp" disk. This space is large enough to hold the input files from mass storage, as well as large output files.

    • The default directory under which programs will run is: "/ptmp/$USER/PROGRAM" (ex,  /ptmp/mesouser/TERRAIN)

    • At the end of each script all output files will be written to mass storage, but will also be kept on the /ptmp disk (Under /ptmp/$USER/PROGRAM) for easy access.

    • For your convenience these directories will not be removed after the completion of the run. If you no longer need it, please remove it by hand. If not removed, it will be deleted once the scrubber runs.

    • Please do clean up files under /ptmp/$USER which are no longer needed. If you leave to many large files under /ptmp/$USER, you may well run into space problems.

    • NEVER delete the directory /ptmp/$USER - even if you do not use this space, you need it to compile on the NCAR's IBM computers.

 

  • Will I be able to use the new IBM scripts on other IBM computers?

    • The basics of the scripts will work on other IBM's.

    • The LoadLeveler commands, especially the batch classes, could be different.

    • Most of the scripts have been developed around the NCAR MSS facilities, and those are only available on NCAR computers.


  • Where can I find the IBM scripts?

    • Most of the scripts are available in the tar files.

    • As new scripts are being made available, they will be place in "~mesouser/MM5V3/IBM"

     

  • How do I get the data back to my local computer?

    Since version 3.6 all the batch job scripts has been made more user friendly. All "rcp" commands has therefore been removed from the scripts to prevent the script from stopping due to firewall problems. If you need the data on your local machine, please ftp it over manually.

 

Thanks to Dick Valent from NCAR/SCD that helped with answers to these questions

TERRAIN

  • How does one run TERRAIN on the IBM?
    • Copy the script terrain.deck from ~mesouser/MM5V3/IBM to your working directory.
    • Add your changes to the script
      • Pay special attention to parameters maked in red in the terrain.deck
      • Parameters in green is worth looking at
      • [comment] is used for extra information
    • Run script:
      • Run script interactively (this prevent problems with ftp and mass store access)
      • TYPE:     ./terrain.deck
          
  • Do I have to use this script to run TERRAIN on NCAR IBM computers?
    • No
      • you can download the source code from ~mesouser/MM5V3
      • unzip and untar the file in /ptmp/$USER
      • run code manually
          
  • I get the followining message whne running TERRAIN:
    "Cannot load program ./terrain.exe because of the following error
      System error: There is not enough memory available now    "
    • Edit the TERRAIN Makefile and under the IBM options, change the following line from:
      "LDOPTIONS = "
      to:
      "LDOPTIONS = -bmaxdata:2000000000"

      Then recompile the code, before running the script again.

REGRID

  • How does one run REGRID on the IBM?
    • Copy the script regrid.deck from ~mesouser/MM5V3/IBM to your working directory.
    • Add your changes to the script
      • Pay special attention to parameters maked in red in the regrid.deck
      • Parameters in green is worth looking at
      • [comment] is used for extra information
    • Run script:
      • Run script interactively (this prevent problems with ftp and mass store access)
      • TYPE:     ./regrid.deck
    •  
    • This script only works for input data that is available on NCAR mass storage (NCEP GDAS data in ON84 or GRIB format, NCEP/NCAR Reanalysis data, ECMWF Reanalysis data, FNL Data, Eta AWIP DATA and TOGA data).
    • The script expects to find a TERRAIN_DOMAINn file on mass storage for input to the "regridder" part of the run.
    • All output files ("pregrid" & "regridder" output files) will be written to mass storage.
    • Do not run script as a batch job
    • The new NISE data is not yet available via the regrid.deck script.
  • Can I download data without running the regrid.deck script?
    • Yes, all the "get" routines are available on the IBM (for instance, get_nnrp.csh, to download NNRP data).
    • You can copy the script from ~mesouser/MM5V3/Util
    • Copy script to your working directory to run.
    • To run you have 3 options:
      • Edit startdate ; enddate and itimint inside script, run script by typing (eg):
        get_nnrp.csh
        [Data will be downloaded in /ptmp/$USER]
      • OR type (eg - for Mar 13-14, 1993, evry 12 hours):
        get_nnrp.csh   1994-03-13_00   1994-03-14_00   12
        [Data will be downloaded in /ptmp/$USER]
      • OR type (eg - for Mar 13-14, 1993, evry 12 hours):
        get_nnrp.csh 1994-03-13_00 1994-03-14_00 12 /ptmp/$USER/myDATA
        [Data will be downloaded in /ptmp/$USER/myData]
         
    • Since version 3.6, the scripts no longer provide the option to automatically copy the files to your local computer. This creates to many problems with user computer firewalls. If you need the data on your local computer, please ftp the files over manually.
        
  • Do I have to use this script to run REGRID on NCAR IBM computers?
    • No
      • you can download teh source code from ~mesouser/MM5V3
      • unzip and untar the file in /ptmp/$USER
      • get data using "get" script above (or use your own data)
      • run code manually

 

RAWINS [using LITTLE_R is recommended]

  • How does one run RAWINS on the IBM?
    • Copy the script rawins.deck from ~mesouser/MM5V3/IBM to your working directory.
    • Add your changes to the script
      • Pay special attention to parameters maked in red in the rawins.deck
      • Parameters in green is worth looking at
      • [comment] is used for extra information
    • Run script:
      • Run script interactively (this prevent problems with ftp and mass store access)
      • TYPE:     ./rawins.deck
         
    • The script will download the necessary observational files from the mass store and run RAWINS
    • The script expects to find the input files on mass store
        
  • Can I download data without running the rawins.deck script?
    • Yes, a "fetch" routine is available on the IBM
    • You can copy the script from ~mesouser/MM5V3/Util/fetch-rawins-data.csh
    • Copy script to your working directory to run.
      • Edit startdate and enddate, run script by typing:
        fetch-rawins-data.csh
        [Data will be downloaded in /ptmp/$USER]
         
  • Do I have to use this script to run RAWINS on NCAR IBM computers?
    • No
      • you can download teh source code from ~mesouser/MM5V3
      • unzip and untar the file in /ptmp/$USER
      • get data using "fetch" script above (or use your own data)
      • run code manually

 

LITTLE_R

  • How does one run LITTLE_R on the IBM?
    • Copy the script little_r.deck from ~mesouser/MM5V3/IBM to your working directory.
    • Add your changes to the script
      • Pay special attention to parameters maked in red in the little_r.deck
      • Parameters in green is worth looking at
      • [comment] is used for extra information
    • Run script:
      • Run script interactively (this prevent problems with ftp and mass store access)
      • TYPE:     ./little_r.deck
         
    • The script will download the necessary observational files from the mass store and run LITTLE_R
    • The script expects to find the input files on mass store
        
  • Can I download data without running the little_r.deck script?
    • Yes, a "fetch" routine is available on the IBM
    • You can copy the script from ~mesouser/MM5V3/Util/fetch-little_r-data.csh
    • Copy script to your working directory to run.
      • Edit startdate and enddate, run script by typing:
        fetch-little_r-data.csh
        [Data will be downloaded in /ptmp/$USER]
    • Three types of data will will be generated,
            surface_obs files
            upper-air_obs file      s
            obs files
      • The surface obs files have the format "surface_obs_r:DATE". They are necessary if f4d are set to TRUE. If FDDA is not done, these files are not needed for input to little_r as this data will also be included in the obs files. At the end of the run, these files will be tarred, zipped and saved in mass storage under $ExpName/SURFACE_OBS_R.TAR.gz
      • The upper_air obs files have the format "upper-air_obs_r:DATE". They are not needed as direct input to little_r, but needed to generate the obs files. For completeness these are tarred, zipped and save in mass storage under $ExpName/UPPER-AIR_OBS_R.TAR.gz
      • The obs files have the format "obs:DATE". They contain both surface and upper-air data. These files are always necessary as input to little_r. At the end of the run, they are tarred, zipped and save in mass storage under $ExpName/OBS.TAR.gz
          
  • Do I have to use this script to run RAWINS on NCAR IBM computers?
    • No
      • you can download teh source code from ~mesouser/MM5V3
      • unzip and untar the file in /ptmp/$USER
      • get data using "fetch" script above (or use your own data)
      • run code manually

     

INTERP

  • INTERPF

    • How does one run INTERPF on the IBM?
      • Copy the script interpf.deck from ~mesouser/MM5V3/IBM to your working directory.
      • Add your changes to the script
        • Pay special attention to parameters maked in red in the interpf.deck
        • Parameters in green is worth looking at
        • [comment] is used for extra information
        • DO NOT ADD SPACES between the new sigma level values - this will prevent the namelist from bring created correctly
      • Run script:
        • Run script interactively (this prevent problems with ftp and mass store access)
        • TYPE:     ./interpf.deck
            
    • Do I have to use this script to run INTERPF on NCAR IBM computers?
      • No
        • you can download the source code from ~mesouser/MM5V3
        • unzip and untar the file in /ptmp/$USER
        • run code manually

  • INTERPB

    • How does one run INTERPB on the IBM?
      • Copy the script interpb.deck from ~mesouser/MM5V3/IBM to your working directory.
      • Add your changes to the script
        • Pay special attention to parameters maked in red in the interpb.deck
        • Parameters in green is worth looking at
        • [comment] is used for extra information
        • DO NOT ADD SPACES between the pressure level values - this will prevent the namelist from bring created correctly
      • Run script:
        • Run script interactively (this prevent problems with ftp and mass store access)
        • TYPE:     ./interpb.deck
           
        • Once the run is complete the output, MMOUTP_DOMAINn, REGRID_DOMAINn and the intermediate Files (FILE_MMOUTP:*), will be written to mass storage. Take care that the new REGRID_DOMAINn file does not overwrite one on the mass store that was created by REGRID.
            
    • Do I have to use this script to run INTERPB on NCAR IBM computers?
      • No
        • you can download the source code from ~mesouser/MM5V3
        • unzip and untar the file in /ptmp/$USER
        • run code manually

 

NESTDOWN

  • How does one run NESTDOWN on the IBM?
    • Copy the script nestdown.deck from ~mesouser/MM5V3/IBM to your working directory.
    • Add your changes to the script
      • Pay special attention to parameters maked in red in the nestdown.deck
      • Parameters in green is worth looking at
      • [comment] is used for extra information
      • If you do not want to make use of a LOWBDY_DOMAINn file, change
        "set input_lowbdy_file = ( LOWBDY_DOMAINn )"         to "set input_lowbdy_file = (  )",   and
        set use_mm5_lowbdy = .FALSE.
      • DO NOT ADD SPACES between the new sigma level values - this will prevent the namelist from bring created correctly
    • Run script:
      • Run script interactively (this prevent problems with ftp and mass store access)
      • TYPE:     ./nestdown.deck
         
      • Once the run is complete the output, MMINPUT_DOMAINn, LOWBDY_DOMAINn and BDYOUT_DOMAINn, will be written to mass storage. Take care that the new files do not overwrite the ones on the mass store that was created by INTERPF.
          
  • Do I have to use this script to run NESTDOWN on NCAR IBM computers?
    • No
      • you can download the source code from ~mesouser/MM5V3
      • unzip and untar the file in /ptmp/$USER
      • run code manually

 

GRAPH

  • Can GRAPH be compiled and run on an IBM?
    • Yes, GRAPH has been updated to compile and run on am IBM.  (Thanks to Dave Gill for the changes).
  • I get a namelist read error, what when wrong?
    • When running on an IBM SP2, change the last line in "g_defaults.nml", from "&END" to "/"

RIP

  • Anything special to look out for the running RIP on an IBM?
    • When running on an IBM SP2, if the namelist in rip.in is in the default format of beginning and ending with a "&", then an environment variable must be set to recognize this.
                setenv XLFRTEOPTS "namelist=old"
    • If one desires to run on the IBM without setting this variable, the namelist delimiters must be modified to begin with "&" and end with "/".
    • When compiling on an IBM SP, replace calls to "flush" in driver.f and pltitle.f to calls to "fflush".

MM5

  • MM5 Serial code
    • It is recommended that you use the MPP code on the NCAR IBM's. Therefore no specific NCAR script is generated when choosing the "IBM" RUNTIME_SYSTEM option.
    • If you would like to do this for testing purposes an example script "mm5.deck.ibm-omp", is available under the directory   MM5/Templates.
    • This script works similar to the MM5 MPP script, but can be submitted to batch or run interactively.
  • MM5 MPP code
    • Since the NCAR IBM computers are multi processor machines, this is recommended.
    • The easiest way to run the code is by making use of the provided script (mm5-mpp.deck). This script is available from ~mesouser/MM5V3/IBM
    • Copy mm5-mpp.deck to your working directory, and edit the script
      • Make sure your "wall_clock_limit" is set large enough to complete you job (the SOC case run in under 1500).
      • This script has been set up to run on 4 processors (on blackforest, the setup will be different for bluesky as it has 8 and 32 processor nodes, whereas blackforest has 4 processor nodes). If you want to run the job on 16 processors, set "node=4" and "task_per_node=4"
        • ALWAYS make sure that PROCMIN_NS and PROCMIN_EW match the number of processors you are running on. For instance if you want to run on 64 processors, set (for blackforest which has 4 processor nodes) node=16, tasks_per_node=4 AND set PROCMIN_NS=8 ; PROCMIN_EW=8
      • The script expects to find all the input files on mass storage.
    • To run TYPE:
      • ./mm5-mpp.deck
      • This will envoke 3 jobs:
        • 1) an interactive job to get the source code, compile the code and get the input data from mass store;
        • 2) a multiple processor job will be submitted to the loadleveler to run the MM5 job; and
        • 3) an interactive job will be submitted to the loadleveler to write the output to mass store. This job will automatically run once the MM5 job has been completed
      • This 3 job approach ensure better resource utilization for IO processors.


Suggestion for changes to any of the above scripts are welcome. If you have any suggestions/comments/problems, please send mail to mesouser at mesouser@ucar.edu.

UCAR Privacy Policy - Terms of Use - ©2003UCAR