[Background on this problem: despite John M's claims to the contrary,
Al Bourgeois, who works with the EPA in NC, could not get the MPP version
of the model to give bit-for-bit agreement with the serial version.
Al found the bug and send the following email.]

Hi John,

I was finally able to get bit-for-bit agreement between serial and
parallel simulations on the T3E (using Blackadar PBL, on a 53x75x30
grid, for 5 hours of simulation time). This requires compilation with
the debug (-g) option. Otherwise, differences in velocities start
accumulating rapidly due to the combined effects of (1) small
optimization differences originating from different loop order and loop
index limits and (2) extreme sensitivity of HIRPBL and SOUND to input.
(I need to study this in more detail.)

However, there is a typo in one of the makefiles that made it appear
that something was amiss. In MPP/RSL/Makefile.RSL:

transm_col.o: transm.F
        $(CUT) -c1-72 transm.F | $(SED) 's/`//' | $(EXPAND) |\
           $(CPP) -DMPP1_COLUMNWISE $(INCLUDES) $(CPP_FLAGS) |\
           $(M4) - > transm_col.b
        $(FLIC) -cm transm_col.b > transm_col.dm
        $(M4_FLIC) transm_col.dm | sed '/^$$/d' \
            | $(CPP) $(INCLUDES) | $(CUTTER) \
            | sed 's/CMPP1//' > transm_col.f
        $(MFC) -c $(FC_FLAGS) transm_col.f
        $(RM) transm_col.b transm_col.dm transm_col.f


FC_FLAGS should be FCFLAGS. This was causing compiler optimization to
occur in transm_col, even though I was compiling the code with the "-g"
option.

So ends the mystery! In hindsight, I should have suspected this and done
a "grep f90" on the compilation output and checked compiler options. Oh
well, the pays the same.

Regards,

-Al

-- 

Al Bourgeois                           Lockheed Martin
ajb@vislab.epa.gov                     Research Triangle Park, NC