From dudhia@ucar.edu Mon Apr 28 12:13:17 2014
Return-Path: <dudhia@ucar.edu>
Received: from vscan4.ucar.edu ([128.117.64.124] verified)
  by mail.mmm.ucar.edu (CommuniGate Pro SMTP 6.0.7)
  with ESMTP id 51944186; Mon, 28 Apr 2014 12:13:17 -0600
Received: from localhost (localhost.localdomain [127.0.0.1])
	by vscan4.ucar.edu (Postfix) with ESMTP id 51ECB3EFE;
	Mon, 28 Apr 2014 12:13:17 -0600 (MDT)
Received: from vscan4.ucar.edu ([127.0.0.1])
 by localhost (vscan4.ucar.edu [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 20960-05; Mon, 28 Apr 2014 12:13:13 -0600 (MDT)
Received: from mail.ucar.edu (mail.ucar.edu [128.117.64.220])
	by vscan4.ucar.edu (Postfix) with ESMTP id EFC393E06;
	Mon, 28 Apr 2014 12:13:12 -0600 (MDT)
Received: from plane.mmm.ucar.edu ([128.117.123.56] verified)
  by mail.mmm.ucar.edu (CommuniGate Pro SMTP 6.0.7)
  with ESMTPS id 51944204; Mon, 28 Apr 2014 12:13:13 -0600
Subject: Re: [Wrf-users] Different results with different processor count with WRF
Mime-Version: 1.0 (Apple Message framework v1283)
Content-Type: multipart/alternative; boundary="Apple-Mail=_133F3D4D-48E7-4069-9A1A-AD10E986CD46"
From: Jim Dudhia <dudhia@ucar.edu>
In-Reply-To: <B5853D9A-9775-4B3B-BDD7-DC11D8921EDB@ucar.edu>
Date: Mon, 28 Apr 2014 12:13:12 -0600
Cc: Jim Dudhia <dudhia@ucar.edu>,
 wrfhelp <wrfhelp@ucar.edu>
Message-Id: <8ED18A81-4DF5-4CC5-ABFC-9162F6FB9BEC@ucar.edu>
References: <CAHTPDyTF5YBehJmvfU0DW35wn3MqtCxU+MhT7_rMPFfoQ66VsQ@mail.gmail.com> <B5853D9A-9775-4B3B-BDD7-DC11D8921EDB@ucar.edu>
To: David Gill <gill@ucar.edu>
X-Mailer: Apple Mail (2.1283)
X-Virus-Scanned: amavisd-new at ucar.edu


--Apple-Mail=_133F3D4D-48E7-4069-9A1A-AD10E986CD46
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Maybe wrfhelp could tell them that some difference is expected when =
optimization is used,
but it should be random and not affect rainfall totals over large areas =
much. This is just looking at a point.
Jimy

On Apr 28, 2014, at 12:02 PM, Jim Dudhia wrote:

> Dave,
>    Since you don't get these, I will forward this one.
> Jimy
>=20
>=20
> Begin forwarded message:
>=20
>> From: Vaibhav Saxena <saxena.vaibhav@gmail.com>
>> Subject: [Wrf-users] Different results with different processor count =
with WRF
>> Date: April 25, 2014 4:46:45 AM MDT
>> To: wrf-users@ucar.edu
>>=20
>> Hi All,=20
>>=20
>> I am seeing a large difference in results when running WRF on =
different processor counts. I am running WRF 3.5.1 in (dm+sm) mode with =
3 nested domains on Blue Gene/P. WRF is run in SMP mode on BG/P with 4 =
threads per MPI process and 1 MPI process per node. The model is run to =
produce a 48-hour forecast. Using the innermost nest forecast files =
(wrfout), I am comparing the accumulated precipitation forecast for a =
particular lat/long at the end of simulation (48 hours) when using 512, =
256 and 128 BGP nodes. =20
>>=20
>> Forecasted values for precip look like this:
>> 512 nodes - 54 mm
>> 256 nodes - 16 mm
>> 128 nodes - 7 mm
>> =20
>> I have following Qs:
>> 1. Is it true that WRF may produce incorrect results when the number =
of grid points per patch (assigned to an MPI process) or per tile =
(assigned to a thread) is less than 10x10 ? If this is correct, is it =
per patch or per tile ? The question arises because for 512 nodes, for =
our domain size, each patch has 8x17 grid points but for 256 and 128 =
nodes, there are more than 10x10 points per patch.=20
>>=20
>> 2. Are issues mentioned in =
http://mailman.ucar.edu/pipermail/wrf-users/2012/002944.html still valid =
in WRF 3.5.1 ? Can these issues produce such a large difference ?=20
>>=20
>> Thanks in advance.=20
>> - Vaibhav
>>=20
>>=20
>> --=20
>> email: saxena.vaibhav@gmail.com
>> _______________________________________________
>> Wrf-users mailing list
>> Wrf-users@ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/wrf-users
>=20


--Apple-Mail=_133F3D4D-48E7-4069-9A1A-AD10E986CD46
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Maybe =
wrfhelp could tell them that some difference is expected when =
optimization is used,<div>but it should be random and not affect =
rainfall totals over large areas much. This is just looking at a =
point.</div><div>Jimy</div><div><br><div><div>On Apr 28, 2014, at 12:02 =
PM, Jim Dudhia wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">Dave,<div>&nbsp; &nbsp;Since =
you don't get these, I will forward this =
one.</div><div>Jimy</div><div><br><div><br><div>Begin forwarded =
message:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px;"><span =
style=3D"font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, =
1.0);"><b>From: </b></span><span style=3D"font-family:'Helvetica'; =
font-size:medium;">Vaibhav Saxena &lt;<a =
href=3D"mailto:saxena.vaibhav@gmail.com">saxena.vaibhav@gmail.com</a>&gt;<=
br></span></div><div style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px;"><span =
style=3D"font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, =
1.0);"><b>Subject: </b></span><span style=3D"font-family:'Helvetica'; =
font-size:medium;"><b>[Wrf-users] Different results with different =
processor count with WRF</b><br></span></div><div style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span =
style=3D"font-family:'Helvetica'; font-size:medium; color:rgba(0, 0, 0, =
1.0);"><b>Date: </b></span><span style=3D"font-family:'Helvetica'; =
font-size:medium;">April 25, 2014 4:46:45 AM MDT<br></span></div><div =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px;"><span style=3D"font-family:'Helvetica'; =
font-size:medium; color:rgba(0, 0, 0, 1.0);"><b>To: </b></span><span =
style=3D"font-family:'Helvetica'; font-size:medium;"><a =
href=3D"mailto:wrf-users@ucar.edu">wrf-users@ucar.edu</a><br></span></div>=
<br><div dir=3D"ltr"><div><div><div>Hi All, <br><br>I am seeing a large =
difference in results when running WRF on different processor counts. I =
am running WRF 3.5.1 in (dm+sm) mode with 3 nested domains on Blue =
Gene/P. WRF is run in SMP mode on BG/P with 4 threads per MPI process =
and 1 MPI process per node. The model is run to produce a 48-hour =
forecast. Using the innermost nest forecast files (wrfout), I am =
comparing the accumulated precipitation forecast for a particular =
lat/long at the end of simulation (48 hours) when using 512, 256 and 128 =
BGP nodes.&nbsp; <br>
<br>Forecasted values for precip look like this:<br></div>512 nodes - 54 =
mm<br></div>256 nodes - 16 mm<br></div>128 nodes - 7 =
mm<br><div><div><div><div><div>&nbsp;<br></div><div>I have following =
Qs:<br></div><div>1. Is it true that WRF may produce incorrect results =
when the number of grid points per patch (assigned to an MPI process) or =
per tile (assigned to a thread) is less than 10x10 ? If this is correct, =
is it per patch or per tile ? The question arises because for 512 nodes, =
for our domain size, each patch has 8x17 grid points but for 256 and 128 =
nodes, there are more than 10x10 points per patch. <br>
</div><div><br>2. Are issues mentioned in <a =
href=3D"http://mailman.ucar.edu/pipermail/wrf-users/2012/002944.html">http=
://mailman.ucar.edu/pipermail/wrf-users/2012/002944.html</a> still valid =
in WRF 3.5.1 ? Can these issues produce such a large difference ? <br>
<br></div><div>Thanks in advance. <br></div><div>- =
Vaibhav<br></div><div><br clear=3D"all"></div><div><div><br>-- =
<br>email: <a =
href=3D"mailto:saxena.vaibhav@gmail.com">saxena.vaibhav@gmail.com</a>
</div></div></div></div></div></div></div>
_______________________________________________<br>Wrf-users mailing =
list<br><a href=3D"mailto:Wrf-users@ucar.edu">Wrf-users@ucar.edu</a><br><a=
 =
href=3D"http://mailman.ucar.edu/mailman/listinfo/wrf-users">http://mailman=
.ucar.edu/mailman/listinfo/wrf-users</a><br></blockquote></div><br></div><=
/div></blockquote></div><br></div></body></html>=

--Apple-Mail=_133F3D4D-48E7-4069-9A1A-AD10E986CD46--

