Index for this report.

• Jim Dye's Synthesis of the Discussions as of March 20, 2004
• EXAMINING CANDIDATE RADAR VARIABLES by F. Merceret    
  • CANDIDATE RADAR VARIABLES SUPPLEMENT (March 22, 2004)
  • Addendum 22Mar Suppl
  • More ROC
  (Summary on this page)
• Choosing an Algorithm (Summary on this page) by M. Bateman and D. Mach
  • Choosing An Algorith I
  • Choosing An Algorith II
  • Choosing An Algorith III
• EFFECTS OF TRUNCATION THRESHOLD (CUTOFF) ON RADAR PARAMETERS by F. Merceret     (Summary on this page)
• Boccippio's Modeling Analysis
  • BOCCIPPIO'S ROC ANALYSIS M. Bateman (Sumary on this page)
  • LCC Anvil Electrification Hazard - Preliminary Analysis D. Boccippio
  • Bivariate Illustration (Apr 13, 2004) D. Boccippio
• Reflectivity Averaging: A Monte Carlo Study by F. Merceret
• Why we average dBZ rather than Z by S. Lewis

Jim Dye's Synthesis of the Discussions as of March 20, 2004

After reading and thinking carefully about the recent exchanges and writeups primarily between Frank and Monte/Doug regarding which Reflectivity Parameter might be best to use for an LLCC, I'd like to offer some thoughts.

I think that in the final analysis it will be Harry Koons' extreme value approach that will help us decide which parameter should be used. [Although Dennis Boccippio's and Frank's ROC analysis is proving to be very revealing.]

The recent discussion has illuminated some issues and should help us to decide the parameter(s) that might be most suitable for a radar rule for anvils.

I do need to point out that the results that Frank and Monte have presented so far are based on only the 2001 anvil cases, because that was what could be provided to them quickly. Also the 1x1 anvil thicknesses included in those files were from anvil bottom (regardless of altitude) to top of the anvil and were not constrained to start at 5 km (~0C) or higher as was done for the other calculated parameters. We are now redoing our calculations for the entire anvil data set (2000 and 2001) and using a lower limit of 5 km altitude for all calculated parameters. In addition we are also calculating parameters for a cutoff reflectivity of 5 dBZ. Monte's review of Dennis Boccippio's work showed an improvement in POD by using a cutoff of 0 dBZ rather than -10 dBZ. I felt that 5 dBZ might show even more improvement, because that's often where the transition from weak to strong E field occurs for individual aircraft passes. Unfortunately these new data products will not be available in time for the Conf. call this Thursday, Mar 25th.

MAIN CONCLUSIONS TO DATE:

*** BOTH FRANK'S AND MONTE'S RESULTS ARE FINDING THAT VOLUME INTEGRAL (in some form) IS A BETTER PARAMETER THAN AVERAGE, OR THICKNESS OR SUM.

*** ROC ANALYSIS AND PLOTS CAN HELP US IN SELECTING A VARIABLE (OR VARIABLES).

*** A CUTOFF AT 0 dBZ RATHER THAN -10 dBZ STATISTICALLY BIASES PARAMETERS SUCH AS AVERAGE AND THICKNESS.

HOWEVER,

*** ROC ANALYSIS SHOWS THAT A RADAR CUTOFF OF 0 INSTEAD OF -10 dBZ GIVES A HIGHER PROBABLILITY OF DETECTION (POD) WITH FEWER FALSE ALARMS (FAR).

*** THIS IMPLIES THAT CHARGE CONTRIBUTING TO INTENSE E FIELD RESIDES IN REGIONS WITH REFLECTIVITY > 0 dBZ. BY USING -10 dBZ CUTOFF IN OUR CALCULATIONS WE MAY BE DILUTING THE SIGNAL IN OUR REFLECTIVITY PARAMETER TO DETECT REGIONS OF STRONGER E FIELD.

By way of summary and paraphasing in my own words the main conclusions (but not inclusive) that I drew from recent reports are below.

Return to Top of Page

Frank's report on

EXAMINING CANDIDATE RADAR VARIABLES:

CANDIDATE RADAR VARIABLES SUPPLEMENT (March 22, 2004)

[Frank used a -10 dBZ cutoff of dBZ in all this work]

• Average, Thickness and Volume Integral(Average x Thickness), produce scatter plots that have similar behaviour as a fn. of E.
• The scatter plot for Thickness is very similar to those for Average and Volume Integral.
• All three variables are highly correlated with each other.
• Because of similar behavior there is not much value to a rule by using 2 of these variables via contour plots versus one variable.
• Examining Probablility of Detection (POD) vs. False Alarm Rate (FAR) using Relative Operating Characteristics (ROC) plots shows that all 3 variables have a sharp cutoff as a fn. of E and could potentially be used for a rule.
• All of the 74C curves reach POD = 1 with fewer FARs than NEXRAD.

**** Volume Integral reaches a POD = 1 faster with fewer FARs than average or thickness, hence is a still better variable. ****

Return to Top of Page

Monte and Doug's Paper(s)

• Choosing An Algorith I
• Choosing An Algorith II
• Choosing An Algorith III

CHOOSING AN ALGORITHM PART III

• Interpolating radar data between scan gaps comes closer to giving the "truth" for Sums, Averages and Thickness (they use the word depth) than the radar data with no interpolation for the scan gaps.
• Vertically Integrated Reflectivity (VIR) is a better variable than Average or Sum to characterize the reflectivity of the cloud.
• Thickness should be calculated for each 1x1 column rather than the average over the 11x11 column.

**** Average x Thickness (without interpolation) gives a reasonable approximation to true VIR. [This is what NCAR and now Frank is calling Volume Integral.]****

Return to Top of Page

Frank's report on

EFFECTS OF TRUNCATION THRESHOLD (CUTOFF) ON RADAR PARAMETERS

• Using 0 dBZ compared to -10 dBZ cutoff produces clear cut differences in all 3 variables (average, thickness, volume integral).
• Use of 0 dBZ creates a bias in our determination of these variables from a statistical consideration.
• Volume integral shows a little less difference between a 0 and -10 dBZ cutoff than the other variables, because effects on average and thickness are of opposite sign.

Return to Top of Page

Monte's report on

BOCCIPPIOS'S ROC ANALYSIS

Below I'm including all of Monte's recent email regarding Dennis Boccippio's ROC analysis, because it is new. You need to view the ROC plot that is also a link to see the results he describes.

Here is some preliminary output from Dr. Boccippio:

He did several things:

• Made sure that half the data were used to construct the model and the other half were reserved to test the model. This reduces the risk of "overtraining" the model, which can happen if you use all the data to construct the model. These 50/50 splits were randomly chosen, and repeated 50 times.
• Looked at all the variables "of interest", i.e.,

  Em_m	The magnitude of E at the aircraft
  AVG11X11	Average reflectivity
  thick11x11	Average cloud thickness
  TotSum11x11	Sum of refl (integrated refl.)
  ACintSum11x11	Sum of (Col avg refl.) x (col. thick)

• Used all 30 radar-related variables and "competed" them for relevancy to Em. In every case where a "_0" variable existed in the file, it was better performing than the "non _0" version of the same variable. Then, the variables that came out with a >10% relevancy were then run in simple linear models and ROC curves were calculated. They are displayed in the attached graph. Some things to note:
  • These ROC curves are from the WSR data file.
  • These simple linear models nicely reproduce Frank's rule-based models.
  • The average is the poorest performer of the group.
  • The cloud thickness is a good performer until you look at the very top, and the FAR jumps from about 30% to about 47% -- equal to the FAR for the average, at POD = 1.0.
  • The TotSum11x11_0 has about 10% lower FAR than TotSum11x11 at a POD of 1.0.
  • ACint11x11 -- the product of (col avg) x (col thick) -- is very close in performance to TotSum11x11. This confirms our earlier conclusion that (col avg) x (col thick) is a good substitute for col. integral.
  • The same should hold for ACint11x11_0 and TotSum11x11_0, but they show more differences than the previous pair. We think this is a result of lots of missing data in the ACint11x11_0 column.

    However,

    both the "_0" versions outperform the non-thresholded versions.

  • TotSum11x11_0 (in red) is the best single-variable performer in the set.

Return to Top of Page