P71     Identifying areas of model differences using the Method for Object-based Diagnostic Evaluation (MODE)

Hertneky, Tracy, Tressa Fowler, and Randy Bullock, National Center for Atmospheric Research and the Developmental Testbed Center

A method for identifying spatial regions of differences in model forecasts was assessed, with the goal to develop a more systematic approach for this process, using the Model Evaluation Tools (MET) Method for Object-based Diagnostic Evaluation (MODE). Often times during model development, validation analyses are performed to assess correct implementation of the model. Model implementation differences are not  'forecasting errors' as in model evaluation, and some noise is expected due to differences in precision, operating system, etc. Direct model-to-model comparisons can detect biases, implementation errors, or other problems while ignoring the small, random errors typical of a new model implementation. Examination of these comparisons can be a tedious process and generally does not involve checking each individual field, level, and forecast time period for large differences. By running the MODE tool on fields of model differences, objects can be identified by using thresholds to identify areas that exceed those limits set by the user, creating an automated approach for the user.

To test the functionality of this method, the Air Force’s Global Air-Land Weather Exploitation Model (GALWEM) 10.4 implementation was compared against the UK MET Office’s (UKMO) Unified Model (UM) 10.4 system. For simplicity, the two models were compared directly, without the use of observations, with the objective of identifying model differences to ensure correct implementation. Fields of model differences (GALWEM - UM) were computed for each individual forecast as well as aggregated across series of time and height using the MET grid-stat and series-analysis tools, respectively. The output from these tools was then ingested into MODE for identifying areas of large differences exceeding some threshold. While running MODE on each individual forecast is more computationally expensive, it provides more thorough results. Running MODE on a series of aggregated differences is faster, and can eliminate the need to look at individual forecasts if no objects are found, but can also smooth out smaller, inconsistent differences, which may not be of interest anyways. Output from MODE includes a list of all identified objects and their attributes meeting the user criteria as well as images of the objects. Using the list, users can look closely at select fields where large, spatially coherent differences were identified rather than needing to manually look at each field. This presentation will provide a detailed description of this method and review results over a month long period.