Notes for various cycling runs

Spack-stack (SS) version comparisons

1. To compare SS 1.8.0 with SS 1.6.0 click the Graphs vs Previous for the 2025-04-14 entry. 2025-04-14 run uses SS 1.8.0, 2025-04-13 uses SS 1.6.0
• results are the same.
• Run time speed and memory use was comparable.
• runs from 2025-05-18 on use SS 1.8.0.
2. To compare SS 1.9.1 with SS 1.6.0 click the Graphs vs Previous for the 2025-03-30 entry. 2025-03-30 run uses SS 1.6.0, 2025-03-28 uses SS 1.9.1
• results are the same.
• Run time memory use was significantly greater using SS 1.9.1. SS 1.6.0 OOPS stats, SS 1.9.1 OOPS stats

GPU runs

1. 2025-05-30 entry compares the GPU run of 2025-05-30 with the cpu run of 2025-05-29. The results diverge.
2. 2025-05-31 entry compares the GPU run of 2025-05-31 with the GPU run of 2025-05-30.
3. 2025-06-01 entry compares the cpu run of 2025-06-01 with the GPU run of 2025-05-31. The results diverge.
4. 2025-06-15 entry compares the cpu run of 2025-06-15 with the GPU run of 2025-06-01. The results diverge.

Various deviations

1. 2025-03-11 entry compares the weekly 2025-03-09 run with an mpas-bundle 3.0.2 build using crtm bug fix PR 209
2. 2025-03-07 entry compares the weekly 2025-02-16 run with an mpas-bundle 3.0.2 build using crtm 2.4.1
3. 2025-03-09 entry compares to the weekly 2025-02-16 run. The results are the same.
4. 2025-01-06 run compares MPAS-Model using SMIOL to the weekly 2025-01-05 run with MPAS-Model using PIO. The results are the same.
   • All runs after 2025-01-19 use MPAS-Model with SMIOL
5. 2025-01-05 entry compares to the weekly 2024-12-31 run. The results diverged.
• 2025-01-05 was the first build using the develop branch of mpas-model. Previous builds used hotfix-v8.2.2
• After investigating, the mpas-model branch change did not cause the diversion, see below.
6. 2025-mpas8-2-2 entry compares a new mpas bundle with MPAS-Model 8.2.2 to the weekly 2024-12-31 run. The results diverged.
• The new mpas-bundle build used all of the same versions of the packages as the 2024-12-31 build.
• The new version used the same version of spack-stack (1.6), gcc tools (12.2.0), and mpich (8.1.25)
• The new version was built on 2025-04-21
• The results of the new version appear to be the same as the 2025-01-05 run.
• It is curently not understood what is causing the variances.
• There are no comparison graphs for the new version and the mpas-bundle 8.0.2 baseline.