Notes for various cycling runs

Spack-stack (SS) version comparisons

1. To compare SS 1.8.0 with SS 1.6.0 click the Graphs vs Previous for the 2025-04-14 entry. 2025-04-14 run uses SS 1.8.0, 2025-04-13 uses SS 1.6.0
• results are the same.
• Run time speed and memory use was comparable.
• runs from 2025-05-18 on use SS 1.8.0.
2. To compare SS 1.9.1 with SS 1.6.0 click the Graphs vs Previous for the 2025-03-30 entry. 2025-03-30 run uses SS 1.6.0, 2025-03-28 uses SS 1.9.1
• results are the same.
• Run time memory use was significantly greater using SS 1.9.1. SS 1.6.0 OOPS stats, SS 1.9.1 OOPS stats
3. To compare SS 1.9.3 with SS 1.8.0 click the Graphs vs Previous for the 2025-10-29 entry. 2025-10-26 run uses SS 1.8.0, 2025-10-29 uses SS 1.9.3
• results are the same.
• Run time speed and memory use was comparable.

Derecho OS upgrade to SLES 15-SP6

1. Derecho's OS was upgraded on Nov 4-5 2025
2. New runtime libraries cause deviations in results
3. 2025-11-12 entry compares 11/12 mpas-bundle code built on upgraded Derecho with 11/02 mpas-bundle code built on pre-upgraded Derecho.
   • Results vary +- 3 to 3.5%
   • Results tend to mostly improve
4. 2025-11-02-sp6 entry compares 11/12 mpas-bundle code built on upgraded Derecho with 11/02 mpas-bundle code built on upgraded Derecho.
   • Results are the same
   • this indicates the variances in the 2025-11-12 graphs were caused by upgraded libraries, not by changes to the mpas-bundle code between 11/02 and 11/12

Various deviations

1. 2025-12-04 entry compares a week long simulation using IAU
• There are significant variations
2. 2025-06-29-old entry compares the 6/29 code built with the 6/22 version of crtm with the 6/22 run
3. 2025-06-29 entry compares with the weekly 2025-06-22 run
• There is significant degradation due to crtm changes
4. 2025-03-11 entry compares the weekly 2025-03-09 run with an mpas-bundle 3.0.2 build using crtm bug fix PR 209
5. 2025-03-07 entry compares the weekly 2025-02-16 run with an mpas-bundle 3.0.2 build using crtm 2.4.1
6. 2025-03-09 entry compares to the weekly 2025-02-16 run. The results are the same.
7. 2025-01-06 run compares MPAS-Model using SMIOL to the weekly 2025-01-05 run with MPAS-Model using PIO. The results are the same.
• All runs after 2025-01-19 use MPAS-Model with SMIOL
8. 2025-01-05 entry compares to the weekly 2024-12-31 run. The results diverged.
• 2025-01-05 was the first build using the develop branch of mpas-model. Previous builds used hotfix-v8.2.2
• After investigating, the mpas-model branch change did not cause the diversion, see below.
9. 2025-mpas8-2-2 entry compares a new mpas bundle with MPAS-Model 8.2.2 to the weekly 2024-12-31 run. The results diverged.
• The new mpas-bundle build used all of the same versions of the packages as the 2024-12-31 build.
• The new version used the same version of spack-stack (1.6), gcc tools (12.2.0), and mpich (8.1.25)
• The new version was built on 2025-04-21
• The results of the new version appear to be the same as the 2025-01-05 run.
• It is curently not understood what is causing the variances.
• There are no comparison graphs for the new version and the mpas-bundle 8.0.2 baseline.

GPU runs

1. 2025-05-30 entry compares the GPU run of 2025-05-30 with the cpu run of 2025-05-29. The results diverge.
2. 2025-05-31 entry compares the GPU run of 2025-05-31 with the GPU run of 2025-05-30.
3. 2025-06-01 entry compares the cpu run of 2025-06-01 with the GPU run of 2025-05-31. The results diverge.
4. 2025-06-15 entry compares the cpu run of 2025-06-15 with the GPU run of 2025-06-01. The results diverge.