Appendix B: Model Weighting Strategy

Model Weighting Strategy

Introduction

This document briefly describes a weighting strategy for use with the Climate Model Intercomparison Project, Phase 5 (CMIP5) multimodel archive in the Fourth National Climate Assessment (NCA4). This approach considers both skill in the climatological performance of models over North America and the interdependency of models arising from common parameterizations or tuning practices. The method exploits information relating to the climatological mean state of a number of projection-relevant variables as well as long-term metrics representing long-term statistics of weather extremes. The weights, once computed, can be used to simply compute weighted mean and significance information from an ensemble containing multiple initial condition members from co-dependent models of varying skill.

Our methodology is based on the concepts outlined in Sanderson et al. 2015,¹ and the specific application to the NCA4 is also described in that paper. The approach produces a single set of model weights that can be used to combine projections into a weighted mean result, with significance estimates which also treat the weighting appropriately.

The method, ideally, would seek to have two fundamental characteristics:

• If a duplicate of one ensemble member is added to the archive, the resulting mean and significance estimate for future change computed from the ensemble should not change.

• If a demonstrably unphysical model is added to the archive, the resulting mean and significance estimates should also not change.

Method

The analysis requires an assessment of both model skill and an estimate of intermodel relationships—for which intermodel root mean square difference is taken as a proxy. The model and observational data used here is for the contiguous United States (CONUS), and most of Canada, using high-resolution data where available. Intermodel distances are computed as simple root mean square differences. Data is derived from a number of mean state fields and a number of fields that represent extreme behavior—these are listed in Table B.1. All fields are masked to only include information from CONUS/Canada.

The root mean square error (RMSE) between observations and each model can be used to produce an overall ranking for model simulations of the North American climate. Figure B.1 shows how this metric is influenced by different component variables.

 

Figure 1

A graphical representation of the intermodel distance matrix for CMIP5 and a set of observed values. Each row and column represents a single climate model (or observation). All scores are aggregated over seasons (individual seasons are not shown). Each box represents a pairwise distance, where warm (red) colors indicate a greater distance. Distances are measured as a fraction of the mean intermodel distance in the CMIP5 ensemble. (Figure source: Sanderson et al. 2017² ).

 

Figure 2

Model skill and independence weights for the CMIP5 archive evaluated over the North American domain. Contours show the overall weighting, which is the product of the two individual weights. (Figure source: Sanderson et al. 2017² ).

Models are downweighted for poor skill if their multivariate combined error is significantly greater than a “skill radius” term, which is a free parameter of the approach. The calibration of this parameter is determined through a perfect model study.² A pairwise distance matrix is computed to assess intermodel RMSE values for each model pair in the archive, and a model is downweighted for dependency if there exists another model with a pairwise distance to the original model significantly smaller than a “similarity radius.” This is the second parameter of the approach, which is calibrated by considering known relationships within the archive. The resulting skill and independence weights are multiplied to give an overall “combined” weight—illustrated in Figure B.2 for the CMIP5 ensemble and listed in Table B.2.

The weights are used in the Climate Science Special Report (CSSR) to produce weighted mean and significance maps of future change, where the following protocol is used:

• Stippling—large changes, where the weighted multimodel average change is greater than double the standard deviation of the 20-year mean from control simulations runs, and 90% of the weight corresponds to changes of the same sign.

• Hatching—No significant change, where the weighted multimodel average change is less than the standard deviation of the 20-year means from control simulations runs.

• Whited out—Inconclusive, where the weighted multimodel average change is greater than double the standard deviation of the 20-year mean from control runs and less than 90% of the weight corresponds to changes of the same sign.

 

Figure 3

VIEW

Projections of precipitation change over North America in 2080–2100, relative to 1980–2000 under the higher scenario (RCP8.5). (a) Shows the simple unweighted CMIP5 multimodel average, using the significance methodology from IPCC⁹ ; (b) shows the weighted results as outlined in Section 3 for models weighted by uniqueness only; and (c) shows weighted results for models weighted by both uniqueness and skill. (Figure source: Sanderson et al. 2017² ).

We illustrate the application of this method to future projections of precipitation change under the higher scenario (RCP8.5) in Figure B.3. The weights used in the report are chosen to be conservative, minimizing the risk of overconfidence and maximizing out-of-sample predictive skill for future projections. This results (as in Figure B.3) in only modest differences in the weighted and unweighted maps. It is shown in Sanderson et al. 2017² that a more aggressive weighting strategy, or one focused on a particular variable, tends to exhibit a stronger constraint on future change relative to the unweighted case. It is also notable that tradeoffs exist between skill and replication in the archive (evident in Figure B.2), such that the weighting for both skill and uniqueness has a compensating effect. As such, mean projections using the CMIP5 ensemble are not strongly influenced by the weighting. However, the establishment of the weighting strategy used in the CSSR provides some insurance against a potential case in future assessments where there is a highly replicated, but poorly performing model.

References