Model Development, Testing and Validation

Background Comments on Model Evaluation

Forest researchers and managers often have different requirements and expectations from forest simulation models which, in turn, can profoundly influence their thoughts on model development, testing and validation. There is considerable value to promoting a suite of models with different modeling approaches, structures, capabilities and strengths. We have specifically developed SORTIE-ND to be a research model that has considerable flexibility to incorporated new research finding as new behaviours. In this way the model can keep evolving as new research finding emerge. SORTIE-ND can be thought of as a modeling framework for research into neighbourhood dynamics. Our approach is very powerful from a research program and research model development perspective, but has limitations from a management applications perspective. To move from a research model emphasis to a management model emphasis requires extensive testing of model predictions. This must be done using a static version of the model. Adding a new behaviour, for example, in the middle of model testing is a nightmare. This is but one small example of the many issues that must be considered in model development, testing and validation.

How to test and evaluate SORTIE-ND model predictions is an issue we are currently grappling with. If you are interested in this topic (you are after all reading this), then keep checking this site. You can expect to see considerable changes to this web page over the next year or so as we try, as openly as possible, to provide our findings and opinions on model development, testing and validation. We will be testing the sub-boreal spruce forest version of the model developed to simulate forests impacted by the massive mountain pine beetle (MPB) epidemic here in northcentral BC. The MPB epidemic has affected millions of hectares of lodgepole pine dominated stands in the region. Vast areas will never be salvaged resulting in structurally complex stands. How these stand types will perform in the future is critical to timber supply in the region. SORTIE-ND is ideally suited to model such stand types, however, for model predictions to be used in timber supply analysis requires that model predictions are evaluated and that efforts are made to obtain unbiased model predictions. Evaluation is defined as a process in which a model's conceptual structure and predictions are described and assessed with regard to a specific purpose, for example, volume development of MPB affected stands over time. Consequently, this definition encompasses what is often referred to as validation and verification in the modeling community.

Currently, SORTIE-ND is parameterized, but not calibrated. Parameterization is the process in which the parameters in an equation are fitted to a dataset, in this case, data from sub-boreal forests on northern BC. Calibration is the processes in which the predictions from a model are compared with observations and afterwards one or more parameters in the model are changed to produce predictions that match the observations. Most traditional growth and yield models have some kind of calibration performed in order to make the predictions realistic. Calibration, however, goes against the basic ideals of a research model. The danger with calibration is that you will not necessarily allocate the changes to the correct processes. A calibrated model is not a research model. For example, if a research model under predicts growth of one tree species, then this suggests there may be a factor controlling growth of that species that is not understood or for that particular species you have an inadequate dataset, or some combination of the two. Further experimentation or sampling may solve the problem and lead to a better understanding of forest dynamics. Calibration to solve the problem leads to little understanding.

For these reasons we will maintain two versions of parameter files that drive the model: a research parameter file based on parameterization of field data and a management parameter file that will undergo calibration based on comparing model predictions to independent data, for example re-measured permanent sample plots (PSP). For predicting growth for timber supply analysis the calibration of SORTIE-ND is appealing since it could reduce biases in critical predictions, for example, individual tree and stand growth development over time.

Evaluation of the Complex Stand Simulation Model SORTIE-ND for Timber Supply Review in Sub-Boreal Forests of Northern BC
Coates, K. David, Marius Boldor, Erin Hall, Rasmus Astrup. 2009.

This Forest Science Program Project (Y103187) started in April, 2007 and is now in its third and final year ending March, 2010. The study evaluates and assesses the SORTIE-ND model in sub-boreal forests as a timber supply analysis management tool especially for stands impacted by the mountain pine beetle. The linked report briefly summarizes progress to the end of the second year of the project. This FSP project funded the development of this web site.

There are many tests designed for model validation (Yang et al. 2004). Most tests are designed to compare model predictions with independent observations. Since we know a priori that a model is false it makes little sense to test that the two are the same (Reynolds and Chung 1986). Statistical tests will have a limited role in our evaluation. The alternative to statistical tests is statistical estimation and description. Statistical description can be more informative than statistical tests and it leaves the choice of acceptability up to the individual user. We will use different forms of statistical and graphic description to characterize how the SORTIE-ND model predictions conform to independent data. The evaluation proposed in this study will not result in a simple "yes or no" answer.

We will evaluate the model in terms of its logic and conceptual structure to ensure realistic representations of forest stands. This will be done by identifying model structures that result in counterintuitive growth patterns. Model predictions will be compared to general expectations for the stand development in sub-boreal spruce forests. This will be done to identify if obvious structural limitations exist in the model. A sensitivity analysis will also be performed. Sensitivity analysis is an important part of model evaluation (Vanclay and Skovsgaard 1997). This will provide information on the model's sensitivity to uncertainty about the parameter estimates and will indicate the parameters with greatest influence on predictions. The sensitivity analysis will be performed with a Monte Carlo approach where all parameters are varied simultaneously and the parameter value for each model run is determined by a random draw from a distribution (e.g. Lexer and Hönninger 2004). The span of predicted values following a Monte Carlo style sensitivity analysis gives an indication of the error in the model predictions created due to uncertainty about the input parameters, however, this result does not directly yield a ranking of the importance of input parameters. To get at this issue, we will use regression analysis to identify and rank the most important parameters following a Monte Carlo type sensitivity analysis. The results from the sensitivity analysis will be assessed in two steps. First, an analysis will assess the amplitude of variability caused by uncertainty in parameter estimates. This will be performed through visual assessment of species-specific plot results (e.g., basal area, stems/ha). The second step of the analysis will rank the relative importance of parameters on, for example, final basal area or stems/ha. Each of the output variables will be regressed against the predictor variables and the parameters ranked according to R2.

We will compare model predictions will independent observations of growth from permanent sample plots (PSP) located in the sub-boreal spruce zone. This will provide graphic description and summary statistics on the ranges of accuracy and precision of individual tree and stand level predictions. We will use the BC Forest Service PSP database to find independent repeatedly measured plots to compare with SORTIE-ND predictions. Selected plots must: be located in the sub-boreal spruce zone; have data for at least 20-yrs; be dominated by spruce, subalpine fir, lodgepole pine or aspen. SORTIE-ND field studies used to parameterize the model were predominately from mesic sites. Thus, initial comparisons will be based on PSP data and SORTIE-ND predictions on mesic sites followed by dryer and then wetter sites.

Lastly, we will perform mini-sensitivity analysis by altering key parameter values. This will allow us to document the effects of different types of calibration with the intent of making model predictions better align with independent data. Choice of parameters to be altered will be partly based on the sensitivity analysis and partly on knowledge of the model structure. For the calibrated model, we will calculate the summary statistics and redo the graphics that were produced for the non-calibrated model. The change in predictions due to calibration can then be assessed by comparison of summary statistics and graphical products.

Finding the appropriate level of complexity for a simulation model: an example with a forest growth model
Astrup, R., K. D. Coates, and E. Hall. 2008.

In this study, part of FSP Project Y103187, we use five growth functions in SORTIE-ND of increasing complexity based on data from our sub-boreal spruce forest tree growth studies. We predict growth and compare results against independent data. The topic of model complexity is fundamental to model developers and model users. We investigate how over- and under-fitting of a driving function in a simulation model influences the predictive ability of the model and we investigate whether model selection approaches succeed in selecting driving functions with the best predictive ability. Compared to the independent data, the simplest and the most complex growth functions had the poorest predictive ability while functions of intermediate complexity had the best predictive ability. A growth function of intermediate complexity was the most parsimonious model where error due to approximation and error due to estimation were simultaneously minimized.

Models of individual tree mortality for trembling aspen, lodgepole pine, hybrid spruce and subalpine fir in northwestern British Columbia
Pedersen, S.M. 2007.

This thesis was initiated in the summer of 2005 for the project "Modelling individual tree mortality for northern mixed-species stands" (Forest Science Program Project Y061012) by the Department of Forest Sciences at the University of British Columbia. The study used sub-boreal spruce forest sampling sites near Smithers. The objective of this study was to examine density dependent mortality in adult trees with methods that had been previously been used for modelling density dependent mortality for saplings with good results. Mortality was predicted as a function of recent diameter growth and we tested if incorporating tree size into the mortality model improved it. The models were parameterized from field data using a maximum-likelihood method. Field data was gathered from 16 stands comprised of 337 live and 345 recently dead trees in total. Incorporating tree size into the mortality models gave better fits to the field data. Tolerance to low growth decreases to a minimum at intermediate trees size for all species except for subalpine fir, where it decreases and remains low as trees growth larger. Testing the mortality models in SORTIE-ND showed that they contribute to realistic thinning patterns in simulations of both pure even-aged stands and complex stands. However, it was evident that the performance of the mortality models is highly dependent on the underlying growth models as well as mortality models accounting for random mortality. Discrepancies in modelling results were linked to over- and underestimation of growth or inappropriate random mortality rates. Overall the tested method provides a straight forwards approach to parameterizing growth based mortality models from field data which is relatively easy to obtain.

Evaluation of SORTIE-ND for growth prediction in mixed boreal stands
Astrup, R. 2006.

In Chapter 5 of Rasmus Astrup's Ph.D thesis at UBC he performed an evaluation of SORTIE-ND as a growth model for mixed aspen-spruce stands. SORTIE-ND was evaluated in terms of its conceptual structure, a sensitivity analysis was performed, and the model predictions were compared to independent permanent sample plot data. The evaluation suggested that SORTIE-ND is a suitable model for growth prediction in complex mixed-species stands. However, the evaluation also illustrated topics where further model development can improve the models robustness and predictive ability. A major conclusion of this chapter was the critical need for a better crown behaviour (crown diameter and length allometry and the broader topic of crown shyness). The BV Centre has a crown structure project underway to address this issue - see Current Research Projects.

Implications of Alternate Silvicultural Strategies in Mountain Pine Beetle Damaged Stands
Coates, K. David, Erin Hall. 2005.

This report is one of our first attempts to test model performance in sub-boreal forests damaged by the mountain pine beetle (MPB). We also describe our early work to incorporate a robust snag dynamics behaviour into the SORTIE-ND model. This Technical Report is also summarized in an Extension Note. We use the model to simulate future individual tree and stand growth after different silviculture strategies in MPB attacked stands. We selected four stand types to represent MPB susceptible stands: Pine Minor Spruce, Mixed Pine-Spruce, Spruce Minor Pine, and Pine Dominant. Results of model simulations of understory light environments, natural regeneration survival, and effects of underplanting and delaying planting are reported. For comparision, we present TASS (Tree and Stand Simulator) model projections for single species, even-aged plantations.


Use of a spatially explicit individual-tree model (SORTIE/BC) to explore the implications of patchiness in structurally complex forests
Coates, K.D., C.D. Canham, M. Beaudet, D.L. Sachs and C. Messier. 2003.

This paper summarizes the early SORTIE-related research undertaken as part of the modeling component of the Date Creek Experiment. It also includes aspects of early SORTIE-related research in Quebec. We explain how the discipline of silviculture is evolving rapidly, moving from an agricultural model that emphasized simple stand structures toward a natural disturbance- or ecosystem-based model where stands are managed for multiple species and complex structures. We argue that predicting stand dynamics and future yields in mixed-species complex structured stands cannot be easily accomplished with traditional field experimaents. We outline the development and structure of SORTIE/BC, a descendent of the SORTIE model.  We use the early SORTIE/BC model to simulate partial cutting prescriptions in temperate deciduous, boreal and temperate coniferous mixed-species forests.  We conclude that SORTIE/BC can be very useful to explore and explain the silvicultural implications of complex silicultural prescriptions for which there are no long-term experiments.