epa appeal

Attempts to evaluate CALM’s process of predicting a non-declining yield in jarrah forest

Report to the Conservation Council of WA

Paul Davis BSc (Hons), PhD; Director, Configurable Software Solutions
August 2003

Disclaimer: CALM has not seen a copy of this report, and therefore it has not been reviewed or approved by CALM. I offered to send a draft to CALM for comment, but that offer was declined.

Preamble
Several years ago, I attended an information session presented by CALM, which explained in general terms the basis of the models CALM uses to ascertain growth and increment rates, sustained (or sustainable) yield, and the allowable cut of logs from Western Australia’s State forests. Since then, I have kept abreast of all the available reports, reviews and plans that have involved or used results from the models.

In the past two years, I have been tangentially involved in various FOI applications by the Conservation Council which tried and failed to access sufficient information to evaluate the process whereby the modelling process predicts a sustained yield.

Recently, the Conservation Council and CALM agreed that if CALM made some time available to inform me in an adequately detailed fashion of the workings of the models, I would then be in a position to evaluate the concerns about the sustained yield calculation process.

The meetings
The agreement between CALM and the Conservation Council resulted in three meetings between CALM and me. All meetings were cordial and interesting, if not always as informative as I would have hoped. First CALM presented a general overview of the entire process, and recommended some reading material to refresh my knowledge. The importance of silvicultural practices, and how these are incorporated in the modelling process, was then explained to me. After this, it was agreed that JARSIM was the component of the model where most of my concerns about predictions of sustainability were centred, and that I would therefore be given detailed information about this

The last session was, I understood, meant to be an in-depth exposition on JARSIM at a technical level, so that I, as a software designer and a mathematical ecologist, could appreciate the detail, or at least taste the flavour of the detail, of the model. This meeting was not as informative as I had hoped and expected.

My concerns about CALM’s process of predicting yield in jarrah forest

Fact:
CALM did not show me a single line of computer code, nor a single actual equation. Nor did it permit me to take away with me some graphs which were printed out which showed relationships used by the model. The reasons given for this were that the work was unpublished and therefore couldn’t be made public until it was published; and information could be taken and re-presented elsewhere out of context and therefore erroneously.

The Ferguson Panel recommended publication, stating that: “The [Ferguson] Panel urges that new resources be found to support public education and dissemination (including peer-reviewed publication) programmes.”1

My comment:
I find it difficult to accept or trust CALM’s modelling process because I have not been given sufficient information to assess whether the modelling process is appropriate.

I understand that CALM has refused to provide me the information I sought because that information has not been published. If that information remains unpublished and CALM continues to refuse to provide the information, I, and everyone else outside CALM, will continue to lack the information needed to assess the modelling process.

Fact:
CALM stated that silvicultural practices in their own right guarantee sustainability, and the modelling process simply gives an estimate of a reasonable yield that the available area can provide in any one year, and if it's too high this 10-year period, it can be changed in the next. The 'sustainability' is really because the reserve system ensures that an appreciable area is not logged at all, and even with an area that is to be logged, the silvicultural practices ensure that some areas of it are in fact not logged, for ecological, environmental and social reasons. CALM told me that all the model does (or attempts to do) is determine how much timber the area to be logged might yield.

My comment:
From what I was shown, I believe that CALM’s modelling does not amount to a valid computer or mathematical model predicting a non-declining yield. In my opinion, at the very best it is a modelling process predicting a yield, which may be sustained if the practices are indeed sustainable. My concern here is that the apparent science of a 'computer model' may lend undue credibility to the concept of sustainability possibly derived from a set of practices, and then those practices are used to justify the 'sustainable' prediction of the model! In my view, the practices (sustainable or not) and the model's predictions (sustainable or not) need to be viewed separately so they do not let each apparently justify the other in a circular argument. In my view, the results of the CALM modelling process cannot justify any comment on whether or not the practices as documented are sustainable (which a more ecological model could do), nor can it incorporate the adequacy of the enforcement of those documented practices (which the model apparently assumes to be perfect).

Fact:
CALM’s ‘validation’ of the modelling process relies on the results being deemed ‘reasonable’ by those using the model, or of some comparison of the output of the model compared with that observed for a trial plot.

My comment:
In my opinion this is problematic because if the parameters or structures of the modelling process are simply modified to remove predictions deemed ‘unreasonable’, then the basis of the assumption of reasonability will become critical. A forester, an ecologist, an environmentalist and a logger, for instance, might all have very different definitions of ‘reasonable’, and the legal definition – the opinion of the ordinary person on the Clapham omnibus – is hardly pertinent here. In my view, the process as it has been explained to me may well simply result in a prediction which the designers or operators deem ‘reasonable’. This seems to me to be an entirely subjective opinion, albeit a well informed subjective opinion, apparently (but only apparently) supported by the trappings and mystique of a computer modelling process.

Fact:
I have been told by CALM that frequently the data from trial plots is used to estimate parameters for the model, and then the model output is compared with those very same data. However, it is common scientific practice to compare the output of a model with data that were NOT used in the development of the model.

My comment:
In my opinion, to compare a models output with the data used to estimate its parameters may confirm that the process is a competent numeric mimic, but in no way validates the model’s assumptions, nor its predictions.

Fact:
The literature is rife with inconsistent results from different trial plots and different experimenters.

My comment:
Whilst some inconsistencies can be explained away by untested but ‘reasonable’ assumptions, the possibility that there are major influences not included in the observation and modelling processes cannot be discounted.

Fact:
A significant component of the modelling process (see Appendix A) consists of the Area/Objective Allocation, whereby the outputs of SILVIA and JARSIM are input into FORSCHED. I was told by CALM that this module of the process is not a computer program or system, but a subjective evaluation by a group of knowledgeable individuals.

My comment:
Having been given no insight into this process whatsoever, I cannot comment on it other than I find it difficult to see how it can be anything other than subjective, no matter how experienced, well meaning, well qualified or well informed those individuals are. It seems to me that this is an Achilles heel in any claim of ‘proof’ of sustainability based on a computer modelling process.

Based on the limited information CALM provided me, in my opinion each individual sub-component of the modelling process can both be justified (usually on a best guess basis), and be challenged (usually on a “this is untested” or “this is not based on ecological or biological theory” basis), but that the overall complex mass (morass?) of the modelling process is both unjustifiable and unchallengeable. It is unjustifiable in a theoretical sense because the myriad of assumptions are not externally explicit, some are clearly untrue (even if claimed to be conservative), and any interactions between them are unknown and possibly unknowable; and unchallengeable in a practical sense for pretty much the same reasons. I don’t believe I could accurately say "the model is over-predicting because of x and y". All I can say is "I do not believe the model’s ability to predict is reliable in a scientific sense because much of the 'model' contains untested compromises and unanalysed best guesses massaged by subjective opinion."

Fact:
The average annual sustained yield for 10 years as stated in the proposed Forest Management Plan (pp. 30-31) is:
Jarrah 1st and 2nd grade sawlogs: 131,000 cubic metres
Jarrah bole logs other than 1st and 2nd grade sawlogs: 534,000 cubic metres
Total jarrah logs: 665,000 cubic metres

My comment:
In my opinion, the precautionary principle requires the setting of the cut at the lowest possible socially acceptable level until the most important recommendation of the Ferguson report, “The Panel recommends that alternative approaches be developed to determine the sustainable yields of a range of forest values while maintaining critical elements of ecosystem function”, is implemented.1

Fact:
The entire modelling process (see Appendix A) is formulated as a series of at least six independent processes: five are computer based, four are written in a version of Fortran, three use an Oracle database, and at least two (I think more) are running on the VMS operating system.

My comment:
I do not believe that in 2003 these are appropriate tools to implement an ecological computer model for the following reasons:

Fact:
VMS is a very aged operating system.

My opinion:
If not yet generally unsupported, then it will be generally unsupported soon.

Fact:
Fortran has an on-going value in supplying well tested mathematical and statistical sub-routines, but these sub-routines can also be called from much more sophisticated, more easily maintained and hugely more functional computer languages. My post doctoral fellowship research in the 1980’s involved re-casting a Fortran based bog-growth model into a more modern simulation language.

My opinion:
To find Fortran the language of choice some 25 years later was very surprising to me.

Fact:
Oracle is a well established Relational Data Base Management System often used in commerce in the past decade. Structured Query Language is a language developed, as the name implies, for querying data, not implementing models. There are several newer, arguably more cost-effective and in my opinion in many respects more functional RDBMSs available – indeed, I am currently replacing Oracle for a client with a different RDBMS for commercial, performance and functionality reasons.

My opinion:
I find it very difficult to think that anyone would choose to develop a scientific model in Oracle/SQL unless they had absolutely no other option.

The unconnected modules and complex nature of the process cause what I believe to be the following problems:
• there can be no two-way interaction between the modules;
• many assumptions are implicit, not explicit;
• there may be inconsistent, even contradictory, assumptions in different modules;
• any pathological interaction between module assumptions or formulation would be very difficult to detect.
• testing, sensitivity analysis and optimisation activities can only be carried out on a piecemeal basis;
• an attempt to repeatedly run the modelling process with all possible input variables combined in all possible ways would simply be impossible. But this is precisely what I would like to do to investigate and verify the model's behaviour!

Fact:
The models within the process rely almost completely on estimating relationships between observed variables based on some form of statistical regression analysis, using selected data. I have seen no evidence of a single genuine biological, ecological or physiological assumption implemented in the computer modelling. (I was told that by CALM that such concepts are all subsumed in the belief that predictions based on past observations are valid into the future.

My comment:
I am concerned about the predictions of the model because, in my opinion, the approach described above is questionable when considered in light of the following facts:
• The original choice of trial plots was not randomised.
• The maintenance (or not) of trial plots is also not randomised.
• The acceptance or rejection of data is based on the ‘reasonableness’ of the data. Again, as with the output of the model, true but challenging data could be rejected as part of the process.
• Trial plots have been ‘cut over’ at the most twice, many only once, some not at all. The exact situation is not available to me. But the modelling process is predicting sustainability out for 200 years, that is several more cuts, making the implicit but untested (and to my thinking unlikely) assumption that cutting and re-cutting does not affect yield.
• Factors such as Greenhouse effect (decreased total rainfall, increased summer rainfall, increased temperature, increased carbon dioxide, etc) will have impacts on the jarrah trees and on the jarrah ecosystem - some possibly affecting growth rates positively, some possibly not - but none of these can be allowed for in a model based on past data and minimal biology.
• The model accounts for only the two commercial species, jarrah and marri, and takes absolutely no account of any other component of the ecosystem. The implicit assumption that yield is completely independent of the ecosystem of the jarrah forest may make the modelling simpler, but it is hardly persuasive when presenting a model predicting 200-year sustainability.

Conclusion
In my opinion the CALM model is probably a reasonable short-term logging scheduler, which is what I understand it to have been designed for. It is not, in my opinion, a reasonable long-term sustainability predictor. As I understand it, that is not what it was designed for, but that is what it seems to be being used for. I believe that therein lies a major problem for the Government, industry, the WA community and the forests.

Ferguson, I.F., Adams, M., Bradshaw, J., Davies, S., McCormack, R., Young, J. (2003). “Calculating Sustained Yield for the Forest Management Plan (2004-2013).” Stage 3 Report. Report for the Conservation Commission of Western Australia by the Independent Panel.

Appendix A: Modelling process overview (necessarily brief and incomplete)

1 Area FMIS (also MAPInfo and ArcInfo) Fortran
2 Standing Volume IRIS Fortran/Oracle
3a Available Volume SILVIA Fortran/Oracle
3b Available Volume JARSIM Fortran
4 Area/Objective Allocation People
5 Wood Form FORSCHED SQL/Oracle

Copies of this report are available from the Conservation Council of WA call 9420 7266