A bricolage-style exploratory scenario analysis to manage uncertainty in socio-environmental systems modeling: investigating integrated water management options
Article Full Text (PDF)
Model

Supplementary Files

Supplementary Material (PDF)

Keywords

exploratory analysis
exploratory modeling
water management
groundwater
integration

How to Cite

Fu, B., Guillaume, J. H., Jakeman, A. J., & Asher, M. J. (2020). A bricolage-style exploratory scenario analysis to manage uncertainty in socio-environmental systems modeling: investigating integrated water management options. Socio-Environmental Systems Modelling, 2, 16227. https://doi.org/10.18174/sesmo.2020a16227

Abstract

Exploratory analysis, while useful in assessing the implications of model assumptions under large uncertainty, is considered at best a semi-structured activity. There is no algorithmic way for performing exploratory analysis and the existing canonical techniques have their own limitations. To overcome this, we advocate a bricolage-style exploratory scenario analysis, which can be crafted by pragmatically and strategically combining different methods and practices. Our argument is illustrated using a case study in integrated water management in the Murray-Darling Basin, Australia. Scenario ensembles are generated to investigate potential policy innovations, climate and crop market conditions, as well as the effects of uncertainties in model components and parameters. Visualizations, regression trees and marginal effect analyses are exploited to make sense of the ensemble of scenarios. The analysis includes identifying patterns within a scenario ensemble, by visualizing initial hypotheses that are informed by prior knowledge, as well as by visualizing new hypotheses based on identified influential variables. Context-specific relationships are explored by analyzing which values of drivers and management options influence outcomes. Synthesis is achieved by identifying context-specific solutions to consider as part of policy design. The process of analysis is cast as a process of finding patterns and formulating questions within the ensemble of scenarios that merit further examination, allowing end-users to make the decision as to what underlying assumptions should be accepted, and whether uncertainties have been sufficiently explored. This approach is especially advantageous when the precise intentions of management are still subject to deliberations. By describing the reasoning and steps behind a bricolage-style exploratory analysis, we hope to raise awareness of the value of sharing this kind of (common but not often documented) analysis process, and motivate further work to improve sharing of know-how about bricolage in practice.

Article Full Text (PDF)
Model

References

Badham, J., Elsawah, S., Guillaume, J.H.A., Hamilton, S.H., Hunt, R.J., Jakeman, A.J., Pierce, S.A., Snow, V.O., Babbar-Sebens, M., Fu, B., Gober, P., Hill, M.C., Iwanaga, T., Loucks, D.P., Merritt, W.S., Peckham, S.D., Richmond, A.K., Zare, F., Ames, D., & Bammer. G. (2019). Effective modeling for Integrated Water Resource Management: a guide to contextual practices by phases and steps and future opportunities. Environmental Modelling & Software, 116, 40-56. doi.org/10.1016/j.envsoft.2019.02.013.

Bankes, S. (1993). Exploratory Modeling for Policy Analysis. Operations Research, 41(3), 435-449. doi:10.1287/opre.41.3.435

Bankes, S., Walker, W. E., & Kwakkel, J. H. (2013). Exploratory modeling and analysis. In Encyclopedia of operations research and management science (pp. 532-537): Springer.

Bankes, S. C. (2002). Tools and techniques for developing policies for complex and uncertain systems. Proceedings of the National Academy of Sciences, 99(suppl 3), 7263-7266. doi:10.1073/pnas.092081399

Beck, M. B. (1987). Water quality modeling: a review of the analysis of uncertainty. Water Resources Research, 23(8), 1393-1442. doi:10.1029/WR023i008p01393

Beh, E. H. Y., Maier, H. R., & Dandy, G. C. (2015). Adaptive, multiobjective optimal sequencing approach for urban water supply augmentation under deep uncertainty. Water Resources Research, 51(3), 1529-1551. doi:10.1002/2014WR016254

Ben-Haim, Y. (2006). Info-Gap Decision Theory (Second Edition). Oxford: Academic Press.

Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J.,Marsili-Libelli, S., Newham, L.T.H., Norton, J.P., Perrin, C., Pierce, S.A., Robson, B., Seppelt, R., Voinov, A.A., Fath, B., & Andreassian, V. (2013). Characterising performance of environmental models. Environmental Modelling & Software, 40, 1-20. doi:10.1016/j.envsoft.2012.09.011

Blakers, R., Croke, B., & Jakeman, A. (2011). The influence of model simplicity on uncertainty in the context of surface–groundwater modelling and integrated assessment. Paper presented at the 19th International Congress on Modelling and Simulation, Perth, Australia.

Brown, C., Ghile, Y., Laverty, M., & Li, K. (2012). Decision scaling: Linking bottom-up vulnerability analysis with climate projections in the water sector. Water Resources Research, 48(9), n/a-n/a. doi:10.1029/2011WR011212

Bryant, B. P. (2014). sdtoolkit: Scenario Discovery Tools to Support Robust Decision Making (v2.33-1). Retrieved from cran.r-project.org/web/packages/sdtoolkit/index.html

Bryant, B. P., & Lempert, R. J. (2010). Thinking inside the box: A participatory, computer-assisted approach to scenario discovery. Technological Forecasting and Social Change, 77(1), 34-49. doi:10.1016/j.techfore.2009.08.002

Buurman, J., & Babovic, V. (2016). Adaptation Pathways and Real Options Analysis: An approach to deep uncertainty in climate change adaptation policies. Policy and Society, 35(2), 137-150. doi:10.1016/j.polsoc.2016.05.002

Chiew, F. H. S., Zhou, S. L., & McMahon, T. A. (2003). Use of seasonal streamflow forecasts in water resources management. Journal of Hydrology, 270(1–2), 135-144. doi:10.1016/S0022-1694(02)00292-5

Croke, B., Ticehurst, J. L., Letcher, R., Norton, J., Newham, L., & Jakeman, A. (2007). Integrated assessment of water resources: Australian experiences. Water Resources Management, 21(1), 351-373. doi:10.1007/s11269-006-9057-8

Croke, B. F., & Jakeman, A. J. (2004). A catchment moisture deficit module for the IHACRES rainfall-runoff model. Environmental Modelling & Software, 19(1), 1-5. doi:10.1016/j.envsoft.2003.09.001

D’Angelo, G., & Marzolla, M. (2014). New trends in parallel and distributed simulation: From many-cores to Cloud Computing. Simulation Modelling Practice and Theory, 49, 320-335. doi:10.1016/j.simpat.2014.06.007

Daggupati, P., Pai, N., Ale, S., Douglas-Mankin, K. R., Zeckoski, R. W., Jeong, J., . . . Youssef, M. A. (2015). A recommended calibration and validation strategy for hydrologic and water quality models. Transactions of the ASABE, 58(6), 1705-1719. doi:10.13031/trans.58.10712

De'Ath, G., & Fabricius, K. E. (2000). Classification and regression trees: a powerful yet simple technique for ecological data analysis. Ecology, 81, 3178-3192. doi:10.1890/0012-9658(2000)081[3178:CARTAP]2.0.CO

Fayyad, U. M., Wierse, A., & Grinstein, G. G. (2002). Information visualization in data mining and knowledge discovery: Morgan Kaufmann.

Fu, B., Dyer, F., Kravchenko, A., Dyack, B., Merritt, W., & Scarpa, R. (2017). A note on communicating environmental change for non-market valuation. Ecological Indicators, 72, 165-172. doi:10.1016/j.ecolind.2016.08.018

Fu, B., & Guillaume, J. (2018). A Bricolage Style Exploratory Scenario Analysis to Manage Uncertainty - Analysis Code and Data (Version v1.0). Retrieved from http://doi.org/10.5281/zenodo.1465093

Fu, B., & Guillaume, J. H. (2014). Assessing certainty and uncertainty in riparian habitat suitability models by identifying parameters with extreme outputs. Environmental Modelling & Software, 60, 277-289. doi:10.1016/j.envsoft.2014.06.015

Fu, B., Guillaume, J. H. A., & Jakeman, A. J. (2015). An iterative method for discovering feasible management interventions and targets conjointly using uncertainty visualizations. Environmental Modelling & Software, 71, 159-173. doi:10.1016/j.envsoft.2015.05.017

Gersonius, B., Ashley, R., Jeuken, A., Pathinara, A., & Zevenbergen, C. (2015). Accounting for uncertainty and flexibility in flood risk management: comparing R eal‐I n‐O ptions optimisation and A daptation T ipping P oints. Journal of Flood Risk Management, 8(2), 135-144.

Gil, Y., David, C. H., Demir, I., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., . . . Yu, X. (2016). Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance. Earth and Space Science, 3(10), 388-415. doi:doi:10.1002/2015EA000136

Giuliani, M., & Castelletti, A. (2016). Is robustness really robust? How different definitions of robustness impact decision-making under climate change. Climatic Change, 135(3-4), 409-424. doi:10.1007/s10584-015-1586-9

Giuliani, M., Herman, J., Castelletti, A., & Reed, P. (2014). Many‐objective reservoir policy identification and refinement to reduce policy inertia and myopia in water management. Water Resources Research, 50(4), 3355-3377. doi:10.1002/2013WR014700

Guillaume, J. H. A., Arshad, M., Jakeman, A. J., Jalava, M., & Kummu, M. (2016). Robust discrimination between uncertain management alternatives by iterative reflection on crossover point scenarios: Principles, design and implementations. Environmental Modelling & Software, 83, 326-343. doi:10.1016/j.envsoft.2016.04.005

Guillaume, J. H. A., Kummu, M., Räsänen, T. A., & Jakeman, A. J. (2015). Prediction under uncertainty as a boundary problem: A general formulation using Iterative Closed Question Modelling. Environmental Modelling & Software, 70, 97-112. doi:10.1016/j.envsoft.2015.04.004

Guzman, J. A., Shirmohammadi, A., Sadeghi, A. M., Wang, X., Chu, M. L., Jha, M. K., . . . Hernandez, J. E. (2015). Uncertainty considerations in calibration and validation of hydrologic and water quality models. Transactions of the ASABE, 58(6), 1745-1762. doi:10.13031/trans.58.10710

Haasnoot, M., van Deursen, W. P. A., Guillaume, J. H. A., Kwakkel, J. H., van Beek, E., & Middelkoop, H. (2014). Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways. Environmental Modelling & Software, 60, 99-120. doi:10.1016/j.envsoft.2014.05.020

Hadka, D. (2016). High Performance Computing with the MOEA Framework and Ignite: CreateSpace Independent Publishing Platform.

Hadka, D. (2019). Project Platypus - A Collection of Libraries for Optimization, Data Analysis, and Decision Making. Retrieved from https://github.com/Project-Platypus

Hadka, D., Herman, J., Reed, P., & Keller, K. (2015). An open source framework for many-objective robust decision making. Environmental Modelling & Software, 74, 114-129. doi:10.1016/j.envsoft.2015.07.014

Hall, J. W., Lempert, R. J., Keller, K., Hackbarth, A., Mijere, C., & McInerney, D. J. (2012). Robust Climate Policies Under Uncertainty: A Comparison of Robust Decision Making and Info-Gap Methods. Risk Analysis, 32(10), 1657-1672. doi:10.1111/j.1539-6924.2012.01802.x

Hamarat, C., Kwakkel, J. H., & Pruyt, E. (2013). Adaptive robust design under deep uncertainty. Technological Forecasting and Social Change, 80(3), 408-418. doi:10.1016/j.techfore.2012.10.004

Herman, J., Reed, P., Zeff, H., & Characklis, G. (2015). How Should Robustness Be Defined for Water Systems Planning under Change? Journal of Water Resources Planning and Management, 04015012. doi:10.1061/(ASCE)WR.1943-5452.0000509

Hodges, J. S. (1991). Six (or so) things you can do with a bad model. Operations Research, 39(3), 355-365. doi:10.1287/opre.39.3.355

Hodges, J. S., & Dewar, J. A. (1992). Is it you or your model talking?: A framework for model validation: Rand Santa Monica, CA.

Hothorn, T., Hornik, K., & Zeileis, A. (2006). Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical statistics, 15(3), 651-674. doi:10.1198/106186006X133933

Hu, Y., Garcia-Cabrejo, O., Cai, X., Valocchi, A. J., & DuPont, B. (2015). Global sensitivity analysis for large-scale socio-hydrological models using Hadoop. Environmental Modelling & Software, 73, 231-243. doi:10.1016/j.envsoft.2015.08.015

Ivkovic, K., Croke, B., Letcher, R., & Evans, W. (2005). The development of a simple model to investigate the impact of groundwater extraction on river flows in the Namoi catchment, NSW, Australia. Paper presented at the Proceedings of New Zealand Hydrological Society-IAH-NSSSS Conference 2005.

Jakeman, A., & Hornberger, G. (1993). How much complexity is warranted in a rainfall‐runoff model? Water Resources Research, 29(8), 2637-2649. doi:10.1029/93WR00877

Jakeman, A., Littlewood, I., & Whitehead, P. (1990). Computation of the instantaneous unit hydrograph and identifiable component flows with application to two small upland catchments. Journal of Hydrology, 117(1-4), 275-300. doi:10.1016/0022-1694(90)90097-H

Jaxa-Rozen, M., & Kwakkel, J. H. (2018). PyNetLogo: Linking NetLogo with Python. Journal of Artificial Societies and Social Simulation, 21(2), 4. doi:10.18564/jasss.3668

Kasprzyk, J., Guillaume, J., Kollat, J., & Danilo, C. (2014, 2014). Hypothesis Testing for Management: Evolving and Answering Closed Questions Using Multiobjective Visualization. Paper presented at the International Congress on Environmental Modelling and Software, San Diego, California, USA.

Kasprzyk, J. R., Nataraj, S., Reed, P. M., & Lempert, R. J. (2013). Many objective robust decision making for complex environmental systems undergoing change. Environmental Modelling & Software, 42, 55-71. doi:10.1016/j.envsoft.2012.12.007

Kehrer, J., & Hauser, H. (2013). Visualization and Visual Analysis of Multifaceted Scientific Data: A Survey. IEEE Transactions on Visualization and Computer Graphics, 19(3), 495-513. doi:10.1109/TVCG.2012.110

Kincheloe, J. L. (2001). Describing the Bricolage: Conceptualizing a New Rigor in Qualitative Research. Qualitative inquiry, 7(6), 679-692. doi:10.1177/107780040100700601

Kincheloe, J. L. (2005). On to the next level: Continuing the conceptualization of the bricolage. Qualitative inquiry, 11(3), 323-350. doi:10.1177/1077800405275056

Kwadijk, J. C., Haasnoot, M., Mulder, J. P., Hoogvliet, M., Jeuken, A., van der Krogt, R. A., . . . van Waveren, H. (2010). Using adaptation tipping points to prepare for climate change and sea level rise: a case study in the Netherlands. Wiley Interdisciplinary Reviews: Climate Change, 1(5), 729-740. doi:10.1002/wcc.64

Kwakkel, J. H. (2017). The Exploratory Modeling Workbench: An open source toolkit for exploratory modeling, scenario discovery, and (multi-objective) robust decision making. Environmental Modelling & Software, 96, 239-250. doi:10.1016/j.envsoft.2017.06.054

Kwakkel, J. H., Eker, S., & Pruyt, E. (2016). How Robust is a Robust Policy? Comparing Alternative Robustness Metrics for Robust Decision-Making. In M. Doumpos, C. Zopounidis, & E. Grigoroudis (Eds.), Robustness Analysis in Decision Aiding, Optimization, and Analytics (pp. 221-237). Cham: Springer International Publishing.

Kwakkel, J. H., & Haasnoot, M. (2019). Supporting DMDU: A taxonomy of approaches and tools. In Decision Making under Deep Uncertainty (pp. 355-374): Springer.

Kwakkel, J. H., Haasnoot, M., & Walker, W. E. (2015). Developing dynamic adaptive policy pathways: a computer-assisted approach for developing adaptive strategies for a deeply uncertain world. Climatic Change, 132(3), 373-386. doi:10.1007/s10584-014-1210-4

Kwakkel, J. H., Haasnoot, M., & Walker, W. E. (2016). Comparing robust decision-making and dynamic adaptive policy pathways for model-based decision support under deep uncertainty. Environmental Modelling & Software, 86, 168-183. doi:10.1016/j.envsoft.2016.09.017

Kwakkel, J. H., & Pruyt, E. (2013). Exploratory Modeling and Analysis, an approach for model-based foresight under deep uncertainty. Technological Forecasting and Social Change, 80(3), 419-431. doi:10.1016/j.techfore.2012.10.005

Lahtinen, T. J., Guillaume, J. H., & Hämäläinen, R. P. (2017). Why pay attention to paths in the practice of environmental modelling? Environmental Modelling & Software, 92, 74-81. doi:10.1016/j.envsoft.2017.02.019

Landry, M., Banville, C., & Oral, M. (1996). Model legitimisation in operational research. European Journal of Operational Research, 92(3), 443-457. doi:10.1016/0377-2217(96)00003-3

Lempert, R. J. (2002). A new decision sciences for complex systems. Proceedings of the National Academy of Sciences, 99(suppl 3), 7309-7313. doi:10.1073/pnas.082081699

Lempert, R. J., Bryant, B. P., & Bankes, S. C. (2008). Comparing algorithms for scenario discovery. RAND, Santa Monica, CA.

Lempert, R. J., & Collins, M. T. (2007). Managing the risk of uncertain threshold responses: comparison of robust, optimum, and precautionary approaches. Risk Analysis, 27(4), 1009-1026. doi:10.1111/j.1539-6924.2007.00940.x

Lempert, R. J., Popper, S. W., & Bankes, S. C. (2003). Shaping the next one hundred years: new methods for quantitative, long-term policy analysis (0833034855). Retrieved from Santa Monica, California, USA:

Letcher, R., Jakeman, A., & Croke, B. (2004). Model development for integrated assessment of water allocation options. Water Resources Research, 40(5). doi:10.1029/2003WR002933

Maier, H. R., Guillaume, J. H. A., van Delden, H., Riddell, G. A., Haasnoot, M., & Kwakkel, J. H. (2016). An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together? Environmental Modelling & Software, 81, 154-164. doi:10.1016/j.envsoft.2016.03.014

Matrosov, E. S., Padula, S., & Harou, J. J. (2013). Selecting portfolios of water supply and demand management strategies under uncertainty—contrasting economic optimisation and ‘robust decision making’approaches. Water Resources Management, 27(4), 1123-1148. doi:10.1007/s11269-012-0118-x

McPhail, C., Maier, H., Kwakkel, J., Giuliani, M., Castelletti, A., & Westra, S. (2018). Robustness metrics: How are they calculated, when should they be used and why do they give different results? Earth's Future, 6(2), 169-191. doi:10.1002/2017EF000649

Moallemi, E. A., Elsawah, S., & Ryan, M. J. (2018). Model-based multi-objective decision making under deep uncertainty from a multi-method design lens. Simulation Modelling Practice and Theory, 84, 232-250. doi:10.1016/j.simpat.2018.02.009

Moriasi, D. N., Arnold, J. G., Van Liew, M. W., Bingner, R. L., Harmel, R. D., & Veith, T. L. (2007). Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Transactions of the ASABE, 50(3), 885-900. doi:10.13031/2013.23153

Moriasi, D. N., Zeckoski, R. W., Arnold, J. G., Baffaut, C., Malone, R. W., Daggupati, P., . . . Wilson, B. N. (2015). Hydrologic and water quality models: Key calibration and validation topics. Transactions of the ASABE, 58(6), 1609-1618. doi:10.13031/trans.58.11075

Murray–Darling Basin Authority. (2010). Guide to the proposed Basin Plan. Retrieved from Canberra: https://www.mdba.gov.au/publications/archived-information/basin-plan-archives/guide-proposed-basin-plan

Murray Irrigation Limited. (2010). Response to the Guide to the Proposed Murray-Darling Basin Plan. Retrieved from http://www.murrayirrigation.com.au/media/2503/Submission%20to%20the%20MDBA%20re%20Guide%20to%20the%20Proposed%20Murray-Darling%20Basin%20Plan.pdf

Norton, J. (2015). An introduction to sensitivity assessment of simulation models. Environmental Modelling & Software, 69, 166-174. doi:10.1016/j.envsoft.2015.03.020

Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. Science, 263(5147), 641-646. doi:10.1126/science.263.5147.641

Pappenberger, F., & Beven, K. J. (2006). Ignorance is bliss: Or seven reasons not to use uncertainty analysis. Water Resources Research, 42(5), n/a-n/a. doi:10.1029/2005WR004820

Pollino, C., Lester, R., Podger, G., Black, D., & Overton, I. (2011). Analysis of South Australia’s environmental water and water quality requirements and their delivery under the Guide to the proposed Basin Plan. Retrieved from Adelaide:

Refsgaard, J. C., & Knudsen, J. (1996). Operational Validation and Intercomparison of Different Types of Hydrological Models. Water Resources Research, 32(7), 2189-2202. doi:10.1029/96WR00896

Refsgaard, J. C., van der Sluijs, J. P., Højberg, A. L., & Vanrolleghem, P. A. (2007). Uncertainty in the environmental modelling process–a framework and guidance. Environmental Modelling & Software, 22(11), 1543-1556. doi:10.1016/j.envsoft.2007.02.004

Reichert, P., & Borsuk, M. E. (2005). Does high forecast uncertainty preclude effective decision support? Environmental Modelling & Software, 20(8), 991-1001. doi:10.1016/j.envsoft.2004.10.005

Roach, T., Kapelan, Z., & Ledbetter, R. (2015). Comparison of info-gap and robust optimisation methods for integrated water resource management under severe uncertainty. Procedia Engineering, 119, 874-883. doi:10.1016/j.proeng.2015.08.955

Roach, T., Kapelan, Z., Ledbetter, R., & Ledbetter, M. (2016). Comparison of robust optimization and info-gap methods for water resource management under deep uncertainty. Journal of Water Resources Planning and Management, 142(9), 04016028. doi:10.1061/(ASCE)WR.1943-5452.0000660

Roberts, J., & Marston, F. (2011). Water Regime of Wetland and Floodplain Plants: a Source Book for the Murray-Darling Basin. Canberra: National Water Commission.

Rogers, K., & Ralph, T. (2010). Floodplain Wetland Biota in the Murray-Darling Basin. Collingwood: CSIRO Publishing.

Saltelli, A., Chan, K., & Scott, E. M. (2000). Sensitivity analysis (Vol. 1): Wiley New York.

Saltelli, A., Tarantola, S., & Campolongo, F. (2000). Sensitivity analysis as an ingredient of modeling. Statistical Science, 15(4), 377-395.

Schoemaker, P. J. (1995). Scenario planning: a tool for strategic thinking. Sloan management review, 36(2), 25.

Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8(1), 25. doi:10.1186/1471-2105-8-25

Ticehurst, J. L., & Curtis, A. L. (2015). Can existing practices expected to lead to improved on-farm water use efficiency enable irrigators to effectively respond to reduced water entitlements in the Murray–Darling Basin? Journal of Hydrology, 528, 613-620. doi:10.1016/j.jhydrol.2015.06.055

Tsoukiàs, A. (2008). From decision theory to decision aiding methodology. European Journal of Operational Research, 187(1), 138-161.

Tufte, E. R. (2001). The visual display of quantitative information (Vol. 2): Graphics press Cheshire, CT.

Vaishnavi, V. K., & Kuechler, W. (2007). Using Patterns to Illuminate Research Practice. In Design Science Research Methods and Patterns (pp. 57-73): Auerbach Publications.

van Asselt, M., & Rotmans, J. (1996). Uncertainty in perspective. Global Environmental Change, 6(2), 121-157. doi:10.1016/0959-3780(96)00015-5

Walker, W., Haasnoot, M., & Kwakkel, J. (2013). Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty. Sustainability, 5(3), 955. doi:10.3390/su5030955

Weimer-Jehle, W. (2006). Cross-impact balances: A system-theoretical approach to cross-impact analysis. Technological Forecasting and Social Change, 73(4), 334-361. doi:10.1016/j.techfore.2005.06.005

Wickham, H. (2009). ggplot2: Elegant Graphics for Data Analysis: Springer-Verlag New York,.

Williams, B. K. (2011). Passive and active adaptive management: Approaches and an example. Journal of Environmental Management, 92(5), 1371-1378. doi:10.1016/j.jenvman.2010.10.039

Yee, J. S. R. (2010). Methodological Innovation in Practice-Based Design Doctorates. Journal of Research Practice, 6(2), 15-15.

Zhang, M., Yue, P., Wu, Z., Ziebelin, D., Wu, H., & Zhang, C. (2017). Model provenance tracking and inference for integrated environmental modelling. Environmental Modelling & Software, 96, 95-105. doi:https://doi.org/10.1016/j.envsoft.2017.06.051

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright (c) 2020 Baihua Fu, Joseph H.A. Guillaume, Anthony J Jakeman, Michael J. Asher