Assessing the quality of land system models: moving from valibration to evaludation
Article Full Text (PDF)

Supplementary Files

Supplementary Material (PDF)

Keywords

validation
evaluation
CRAFTY
agent-based model
land use change

How to Cite

Assessing the quality of land system models: moving from valibration to evaludation. (2023). Socio-Environmental Systems Modelling, 5, 18434. https://doi.org/10.18174/sesmo.18434

Abstract

Reviews suggest that evaluation of land system models is largely inadequate, with undue reliance on a vague concept of validation. Efforts to improve and standardise evaluation practices have so far had limited effect. In this article we examine the issues surrounding land system model evaluation and consider the relevance of the TRACE framework for environmental model documentation. In doing so, we discuss the application of a comprehensive range of evaluation procedures to existing models, and the value of each specific procedure. We develop a tiered checklist for going beyond what seems to be a common practice of ‘valibration’ (the repeated variation of model parameter values to achieve agreement with data) to achieving ‘evaludation’ (the rigorous, broad-based assessment of model quality and validity). We propose the Land Use Change – TRACE (LUC-TRACE) model evaludation protocol and argue that engagement with a comprehensive protocol of this kind (even if not this particular one) is valuable in ensuring that land system model results are interpreted appropriately. We also suggest that the main benefit of such formalised structures is to assist the process of critical thinking about model utility, and that the variety of legitimate modelling approaches precludes universal tests of whether a model is ‘valid’. Evaludation is therefore a detailed and subjective process requiring the sustained intellectual engagement of model developers and users.

Article Full Text (PDF)

References

An, L., Grimm, V., Sullivan, A., Turner, B. L., II, Malleson, N., Heppenstall, A., Vincenot, C., Robinson, D., Ye, X., Liu, J., Lindkvist, E., & Tang, W. (2021). Challenges, tasks, and opportunities in modeling agent-based complex systems. Ecological Modelling, 457, 109685. https://doi.org/10.1016/j.ecolmodel.2021.109685

Angione, C., Silverman, E., & Yaneske, E. (2022). Using machine learning as a surrogate model for agent-based simulations. PloS One, 17(2), e0263150. https://doi.org/10.1371/journal.pone.0263150

Arneth, A., Brown, C., & Rounsevell, M. D. A. (2014). Global models of human decision-making for land-based mitigation and adaptation assessment. Nature Climate Change, 4(7), 550–557. https://doi.org/10.1038/nclimate2250

Arsenault, R., Brissette, F., & Martel, J.-L. (2018). The hazards of split-sample validation in hydrological model calibration. Journal of Hydrology, 566, 346–362. https://doi.org/10.1016/j.jhydrol.2018.09.027

Augusiak, J., Van den Brink, P. J., & Grimm, V. (2014). Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach. Ecological Modelling, 280, 117–128. https://doi.org/10.1016/j.ecolmodel.2013.11.009

Ayllón, D., Railsback, S. F., Gallagher, C., Augusiak, J., Baveco, H., Berger, U., Charles, S., Martin, R., Focks, A., Galic, N., Liu, C., van Loon, E. E., Nabe-Nielsen, J., Piou, C., Polhill, J. G., Preuss, T. G., Radchuk, V., Schmolke, A., Stadnicka-Michalak, J., … Grimm, V. (2021). Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support. Environmental Modelling & Software, 136, 104932. https://doi.org/10.1016/j.envsoft.2020.104932

Baldos, C., & Hertel, T. W. (2013). Looking back to move forward on model validation: insights from a global model of agricultural land use Related content Climate adaptation as mitigation: the case of agricultural investments. Environmental Research Letters: ERL [Web Site], 8, 34024. https://doi.org/10.1088/1748-9326/8/3/034024

Barlas, Y. (1996). Formal aspects of model validity and validation in system dynamics. System Dynamics Review, 12(3), 183–210. https://doi.org/10.1002/(sici)1099-1727(199623)12:3<183::aid-sdr103>3.0.co;2-4

Barnaud, C., Le Page, C., Dumrongrojwatthana, P., & Trébuil, G. (2013). Spatial representations are not neutral: Lessons from a participatory agent-based modelling process in a land-use conflict. Environmental Modelling & Software, 45, 150–159. https://doi.org/10.1016/j.envsoft.2011.11.016

Bianchi, F., & Squazzoni, F. (2019). Modelling and social science. In Modelling Transitions (pp. 60–74). https://doi.org/10.4324/9780429056574-5

Boone, R. B., & Galvin, K. A. (2014). Simulation as an approach to social-ecological integration, with an emphasis on agent-based modeling. In Understanding Society and Natural Resources: Forging New Strands of Integration Across the Social Sciences (pp. 179–202). Springer Netherlands. https://doi.org/10.1007/978-94-017-8959-2_9

Brown, C., Alexander, P., Holzhauer, S., & Rounsevell, M. D. A. (2017). Behavioral models of climate change adaptation and mitigation in land-based sectors. Wiley Interdisciplinary Reviews: Climate Change, 8 (2), e448. https://doi.org/10.1002/wcc.448

Brown, C., Brown, K., & Rounsevell, M. (2016). A philosophical case for process-based modelling of land use change. Modeling Earth Systems and Environment, 2(2), 50. https://doi.org/10.1007/s40808-016-0102-1

Brown, C., Holman, I., & Rounsevell, M. (2021). How modelling paradigms affect simulated future land use change. Earth System Dynamics, 12, 211–231. https://doi.org/10.5194/esd-12-211-2021

Brown, C., & Rounsevell, M. (2021). How can social–ecological system models simulate the emergence of social–ecological crises? People and Nature, 3(1), 88–103. https://doi.org/10.1002/pan3.10167

Brown, C., Seo, B., Alexander, P., Burton, V., Chacón-Montalván, E. A., Dunford, R., Merkle, M., Harrison, P. A., Prestele, R., Robinson, E. L., & Rounsevell, M. (2022). Agent‐based modeling of alternative futures in the British land use system. Earth’s Future, 10(11), e2022EF002905. https://doi.org/10.1029/2022ef002905

Brown, C., Seo, B., & Rounsevell, M. (2019). Societal breakdown as an emergent property of large-scale behavioural models of land use change. Earth System Dynamics Discussions, May, 1–49. https://doi.org/10.5194/esd-2019-24

Brown, D. G., Page, S., Riolo, R., Zellner, M., & Rand, W. (2005). Path dependence and the validation of agent‐based spatial models of land use. International Journal of Geographical Information Science: IJGIS, 19(2), 153–174. https://doi.org/10.1080/13658810410001713399

Brown, D. G., Verburg, P. H., Pontius, R. G., & Lange, M. D. (2013). Opportunities to improve impact, integration, and evaluation of land change models. Current Opinion in Environmental Sustainability 5 (5), 452–457. https://doi.org/10.1016/j.cosust.2013.07.012

Burton, V., Metzger, M. J., Brown, C., & Moseley, D. (2018). Green Gold to Wild Woodlands; understanding stakeholder visions for woodland expansion in Scotland. Landscape Ecology, 34, 1693-1713. https://doi.org/10.1007/s10980-018-0674-4

Center for Open Science. (2022). Registered Reports. https://www.cos.io/initiatives/registered-reports

Clemen, T., Lenfers, U. A., Dybulla, J., Ferreira, S. M., Kiker, G. A., Martens, C., & Scheiter, S. (2021). A cross-scale modeling framework for decision support on elephant management in Kruger National Park, South Africa. Ecological Informatics, 62, 101266. https://doi.org/10.1016/j.ecoinf.2021.101266

Collier, N., Hoffman, F. M., Lawrence, D. M., Keppel-Aleks, G., Koven, C. D., Riley, W. J., Mu, M., & Randerson, J. T. (2018). The international land model benchmarking (ILAMB) system: Design, theory, and implementation. Journal of Advances in Modeling Earth Systems, 10(11), 2731–2754. https://doi.org/10.1029/2018ms001354

Edmonds, B. (2017). Different Modelling Purposes. In B. Edmonds & R. Meyer (Eds.), Simulating Social Complexity: A Handbook (pp. 39–58). Springer International Publishing. https://doi.org/10.1007/978-3-319-66948-9_4

Edmonds, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., & Squazzoni, F. (2019). Different modelling purposes. Journal of Artificial Societies and Social Simulation, 22(3), 6. https://doi.org/10.18564/jasss.3993

Edmonds, B., & ní Aodha, L. (2019). Using Agent-Based Modelling to Inform Policy – What Could Possibly Go Wrong? Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11463 LNAI, 1–16. https://doi.org/10.1007/978-3-030-22270-3_1

Epstein, J. M. (1999). Agent-based computational models and generative social science. Complexity, 4(5), 41–60. https://onlinelibrary.wiley.com/doi/abs/10.1002/(SICI)1099-0526(199905/06)4:5%3C41::AID-CPLX9%3E3.0.CO;2-F

Epstein, J. M. (2008). Why Model? Journal of Artificial Societies and Social Simulation, 11(4)12.

Essawy, B. T., Goodall, J. L., Voce, D., Morsy, M. M., Sadler, J. M., Choi, Y. D., Tarboton, D. G., & Malik, T. (2020). A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling & Software, 134, 104753. https://doi.org/10.1016/j.envsoft.2020.104753

Feyerabend, P. (1993). Against Method. Verso. https://play.google.com/store/books/details?id=8y-FVtrKeSYC

Gambhir, A., Butnar, I., Li, P. H., Smith, P., & Strachan, N. (2019). A review of criticisms of integrated assessment models and proposed approaches to address these, through the lens of BECCs. Energies, 12(9), 1–21. https://doi.org/10.3390/en12091747

Gary Polhill, J., Parker, D., Brown, D., & Grimm, V. (2008). Using the ODD protocol for describing three agent-based social simulation models of land-use change. JASSS. https://www.jasss.org/11/2/3.html Journal of Artificial Societies and Social Simulation, 11(2)3.

Giupponi, C., Ausseil, A.-G., Balbi, S., Cian, F., Fekete, A., Gain, A. K., Essenfelder, A. H., Martínez-López, J., Mojtahed, V., Norf, C., & Others. (2022). Integrated modelling of social-ecological systems for climate change adaptation. Socio-Environmental Systems Modelling, 3, 18161–18161. https://doi.org/10.18174/sesmo.18161

Gostoli, U., & Silverman, E. (2020). Sound behavioural theories, not data, is what makes computational models useful. Review of Artificial Societies and Social Simulation. https://rofasss.org/2020/04/22/sound-behavioural-theories/

Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P., & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129–139. https://doi.org/10.1016/j.ecolmodel.2014.01.018

Grimm, V., & Berger, U. (2016). Robustness analysis: Deconstructing computational models for ecological theory and applications. Ecological Modelling, 326, 162–167. https://doi.org/10.1016/j.ecolmodel.2015.07.018

Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768. https://doi.org/10.1016/J.ECOLMODEL.2010.08.019

Grimm, V., & Railsback, S. F. (2012). Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 367(1586), 298–310. https://doi.org/10.1098/rstb.2011.0180

Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A. S. A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M.-S., Stillman, R. A., Thiele, J. C., & Ayllón, D. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. Journal of Artificial Societies and Social Simulation, 23(2), 7. https://doi.org/10.18564/jasss.4259

Groeneveld, J., Müller, B., Buchmann, C. M., Dressler, G., Guo, C., Hase, N., Hoffmann, F., John, F., Klassert, C., Lauf, T., Liebelt, V., Nolzen, H., Pannicke, N., Schulze, J., Weise, H., & Schwarz, N. (2017). Theoretical foundations of human decision-making in agent-based land use models – A review. Environmental Modelling & Software, 87, 39–48. https://doi.org/10.1016/J.ENVSOFT.2016.10.008

Guzy, M., Smith, C., Bolte, J., Hulse, D., & Gregory, S. (2008). Policy Research Using Agent-Based Modeling to Assess Future Impacts of Urban Expansion into Farmlands and Forests. Ecology and Society, 13(1) 37. https://doi.org/10.5751/ES-02388-130137

Hamilton, S. H., Fu, B., Guillaume, J. H. A., Badham, J., Elsawah, S., Gober, P., Hunt, R. J., Iwanaga, T., Jakeman, A. J., Ames, D. P., Curtis, A., Hill, M. C., Pierce, S. A., & Zare, F. (2019). A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental Modelling & Software, 118, 83–98. https://doi.org/10.1016/j.envsoft.2019.04.008

Hamilton, S. H., Pollino, C. A., Stratford, D. S., Fu, B., & Jakeman, A. J. (2022). Fit-for-purpose environmental modeling: Targeting the intersection of usability, reliability and feasibility. Environmental Modelling & Software, 148, 105278. https://doi.org/10.1016/j.envsoft.2021.105278

Hoffman, F. M., Vertenstein, M., Kitabata, H., & White, J. B. (2005). Vectorizing the Community Land Model. The International Journal of High Performance Computing Applications, 19(3), 247–260. https://doi.org/10.1177/1094342005056113

Holman, I. P., Harrison, P. A., & Metzger, M. J. (2016). Cross-sectoral impacts of climate and socio-economic change in Scotland: implications for adaptation policy. Regional Environmental Change, 16(1), 97–109. https://doi.org/10.1007/s10113-014-0679-8

Holzworth, D., Huth, N. I., Fainges, J., Brown, H., Zurcher, E., Cichota, R., Verrall, S., Herrmann, N. I., Zheng, B., & Snow, V. (2018). APSIM Next Generation: Overcoming challenges in modernising a farming systems model. Environmental Modelling & Software, 103, 43–51. https://doi.org/10.1016/j.envsoft.2018.02.002

Huber, R., Bakker, M., Balmann, A., Berger, T., Bithell, M., Brown, C., Grêt-Regamey, A., Xiong, H., Le, Q. B., Mack, G., Meyfroidt, P., Millington, J., Müller, B., Polhill, J. G., Sun, Z., Seidl, R., Troost, C., & Finger, R. (2018). Representation of decision-making in European agricultural agent-based models. Agricultural Systems, 167, 143–160. https://doi.org/10.1016/j.agsy.2018.09.007

Hughes, H. R., & Paterson, M. (2017). Narrowing the climate field: The symbolic power of authors in the IPCC’s assessment of mitigation. The Review of Policy Research, 34(6), 744–766. https://doi.org/10.1111/ropr.12255

Hüning, C., Adebahr, M., Thiel-Clemen, T., Dalski, J., Lenfers, U., & Grundmann, L. (2016). Modeling & simulation as a service with the massive multi-agent system MARS. Proceedings of the Agent-Directed Simulation Symposium, 1, 1–8. https://dl.acm.org/doi/10.5555/2972193.2972194

Hunka, A. D., Meli, M., Thit, A., Palmqvist, A., Thorbek, P., & Forbes, V. E. (2013). Stakeholders’ Perspective on Ecological Modeling in Environmental Risk Assessment of Pesticides: Challenges and Opportunities. Risk Analysis: An Official Publication of the Society for Risk Analysis, 33(1), 68–79. https://doi.org/10.1111/j.1539-6924.2012.01835.x

Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., Voinov, A., Polhill, G., Sun, Z., & Parker, D. C. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4)4. https://doi.org/10.18564/jasss.2897

Lenfers, U. A., Ahmady-Moghaddam, N., Glake, D., Ocker, F., Osterholz, D., Ströbele, J., & Clemen, T. (2021). Improving Model Predictions—Integration of Real-Time Sensor Data into a Running Simulation of an Agent-Based Model. Sustainability: Science Practice and Policy, 13(13), 7000. https://doi.org/10.3390/su13137000

Ligmann-Zielinska, A., Siebers, P.-O., Magliocca, N., Parker, D. C., Grimm, V., Du, J., Cenek, M., Radchuk, V., Arbab, N. N., Li, S., Berger, U., Paudel, R., Robinson, D. T., Jankowski, P., An, L., & Ye, X. (2020). ‘one size does not fit all’: A roadmap of purpose-driven mixed-method pathways for sensitivity analysis of agent-based models. Journal of Artificial Societies and Social Simulation, 23(1), 6. https://doi.org/10.18564/jasss.4201

Lorscheid, I., Berger, U., Grimm, V., & Meyer, M. (2019). From cases to general principles: A call for theory development through agent-based modeling. Ecological Modelling, 393, 153–156. https://doi.org/10.1016/j.ecolmodel.2018.10.006

Low, S., & Schäfer, S. (2020). Is bio-energy carbon capture and storage (BECCS) feasible? The contested authority of integrated assessment modeling. Energy Research and Social Science, 60, 101326. https://doi.org/10.1016/j.erss.2019.101326

Magliocca, N. R., Brown, D. G., & Ellis, E. C. (2014). Cross-site comparison of land-use decision-making and its consequences across land systems with a generalized agent-based model. PloS One, 9(1), e86179. https://doi.org/10.1371/journal.pone.0086179

Malek, Ž., & Verburg, P. H. (2020). Mapping global patterns of land use decision-making. Global Environmental Change: Human and Policy Dimensions, 65, 102170. https://doi.org/10.1016/j.gloenvcha.2020.102170

McCulloch, J., Ge, J., Ward, J. A., Heppenstall, A., Polhill, J. G., & Malleson, N. (2022). Calibrating agent-based models using uncertainty quantification methods. Journal of Artificial Societies and Social Simulation, 25 (2), 1. https://doi.org/10.18564/jasss.4791

Millington, J. D. A., Demeritt, D., & Romero-Calcerrada, R. (2011). Participatory evaluation of agent-based land-use models. Journal of Land Use Science, 6(2–3), 195–210. https://doi.org/10.1080/1747423X.2011.558595

Millington, J. D. A., Katerinchuk, V., Bicudo da Silva, R. F., de Castro Victoria, D., & Batistella, M. (2021). Modelling drivers of Brazilian agricultural change in a telecoupled world. Environmental Modelling & Software, 105024. https://doi.org/10.1016/j.envsoft.2021.105024

Millington, J. D. A., & Wainwright, J. (2017). Mixed qualitative-simulation methods: Understanding geography through thick and thin. Progress in Human Geography, 41(1), 68–88. https://doi.org/10.1177/0309132515627021

Mohd, M. H. (2022). Revisiting discrepancies between stochastic agent-based and deterministic models. Community Ecology, 23(3), 453–468. https://doi.org/10.1007/s42974-022-00118-2

Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., Schlüter, M., Schulze, J., Weise, H., & Schwarz, N. (2013). Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48. https://doi.org/10.1016/j.envsoft.2013.06.003

Murray-Rust, D., Brown, C., van Vliet, J., Alam, S. J., Robinson, D. T., Verburg, P. H., & Rounsevell, M. (2014). Combining agent functional types, capitals and services to model land use dynamics. Environmental Modelling & Software, 59, 187–201. https://doi.org/10.1016/j.envsoft.2014.05.019

Naivinit, W., Le Page, C., Trébuil, G., & Gajaseni, N. (2010). Participatory agent-based modeling and simulation of rice production and labor migrations in Northeast Thailand. Environmental Modelling & Software, 25(11), 1345–1358. https://doi.org/10.1016/j.envsoft.2010.01.012

Newland, C. P., Maier, H. R., Zecchin, A. C., Newman, J. P., & van Delden, H. (2018). Multi-objective optimisation framework for calibration of Cellular Automata land-use models. Environmental Modelling and Software, 100, 175–200. https://doi.org/10.1016/j.envsoft.2017.11.012

Ngo, The An, & See, L. (2012). Calibration and Validation of Agent-Based Models of Land Cover Change. In A. J. Heppenstall, A. T. Crooks, L. M. See, & M. Batty (Eds.), Agent-Based Models of Geographical Systems (pp. 181–197). Springer Netherlands. https://doi.org/10.1007/978-90-481-8927-4_10

Niamir, L., Ivanova, O., & Filatova, T. (2020). Economy-wide impacts of behavioral climate change mitigation: Linking agent-based and computable general equilibrium models. Environmental Modelling & Software, 134, 104839. https://doi.org/10.1016/j.envsoft.2020.104839

Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), 641–646. https://doi.org/10.1126/science.263.5147.641

O’Sullivan, D., Evans, T., Manson, S., Metcalf, S., Ligmann-Zielinska, A., & Bone, C. (2016). Strategic directions for agent-based modeling: avoiding the YAAWN syndrome. Journal of Land Use Science, 11(2), 177–187. https://doi.org/10.1080/1747423X.2015.1030463

Pedde, S., Kok, K., Hölscher, K., Frantzeskaki, N., Holman, I., Dunford, R., Smith, A., & Jäger, J. (2019). Advancing the use of scenarios to understand society’s capacity to achieve the 1.5 degree target. Global Environmental Change: Human and Policy Dimensions, 56, 75–85. https://doi.org/10.1016/J.GLOENVCHA.2019.03.010

Polhill, G. (2018). Why the social simulation community should tackle prediction. Review of Artificial Societies and Social Simulation. https://rofasss.org/2018/08/17/gp/

Polhill, G., Izquierdo, L. R., & Gotts, N. M. (2005). The Ghost in the Model (and Other Effects of Floating Point Arithmetic). Journal of Artificial Societies and Social Simulation, 8(1), 1–24. http://jasss.soc.surrey.ac.uk/8/1/5.html

Polhill, Gary, & Salt, D. (2017). The Importance of ontological structure: Why validation by ‘fit-to-data’ is insufficient. In Understanding Complex Systems (pp. 141–172). Springer Verlag. https://doi.org/10.1007/978-3-319-66948-9_8

Railsback, S. F., & Grimm, V. (2019). Agent-Based and Individual-Based Modeling: A Practical Introduction. 2nd Edition. Princeton University Press. https://play.google.com/store/books/details?id=VQOjinaRG9cC

Rand, W., Brown, D. G., Page, S. E., Riolo, R., Fernandez, L. E., & Zellner, M. (2003). Statistical validation of spatial patterns in agent-based models. Proceedings of Agent Based Simulation 2003, Vol. 4. https://pdfs.semanticscholar.org/c2da/2de18efa2300bf3cdfdf86b51963480d1804.pdf

Roe, S., Streck, C., Obersteiner, M., Frank, S., Griscom, B., Drouet, L., Fricko, O., Gusti, M., Harris, N., Hasegawa, T., Hausfather, Z., Havlík, P., House, J., Nabuurs, G. J., Popp, A., Sánchez, M. J. S., Sanderman, J., Smith, P., Stehfest, E., & Lawrence, D. (2019). Contribution of the land sector to a 1.5 °C world. Nature Climate Change, 9(11), 817–828. https://doi.org/10.1038/s41558-019-0591-9

Rogelj, J., Shindell, D., Jiang, K., Fifita, S., Forster, P., Ginzburg, V., Handa, C., Kheshgi, H., Kobayashi, S., Kriegler, E., Mundaca, L., Séférian, R., & Vilariño, M. V. (2018). Mitigation Pathways Compatible with 1.5°C in the Context of Sustainable Development. In V. Masson-Delmotte, P. Zhai, H.-O. Pörtner, D. Roberts, P. R. Skea, J. Shukla, A. Pirani, W. Moufouma-Okia, C. Péan, R. Pidcock, S. Connors, J. B. R. Matthews, Y. Chen, X. Zhou, M. I. Gomis, E. Lonnoy, T. Maycock, V. Tignor, & T. Waterfield (Eds.), Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, (p. 82). IPCC Secretariat. https://www.ipcc.ch/site/assets/uploads/sites/2/2019/02/SR15_Chapter2_Low_Res.pdf

Rosa, I. M. D., Ahmed, S. E., & Ewers, R. M. (2014). The transparency, reliability and utility of tropical rainforest land-use and land-cover change models. Global Change Biology, 20(6), 1707–1722. https://doi.org/10.1111/gcb.12523

Rounsevell, M., Arneth, A., Brown, C., Cheung, W. W. L., Gimenez, O., Holman, I., Leadley, P., Luján, C., Mahevas, S., Maréchaux, I., Pélissier, R., Verburg, P. H., Vieilledent, G., Wintle, B. A., & Shin, Y.-J. (2021). Identifying uncertainties in scenarios and models of socio-ecological systems in support of decision-making. One Earth, 4(7), 967–985. https://doi.org/10.1016/j.oneear.2021.06.003

Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019). Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environmental Modelling & Software, 114, 29–39. https://doi.org/10.1016/j.envsoft.2019.01.012

Schmolke, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: a strategy for the future. Trends in Ecology & Evolution, 25(8), 479–486. https://doi.org/10.1016/J.TREE.2010.05.001

Schulze, J., Müller, B., Groeneveld, J., & Grimm, V. (2017). Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward. Journal of Artificial Societies and Social Simulation, 20(2), 8. https://doi.org/10.18564/jasss.3423

Schwarz, N., Dressler, G., Frank, K., Jager, W., Janssen, M., Müller, B., Schlüter, M., Wijermans, N., & Groeneveld, J. (2020). Formalising theories of human decision-making for agent-based modelling of social-ecological systems: practical lessons learned and ways forward. Socio-Environmental Systems Modelling, 2, 16340. https://doi.org/10.18174/sesmo.2020a16340

Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. https://doi.org/10.1098/rsos.160384

Sohl, T. L., & Claggett, P. R. (2013). Clarity versus complexity: Land-use modeling as a practical tool fordecision-makers. In Journal of Environmental Management, 129, 235–243. https://doi.org/10.1016/j.jenvman.2013.07.027

Srikrishnan, V., & Keller, K. (2021). Small increases in agent-based model complexity can result in large increases in required calibration data. Environmental Modelling & Software, 138, 104978. https://doi.org/10.1016/j.envsoft.2021.104978

Steel, D. (2007). Across the boundaries: Extrapolation in biology and social science. Oxford, UK: Oxford University Press.

Steinmann, P., Wang, J. R., van Voorn, G. A. K., & Kwakkel, J. H. (2020). Don’t try to predict COVID-19. If you must, use Deep Uncertainty methods. Review of Artificial Societies and Social Simulation. https://rofasss.org/2020/04/17/deep-uncertainty/

Stepanyan, D., Grethe, H., Zimmermann, G., Siddig, K., Deppermann, A., Feuerbacher, A., Luckmann, J., Valin, H., Nishizawa, T., Ermolieva, T., & Havlik, P. (2021). Multiple rotations of Gaussian quadratures: An efficient method for uncertainty analyses in large-scale simulation models. Environmental Modelling & Software, 136, 104929. https://doi.org/10.1016/j.envsoft.2020.104929

Storm, H., Baylis, K., & Heckelei, T. (2020). Machine learning in agricultural and applied economics. European Review of Agricultural Economics, 47(3), 849–892. https://doi.org/10.1093/erae/jbz033

Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., Balbi, S., Nolzen, H., Müller, B., Schulze, J., & Buchmann, C. M. (2016). Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67. https://doi.org/10.1016/j.envsoft.2016.09.006

Taghikhah, F., Filatova, T., & Voinov, A. (2021). Where does theory have it right? A comparison of theory-driven and empirical agent based models. Journal of Artificial Societies and Social Simulation, 24(2), 4. https://doi.org/10.18564/jasss.4573

ten Broeke, G., van Voorn, G., Ligtenberg, A., & Molenaar, J. (2021). The use of surrogate models to Analyse agent-based models. Journal of Artificial Societies and Social Simulation, 24(2), 3. https://doi.org/10.18564/jasss.4530

Tesfatsion, L. (2017). Modeling economic systems as locally-constructive sequential games. Journal of Economic Methodology, 24(4), 384–409. https://doi.org/10.1080/1350178X.2017.1382068

Troost, C., & Berger, T. (2020). Formalising validation? Towards criteria for valid conclusions from agent-based simulation. In A. van Griensven Jiri Nossent Daniel P. Ames (Ed.), 10th International Congress on Environmental Modelling and Software.

Troost, C., Huber, R., Bell, A. R., van Delden, H., Filatova, T., Le, Q. B., Lippe, M., Niamir, L., Polhill, J. G., Sun, Z., & Berger, T. (2023). How to keep it adequate: A protocol for ensuring validity in agent-based simulation. Environmental Modelling & Software, 159, 105559. https://doi.org/10.1016/j.envsoft.2022.105559

Urban, M. C., Travis, J. M. J., Zurell, D., Thompson, P. L., Synes, N. W., Scarpa, A., Peres-Neto, P. R., Malchow, A.-K., James, P. M. A., Gravel, D., De Meester, L., Brown, C., Bocedi, G., Albert, C. H., Gonzalez, A., & Hendry, A. P. (2021). Coding for Life: Designing a Platform for Projecting and Protecting Global Biodiversity. Bioscience 72, 1. https://doi.org/10.1093/biosci/biab099

van Soesbergen, A. (2016). A Review of Land-Use Change Models. Cambridge, UK: UNEP World Conservation Monitoring Centre

van Vliet, J., Bregt, A. K., Brown, D. G., van Delden, H., Heckbert, S., & Verburg, P. H. (2016). A review of current calibration and validation practices in land-change modeling. Environmental Modelling & Software, 82, 174–182. https://doi.org/10.1016/j.envsoft.2016.04.017

Verburg, P. H., Alexander, P., Evans, T., Magliocca, N. R., Malek, Z., Rounsevell, M. D. A., & van Vliet, J. (2019). Beyond land cover change: towards a new generation of land use models. Current Opinion in Environmental Sustainability, 38, 77–85. https://doi.org/10.1016/j.cosust.2019.05.002

Verburg, P. H., Neumann, K., & Nol, L. (2011). Challenges in using land use and land cover data for global change studies. Global Change Biology, 17(2), 974–989. https://doi.org/10.1111/j.1365-2486.2010.02307.x

Vermeer, W. H., Smith, J. D., Wilensky, U., & Brown, C. H. (2022). High-Fidelity Agent-Based Modeling to Support Prevention Decision-Making: an Open Science Approach. Prevention Science: The Official Journal of the Society for Prevention Research, 23(5), 832–843. https://doi.org/10.1007/s11121-021-01319-3

Williams, T. G., Guikema, S. D., Brown, D. G., & Agrawal, A. (2020). Assessing model equifinality for robust policy analysis in complex socio-environmental systems. Environmental Modelling & Software, 104831. https://doi.org/10.1016/j.envsoft.2020.104831

Zhang, J., & Robinson, D. T. (2021). Replication of an agent-based model using the Replication Standard. Environmental Modelling & Software, 139, 105016. https://doi.org/10.1016/j.envsoft.2021.105016

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright (c) 2023 Calum Brown, James Millington, Mark Rounsevell