Abstract
Reviews suggest that evaluation of land system models is largely inadequate, with undue reliance on a vague concept of validation. Efforts to improve and standardise evaluation practices have so far had limited effect. In this article we examine the issues surrounding land system model evaluation and consider the relevance of the TRACE framework for environmental model documentation. In doing so, we discuss the application of a comprehensive range of evaluation procedures to existing models, and the value of each specific procedure. We develop a tiered checklist for going beyond what seems to be a common practice of ‘valibration’ (the repeated variation of model parameter values to achieve agreement with data) to achieving ‘evaludation’ (the rigorous, broad-based assessment of model quality and validity). We propose the Land Use Change – TRACE (LUC-TRACE) model evaludation protocol and argue that engagement with a comprehensive protocol of this kind (even if not this particular one) is valuable in ensuring that land system model results are interpreted appropriately. We also suggest that the main benefit of such formalised structures is to assist the process of critical thinking about model utility, and that the variety of legitimate modelling approaches precludes universal tests of whether a model is ‘valid’. Evaludation is therefore a detailed and subjective process requiring the sustained intellectual engagement of model developers and users.
References
An, L., Grimm, V., Sullivan, A., Turner, B. L., II, Malleson, N., Heppenstall, A., Vincenot, C., Robinson, D., Ye, X., Liu, J., Lindkvist, E., & Tang, W. (2021). Challenges, tasks, and opportunities in modeling agent-based complex systems. Ecological Modelling, 457, 109685. https://doi.org/10.1016/j.ecolmodel.2021.109685
Angione, C., Silverman, E., & Yaneske, E. (2022). Using machine learning as a surrogate model for agent-based simulations. PloS One, 17(2), e0263150. https://doi.org/10.1371/journal.pone.0263150
Arneth, A., Brown, C., & Rounsevell, M. D. A. (2014). Global models of human decision-making for land-based mitigation and adaptation assessment. Nature Climate Change, 4(7), 550–557. https://doi.org/10.1038/nclimate2250
Arsenault, R., Brissette, F., & Martel, J.-L. (2018). The hazards of split-sample validation in hydrological model calibration. Journal of Hydrology, 566, 346–362. https://doi.org/10.1016/j.jhydrol.2018.09.027
Augusiak, J., Van den Brink, P. J., & Grimm, V. (2014). Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach. Ecological Modelling, 280, 117–128. https://doi.org/10.1016/j.ecolmodel.2013.11.009
Ayllón, D., Railsback, S. F., Gallagher, C., Augusiak, J., Baveco, H., Berger, U., Charles, S., Martin, R., Focks, A., Galic, N., Liu, C., van Loon, E. E., Nabe-Nielsen, J., Piou, C., Polhill, J. G., Preuss, T. G., Radchuk, V., Schmolke, A., Stadnicka-Michalak, J., … Grimm, V. (2021). Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support. Environmental Modelling & Software, 136, 104932. https://doi.org/10.1016/j.envsoft.2020.104932
Baldos, C., & Hertel, T. W. (2013). Looking back to move forward on model validation: insights from a global model of agricultural land use Related content Climate adaptation as mitigation: the case of agricultural investments. Environmental Research Letters: ERL [Web Site], 8, 34024. https://doi.org/10.1088/1748-9326/8/3/034024
Barlas, Y. (1996). Formal aspects of model validity and validation in system dynamics. System Dynamics Review, 12(3), 183–210. https://doi.org/10.1002/(sici)1099-1727(199623)12:3<183::aid-sdr103>3.0.co;2-4
Barnaud, C., Le Page, C., Dumrongrojwatthana, P., & Trébuil, G. (2013). Spatial representations are not neutral: Lessons from a participatory agent-based modelling process in a land-use conflict. Environmental Modelling & Software, 45, 150–159. https://doi.org/10.1016/j.envsoft.2011.11.016
Bianchi, F., & Squazzoni, F. (2019). Modelling and social science. In Modelling Transitions (pp. 60–74). https://doi.org/10.4324/9780429056574-5
Boone, R. B., & Galvin, K. A. (2014). Simulation as an approach to social-ecological integration, with an emphasis on agent-based modeling. In Understanding Society and Natural Resources: Forging New Strands of Integration Across the Social Sciences (pp. 179–202). Springer Netherlands. https://doi.org/10.1007/978-94-017-8959-2_9
Brown, C., Alexander, P., Holzhauer, S., & Rounsevell, M. D. A. (2017). Behavioral models of climate change adaptation and mitigation in land-based sectors. Wiley Interdisciplinary Reviews: Climate Change, 8 (2), e448. https://doi.org/10.1002/wcc.448
Brown, C., Brown, K., & Rounsevell, M. (2016). A philosophical case for process-based modelling of land use change. Modeling Earth Systems and Environment, 2(2), 50. https://doi.org/10.1007/s40808-016-0102-1
Brown, C., Holman, I., & Rounsevell, M. (2021). How modelling paradigms affect simulated future land use change. Earth System Dynamics, 12, 211–231. https://doi.org/10.5194/esd-12-211-2021
Brown, C., & Rounsevell, M. (2021). How can social–ecological system models simulate the emergence of social–ecological crises? People and Nature, 3(1), 88–103. https://doi.org/10.1002/pan3.10167
Brown, C., Seo, B., Alexander, P., Burton, V., Chacón-Montalván, E. A., Dunford, R., Merkle, M., Harrison, P. A., Prestele, R., Robinson, E. L., & Rounsevell, M. (2022). Agent‐based modeling of alternative futures in the British land use system. Earth’s Future, 10(11), e2022EF002905. https://doi.org/10.1029/2022ef002905
Brown, C., Seo, B., & Rounsevell, M. (2019). Societal breakdown as an emergent property of large-scale behavioural models of land use change. Earth System Dynamics Discussions, May, 1–49. https://doi.org/10.5194/esd-2019-24
Brown, D. G., Page, S., Riolo, R., Zellner, M., & Rand, W. (2005). Path dependence and the validation of agent‐based spatial models of land use. International Journal of Geographical Information Science: IJGIS, 19(2), 153–174. https://doi.org/10.1080/13658810410001713399
Brown, D. G., Verburg, P. H., Pontius, R. G., & Lange, M. D. (2013). Opportunities to improve impact, integration, and evaluation of land change models. Current Opinion in Environmental Sustainability 5 (5), 452–457. https://doi.org/10.1016/j.cosust.2013.07.012
Burton, V., Metzger, M. J., Brown, C., & Moseley, D. (2018). Green Gold to Wild Woodlands; understanding stakeholder visions for woodland expansion in Scotland. Landscape Ecology, 34, 1693-1713. https://doi.org/10.1007/s10980-018-0674-4
Center for Open Science. (2022). Registered Reports. https://www.cos.io/initiatives/registered-reports
Clemen, T., Lenfers, U. A., Dybulla, J., Ferreira, S. M., Kiker, G. A., Martens, C., & Scheiter, S. (2021). A cross-scale modeling framework for decision support on elephant management in Kruger National Park, South Africa. Ecological Informatics, 62, 101266. https://doi.org/10.1016/j.ecoinf.2021.101266
Collier, N., Hoffman, F. M., Lawrence, D. M., Keppel-Aleks, G., Koven, C. D., Riley, W. J., Mu, M., & Randerson, J. T. (2018). The international land model benchmarking (ILAMB) system: Design, theory, and implementation. Journal of Advances in Modeling Earth Systems, 10(11), 2731–2754. https://doi.org/10.1029/2018ms001354
Edmonds, B. (2017). Different Modelling Purposes. In B. Edmonds & R. Meyer (Eds.), Simulating Social Complexity: A Handbook (pp. 39–58). Springer International Publishing. https://doi.org/10.1007/978-3-319-66948-9_4
Edmonds, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., & Squazzoni, F. (2019). Different modelling purposes. Journal of Artificial Societies and Social Simulation, 22(3), 6. https://doi.org/10.18564/jasss.3993
Edmonds, B., & ní Aodha, L. (2019). Using Agent-Based Modelling to Inform Policy – What Could Possibly Go Wrong? Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11463 LNAI, 1–16. https://doi.org/10.1007/978-3-030-22270-3_1
Epstein, J. M. (1999). Agent-based computational models and generative social science. Complexity, 4(5), 41–60. https://onlinelibrary.wiley.com/doi/abs/10.1002/(SICI)1099-0526(199905/06)4:5%3C41::AID-CPLX9%3E3.0.CO;2-F
Epstein, J. M. (2008). Why Model? Journal of Artificial Societies and Social Simulation, 11(4)12.
Essawy, B. T., Goodall, J. L., Voce, D., Morsy, M. M., Sadler, J. M., Choi, Y. D., Tarboton, D. G., & Malik, T. (2020). A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling & Software, 134, 104753. https://doi.org/10.1016/j.envsoft.2020.104753
Feyerabend, P. (1993). Against Method. Verso. https://play.google.com/store/books/details?id=8y-FVtrKeSYC
Gambhir, A., Butnar, I., Li, P. H., Smith, P., & Strachan, N. (2019). A review of criticisms of integrated assessment models and proposed approaches to address these, through the lens of BECCs. Energies, 12(9), 1–21. https://doi.org/10.3390/en12091747
Gary Polhill, J., Parker, D., Brown, D., & Grimm, V. (2008). Using the ODD protocol for describing three agent-based social simulation models of land-use change. JASSS. https://www.jasss.org/11/2/3.html Journal of Artificial Societies and Social Simulation, 11(2)3.
Giupponi, C., Ausseil, A.-G., Balbi, S., Cian, F., Fekete, A., Gain, A. K., Essenfelder, A. H., Martínez-López, J., Mojtahed, V., Norf, C., & Others. (2022). Integrated modelling of social-ecological systems for climate change adaptation. Socio-Environmental Systems Modelling, 3, 18161–18161. https://doi.org/10.18174/sesmo.18161
Gostoli, U., & Silverman, E. (2020). Sound behavioural theories, not data, is what makes computational models useful. Review of Artificial Societies and Social Simulation. https://rofasss.org/2020/04/22/sound-behavioural-theories/
Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P., & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129–139. https://doi.org/10.1016/j.ecolmodel.2014.01.018
Grimm, V., & Berger, U. (2016). Robustness analysis: Deconstructing computational models for ecological theory and applications. Ecological Modelling, 326, 162–167. https://doi.org/10.1016/j.ecolmodel.2015.07.018
Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768. https://doi.org/10.1016/J.ECOLMODEL.2010.08.019
Grimm, V., & Railsback, S. F. (2012). Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 367(1586), 298–310. https://doi.org/10.1098/rstb.2011.0180
Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A. S. A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M.-S., Stillman, R. A., Thiele, J. C., & Ayllón, D. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. Journal of Artificial Societies and Social Simulation, 23(2), 7. https://doi.org/10.18564/jasss.4259
Groeneveld, J., Müller, B., Buchmann, C. M., Dressler, G., Guo, C., Hase, N., Hoffmann, F., John, F., Klassert, C., Lauf, T., Liebelt, V., Nolzen, H., Pannicke, N., Schulze, J., Weise, H., & Schwarz, N. (2017). Theoretical foundations of human decision-making in agent-based land use models – A review. Environmental Modelling & Software, 87, 39–48. https://doi.org/10.1016/J.ENVSOFT.2016.10.008
Guzy, M., Smith, C., Bolte, J., Hulse, D., & Gregory, S. (2008). Policy Research Using Agent-Based Modeling to Assess Future Impacts of Urban Expansion into Farmlands and Forests. Ecology and Society, 13(1) 37. https://doi.org/10.5751/ES-02388-130137
Hamilton, S. H., Fu, B., Guillaume, J. H. A., Badham, J., Elsawah, S., Gober, P., Hunt, R. J., Iwanaga, T., Jakeman, A. J., Ames, D. P., Curtis, A., Hill, M. C., Pierce, S. A., & Zare, F. (2019). A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental Modelling & Software, 118, 83–98. https://doi.org/10.1016/j.envsoft.2019.04.008
Hamilton, S. H., Pollino, C. A., Stratford, D. S., Fu, B., & Jakeman, A. J. (2022). Fit-for-purpose environmental modeling: Targeting the intersection of usability, reliability and feasibility. Environmental Modelling & Software, 148, 105278. https://doi.org/10.1016/j.envsoft.2021.105278
Hoffman, F. M., Vertenstein, M., Kitabata, H., & White, J. B. (2005). Vectorizing the Community Land Model. The International Journal of High Performance Computing Applications, 19(3), 247–260. https://doi.org/10.1177/1094342005056113
Holman, I. P., Harrison, P. A., & Metzger, M. J. (2016). Cross-sectoral impacts of climate and socio-economic change in Scotland: implications for adaptation policy. Regional Environmental Change, 16(1), 97–109. https://doi.org/10.1007/s10113-014-0679-8
Holzworth, D., Huth, N. I., Fainges, J., Brown, H., Zurcher, E., Cichota, R., Verrall, S., Herrmann, N. I., Zheng, B., & Snow, V. (2018). APSIM Next Generation: Overcoming challenges in modernising a farming systems model. Environmental Modelling & Software, 103, 43–51. https://doi.org/10.1016/j.envsoft.2018.02.002
Huber, R., Bakker, M., Balmann, A., Berger, T., Bithell, M., Brown, C., Grêt-Regamey, A., Xiong, H., Le, Q. B., Mack, G., Meyfroidt, P., Millington, J., Müller, B., Polhill, J. G., Sun, Z., Seidl, R., Troost, C., & Finger, R. (2018). Representation of decision-making in European agricultural agent-based models. Agricultural Systems, 167, 143–160. https://doi.org/10.1016/j.agsy.2018.09.007
Hughes, H. R., & Paterson, M. (2017). Narrowing the climate field: The symbolic power of authors in the IPCC’s assessment of mitigation. The Review of Policy Research, 34(6), 744–766. https://doi.org/10.1111/ropr.12255
Hüning, C., Adebahr, M., Thiel-Clemen, T., Dalski, J., Lenfers, U., & Grundmann, L. (2016). Modeling & simulation as a service with the massive multi-agent system MARS. Proceedings of the Agent-Directed Simulation Symposium, 1, 1–8. https://dl.acm.org/doi/10.5555/2972193.2972194
Hunka, A. D., Meli, M., Thit, A., Palmqvist, A., Thorbek, P., & Forbes, V. E. (2013). Stakeholders’ Perspective on Ecological Modeling in Environmental Risk Assessment of Pesticides: Challenges and Opportunities. Risk Analysis: An Official Publication of the Society for Risk Analysis, 33(1), 68–79. https://doi.org/10.1111/j.1539-6924.2012.01835.x
Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., Voinov, A., Polhill, G., Sun, Z., & Parker, D. C. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4)4. https://doi.org/10.18564/jasss.2897
Lenfers, U. A., Ahmady-Moghaddam, N., Glake, D., Ocker, F., Osterholz, D., Ströbele, J., & Clemen, T. (2021). Improving Model Predictions—Integration of Real-Time Sensor Data into a Running Simulation of an Agent-Based Model. Sustainability: Science Practice and Policy, 13(13), 7000. https://doi.org/10.3390/su13137000
Ligmann-Zielinska, A., Siebers, P.-O., Magliocca, N., Parker, D. C., Grimm, V., Du, J., Cenek, M., Radchuk, V., Arbab, N. N., Li, S., Berger, U., Paudel, R., Robinson, D. T., Jankowski, P., An, L., & Ye, X. (2020). ‘one size does not fit all’: A roadmap of purpose-driven mixed-method pathways for sensitivity analysis of agent-based models. Journal of Artificial Societies and Social Simulation, 23(1), 6. https://doi.org/10.18564/jasss.4201
Lorscheid, I., Berger, U., Grimm, V., & Meyer, M. (2019). From cases to general principles: A call for theory development through agent-based modeling. Ecological Modelling, 393, 153–156. https://doi.org/10.1016/j.ecolmodel.2018.10.006
Low, S., & Schäfer, S. (2020). Is bio-energy carbon capture and storage (BECCS) feasible? The contested authority of integrated assessment modeling. Energy Research and Social Science, 60, 101326. https://doi.org/10.1016/j.erss.2019.101326
Magliocca, N. R., Brown, D. G., & Ellis, E. C. (2014). Cross-site comparison of land-use decision-making and its consequences across land systems with a generalized agent-based model. PloS One, 9(1), e86179. https://doi.org/10.1371/journal.pone.0086179
Malek, Ž., & Verburg, P. H. (2020). Mapping global patterns of land use decision-making. Global Environmental Change: Human and Policy Dimensions, 65, 102170. https://doi.org/10.1016/j.gloenvcha.2020.102170
McCulloch, J., Ge, J., Ward, J. A., Heppenstall, A., Polhill, J. G., & Malleson, N. (2022). Calibrating agent-based models using uncertainty quantification methods. Journal of Artificial Societies and Social Simulation, 25 (2), 1. https://doi.org/10.18564/jasss.4791
Millington, J. D. A., Demeritt, D., & Romero-Calcerrada, R. (2011). Participatory evaluation of agent-based land-use models. Journal of Land Use Science, 6(2–3), 195–210. https://doi.org/10.1080/1747423X.2011.558595
Millington, J. D. A., Katerinchuk, V., Bicudo da Silva, R. F., de Castro Victoria, D., & Batistella, M. (2021). Modelling drivers of Brazilian agricultural change in a telecoupled world. Environmental Modelling & Software, 105024. https://doi.org/10.1016/j.envsoft.2021.105024
Millington, J. D. A., & Wainwright, J. (2017). Mixed qualitative-simulation methods: Understanding geography through thick and thin. Progress in Human Geography, 41(1), 68–88. https://doi.org/10.1177/0309132515627021
Mohd, M. H. (2022). Revisiting discrepancies between stochastic agent-based and deterministic models. Community Ecology, 23(3), 453–468. https://doi.org/10.1007/s42974-022-00118-2
Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., Schlüter, M., Schulze, J., Weise, H., & Schwarz, N. (2013). Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48. https://doi.org/10.1016/j.envsoft.2013.06.003
Murray-Rust, D., Brown, C., van Vliet, J., Alam, S. J., Robinson, D. T., Verburg, P. H., & Rounsevell, M. (2014). Combining agent functional types, capitals and services to model land use dynamics. Environmental Modelling & Software, 59, 187–201. https://doi.org/10.1016/j.envsoft.2014.05.019
Naivinit, W., Le Page, C., Trébuil, G., & Gajaseni, N. (2010). Participatory agent-based modeling and simulation of rice production and labor migrations in Northeast Thailand. Environmental Modelling & Software, 25(11), 1345–1358. https://doi.org/10.1016/j.envsoft.2010.01.012
Newland, C. P., Maier, H. R., Zecchin, A. C., Newman, J. P., & van Delden, H. (2018). Multi-objective optimisation framework for calibration of Cellular Automata land-use models. Environmental Modelling and Software, 100, 175–200. https://doi.org/10.1016/j.envsoft.2017.11.012
Ngo, The An, & See, L. (2012). Calibration and Validation of Agent-Based Models of Land Cover Change. In A. J. Heppenstall, A. T. Crooks, L. M. See, & M. Batty (Eds.), Agent-Based Models of Geographical Systems (pp. 181–197). Springer Netherlands. https://doi.org/10.1007/978-90-481-8927-4_10
Niamir, L., Ivanova, O., & Filatova, T. (2020). Economy-wide impacts of behavioral climate change mitigation: Linking agent-based and computable general equilibrium models. Environmental Modelling & Software, 134, 104839. https://doi.org/10.1016/j.envsoft.2020.104839
Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), 641–646. https://doi.org/10.1126/science.263.5147.641
O’Sullivan, D., Evans, T., Manson, S., Metcalf, S., Ligmann-Zielinska, A., & Bone, C. (2016). Strategic directions for agent-based modeling: avoiding the YAAWN syndrome. Journal of Land Use Science, 11(2), 177–187. https://doi.org/10.1080/1747423X.2015.1030463
Pedde, S., Kok, K., Hölscher, K., Frantzeskaki, N., Holman, I., Dunford, R., Smith, A., & Jäger, J. (2019). Advancing the use of scenarios to understand society’s capacity to achieve the 1.5 degree target. Global Environmental Change: Human and Policy Dimensions, 56, 75–85. https://doi.org/10.1016/J.GLOENVCHA.2019.03.010
Polhill, G. (2018). Why the social simulation community should tackle prediction. Review of Artificial Societies and Social Simulation. https://rofasss.org/2018/08/17/gp/
Polhill, G., Izquierdo, L. R., & Gotts, N. M. (2005). The Ghost in the Model (and Other Effects of Floating Point Arithmetic). Journal of Artificial Societies and Social Simulation, 8(1), 1–24. http://jasss.soc.surrey.ac.uk/8/1/5.html
Polhill, Gary, & Salt, D. (2017). The Importance of ontological structure: Why validation by ‘fit-to-data’ is insufficient. In Understanding Complex Systems (pp. 141–172). Springer Verlag. https://doi.org/10.1007/978-3-319-66948-9_8
Railsback, S. F., & Grimm, V. (2019). Agent-Based and Individual-Based Modeling: A Practical Introduction. 2nd Edition. Princeton University Press. https://play.google.com/store/books/details?id=VQOjinaRG9cC
Rand, W., Brown, D. G., Page, S. E., Riolo, R., Fernandez, L. E., & Zellner, M. (2003). Statistical validation of spatial patterns in agent-based models. Proceedings of Agent Based Simulation 2003, Vol. 4. https://pdfs.semanticscholar.org/c2da/2de18efa2300bf3cdfdf86b51963480d1804.pdf
Roe, S., Streck, C., Obersteiner, M., Frank, S., Griscom, B., Drouet, L., Fricko, O., Gusti, M., Harris, N., Hasegawa, T., Hausfather, Z., Havlík, P., House, J., Nabuurs, G. J., Popp, A., Sánchez, M. J. S., Sanderman, J., Smith, P., Stehfest, E., & Lawrence, D. (2019). Contribution of the land sector to a 1.5 °C world. Nature Climate Change, 9(11), 817–828. https://doi.org/10.1038/s41558-019-0591-9
Rogelj, J., Shindell, D., Jiang, K., Fifita, S., Forster, P., Ginzburg, V., Handa, C., Kheshgi, H., Kobayashi, S., Kriegler, E., Mundaca, L., Séférian, R., & Vilariño, M. V. (2018). Mitigation Pathways Compatible with 1.5°C in the Context of Sustainable Development. In V. Masson-Delmotte, P. Zhai, H.-O. Pörtner, D. Roberts, P. R. Skea, J. Shukla, A. Pirani, W. Moufouma-Okia, C. Péan, R. Pidcock, S. Connors, J. B. R. Matthews, Y. Chen, X. Zhou, M. I. Gomis, E. Lonnoy, T. Maycock, V. Tignor, & T. Waterfield (Eds.), Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, (p. 82). IPCC Secretariat. https://www.ipcc.ch/site/assets/uploads/sites/2/2019/02/SR15_Chapter2_Low_Res.pdf
Rosa, I. M. D., Ahmed, S. E., & Ewers, R. M. (2014). The transparency, reliability and utility of tropical rainforest land-use and land-cover change models. Global Change Biology, 20(6), 1707–1722. https://doi.org/10.1111/gcb.12523
Rounsevell, M., Arneth, A., Brown, C., Cheung, W. W. L., Gimenez, O., Holman, I., Leadley, P., Luján, C., Mahevas, S., Maréchaux, I., Pélissier, R., Verburg, P. H., Vieilledent, G., Wintle, B. A., & Shin, Y.-J. (2021). Identifying uncertainties in scenarios and models of socio-ecological systems in support of decision-making. One Earth, 4(7), 967–985. https://doi.org/10.1016/j.oneear.2021.06.003
Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019). Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environmental Modelling & Software, 114, 29–39. https://doi.org/10.1016/j.envsoft.2019.01.012
Schmolke, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: a strategy for the future. Trends in Ecology & Evolution, 25(8), 479–486. https://doi.org/10.1016/J.TREE.2010.05.001
Schulze, J., Müller, B., Groeneveld, J., & Grimm, V. (2017). Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward. Journal of Artificial Societies and Social Simulation, 20(2), 8. https://doi.org/10.18564/jasss.3423
Schwarz, N., Dressler, G., Frank, K., Jager, W., Janssen, M., Müller, B., Schlüter, M., Wijermans, N., & Groeneveld, J. (2020). Formalising theories of human decision-making for agent-based modelling of social-ecological systems: practical lessons learned and ways forward. Socio-Environmental Systems Modelling, 2, 16340. https://doi.org/10.18174/sesmo.2020a16340
Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. https://doi.org/10.1098/rsos.160384
Sohl, T. L., & Claggett, P. R. (2013). Clarity versus complexity: Land-use modeling as a practical tool fordecision-makers. In Journal of Environmental Management, 129, 235–243. https://doi.org/10.1016/j.jenvman.2013.07.027
Srikrishnan, V., & Keller, K. (2021). Small increases in agent-based model complexity can result in large increases in required calibration data. Environmental Modelling & Software, 138, 104978. https://doi.org/10.1016/j.envsoft.2021.104978
Steel, D. (2007). Across the boundaries: Extrapolation in biology and social science. Oxford, UK: Oxford University Press.
Steinmann, P., Wang, J. R., van Voorn, G. A. K., & Kwakkel, J. H. (2020). Don’t try to predict COVID-19. If you must, use Deep Uncertainty methods. Review of Artificial Societies and Social Simulation. https://rofasss.org/2020/04/17/deep-uncertainty/
Stepanyan, D., Grethe, H., Zimmermann, G., Siddig, K., Deppermann, A., Feuerbacher, A., Luckmann, J., Valin, H., Nishizawa, T., Ermolieva, T., & Havlik, P. (2021). Multiple rotations of Gaussian quadratures: An efficient method for uncertainty analyses in large-scale simulation models. Environmental Modelling & Software, 136, 104929. https://doi.org/10.1016/j.envsoft.2020.104929
Storm, H., Baylis, K., & Heckelei, T. (2020). Machine learning in agricultural and applied economics. European Review of Agricultural Economics, 47(3), 849–892. https://doi.org/10.1093/erae/jbz033
Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., Balbi, S., Nolzen, H., Müller, B., Schulze, J., & Buchmann, C. M. (2016). Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67. https://doi.org/10.1016/j.envsoft.2016.09.006
Taghikhah, F., Filatova, T., & Voinov, A. (2021). Where does theory have it right? A comparison of theory-driven and empirical agent based models. Journal of Artificial Societies and Social Simulation, 24(2), 4. https://doi.org/10.18564/jasss.4573
ten Broeke, G., van Voorn, G., Ligtenberg, A., & Molenaar, J. (2021). The use of surrogate models to Analyse agent-based models. Journal of Artificial Societies and Social Simulation, 24(2), 3. https://doi.org/10.18564/jasss.4530
Tesfatsion, L. (2017). Modeling economic systems as locally-constructive sequential games. Journal of Economic Methodology, 24(4), 384–409. https://doi.org/10.1080/1350178X.2017.1382068
Troost, C., & Berger, T. (2020). Formalising validation? Towards criteria for valid conclusions from agent-based simulation. In A. van Griensven Jiri Nossent Daniel P. Ames (Ed.), 10th International Congress on Environmental Modelling and Software.
Troost, C., Huber, R., Bell, A. R., van Delden, H., Filatova, T., Le, Q. B., Lippe, M., Niamir, L., Polhill, J. G., Sun, Z., & Berger, T. (2023). How to keep it adequate: A protocol for ensuring validity in agent-based simulation. Environmental Modelling & Software, 159, 105559. https://doi.org/10.1016/j.envsoft.2022.105559
Urban, M. C., Travis, J. M. J., Zurell, D., Thompson, P. L., Synes, N. W., Scarpa, A., Peres-Neto, P. R., Malchow, A.-K., James, P. M. A., Gravel, D., De Meester, L., Brown, C., Bocedi, G., Albert, C. H., Gonzalez, A., & Hendry, A. P. (2021). Coding for Life: Designing a Platform for Projecting and Protecting Global Biodiversity. Bioscience 72, 1. https://doi.org/10.1093/biosci/biab099
van Soesbergen, A. (2016). A Review of Land-Use Change Models. Cambridge, UK: UNEP World Conservation Monitoring Centre
van Vliet, J., Bregt, A. K., Brown, D. G., van Delden, H., Heckbert, S., & Verburg, P. H. (2016). A review of current calibration and validation practices in land-change modeling. Environmental Modelling & Software, 82, 174–182. https://doi.org/10.1016/j.envsoft.2016.04.017
Verburg, P. H., Alexander, P., Evans, T., Magliocca, N. R., Malek, Z., Rounsevell, M. D. A., & van Vliet, J. (2019). Beyond land cover change: towards a new generation of land use models. Current Opinion in Environmental Sustainability, 38, 77–85. https://doi.org/10.1016/j.cosust.2019.05.002
Verburg, P. H., Neumann, K., & Nol, L. (2011). Challenges in using land use and land cover data for global change studies. Global Change Biology, 17(2), 974–989. https://doi.org/10.1111/j.1365-2486.2010.02307.x
Vermeer, W. H., Smith, J. D., Wilensky, U., & Brown, C. H. (2022). High-Fidelity Agent-Based Modeling to Support Prevention Decision-Making: an Open Science Approach. Prevention Science: The Official Journal of the Society for Prevention Research, 23(5), 832–843. https://doi.org/10.1007/s11121-021-01319-3
Williams, T. G., Guikema, S. D., Brown, D. G., & Agrawal, A. (2020). Assessing model equifinality for robust policy analysis in complex socio-environmental systems. Environmental Modelling & Software, 104831. https://doi.org/10.1016/j.envsoft.2020.104831
Zhang, J., & Robinson, D. T. (2021). Replication of an agent-based model using the Replication Standard. Environmental Modelling & Software, 139, 105016. https://doi.org/10.1016/j.envsoft.2021.105016
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2023 Calum Brown, James Millington, Mark Rounsevell