Assessing the quality of land system models: moving from valibration to evaludation
Article Full Text (PDF)

Supplementary Files

Supplementary Material (PDF)


agent-based model
land use change

How to Cite

Brown, C., Millington, J., & Rounsevell, M. (2023). Assessing the quality of land system models: moving from valibration to evaludation. Socio-Environmental Systems Modelling, 5, 18434.


Reviews suggest that evaluation of land system models is largely inadequate, with undue reliance on a vague concept of validation. Efforts to improve and standardise evaluation practices have so far had limited effect. In this article we examine the issues surrounding land system model evaluation and consider the relevance of the TRACE framework for environmental model documentation. In doing so, we discuss the application of a comprehensive range of evaluation procedures to existing models, and the value of each specific procedure. We develop a tiered checklist for going beyond what seems to be a common practice of ‘valibration’ (the repeated variation of model parameter values to achieve agreement with data) to achieving ‘evaludation’ (the rigorous, broad-based assessment of model quality and validity). We propose the Land Use Change – TRACE (LUC-TRACE) model evaludation protocol and argue that engagement with a comprehensive protocol of this kind (even if not this particular one) is valuable in ensuring that land system model results are interpreted appropriately. We also suggest that the main benefit of such formalised structures is to assist the process of critical thinking about model utility, and that the variety of legitimate modelling approaches precludes universal tests of whether a model is ‘valid’. Evaludation is therefore a detailed and subjective process requiring the sustained intellectual engagement of model developers and users.

Article Full Text (PDF)


An, L., Grimm, V., Sullivan, A., Turner, B. L., II, Malleson, N., Heppenstall, A., Vincenot, C., Robinson, D., Ye, X., Liu, J., Lindkvist, E., & Tang, W. (2021). Challenges, tasks, and opportunities in modeling agent-based complex systems. Ecological Modelling, 457, 109685.

Angione, C., Silverman, E., & Yaneske, E. (2022). Using machine learning as a surrogate model for agent-based simulations. PloS One, 17(2), e0263150.

Arneth, A., Brown, C., & Rounsevell, M. D. A. (2014). Global models of human decision-making for land-based mitigation and adaptation assessment. Nature Climate Change, 4(7), 550–557.

Arsenault, R., Brissette, F., & Martel, J.-L. (2018). The hazards of split-sample validation in hydrological model calibration. Journal of Hydrology, 566, 346–362.

Augusiak, J., Van den Brink, P. J., & Grimm, V. (2014). Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach. Ecological Modelling, 280, 117–128.

Ayllón, D., Railsback, S. F., Gallagher, C., Augusiak, J., Baveco, H., Berger, U., Charles, S., Martin, R., Focks, A., Galic, N., Liu, C., van Loon, E. E., Nabe-Nielsen, J., Piou, C., Polhill, J. G., Preuss, T. G., Radchuk, V., Schmolke, A., Stadnicka-Michalak, J., … Grimm, V. (2021). Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support. Environmental Modelling & Software, 136, 104932.

Baldos, C., & Hertel, T. W. (2013). Looking back to move forward on model validation: insights from a global model of agricultural land use Related content Climate adaptation as mitigation: the case of agricultural investments. Environmental Research Letters: ERL [Web Site], 8, 34024.

Barlas, Y. (1996). Formal aspects of model validity and validation in system dynamics. System Dynamics Review, 12(3), 183–210.<183::aid-sdr103>;2-4

Barnaud, C., Le Page, C., Dumrongrojwatthana, P., & Trébuil, G. (2013). Spatial representations are not neutral: Lessons from a participatory agent-based modelling process in a land-use conflict. Environmental Modelling & Software, 45, 150–159.

Bianchi, F., & Squazzoni, F. (2019). Modelling and social science. In Modelling Transitions (pp. 60–74).

Boone, R. B., & Galvin, K. A. (2014). Simulation as an approach to social-ecological integration, with an emphasis on agent-based modeling. In Understanding Society and Natural Resources: Forging New Strands of Integration Across the Social Sciences (pp. 179–202). Springer Netherlands.

Brown, C., Alexander, P., Holzhauer, S., & Rounsevell, M. D. A. (2017). Behavioral models of climate change adaptation and mitigation in land-based sectors. Wiley Interdisciplinary Reviews: Climate Change, 8 (2), e448.

Brown, C., Brown, K., & Rounsevell, M. (2016). A philosophical case for process-based modelling of land use change. Modeling Earth Systems and Environment, 2(2), 50.

Brown, C., Holman, I., & Rounsevell, M. (2021). How modelling paradigms affect simulated future land use change. Earth System Dynamics, 12, 211–231.

Brown, C., & Rounsevell, M. (2021). How can social–ecological system models simulate the emergence of social–ecological crises? People and Nature, 3(1), 88–103.

Brown, C., Seo, B., Alexander, P., Burton, V., Chacón-Montalván, E. A., Dunford, R., Merkle, M., Harrison, P. A., Prestele, R., Robinson, E. L., & Rounsevell, M. (2022). Agent‐based modeling of alternative futures in the British land use system. Earth’s Future, 10(11), e2022EF002905.

Brown, C., Seo, B., & Rounsevell, M. (2019). Societal breakdown as an emergent property of large-scale behavioural models of land use change. Earth System Dynamics Discussions, May, 1–49.

Brown, D. G., Page, S., Riolo, R., Zellner, M., & Rand, W. (2005). Path dependence and the validation of agent‐based spatial models of land use. International Journal of Geographical Information Science: IJGIS, 19(2), 153–174.

Brown, D. G., Verburg, P. H., Pontius, R. G., & Lange, M. D. (2013). Opportunities to improve impact, integration, and evaluation of land change models. Current Opinion in Environmental Sustainability 5 (5), 452–457.

Burton, V., Metzger, M. J., Brown, C., & Moseley, D. (2018). Green Gold to Wild Woodlands; understanding stakeholder visions for woodland expansion in Scotland. Landscape Ecology, 34, 1693-1713.

Center for Open Science. (2022). Registered Reports.

Clemen, T., Lenfers, U. A., Dybulla, J., Ferreira, S. M., Kiker, G. A., Martens, C., & Scheiter, S. (2021). A cross-scale modeling framework for decision support on elephant management in Kruger National Park, South Africa. Ecological Informatics, 62, 101266.

Collier, N., Hoffman, F. M., Lawrence, D. M., Keppel-Aleks, G., Koven, C. D., Riley, W. J., Mu, M., & Randerson, J. T. (2018). The international land model benchmarking (ILAMB) system: Design, theory, and implementation. Journal of Advances in Modeling Earth Systems, 10(11), 2731–2754.

Edmonds, B. (2017). Different Modelling Purposes. In B. Edmonds & R. Meyer (Eds.), Simulating Social Complexity: A Handbook (pp. 39–58). Springer International Publishing.

Edmonds, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., & Squazzoni, F. (2019). Different modelling purposes. Journal of Artificial Societies and Social Simulation, 22(3), 6.

Edmonds, B., & ní Aodha, L. (2019). Using Agent-Based Modelling to Inform Policy – What Could Possibly Go Wrong? Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11463 LNAI, 1–16.

Epstein, J. M. (1999). Agent-based computational models and generative social science. Complexity, 4(5), 41–60.;2-F

Epstein, J. M. (2008). Why Model? Journal of Artificial Societies and Social Simulation, 11(4)12.

Essawy, B. T., Goodall, J. L., Voce, D., Morsy, M. M., Sadler, J. M., Choi, Y. D., Tarboton, D. G., & Malik, T. (2020). A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling & Software, 134, 104753.

Feyerabend, P. (1993). Against Method. Verso.

Gambhir, A., Butnar, I., Li, P. H., Smith, P., & Strachan, N. (2019). A review of criticisms of integrated assessment models and proposed approaches to address these, through the lens of BECCs. Energies, 12(9), 1–21.

Gary Polhill, J., Parker, D., Brown, D., & Grimm, V. (2008). Using the ODD protocol for describing three agent-based social simulation models of land-use change. JASSS. Journal of Artificial Societies and Social Simulation, 11(2)3.

Giupponi, C., Ausseil, A.-G., Balbi, S., Cian, F., Fekete, A., Gain, A. K., Essenfelder, A. H., Martínez-López, J., Mojtahed, V., Norf, C., & Others. (2022). Integrated modelling of social-ecological systems for climate change adaptation. Socio-Environmental Systems Modelling, 3, 18161–18161.

Gostoli, U., & Silverman, E. (2020). Sound behavioural theories, not data, is what makes computational models useful. Review of Artificial Societies and Social Simulation.

Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P., & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129–139.

Grimm, V., & Berger, U. (2016). Robustness analysis: Deconstructing computational models for ecological theory and applications. Ecological Modelling, 326, 162–167.

Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768.

Grimm, V., & Railsback, S. F. (2012). Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 367(1586), 298–310.

Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A. S. A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M.-S., Stillman, R. A., Thiele, J. C., & Ayllón, D. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. Journal of Artificial Societies and Social Simulation, 23(2), 7.

Groeneveld, J., Müller, B., Buchmann, C. M., Dressler, G., Guo, C., Hase, N., Hoffmann, F., John, F., Klassert, C., Lauf, T., Liebelt, V., Nolzen, H., Pannicke, N., Schulze, J., Weise, H., & Schwarz, N. (2017). Theoretical foundations of human decision-making in agent-based land use models – A review. Environmental Modelling & Software, 87, 39–48.

Guzy, M., Smith, C., Bolte, J., Hulse, D., & Gregory, S. (2008). Policy Research Using Agent-Based Modeling to Assess Future Impacts of Urban Expansion into Farmlands and Forests. Ecology and Society, 13(1) 37.

Hamilton, S. H., Fu, B., Guillaume, J. H. A., Badham, J., Elsawah, S., Gober, P., Hunt, R. J., Iwanaga, T., Jakeman, A. J., Ames, D. P., Curtis, A., Hill, M. C., Pierce, S. A., & Zare, F. (2019). A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental Modelling & Software, 118, 83–98.

Hamilton, S. H., Pollino, C. A., Stratford, D. S., Fu, B., & Jakeman, A. J. (2022). Fit-for-purpose environmental modeling: Targeting the intersection of usability, reliability and feasibility. Environmental Modelling & Software, 148, 105278.

Hoffman, F. M., Vertenstein, M., Kitabata, H., & White, J. B. (2005). Vectorizing the Community Land Model. The International Journal of High Performance Computing Applications, 19(3), 247–260.

Holman, I. P., Harrison, P. A., & Metzger, M. J. (2016). Cross-sectoral impacts of climate and socio-economic change in Scotland: implications for adaptation policy. Regional Environmental Change, 16(1), 97–109.

Holzworth, D., Huth, N. I., Fainges, J., Brown, H., Zurcher, E., Cichota, R., Verrall, S., Herrmann, N. I., Zheng, B., & Snow, V. (2018). APSIM Next Generation: Overcoming challenges in modernising a farming systems model. Environmental Modelling & Software, 103, 43–51.

Huber, R., Bakker, M., Balmann, A., Berger, T., Bithell, M., Brown, C., Grêt-Regamey, A., Xiong, H., Le, Q. B., Mack, G., Meyfroidt, P., Millington, J., Müller, B., Polhill, J. G., Sun, Z., Seidl, R., Troost, C., & Finger, R. (2018). Representation of decision-making in European agricultural agent-based models. Agricultural Systems, 167, 143–160.

Hughes, H. R., & Paterson, M. (2017). Narrowing the climate field: The symbolic power of authors in the IPCC’s assessment of mitigation. The Review of Policy Research, 34(6), 744–766.

Hüning, C., Adebahr, M., Thiel-Clemen, T., Dalski, J., Lenfers, U., & Grundmann, L. (2016). Modeling & simulation as a service with the massive multi-agent system MARS. Proceedings of the Agent-Directed Simulation Symposium, 1, 1–8.

Hunka, A. D., Meli, M., Thit, A., Palmqvist, A., Thorbek, P., & Forbes, V. E. (2013). Stakeholders’ Perspective on Ecological Modeling in Environmental Risk Assessment of Pesticides: Challenges and Opportunities. Risk Analysis: An Official Publication of the Society for Risk Analysis, 33(1), 68–79.

Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., Voinov, A., Polhill, G., Sun, Z., & Parker, D. C. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4)4.

Lenfers, U. A., Ahmady-Moghaddam, N., Glake, D., Ocker, F., Osterholz, D., Ströbele, J., & Clemen, T. (2021). Improving Model Predictions—Integration of Real-Time Sensor Data into a Running Simulation of an Agent-Based Model. Sustainability: Science Practice and Policy, 13(13), 7000.

Ligmann-Zielinska, A., Siebers, P.-O., Magliocca, N., Parker, D. C., Grimm, V., Du, J., Cenek, M., Radchuk, V., Arbab, N. N., Li, S., Berger, U., Paudel, R., Robinson, D. T., Jankowski, P., An, L., & Ye, X. (2020). ‘one size does not fit all’: A roadmap of purpose-driven mixed-method pathways for sensitivity analysis of agent-based models. Journal of Artificial Societies and Social Simulation, 23(1), 6.

Lorscheid, I., Berger, U., Grimm, V., & Meyer, M. (2019). From cases to general principles: A call for theory development through agent-based modeling. Ecological Modelling, 393, 153–156.

Low, S., & Schäfer, S. (2020). Is bio-energy carbon capture and storage (BECCS) feasible? The contested authority of integrated assessment modeling. Energy Research and Social Science, 60, 101326.

Magliocca, N. R., Brown, D. G., & Ellis, E. C. (2014). Cross-site comparison of land-use decision-making and its consequences across land systems with a generalized agent-based model. PloS One, 9(1), e86179.

Malek, Ž., & Verburg, P. H. (2020). Mapping global patterns of land use decision-making. Global Environmental Change: Human and Policy Dimensions, 65, 102170.

McCulloch, J., Ge, J., Ward, J. A., Heppenstall, A., Polhill, J. G., & Malleson, N. (2022). Calibrating agent-based models using uncertainty quantification methods. Journal of Artificial Societies and Social Simulation, 25 (2), 1.

Millington, J. D. A., Demeritt, D., & Romero-Calcerrada, R. (2011). Participatory evaluation of agent-based land-use models. Journal of Land Use Science, 6(2–3), 195–210.

Millington, J. D. A., Katerinchuk, V., Bicudo da Silva, R. F., de Castro Victoria, D., & Batistella, M. (2021). Modelling drivers of Brazilian agricultural change in a telecoupled world. Environmental Modelling & Software, 105024.

Millington, J. D. A., & Wainwright, J. (2017). Mixed qualitative-simulation methods: Understanding geography through thick and thin. Progress in Human Geography, 41(1), 68–88.

Mohd, M. H. (2022). Revisiting discrepancies between stochastic agent-based and deterministic models. Community Ecology, 23(3), 453–468.

Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., Schlüter, M., Schulze, J., Weise, H., & Schwarz, N. (2013). Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48.

Murray-Rust, D., Brown, C., van Vliet, J., Alam, S. J., Robinson, D. T., Verburg, P. H., & Rounsevell, M. (2014). Combining agent functional types, capitals and services to model land use dynamics. Environmental Modelling & Software, 59, 187–201.

Naivinit, W., Le Page, C., Trébuil, G., & Gajaseni, N. (2010). Participatory agent-based modeling and simulation of rice production and labor migrations in Northeast Thailand. Environmental Modelling & Software, 25(11), 1345–1358.

Newland, C. P., Maier, H. R., Zecchin, A. C., Newman, J. P., & van Delden, H. (2018). Multi-objective optimisation framework for calibration of Cellular Automata land-use models. Environmental Modelling and Software, 100, 175–200.

Ngo, The An, & See, L. (2012). Calibration and Validation of Agent-Based Models of Land Cover Change. In A. J. Heppenstall, A. T. Crooks, L. M. See, & M. Batty (Eds.), Agent-Based Models of Geographical Systems (pp. 181–197). Springer Netherlands.

Niamir, L., Ivanova, O., & Filatova, T. (2020). Economy-wide impacts of behavioral climate change mitigation: Linking agent-based and computable general equilibrium models. Environmental Modelling & Software, 134, 104839.

Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), 641–646.

O’Sullivan, D., Evans, T., Manson, S., Metcalf, S., Ligmann-Zielinska, A., & Bone, C. (2016). Strategic directions for agent-based modeling: avoiding the YAAWN syndrome. Journal of Land Use Science, 11(2), 177–187.

Pedde, S., Kok, K., Hölscher, K., Frantzeskaki, N., Holman, I., Dunford, R., Smith, A., & Jäger, J. (2019). Advancing the use of scenarios to understand society’s capacity to achieve the 1.5 degree target. Global Environmental Change: Human and Policy Dimensions, 56, 75–85.

Polhill, G. (2018). Why the social simulation community should tackle prediction. Review of Artificial Societies and Social Simulation.

Polhill, G., Izquierdo, L. R., & Gotts, N. M. (2005). The Ghost in the Model (and Other Effects of Floating Point Arithmetic). Journal of Artificial Societies and Social Simulation, 8(1), 1–24.

Polhill, Gary, & Salt, D. (2017). The Importance of ontological structure: Why validation by ‘fit-to-data’ is insufficient. In Understanding Complex Systems (pp. 141–172). Springer Verlag.

Railsback, S. F., & Grimm, V. (2019). Agent-Based and Individual-Based Modeling: A Practical Introduction. 2nd Edition. Princeton University Press.

Rand, W., Brown, D. G., Page, S. E., Riolo, R., Fernandez, L. E., & Zellner, M. (2003). Statistical validation of spatial patterns in agent-based models. Proceedings of Agent Based Simulation 2003, Vol. 4.

Roe, S., Streck, C., Obersteiner, M., Frank, S., Griscom, B., Drouet, L., Fricko, O., Gusti, M., Harris, N., Hasegawa, T., Hausfather, Z., Havlík, P., House, J., Nabuurs, G. J., Popp, A., Sánchez, M. J. S., Sanderman, J., Smith, P., Stehfest, E., & Lawrence, D. (2019). Contribution of the land sector to a 1.5 °C world. Nature Climate Change, 9(11), 817–828.

Rogelj, J., Shindell, D., Jiang, K., Fifita, S., Forster, P., Ginzburg, V., Handa, C., Kheshgi, H., Kobayashi, S., Kriegler, E., Mundaca, L., Séférian, R., & Vilariño, M. V. (2018). Mitigation Pathways Compatible with 1.5°C in the Context of Sustainable Development. In V. Masson-Delmotte, P. Zhai, H.-O. Pörtner, D. Roberts, P. R. Skea, J. Shukla, A. Pirani, W. Moufouma-Okia, C. Péan, R. Pidcock, S. Connors, J. B. R. Matthews, Y. Chen, X. Zhou, M. I. Gomis, E. Lonnoy, T. Maycock, V. Tignor, & T. Waterfield (Eds.), Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, (p. 82). IPCC Secretariat.

Rosa, I. M. D., Ahmed, S. E., & Ewers, R. M. (2014). The transparency, reliability and utility of tropical rainforest land-use and land-cover change models. Global Change Biology, 20(6), 1707–1722.

Rounsevell, M., Arneth, A., Brown, C., Cheung, W. W. L., Gimenez, O., Holman, I., Leadley, P., Luján, C., Mahevas, S., Maréchaux, I., Pélissier, R., Verburg, P. H., Vieilledent, G., Wintle, B. A., & Shin, Y.-J. (2021). Identifying uncertainties in scenarios and models of socio-ecological systems in support of decision-making. One Earth, 4(7), 967–985.

Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., & Wu, Q. (2019). Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environmental Modelling & Software, 114, 29–39.

Schmolke, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: a strategy for the future. Trends in Ecology & Evolution, 25(8), 479–486.

Schulze, J., Müller, B., Groeneveld, J., & Grimm, V. (2017). Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward. Journal of Artificial Societies and Social Simulation, 20(2), 8.

Schwarz, N., Dressler, G., Frank, K., Jager, W., Janssen, M., Müller, B., Schlüter, M., Wijermans, N., & Groeneveld, J. (2020). Formalising theories of human decision-making for agent-based modelling of social-ecological systems: practical lessons learned and ways forward. Socio-Environmental Systems Modelling, 2, 16340.

Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384.

Sohl, T. L., & Claggett, P. R. (2013). Clarity versus complexity: Land-use modeling as a practical tool fordecision-makers. In Journal of Environmental Management, 129, 235–243.

Srikrishnan, V., & Keller, K. (2021). Small increases in agent-based model complexity can result in large increases in required calibration data. Environmental Modelling & Software, 138, 104978.

Steel, D. (2007). Across the boundaries: Extrapolation in biology and social science. Oxford, UK: Oxford University Press.

Steinmann, P., Wang, J. R., van Voorn, G. A. K., & Kwakkel, J. H. (2020). Don’t try to predict COVID-19. If you must, use Deep Uncertainty methods. Review of Artificial Societies and Social Simulation.

Stepanyan, D., Grethe, H., Zimmermann, G., Siddig, K., Deppermann, A., Feuerbacher, A., Luckmann, J., Valin, H., Nishizawa, T., Ermolieva, T., & Havlik, P. (2021). Multiple rotations of Gaussian quadratures: An efficient method for uncertainty analyses in large-scale simulation models. Environmental Modelling & Software, 136, 104929.

Storm, H., Baylis, K., & Heckelei, T. (2020). Machine learning in agricultural and applied economics. European Review of Agricultural Economics, 47(3), 849–892.

Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., Balbi, S., Nolzen, H., Müller, B., Schulze, J., & Buchmann, C. M. (2016). Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67.

Taghikhah, F., Filatova, T., & Voinov, A. (2021). Where does theory have it right? A comparison of theory-driven and empirical agent based models. Journal of Artificial Societies and Social Simulation, 24(2), 4.

ten Broeke, G., van Voorn, G., Ligtenberg, A., & Molenaar, J. (2021). The use of surrogate models to Analyse agent-based models. Journal of Artificial Societies and Social Simulation, 24(2), 3.

Tesfatsion, L. (2017). Modeling economic systems as locally-constructive sequential games. Journal of Economic Methodology, 24(4), 384–409.

Troost, C., & Berger, T. (2020). Formalising validation? Towards criteria for valid conclusions from agent-based simulation. In A. van Griensven Jiri Nossent Daniel P. Ames (Ed.), 10th International Congress on Environmental Modelling and Software.

Troost, C., Huber, R., Bell, A. R., van Delden, H., Filatova, T., Le, Q. B., Lippe, M., Niamir, L., Polhill, J. G., Sun, Z., & Berger, T. (2023). How to keep it adequate: A protocol for ensuring validity in agent-based simulation. Environmental Modelling & Software, 159, 105559.

Urban, M. C., Travis, J. M. J., Zurell, D., Thompson, P. L., Synes, N. W., Scarpa, A., Peres-Neto, P. R., Malchow, A.-K., James, P. M. A., Gravel, D., De Meester, L., Brown, C., Bocedi, G., Albert, C. H., Gonzalez, A., & Hendry, A. P. (2021). Coding for Life: Designing a Platform for Projecting and Protecting Global Biodiversity. Bioscience 72, 1.

van Soesbergen, A. (2016). A Review of Land-Use Change Models. Cambridge, UK: UNEP World Conservation Monitoring Centre

van Vliet, J., Bregt, A. K., Brown, D. G., van Delden, H., Heckbert, S., & Verburg, P. H. (2016). A review of current calibration and validation practices in land-change modeling. Environmental Modelling & Software, 82, 174–182.

Verburg, P. H., Alexander, P., Evans, T., Magliocca, N. R., Malek, Z., Rounsevell, M. D. A., & van Vliet, J. (2019). Beyond land cover change: towards a new generation of land use models. Current Opinion in Environmental Sustainability, 38, 77–85.

Verburg, P. H., Neumann, K., & Nol, L. (2011). Challenges in using land use and land cover data for global change studies. Global Change Biology, 17(2), 974–989.

Vermeer, W. H., Smith, J. D., Wilensky, U., & Brown, C. H. (2022). High-Fidelity Agent-Based Modeling to Support Prevention Decision-Making: an Open Science Approach. Prevention Science: The Official Journal of the Society for Prevention Research, 23(5), 832–843.

Williams, T. G., Guikema, S. D., Brown, D. G., & Agrawal, A. (2020). Assessing model equifinality for robust policy analysis in complex socio-environmental systems. Environmental Modelling & Software, 104831.

Zhang, J., & Robinson, D. T. (2021). Replication of an agent-based model using the Replication Standard. Environmental Modelling & Software, 139, 105016.

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright (c) 2023 Calum Brown, James Millington, Mark Rounsevell