Abstract
Computer models are essential for flood risk decision-making, where documentation plays a key role in ensuring transparency and reproducibility. This study examined how different stakeholders perceive documentation practices by analysing a Dutch flood-risk model case study. We assessed perspectives from external observers (ourselves), the modellers (a consultancy company), and the model end-users (the Water Authority) through documentation analysis, model reruns, and interviews. We found a mismatch between the Water Authority’s documentation goals and what was achieved with the delivered documentation. For instance, we could not reproduce the model results based on the available documentation, as crucial tacit knowledge remained implicit. Despite these shortcomings, both the consultancy company and the Water Authority were satisfied with the final products — illustrating how documentation quality is in the eye of the beholder and in this case shaped by the trust from the Water Authority in the consultancy company. However, when key goals like transparency and reproducibility are not met, accountability for model results becomes a concern, especially in decision-making contexts — highlighting the gap between perceived adequacy and broader expectations of documentation.
References
Adams, W. (2015). Conducting semi-structured interviews. In John Wiley & Sons, Ltd (Chap. 19, pp. 492–505). https://doi.org/10.1002/9781119171386.ch19
Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989. https://doi.org/10.1177/1461444816676645
Arnold, T., Guillaume, J., Lahtinen, T., et al. (2020). From ad-hoc modelling to strategic infrastructure: A manifesto for model management. Environmental Modelling & Software, 123, 104563. https://doi.org/10.1016/j.envsoft.2019.104563
Ayllón, D., Railsback, S. F., Gallagher, C., Augusiak, J., Baveco, H., Berger, U., Charles, S., Martin, R., Focks, A., Galic, N., Liu, C., van Loon, E. E., Nabe-Nielsen, J., Piou, C., Polhill, J. G., Preuss, T. G., Radchuk, V., Schmolke, A., Stadnicka-Michalak, J., Thorbek, P., & Grimm, V. (2021). Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support. Environmental Modelling & Software, 136, 104932. https://doi.org/10.1016/j.envsoft.2020.104932
Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J., Marsili-Libelli, S., Newham, L. T. H., Norton, J. P., Perrin, C., Pierce, S. A., Robson, B., Seppelt, R., Voinov, A. A., Fath, B. D., & Andreassian, V. (2013). Characterising performance of environmental models. Environmental Modelling & Software, 40, 1–20. https://doi.org/10.1016/j.envsoft.2012.09.011
Bonet, F. J., Pérez-Pérez, R., Benito, B. M., Suzart de Albuquerque, F., & Zamora, R. (2014). Documenting, storing, and executing models in ecology: A conceptual framework and real implementation in a global change monitoring program. Environmental Modelling & Software, 52, 192–199. https://doi.org/10.1016/j.envsoft.2013.10.027
Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. The MIT Press. https://doi.org/10.7551/mitpress/6352.001.0001
Calder, M., Craig, C., Culley, D., de Cani, R., Donnelly, C. A., Douglas, R., Edmonds, B., Gascoigne, J., Gilbert, N., Hargrove, C., Hinds, D., Lane, D. C., Mitchell, D., Pavey, G., Robertson, D., Rosewell, B., Sherwin, S., Walport, M., & Wilson, A. (2018). Computational modelling for decision-making: Where, why, what, who and how. Royal Society Open Science, 5(6), 172096. https://doi.org/10.1098/rsos.172096
Collins, H. (1975). The seven sexes: A study in the sociology of a phenomenon or the replication of experiments in physics. Sociology, 9, 205–224.
De Bruijn, K., Wagenaar, D., Slager, K., de Bel, M., & Burzel, A. (2018). Leidraad voor het maken van overstromingssimulaties (Tech. Rep. No. 1200537-007-ZWS-0004). Deltares.
de Moel, H., Bouwer, L., & Aerts, J. (2014). Uncertainty and sensitivity of flood risk calculations for a dike ring in the south of the Netherlands. Science of The Total Environment, 473–474, 224–234. https://doi.org/10.1016/j.scitotenv.2013.12.015
Deitrick, A., Torhan, S., & Grady, C. (2021). Investigating the influence of ethical and epistemic values on decisions in the watershed modeling process. Water Resources Research, 57. https://doi.org/10.1029/2021WR030481
Deltares. (2021). D-HYDRO Suite 1D2D. https://www.deltares.nl/nl/software/d-hydro-suite-1d2d/
Edwards, P. N. (2010). A vast machine: Computer models, climate data, and the politics of global warming. The MIT Press.
Fisher, E., Pascual, P., & Wagner, W. (2010). Understanding environmental models in their legal and regulatory context. Journal of Environmental Law, 22(2), 251–283. https://doi.org/10.1093/jel/eqq012
Funtowicz, S., & Ravetz, J. (1993). Science for the post-normal age. Futures, September, 739–755.
Glynn, P. D., Voinov, A. A., Shapiro, C. D., & White, P. A. (2017). From data to decisions: Processing information, biases, and beliefs for improved management of natural resources and environments. Earth’s Future, 5(4), 356–378. https://doi.org/10.1002/2016EF000487
Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768. https://doi.org/10.1016/j.ecolmodel.2010.08.019
Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P., & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129–139. https://doi.org/10.1016/j.ecolmodel.2014.01.018
Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A. S. A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M. S., Stillman, R. A., Thiele, J. C., & Ayllón, D. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. Journal of Artificial Societies and Social Simulation, 23(2), 7. https://doi.org/10.18564/jasss.4259
Gundersen, E. (2021). The fundamental principles of reproducibility. Philosophical Transactions of the Royal Society A, 379, 20200210. https://doi.org/10.1098/rsta.2020.0210
Hutton, C., Wagener, T., Freer, J., Han, D., Duffy, C., & Arheimer, B. (2016). Most computational hydrology is not reproducible, so is it really science? Water Resources Research, 52(10), 7548–7555. https://doi.org/10.1002/2016WR019285
Jackson, S. J. (2014). Rethinking repair. In Media technologies: Essays on communication, materiality, and society. The MIT Press. https://doi.org/10.7551/mitpress/9042.003.0015
Jakeman, A., Voinov, A., Rizzoli, A., Chen, S. H. (Eds.). (2008). Environmental modelling, software and decision support: State of the art and new perspective. Elsevier.
Jakeman, A. J., Elsawah, S., Wang, H. H., Hamilton, S. H., Iwanaga, T., Grimm, V., Koralewski, T. E., Salado, A., Jordan, R., & Voinov, A. A. (2024). Towards normalizing good practice across the whole modeling cycle: Its instrumentation and future research topics. Socio-Environmental Systems Modelling, 6, 18755. https://doi.org/10.18174/sesmo.18755
Janssen, M. A., Pritchard, C., & Lee, A. (2020). On code sharing and model documentation of published individual and agent-based models. Environmental Modelling & Software, 134, 104873. https://doi.org/10.1016/j.envsoft.2020.104873
Kajko-Mattsson, M. (2005). A survey of documentation practice within corrective maintenance. Empirical Software Engineering, 19(1), 31–55. https://doi.org/10.1023/B:LIDA.0000048322.42751.ca
Kumar, R. (2014). Research methodology: A step-by-step guide for beginners (4th ed.). SAGE Publications Ltd.
Lintsen, H. (2002). Two centuries of central water management in the Netherlands. Technology and Culture, 43(3), 549–568.
Martinez-Moyano, I. J. (2012). Documentation for model transparency. System Dynamics Review, 28(2), 199–208. https://doi.org/10.1002/sdr.1471
Maskrey, S. A., Mount, N. J., & Thorne, C. R. (2022). Doing flood risk modelling differently: Evaluating the potential for participatory techniques to broaden flood risk management decision-making. Journal of Flood Risk Management, 15(1), e12757. https://doi.org/10.1111/jfr3.12757
Mayer, L. A., Loa, K., Cwik, B., Tuana, N., Keller, K., Gonnerman, C., Parker, A. M., & Lempert, R. J. (2017). Understanding scientists’ computational modeling decisions about climate risk management strategies using values-informed mental models. Global Environmental Change, 42, 107–116. https://doi.org/10.1016/j.gloenvcha.2016.12.007
Melsen, L. A. (2022). It takes a village to run a model - the social practices of hydrological modeling. Water Resources Research, 58(2), e2021WR030600. https://doi.org/10.1029/2021WR030600
Melsen, L. A., Teuling, A. J., Torfs, P. J. J. F., & Salvadore, P. A. T. (2025). The rise of the Nash-Sutcliffe efficiency in hydrology. Hydrological Sciences Journal. https://doi.org/10.1080/02626667.2025.2475105
Melsen, L. A., Torfs, P. J. J. F., Uijlenhoet, R., & Teuling, A. J. (2017). Comment on “Most computational hydrology is not reproducible, so is it really science?” by Christopher Hutton et al. Water Resources Research, 53(3), 2568–2569. https://doi.org/10.1002/2016WR020208
Müller, B., Balbi, S., Buchmann, C. M., de Sousa, L., Dressler, G., Groeneveld, J., Klassert, C. J., Le, Q. B., Millington, J. D. A., Nolzen, H., Parker, D. C., Polhill, J. G., Schlüter, M., Schulze, J., Schwarz, N., Sun, Z., Taillandier, P., & Weise, H. (2014). Standardised and transparent model descriptions for agent-based models: Current status and prospects. Environmental Modelling & Software, 55, 156–163. https://doi.org/10.1016/j.envsoft.2014.01.029
O’Keeffe, J., Buytaert, W., Mijic, A., Brozović, N., & Sinha, R. (2016). The use of semi-structured interviews for the characterisation of farmer irrigation practices. Hydrology and Earth System Sciences, 20(5), 1911–1924. https://doi.org/10.5194/hess-20-1911-2016
Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), 641–646. https://doi.org/10.1126/science.263.5147.641
Overeem, I., Berlin, M., & Syvitski, J. P. M. (2013). Strategies for integrated modeling: The community surface dynamics modeling system example. Environmental Modelling & Software, 39, 314–321. https://doi.org/10.1016/j.envsoft.2012.01.012
Peters, E., Bier, G., van Lanen, H. A. J., & Torfs, P. J. J. F. (2006). Propagation and spatial distribution of drought in a groundwater catchment. Journal of Hydrology, 321, 257–275. https://doi.org/10.1016/j.jhydrol.2005.08.004
Polhill, J. G., & Edmonds, B. (2007). Open access for social simulation. Journal of Artificial Societies and Social Simulation, 10(3), 10. https://www.jasss.org/10/3/10.html
Remmers, J., Teuling, A. J., & Melsen, L. A. (2024). A modeller’s fingerprint on hydrodynamic decision support modelling. Environmental Modelling & Software, 106167. https://doi.org/10.1016/j.envsoft.2024.106167
Saltelli, A., Bammer, G., Bruno, I., Charters, E., Di Fiore, M., Didier, E., Espeland, W. N., Kay, J., Lo Piano, S., Mayo, D., Pielke Jr, R., Portaluri, T., Porter, T. M., Puy, A., Rafols, I., Ravetz, J. R., Reinert, E., Sarewitz, D., Stark, P. B., Stirling, A., van der Sluijs, J., & Vineis, P. (2020). Five ways to ensure that models serve society: A manifesto. Nature, 582, 482–484. https://doi.org/10.1038/d41586-020-01812-9
Schmolke, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: A strategy for the future. Trends in Ecology & Evolution, 25(8), 479–486. https://doi.org/10.1016/j.tree.2010.05.001
Star, S. L. (1999). The ethnography of infrastructure. American Behavioral Scientist, 43(3), 377–391. https://doi.org/10.1177/00027649921955326
Todini, E. (2011). History and perspectives of hydrological catchment modelling. Hydrology Research, 42, 73–85. https://doi.org/10.2166/nh.2011.096
van Voorn, G. A. K., Verburg, R. W., Kunseler, E. M., Vader, J., & Janssen, P. H. M. (2016). A checklist for model credibility, salience, and legitimacy to improve information transfer in environmental policy assessments. Environmental Modelling & Software, 83, 224–236. https://doi.org/10.1016/j.envsoft.2016.06.003
van Waveren, R. H., Groot, S., Scholten, H., van Geer, F., Wüsten, H., Koeze, R., & Noort, J. (1999). Vloeiend modelleren in het waterbeheer: Handboek Good Modelling Practice (Tech. Rep. No. 99-05/99.036). STOWA/RIZA. https://edepot.wur.nl/181482
Wiering, M., & Winnubst, M. (2017). The conception of public interest in Dutch flood risk management: Untouchable or transforming? Environmental Science & Policy, 73, 12–19. https://doi.org/10.1016/j.envsci.2017.03.002
Winsberg, E. (2010). Science in the age of computer simulation. University of Chicago Press.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2025 Brit E. van der Meijden and Lieke Melsen