Gaming as a research method in humanitarian logistics

Abstract

Purpose

The purpose of this paper is to present a methodology for research through game design and discuss how simulation games can be used to bridge the gap between operational exercises and simulation or analytical modelling and to provide guidelines on how simulation games can be designed for different research purposes in the context of humanitarian logistics.

Design/methodology/approach

This paper combines a literature review on gaming as a research method with an analysis of requirements for humanitarian logistics research methods. Starting from this theoretical framework, the authors develop a design thinking approach that highlights how games can be used for different research purposes. To illustrate the approach, the authors develop two different game set-ups that are of increasing fidelity and complexity. Finally, the authors discuss the results of the evaluation of both approaches, reflect on the design choices and provide recommendations for research and practice.

Findings

Gaming is a suitable research method to explore and analyse behaviour and decisions in emergent settings that require team work and collaborative problem solving. Especially when safety and security concerns may hinder access and experimentation on site, gaming can offer a realistic and engaging quasi-experimental environment. The aspects of engagement and realism also make gaming a suitable tool to combine training and research.

Originality/value

Although the use of games has attracted some attention in commercial supply chain management and crisis response, there is no systematic overview of gaming as a research method in humanitarian logistics. This paper is set to make a headway in addressing this gap by proposing a concrete approach to design games for humanitarian logistics research.

Citation

Lukosch, H. and Comes, T. (2019), "Gaming as a research method in humanitarian logistics", Journal of Humanitarian Logistics and Supply Chain Management, Vol. 9 No. 3, pp. 352-370. https://doi.org/10.1108/JHLSCM-06-2018-0046

1. Introduction

Today 80 per cent of humanitarian funding goes to conflict-driven disasters (GHA, 2017). At the same time, only a small fraction of humanitarian logistics research explicitly addresses man-made disasters and conflicts (Kunz and Reiner, 2012). Maybe not surprisingly, there are only few empirically grounded studies on conflicts since field studies would expose researchers and partnering organizations to risks.

The example of conflicts is only one example calling for research methods that allow us to capture and explore the richness of the context in a controlled and safe research environment. Despite the many calls for empirically grounded research methods in humanitarian logistics (Baharmand et al., 2017; Kovács and Spens, 2009; Kunz and Reiner, 2012; Van Wassenhove et al., 2012), Jabbour et al. (2017) in a recent survey highlight that almost half of the humanitarian logistics publications do not consider context and type of disaster.

As gaming enables participants to experience first-hand the complexity and the pressure to act that are typical for humanitarian operations (Harteveld and Suarez, 2015), and players experience the consequences of their decisions in a protected environment, gaming approaches are promising to bridge this gap. Gaming has been used successfully for training in disaster management, security or supply chain management with the ambition to equip decision-makers with the skills to handle the situations they will be confronted with (Barbara et al., 2015; Koray et al., 2015; Kurapati et al., 2015; Meesters and Van De Walle, 2013; Noori et al., 2017). While the possibilities to use games for training are therefore uncontested, little has been written about gaming as a research method in humanitarian logistics.

In this paper, we present gaming as a research method and develop a framework that allows researchers to design and develop games for humanitarian logistics research. We start with a background section that provides insights into the requirements for research methods in humanitarian logistics and discusses simulation gaming generally and as a research method. In Section 3, we explain our methodology: research through game design for humanitarian logistics. Subsequently, we provide two examples of increasing complexity and fidelity that illustrate how gaming can be used for a technology innovation case. In Section 4, we present a board game, primarily targeted at exploratory research and awareness creation. In Section 5, we use a large-scale simulation exercise to combine testing of technology innovation with its impact on performance and usefulness. In a synthesis (Section 6), we compare the main differences and rationales of our design choices, and discussion implications for our findings. Finally, we conclude with an overall reflection on gaming as a research method for humanitarian logistics, and present implications for research and practice in Section 7.

2. Background

2.1 The need for relevant and contextualized research in humanitarian logistics

Although many authors have stressed the need for empirical research (Starr and Van Wassenhove, 2014; Van Wassenhove et al., 2012), the most prominent research method in humanitarian logistics is still analytical modelling and simulation (Kunz and Reiner, 2012; Leiras et al., 2014). At the same time, there is increasing concern about the relevance of humanitarian logistics research for practice (Laguna et al., 2015; Leiras et al., 2014).

Kunz et al. (2017) identify eight barriers to relevant research, falling broadly into barriers for research and barriers for communication and data sharing (including trust and competition). We will use the first set of barriers to develop requirements for gaming as a research method, and to identify problem areas for which gaming is a particularly promising method:

1. Problem definition driven by standard problems: much research is dedicated to improving operations research models and simulations (see above). But humanitarian logistics problems are often ill-structured and messy, requiring the use of multi-disciplinary cross-functional approaches (Näslund, 2002). At the same time, Jahre et al. (2009) stress the need for theory building within the field of humanitarian logistics. Particularly to explore and understand the impact of new, emerging and innovative practices and technologies, such as cash transfers, digital identity, block-chain based smart contracts or new monitoring and tracking systems, research methods are required that support theory building and problem formulation across different research areas and disciplines.

2. Lack of contextualization: as humanitarian responders are working in a complex systems of different organizations, mandates, norms and supported by a range of technologies (Van de Walle and Comes, 2014), it is important to consider interventions in the context. To overcome the lack of context, which is crucial for socio-technical systems, case studies for theory building, field work or participatory action research have been advocated in logistics research (Jahre et al., 2009). Being bound to past or on-going cases, however, implies that logistics research is not oriented towards the future and the fundamental changes that it may bring (Näslund, 2002). Maybe most prominent in this context is the discussion on trends such as climate adaptation or urbanization (Kovács and Spens, 2011), and the interplay between response and development via social protection systems or local procurement programmes.

3. Difficult data collection: access to regions that are affected by disasters may put humanitarians, beneficiaries and researchers at risk and adds a burden to already stressed system. The lack of research on conflicts has already been mentioned before. But even if access to selected sites or responders can be achieved, there are often only few data points are interviewees (Chan and Comes, 2014). Here, methods are needed that foster the reproducibility and generalizability of results in a safe environment and allow for comparing different variables and settings.

4. Lack of validation with practice: model validation is most commonly done by comparing computational results with benchmark cases (e.g. performance for a past case) without being reflective on why specific decisions were made and if the underlying modelling assumptions on the constraints or objectives are correct. Here, methods are needed to foster the participation of humanitarians in a way that fosters learning in both academia and practice.

2.2 Simulation gaming

Simulation gaming is an increasingly accepted research method to study complex systems (Kurapati et al., 2018). Simulation games can be described as experimental, rule-based, interactive and social organization, constituted by the players themselves, who learn by taking actions and by experiencing their effects through feedback mechanisms within the game (Mayer, 2009; Lukosch et al., 2018). They can be distinguished based on the purpose they serve (Shubik, 1983; Ståhl et al., 1983), leading to following taxonomy:

1. Training games – with a wide use and acceptance in industry, military and education, and the aim to train specific skills without too much conceptual detail.

2. Teaching games – broader in scope than training games, and try to address wider concepts and more abstract ideas.

3. Experimentation games – aim at testing theories or hypotheses, and can also be used to test the effects of certain variables on a given situation.

4. Research games – are designed and/or used to obtain data or empirical material, dealing with a more or less realistic situation or scenario, addressing experienced players.

5. Operational games – to aid decision making, planning and policy implementation in specific situations.

In simulation games, players enact a certain role in a simulated environment (Duke and Geurts, 2004). As pointed out by Klabbers (2018), when games are used for research, they should address players in their capacity of being reflexive actors; while playing being engaged with introspection, allowing for reflection-in, and reflection-on-action. In the language of Sociologist Anthony Giddens, games represent a duality of structure as players of games constitute the emerging social organization, while at the same time the game is the very medium of its constitution (Giddens, 1993; Klabbers, 2018). Thus, while playing a game, a player continuously sets new (social) rules, and observes the consequences of his or her actions and decisions. In games, transfer of knowledge occurs through the player’s background that influences game play, and through the process of acquiring knowledge and skills from the game play that can be useful in a specific real context (Copier, 2007). As Klabbers (2018) discusses, the player is not only a subject, but also on agent or actor, who demonstrates purposeful behaviour, based on a certain set of skills and knowledge. Such behaviour can be observed and analysed in a qualitative way.

As such, simulation gaming has a long tradition and proven value in education, policy making and research (Duke and Geurts, 2004; Klabbers, 2006). While games are clearly distinguished from the real world (Klabbers, 2006), their boundary is open to transfers between game and reality, which makes them applicable tools to understand and design real complex systems.

Nowadays, games come in many forms and are based on different technologies. From role-playing games that are based on (almost) no materials, to physical board games, to high-end games that use advanced digital technology. The choice of the applicable game type is based on factors such as fidelity or level of realism that is needed for the purpose of the game, the size of the target group and the quality and quantity of data to collect. Physical board games, for example, have the advantage to enable direct communication and collaboration amongst a relatively small group of players, and allow for open game play, as often rules can be changed by the players (and/or the facilitator) during game play, if necessary. Disadvantages of this type of games are that usually it is difficult to scale them up to a large group of players, and that data collection is limited to observation and reflection. In comparison, digital games are more expensive in development cost and time. On the other hand, data collection and processing can be automated and can be used for quantitative data collection. The implementation and adjustment of scenarios as context and storyline for the game can be easier than in physical games. In some cases, it is not necessary to design complete games, but to make use of the engaging nature of game elements, in order to, e.g., use scenarios to increase the realism of a given exercise, and to foster more realistic behaviour of test persons.

2.3 Gaming as a research method

While in the past simulation gaming has been largely ignored as a research method (Greenblat, 1975), the increasing need to understand complex phenomena slowly turn simulation games into an accepted research instrument (Klabbers, 2006). Especially the use of simulation games to integrate different perspectives, concepts, theories, data, information, methods and techniques from various disciplines makes this method a valuable tool for meaning making and trans-disciplinary research (Klabbers, 2009). As a research method, simulation games are usually combined with other instruments such as questionnaires, tests and debriefing (Kurapati et al., 2015; Lukosch et al., 2018).

As other methods that help in understanding complex problems, simulation games are based on a model. Bradley et al. (1977) characterize the use of models in providing guidelines to decision-makers for effective decisions as operational research. Gaming can thus be described as an approach of operational research, applicable to complex systems. Bradley et al. (1977) categorize tools for operational research into four types: operational exercise, simulation gaming, simulation and analytical model:

1. In an operational exercise, the (research) experiments are directly implemented in the real environment. As discussed above, this is often not possible in humanitarian logistics research, as the real environment can be very dangerous for both participants and researchers as well as for the technology to be tested. Yet, the advantage of an operational exercise is that it can deliver the highest degree of realism.

2. Simulation games are interactive environments that allow for direct engagement and provide immediate feedback to players and researchers. The level of realism is lower than in operational exercises (Bradley et al., 1977), as they represent a more abstract environment. Based on the technology and game process used, it can be easy or difficult to collect valuable data for research purposes. While a pure simulation aims at representing a system as realistic without human interference (Bailey, 1982), a simulation game is a more simplified, yet realistic representation of a system that can be manipulated by (the decisions of) players. Games are hence characterized by a game layer on top of a simulation.

3. In simulations, an environment is represented by the use of mathematics or objective representation (Feinstein and Cannon, 2002). Simulation models are inductive and can be used to evaluate the performance of a given system (Bradley et al., 1977). A limitation of most forms of simulation is that the human decision-maker is removed from the representation.

4. Analytical models are theoretical constructs of a given system, expressed in mathematical terms, usually based on a simplified framework, to visualize complex processes (Choi et al., 2016). Analytical models are easy und cheap to develop (Bradley et al., 1977). Yet, their level of simplification of a given system is usually the highest, hence their level of realism is very low.

Above, we identified gaps and requirements for research methods in humanitarian logistics. Table I compares aforementioned OR research methods based on these aspects, highlighting that simulation games bridge the gap between operational exercises and simulations or analytical modelling.

As in any other method, the validity of the results from gaming of course depends on defining a rigorous set-up and design. Traditional research designs follow four steps: random selection of subjects; random assignment of subjects to different treatment conditions; experimenter manipulation of treatments and experimenter control over the conduct of the experiment (Bachrach and Bendoly, 2011). The strict control over some of these aspects is not given in simulation games. For example, a random selection of subjects can be difficult, when only a limited number of experts is available for a gaming session. Yet, the rigour of the method can be guaranteed, when the process of experimental research is followed. This process can be formulated as conceptualizing the research question; operationalization and design; methodology and collecting data; validity testing and interpretability; and effect and relationship testing (Bachrach and Bendoly, 2011). All steps can and need to be followed when a game is used as instrument.

One of the main advantages of simulation games is that they can provide rich output and data. For example, reactions and responses of players in scenario-based role-playing exercises can be recorded, quantified and analysed (Rungtusanatham et al., 2011). As in any other experiment, researchers have to check whether the abstraction of a simulation game influences the results, and how to apply the results to the outside world (Deck and Smith, 2013). Compared to simulations and analytical models, however, the level of complexity and realism of games usually higher.

In sum, games can be seen as operational and interactive representations of a real system, allowing participants and researchers to literally “play” with game elements and observe the consequences. We highlighted above, that there are many choices in game design that influence the validity, type and quantity of data that can be collected through simulation-game-based research and the importance of following design guidelines to develop games for given contexts and research aims. In the following section, we show which design methodology we apply for the use of gaming as research method in humanitarian logistics.

2.4 Designing games for research

To design games for research of complex socio-technical systems, it is crucial to address three aspects: the actor layer, the technological layer and the relationships among actors and between actors and technology (Klabbers, 2018). Especially defining the relationships can lead to interesting game mechanisms, as forms and rules of communication, information sharing, or power relationships can be displayed. Games can be used to research these relationships and can motivate players to change them to improve the real system (Lukosch et al., 2018). In a rigorous way, games can be seen as an artefact in a design science cycle, or to facilitate “research through game design”, as Figure 1 illustrates.

The proposed research through game design process starts with identifying a research gap, and formulating related research questions, as shown in the lower left of the figure. This is mainly a deductive activity. From these theoretical considerations, game requirements are formulated based on theoretical insights and practical considerations. Such rather abstract requirements are then further developed in simple, often paper-based game prototypes. In test sessions with game design and content experts, these prototypes are tested and evaluated. After consolidating a final design, the experimental set-up is developed.

The data collected in such research cycle can be very different in nature. Typically, game data are collected either directly from the game (scores, decision, actions) or from (structured) observations (validated) material can be used, such as debriefing material, pre- and post-test surveys and other instruments that are related to the research questions. Various approaches towards data analysis can be used, such as content analysis, coding of verbal feedback or Bayesian analysis.

The cycle illustrates that gaming can be seen as part of a whole research design, where a theoretical gap analysis leads to research questions that inform both the design of a game prototype, and the development of hypotheses or a research framework.

3. Methodology

To understand the specific steps and requirements for designing games as a research instrument for humanitarian logistics, we use a case study approach that allows us to explore the intricacies of the method for one of the most prominent cases that call for gaming as a research method: innovation in conflict situations.

3.1 Case study: information technology for humanitarian logistics in conflicts

To illustrate how gaming can be applied as research method in humanitarian aid logistics, and how this field can benefit from gaming research, we focus on information technology developments. Humanitarian organizations are confronted with mounting tensions as they seek to maintain access to populations in need and provide aid to the most vulnerable populations in crisis regions such as the Middle East or Africa. Technological innovation is seen as one of the major game changers to enable humanitarians to “Stay and Deliver” (Egeland et al., 2011). For example, satellite imagery, real-time analysis of camera footage from cars or UAVs and more generally speaking remote sensing and monitoring are seen as promising approaches to help humanitarian organizations to rapidly evaluate access conditions and risks (Comes and Van de Walle, 2016). While in 2015, cash and markets-based interventions only amounted for about 6 per cent of all aid, relief provision is beginning to shift towards virtual distributions through digital payment systems, or “mobile money” (Logistics Cluster, 2015). Orders and shipment logistics can be complemented by social media feeds and analyses of customer buying patterns (Majewski et al., 2010), and biometric identification technologies are increasingly used for refugee management (Jacobsen, 2015). Real-time tracking and monitoring is expected to improve efficiency and effectiveness of logistics by enabling decision-makers to adapt to the ever-changing context of the operations. As such, technological innovation entails the need to adapt decision processes, coordination and management structures (Comes et al., 2018).

Introducing new technologies or processes in an on-going conflict poses a significant risk, because any deviation from procedures, malfunction, unforeseen use or even possession of technology can cause harm. At the same time, it is important to take into account the specifics of the context, such as the sensitivity of data and information, or the legal and technological constraints, security and logistics guidelines and policies (Van de Walle and Comes, 2015). As such, the challenge of evaluating the impact of monitoring and tracking technologies on humanitarian logistics in conflict is used as example for many other innovations that are currently being explored as new humanitarian technologies.

Van Wassenhove (2006) argues that a humanitarian supply chain needs to be designed to align material and informational flows. To evaluate the impact of innovation on supply chain performance and the use in terms of (improved) decision making and coordination, we therefore analyse a humanitarian supply chain across three layers: in terms of physical movement of goods and people; informational flows; and decision making and planning.

3.2 Game design approach

To explore which technologies could be beneficial for humanitarian operations; and to identify what the impact of the developed solutions is on all the three levels, we developed and used a two games in the context of a European research project that develops technologies and policies for staff safety and logistics support in conflicts. First, we conducted a requirements analysis for the development of a tracking, monitoring and logistics support system. We supported this step with the development and implementation of a board game. Second, we used game elements in a large-scale exercise to evaluate the usability and usefulness of the developed system.

Here, we use the game research cycle (Figure 1) to discuss how the respective games were designed and used. All documentation of the concrete game designs, execution and evaluation are available publicly and openly in the project deliverables on www.itrack-project.eu/page/en/documentation/public-deliverables.php

For the board game, we first conducted a systems analysis of the processes of humanitarian logistics, building upon theoretical knowledge in humanitarian logistics, information management and ICT. Second, we reviewed the proposed solutions from technical partners in the research project. We chose to use the game to explore decisions and preferences of aid workers towards different technical components of the proposed system. This set-up led to a number of game requirements, such as that the game should be placed in the context of a crisis situation, that it should foster the discussion between different humanitarian aid organizations, and that it should be playable within half a day. Based on this identification of requirements, a first paper-based prototype was developed and test-played with game design experts. Based on their feedback, and after some design iterations, the final design of “Plaitra” was developed. As experimental set-up, it was decided to conduct a qualitative study, in which the game served in first instance for making the players aware of the different technical solutions, and to observe their choices in the game. Second, a debriefing structure and questions were developed, following the guidelines by Kriz (2010).

For the exercise, the research aimed at testing a prototype of the technologies and its impact on workflows, communication, decision making, coordination and performance. Here, we used game elements, especially scenario building and role-playing, to support a realistic system test. Again, theoretical articles as well as working reports were analysed to identify current processes and policies in humanitarian logistics. Those were translated into scenarios and role descriptions that related to different system components. A large number of additional research instruments was selected, such as usability test instruments, demographical data collection, questionnaires to evaluate the experienced usefulness of the system, as well as debriefing structures. A combination of qualitative and quantitative data analysis methods was applied to interpret the data collected. The results of this test informed the development of the final, integrated version of the socio-technical system.

4. Plaitra

4.1 Design cycle: needs analysis, prototype development, tests

The physical board game “Plaitra” is the result of an iterative design process, including academics, game designers and experts from the field, following the research design cycle as shown in Figure 2. First, interviews were conducted with humanitarian aid workers to explore what kind of technology is used in conflicts, and to carry out a needs analysis. Via a literature review, we identified requirements for innovations to be of use for humanitarian operations in this context. Based on both empirical work and literature review, research gaps were identified. Second, a game prototype was developed and tested with students of a technical university as well as experts with a background in humanitarian aid (for details, see Schwarz et al., 2017). The main requirement for the game design was that the game session should allow researchers to observe the decisions of the players for a technology. Furthermore, the impact of that choice on logistics operations should immediately be visible both for players as well as for researchers.

In the game, players take over the role of humanitarian organizations, who have to efficiently allocate limited resources while dealing with multiple risks, uncertainties and impediments to communication. Additionally, no single organization can cover all humanitarian assistance needed; instead, organizations have to collaborate. Consequently, humanitarian actors have to share information and coordinate their operations in order to ensure a coherent response.

The game material consists of two boards that represent a crisis region, as illustrated in Figure 3. Four teams of two players play the game, while two researchers/facilitators support them. Players have to carry out assessments to indicate the humanitarian needs in particular places. Players can invest in technologies to gather and share information.

We identified three main categories of information that were translated into the game: humanitarian needs, actions of other humanitarians and security situation. In order to let players experience how technology impacts information flows and thereby logistics processes and coordination, the following goals for the game have been identified:

1. The main purpose of the board game is supporting requirements analysis. After game play, a structured debriefing is used to discuss technologies and policies with the players.

2. The technological components that are available for the players in the game represent the components proposed by the technical partners of the research project, and beyond. They include advanced technologies that are not deployed in the field of humanitarian logistics yet. The extensive list of available technologies in the game is designed to make the player aware of the possibilities of technology for the humanitarian field.

3. Information management workflows and processes have been modelled for implementation in the game, which allows us to explore the processes and policies of their use, too.

4. The role of coordination is addressed in the game. Communication is not allowed in the first rounds of the game. Players need to invest in technologies to be able to share information and coordinate with each other.

5. Humanitarian responders operate with constrained resources. This has been modelled in the game by disruptive events happening during the game play, and the limited availability of technology at the beginning, but also during the game.

4.2 Empirical cycle: experimental set-up, game session and data collection

We decided for qualitative data analysis of the game session, as we expected only a limited number of test persons being able to participate in a game session, and our research was primarily of exploratory nature. We first created an observer protocol and an in-game observer role to make the role less intrusive. During game play, a journalist notes down observations of player actions and decisions along a pre-structured protocol. Furthermore, journalists can provide some information to the players. In addition, the game session includes a debriefing phase, following the structure of debriefing as proposed by Kriz (2010).

We invited 16 humanitarian practitioners with experience in the field to take part in a full-day game workshop. The 16 players were randomly distributed to two parallel sessions that took place at a University in Finland. In each session, a facilitator as well as a journalist joined. The game play started with a briefing, introduction to gaming, purpose and main rules of the game. After that, the players played 16 rounds with the game (see for an impression Plate 1).

The observers stayed during the whole game play and wrote down their observations. After game play, players first filled in a questionnaire about their experience of the game play. Then, all 16 players gathered for a plenary debriefing that was facilitated by a researcher who was not involved in the game play. After the plenary debriefing, the players discussed in smaller groups which technology would be useful and usable in humanitarian aid logistics. The results of these discussions were collected, too.

4.3 Results

Via the questionnaires, we collected experiences with the game, and how useful the players valued the game as instrument itself. The outcome of this research step was foremost positive. This feedback showed that the game design was perceived as engaging, purposeful, and – despite the high level of abstraction – realistic. Some comments from the questionnaires and debrief allowed us to improve the fidelity and realism of the set-up and workflows.

Through a combination of performance measuring of the play itself, observation of the discussion during the games, the de-brief sessions conducted after the game play, a number of issues were raised that are relevant for the further discussion of technology innovation:

• the general scepticism against technology innovation particularly at operational level;

• the need to address secondary impact of using technology in the operating environment;

• the need to support decision making under security threats during convoys; and

• the need for decentralized communication structures among operational staff; and the need to align technology and workflows and processes.

These results show that the research through game design approach allowed for broader insights than the choices of players alone. Especially, the combination of research instruments, namely game play, observation, questionnaire and debriefing, provided rich material that allowed for above mentioned conclusions. To specifically address the last point in the list, the simulation exercise was designed, again using the game design research cycle.

5. Simulation exercise

Based on the requirements from the game, a literature review and a series of interviews, a tracking, monitoring and logistics support system for humanitarians in conflicts was developed[1]. To test the usefulness and impact of this system, the research through game design cycle was conducted a second time, taking the results from the board game as a starting point. For the second iteration, a computer supported simulation exercise was held at the campus of Delft University of Technology in April 2018. This exercise aimed at creating a safe and realistic environment to integrate software testing activities with humanitarian logistics and coordination. Humanitarian logistics professionals as well as technology developers joined the exercise. As for the board game, we will illustrate below how we implemented the research through game design cycle in our design choices for the exercise.

5.1 Design cycle: exercise requirements, tests and overall design

The first step in the design of the simulation exercise was to define the requirements for the exercise. The exercise itself was meant to close the first development cycle of the integrated version of the tracking, monitoring and logistics support system and to inform the second development phase with additional requirements. Two sets of requirements for the exercise were identified: first, an evaluation framework was developed that addressed the individual technological components, as well as the integrated platform. An inventory of all technological components and related hard and software needs was made, and we chose to evaluate their usability with help of existing and accepted instruments.

Second, the purpose was to create a realistic stress test for the system beyond the limitations of a clean, experimental test situation. To this end, scenarios, understood as game elements to increase the fidelity or realism of the exercise, were developed, to enable participants to test the system in a realistic quasi-experimental environment. Different locations were chosen and decorated into realistic mission control room, warehouse and checkpoint (Plate 2). In addition to the experiment materials such as laptops and mobile phones, other facilities used to increase the fidelity included a convoy of three vehicles, radios and a UAV.

The exercise was monitored at both the system and the process level. The system was monitored at the back-end and fine-tuned immediately when necessary. Operations at all three exercise locations were filmed through webcams and all communications made in the system were logged for post exercise analysis.

5.2 Empirical cycle: realization of the exercise and data collection

The simulation exercise was carried out in April 2018 with participants from the project consortium, humanitarian organizations and EU civil protection. The usability tests focussed on testing each component. The participants were asked to fill in various surveys to evaluate the user experience (Laugwitz et al., 2008), the components’ usefulness and ease of use (Davis, 1989), as well as the ethics and privacy issues addressed by the system. The usability test was completed with a short debriefing for direct comments and feedback to the tech partners.

In order to effectively test the integrated system, the simulation exercise adopted scenarios of basic logistics workflows that were combined with a collection of special injects to represent various field situations. The special injects covered issues from the perspectives of logistics procedures, humanitarian information management and security risk management. All exercise data, including communication and system operations, were logged automatically on a secure local server for analysis. A debriefing session was held at the end of each day to discuss issues encountered from the exercise and collect feedback on both the system and user behaviour. At the end of the week, a final reflection meeting was organized to summarize the progress made and the lessons learnt throughout the simulation exercise.

5.3 Results

All participants viewed the simulation exercise valuable and as an insightful way of testing technology. The scenario-based testing approach allowed participants to experiment new technology in a realistic and safe environment. The exercise week also brought the technology developers and end users together and shortened the feedback loop between the test lab and the fields. In addition, playing logistics themed scenarios and injects also provided a platform to reflect specific policies and protocols in carrying out missions. In the exercise, especially the realistic documents used for information management as well as the realistic communication protocols applied added to a realistic feeling. They contributed to a realistic, while safe test environment in the context of humanitarian aid logistics.

6. Synthesis

Humanitarian logistics requires both anticipation-based strategies that rely on extrapolation of trends and forecasted scenarios (Laguna Salvadó et al., 2018) as well as approaches that foster resilient, flexible and agile supply chains (Charles et al., 2010; Oloruntoba and Gray, 2006). The role of (new) information and communication technology (ICT) has widely been described as essential (Altay and Labonte, 2014; Comes et al., 2019) – but it is yet to be explored how ICT can support robustness and resilience in humanitarian operations.

For our case of ICT in conflict situations, we were able to create realistic and at the same time safe research conditions. Table II illustrates how different research questions were addressed for both games. In both games, we were able to address and evaluate the impact of innovation on supply chain performance and on decision making and coordination. Following Van Wassenhove (2006), we did this on and across three layers: in terms of physical movement of goods and people; with regard to informational flows; and in relation to decision making and planning. In the following, we will discuss the most important differences in the design choices and their implications.

In the Plaitra board game, physical movement was represented with lego blocks, both simulating trucks as well as assets of aid supply. Players were able to track their assets and got direct feedback on movements, and aid supplied to people in need. Additionally, players had to handle realistic forms to manage the logistical processes. The movement of lego blocks – assets – was used as direct feedback from the game, but also to show the difference on the assumed and actual situation. This direct feedback of the game enabled the players to immediately evaluate the effects of their decisions. Thus, game play allowed showing consequences of decisions and actions not only on one individual layer, but also across the three layers.

In the simulation exercise, physical movement could be simulated even more realistically. Boxes and sacks had to be carried from a warehouse through a physical checkpoint back to the warehouse. The goal of this exercise element was mainly to increase the fidelity of the exercise, which was appreciated by the participants. Especially the realistic scanning of assets, and again, the combination with realistic forms that had to be used, supported the realism of the exercise. Different technology was used to track assets and people during the exercise. Using technology had the advantage to automatically log information flows to be used for evaluation after the exercise. During the exercise, technology was used in the planning and as decision support tool. It allowed participants as well as research works to immediately observe consequences of their decisions. This second case shows how all three layers of humanitarian aid logistics can be addressed by gaming as a research method.

The use of the board game Plaitra illustrates how a relatively limited and simplified game session can contribute to explorative research. In a qualitative way, choices and preferences of experts towards an innovation in the field were identified. The biggest challenge here was to provide a game that was realistic enough to foster realistic choices, while simplify processes in a way that an engaging game play was realized. Thus, the challenge here lied in the design cycle of the research development.

In the exercise, the greatest challenge we faced was located in the empirical cycle, namely the data collection. The exercise itself could relatively easy be developed based on empirical data and information from the field. Yet, as a very realistic and rich test environment was developed, that required a high number of resources (personnel and materials) for conducting and evaluating the field exercises.

7. Discussion and conclusions

In this paper we argue that (simulation) gaming is a research method that complements the existing OR methods in providing a realistic and experimental environment for problem exploration. Research can be conducted in a contextualized yet safe way, as scenarios and roles are implemented in the game environment. Different types of data can be collected within a quasi-experimental set-up. When experts are involved in the development of the game as well as the game session itself, validation with practice is high. Yet, researchers who would like to use games as a research method have to carefully follow a rigorous design process. Most importantly, balancing the complexity required to provide a realistic and engaging set-up with simplification of tasks for reproducibility and valid data collection.

We chose gaming as a research method approach, to address some of the limitations of current research in humanitarian logistics on the four main aspects as introduced in Section 2.1. In the following, we discuss the implications of our findings from the case along these challenges:

1. Problem definition driven by standard problems: by using gaming as a method, we are able to address ill-defined problems (Westera et al., 2008), and explore and understand the impact and use of innovations and how it translates to coordination and performance. Results of qualitative (game) data analysis can thus inform theory building and testing.

2. Lack of contextualization: games can represent the complexity and uncertainty of a crisis or disaster, yet in a safe environment. A novel technology or policy; or an extreme or dangerous scenario can thus be tested in a safe and at the same time realistic environment that allows for valid results to be translated into the field.

3. Difficult data collection: gaming as a research method allows for quasi-experimental study-set up, and for a controlled data collection that integrates the behavioural aspects of decision making, processes and coordination structures.

4. Lack of validation with practice: games can both be developed and played with experts. An early inclusion of experts in the development process ensures the verification of the design of the game. The participation of experts in the game play process itself guarantees validity of the results collected through game play.

7.1 Implications for practice

As discussed prototypically for Plaitra, games have a great potential to connect research and innovation for practice if they are useful for training purposes. We think that this is a unique way to use the power of engagement and the “fun” factor of the games that players generally agreed on for both research and practice.

However, these games have to go beyond the traditional exercises that focus on situations that can be anticipated or that have occurred before to prepare responders and logisticians for the response. In a more and more uncertain and complex world, the exact conditions that field logisticians will be confronted with are harder and harder to predict, and particularly the impact of new technologies on the future operations is hard to foresee. In such uncertain situations, different types of preparation are necessary, that relates to general skills such as flexibility, adaptability, creativeness, communication and decision making. Results of such empirical research can help to support actors in both ways.

Design recommendations for practice:

1. Be aware of the dynamic and uncertain nature of the field and make these characteristics a vital element of the game to be used.

2. Games are only one method to gather a certain type of data related to the human factor. Make sure to include data from other sources to develop a complete picture of a problem.

3. Games are dynamic and the players create their own reality. Let them explore and observe, use the observations to increase the feedback of the game.

7.2 Outlook and implications for research

We have argued that gaming as a research method has potential to contribute to theory building and exploration of new emerging technologies; in situations of limited access and high risk; or for rare and extreme situations. Gaming as a method particularly allows us to explore the interplay between an intervention (e.g. a policy, process or technology) and work practices, decision making and team performance.

Design recommendations for research:

1. Ensure that design and empirical cycle in the research process are well related.

2. Start with formulating a problem, based on theory (and practice), and define game requirements including data to collect.

3. Go beyond designing a game as isolated tool. In some circumstance, it can be useful to use game elements instead of a complete game. Carefully consider and choose additional research instruments including a reflection moment.

Gaming as a research method can open the path to new insights and models, but because of the natural restrictions in numbers of players and experts, duration of play, facilities and equipment, gaming is an addition to other research instruments such as field studies or optimization and simulation. Here, approaches need to be explored that enable a translation of the data and insights collected through the games into valid theories and models, and that embed these insights back into a new cycle of game design and testing.

Figures

Figure 1

Research through game design cycle

Figure 2

Research through game design in humanitarian aid logistics: board game

Figure 3

Plaitra game board

Plate 1

Game material and actions of a team of two players during Plaitra game play

Plate 2

Exercise location examples: warehouse (left), checkpoint (right)

Table I

Comparison of OR methods based on requirements for humanitarian logistics research (cf. Section 2.1)

	Problem exploration	Type of problem	Ease to reproduce	Contextualization	Data collection	Validation with practice
Operational exercise	Highly realistic	Based on current practice	Low	High	Risky	Very high
Simulation games	Realistic	Flexible integration of behaviour and (new) or standard situations, approaches and technologies	High, if rigorous methodology is implemented (see below)	High	Quasi-experimental	High
Simulation	Abstract	Simulation of new (new) or standard situations, approaches and technologies; assumptions on behaviour	High	Low	Inductive, numeric	Low
Analytical model	Highly simplified	Data-driven; based on past frequent cases in data-rich environments or standard problems	Very high	Very low	Numeric	Low

Table II

Comparison of key choices for the board game and simulation exercise

	Design choices
	Board game: Plaitra	Simulation exercise
Corresponding research gap	Theory building; reproducibility; collaboration with practice	Theory building; contextualization; collaboration with practice
Research framework	Requirements elicitation for technology innovation, such as for tracking and monitoring technologies	Testing and evaluation of impact of innovation on individual and team performance
Assumptions	A board game can represent logistics and communication processes to enable experts to make choices explicit	A realistic exercise allows the actions of humanitarians to be observed so their usability and usefulness in the field can be measured
Fidelity	Low	High
Requirements on scope and constraints in participants and budget	Needs to be run in a limited time; few players; relatively easy to set up and transfer; low cost	Several hours; extensive in set-up, high requirements in preparation, planning and technology, including multiple volunteers; high cost
Data collection	Notes of the players; observations; video; debrief	Logs and messages as documented in the system; observations; video; debrief
Main types of result	System requirements	Performance testing, usability, perceived usefulness

Note

1. See www.itrack-project.eu/page/en/documentation/public-deliverables.php for full documentation.