System dynamics applied in enterprise engineering – a systematic literature review

Abstract

Purpose

This study aims to investigate the combined use of System Dynamics (SD) applications in Enterprise Engineering (EE) research and practice. SD application in EE is becoming widely accepted as a tool to support decision-making processes and for capturing relationships within enterprises.

Design/methodology/approach

A systematic literature review (SLR) is conducted using a standard SLR method to provide a comprehensive review of existing literature. The search was conducted on ten platforms identifying 30 publications which were analysed through the use and development of a codebook.

Findings

The SLR showed that 90% of the result set consisted of peer-reviewed academic conferences and journal papers. The SLR identified a highly dispersed author set of 83 authors. Amongst these authors, Vinay Kulkarni was an active author who has co-authored up to four publications in this research area. The analysis further revealed that the combined use of SD applications and EE is an emerging research area that still needs to develop in maturity. While all phases of EE have received attention, the current research work is more focused on the design phase. The important gap between model development and implementation is identified.

Originality/value

The study elucidates the existing status of interdisciplinary research combining techniques from the SD and EE disciplines, suggesting future research topics that combine the strengths of these existing disciplines.

Citation

Hussain, H. and De Vries, M. (2024), "System dynamics applied in enterprise engineering – a systematic literature review", Journal of Modelling in Management, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JM2-05-2023-0082

1. Introduction

Over the years, there has been an increase in complexity associated with the design and fulfilment of business processes. Enterprises are faced with the challenge of incorporating emerging technologies and market information with existing knowledge to create a balance between control, business agility and innovation (Fayoumi and Loucopoulos, 2022). With the support of systems dynamics techniques, enterprises are equipped to make key decisions associated with business-IT alignment (Bigdeli et al., 2019), resource allocation, process efficiency (Fayoumi and Loucopoulos, 2022) and the design and evaluation of virtual enterprise structures (Assimakopoulos and Riggas, 2006).

During the late 1950s, the concept of systems dynamics (SD) originated and developed from the fields of control engineering and management (Angerhofer and Angelides, 2000). Over time, earlier definitions have been refined, leading to the current characterization of SD as “a rigorous method of system description, which facilitates feedback analysis, usually via a continuous simulation model, of the effects of alternative system structure and control policies on system behaviour” (Wolstenholme, 1982). Early applications of SD were in industrial company problems including inventory management and labour market instability. However, over the years, it has increasingly been applied to a more diverse set of scientific areas such as public policy design, strategic management and environmental sciences and management (Cosenz and Noto, 2016). SD application has also been observed on a larger scale in an enterprise simulation (Affeldt, 1999). SD is a computer-aided approach that has proven to be an effective method for modelling complex dynamic systems characterised by interdependence, mutual interaction, information feedback and circular causality (Sterman, 2000). Research in the field of complexity management has led to the development and application of a number of SD modelling tools and techniques, namely, fuzzy logics (Batur et al., 1991), neural networks (Gardner and Derrida, 1988), Bayesian networks (Pearl, 1985), Petri nets (Gunasekaran and Irani, 2010), causal loop diagrams (CLD) (Agyapong-Kodua and Weston, 2011) and stock and flow diagrams (SFD), among others. The CLD and SFD tools are considered most suitable to understand the complex and dynamic behaviour of enterprises (Oladimeji et al., 2020).

Enterprises are dynamic systems where every change introduced to the system can be represented, understood and analysed through feedback loops and communication paths (Fayoumi and Loucopoulos, 2016). The value of SD, used within the context of enterprise simulation, lies in being able to use assumptions when historical data is not available, too costly or experimentation is impossible and still be able to produce continuous dynamic simulations of non-linear complex systems or conceptual models (Fayoumi and Loucopoulos, 2022).

The discipline of enterprise engineering (EE) is multi-disciplinary in nature (Giachetti, 2010), and was founded in 2013 as a means to create scientific rigor in developing and testing theories within the field, and contributing towards a sound body of knowledge in EE (Dietz et al., 2013). EE is recognised as “the body of knowledge, principles, and practices to design an enterprise” (Giachetti, 2010). An enterprise refers to any socio-technical system, including virtual organisations, government transformations and entire industries (Bernus et al., 2016). The EE domain acknowledges not only the (re)designing processes but also the design and alignment scope, which includes multiple design domains, such as the organisation domain, ICT domain, infrastructure (facilities) domain and human skills and know-how domain (De Vries, 2017).

Researchers identify EE as composed of three key concepts when designing or redesigning an enterprise, namely, enterprise ontology, governance and architecture (Barjis, 2011). Enterprise architecture refers to the design principles that are needed to govern enterprise design (Hoogervorst, 2018). Enterprise ontology refers to a concise conceptual representation of enterprise operation (Dietz and Mulder, 2020). Whereas enterprise ontology focuses on enterprise models that represent the essence of enterprise operation, enterprise modelling is more comprehensive in scope and can be defined as “the art of externalising knowledge in the form of models about the structure, functionality, behaviour, organization, management, operations and maintenance of whole or part of an enterprise, or of an enterprise network, as well as the relationships with its environment” (Vernadat, 2020).

EE aims to identify alternative ways and initiatives to improve existing performance by changing the design of multiple design domains (De Vries et al., 2015). According to Hoogervorst (2018), the enterprise poses high levels of uncertainty, which require additional mechanisms to understand “what to do” to improve performance. SD can provide a structured approach to understanding and modelling the complex interdependencies and feedback loops that exist within and from outside the enterprise system (Haraldsson, 2004). SD techniques, particularly “what-if” analysis, can also offer decision-makers valuable tools for predicting the broader consequences of process execution modifications (Pourbafrani et al., 2019). While alternative techniques exist that require less comprehensive process knowledge, their predictive capabilities are limited to short-term horizons (Tax et al., 2017, 2018) The use of SD can provide a systematic understanding of a system’s behaviour to direct attention to “what to do”, as demonstrated in other areas of engineering, such as risk management in construction projects (Wardito et al., 2021), forecasting (Saraji and Sharifabadi, 2017), freight transport decarbonisation (Ghisolfi et al., 2022) and the application of machine learning in self-adaptive systems (Saputri and Lee, 2020). Some demonstration of using SD use in combination with EE is also observed within the domain of EE, where Schneider et al. (2015) developed design guidelines for developing SD models to enhance enterprise architecture management. Their approach, exemplified through a CLD model and validated through expert interviews, highlights some of the possibilities of using SD within EE.

Examination of EE literature shows that a study is needed to determine the maturity of using SD in combination with EE and to determine to what extent such multidisciplinary research could add value. Taking cognizance of this gap, the present investigation attempts to highlight key researchers who work on these two disciplines concurrently, using the strengths of both disciplines to better understand enterprises, their behaviour and whether new enterprise design initiatives will increase enterprise performance. To the best of the authors’ knowledge, existing knowledge on EE and SD is limited to separate streams of research and their co-evolution has not been analysed systematically, with possible overlap in the use of concepts within these two disciplines. Based on this argument, the main objectives of this SLR are as follows:

• Objective 1: Identify how the literature regarding SD applications in EE has evolved over time and identify the prominent researchers within the combined use of SD and EE.

• Objective 2: Identify the co-occurrence of concepts within SD–EE publications.

• Objective 3: Identify the current state of the SD–EE research area to signify opportunities for further research.

The value of this SLR is justified not only by its address of the gap observed in the literature of SD application in EE, but the study also provides a direction for future research and design in the EE discipline.

The rest of this paper is organised as follows. Section 2 presents the research methodology followed in the study. Section 3 describes the publication trends and authorship observed in the literature. Section 4 defines the main concepts that developed within SD and EE. Section 5 synthesizes the results based on inductive analysis of extracted publications. Section 6 presents the main contribution of the study, namely, a discussion of the research status of the SD–EE research area. Section 7 presents conclusions and indicates limitations of the study and opportunities for future research.

2. Research method

This section presents the existing knowledge and developments in the application of SD tools in EE using an SLR and bibliometric analysis. The SLR, adapted from the guidelines of Okoli and Schabram (2010) and Okoli (2015), addressed the key findings and developments within this research area.

2.1 Research method for the systematic literature review

Steps 3–8 of Okoli and Schabram (2010) were used as a basis for a tailored research methodology, which included search execution, practical screening, quality appraisal, extraction, synthesis and presenting the key findings.

The tailored research methodology consists of the following steps:

• A general search was performed to identify any previous SLRs on the topic, using a scoping review as defined by Pham et al. (2014). Forward and reverse snowballing could then be applied to supplement the results with relevant literature. If the search suggested that no SLRs have been published on the topic, good search strings based on the research questions of the study were formulated.

• Database sources are selected to search for primary publications followed by a search.

• A practical screening is performed to include or exclude certain publications and limit the result set. This study focuses on publications which include journals, conference proceedings, book sections, books and practice-focused publications. Okoli (2015) provided a list of criteria that is used to further exclude some publications.

• To identify additional relevant publications, iterative forward and reverse snowballing is applied to the accumulated publications until no additional publications are found.

• To improve reliability of the findings, a quality appraisal based on the SLR guidelines presented by Kitchenham and Charters (2007) is performed.

• A codebook is developed using the guidelines provided by Guest et al. (2012) to extract data from the final paper set.

• The qualitative analysis software ATLAS.ti is used to synthesise the data extracted to make comprehensive sense of the large number of shortlisted publications.

• Key findings are reported based on the synthesized data and linked back to the main objectives of the study.

2.2 Protocol applied

This section refers to the research methodology steps defined in Section 2.1.

2.2.1 Steps 1 and 2: performing a general search on selected platforms.

A general search was conducted to identify any existing SLRs and to provide input for the development of a search strategy. The scoping study was performed on the following 10 platforms: EBSCO, Emerald Journals, IEEE Xplore, ProQuest, ScienceDirect, SpringerLink, Web of Science, Compendex, Scopus and JSTOR. These databases were chosen as they have already been carefully selected by Oladimeji et al. (2020) in their SLR after consulting with information specialists, and industrial and system engineering experts. The initial scoping review was performed using the search terms “system dynamics”, and “enterprise engineering” or “enterprise architecture”. Even though Hoogervorst (2018) considered enterprise architecture as a set of design principles, the terms “enterprise engineering” and “enterprise architecture” are still used interchangeably in the literature. As a result, both phrases were used in the scoping review. The search results included a large number of publications with platforms such as JSTOR giving 35,574 raw results. Due to the large number of results, the search was repeated with limiters being applied, significantly reducing the search results, but no existing SLR in the SD–EE research area was found.

The papers found, generally focused on implementing SD modelling tools at enterprises with the intention of improving business performance. Some of the publications that were found on more than one of the databases include the following: Roychoudhury et al. (2014b), Sousa et al. (2005), Assimakopoulos and Riggas (2006) and Yu et al. (2012). Other topics were also observed, such as simulation modelling with the use of software (e.g. I-think, ArchiMate, etc.), systems engineering and the discussion of virtual enterprises. Therefore, to further improve the search and reduce the results, a search strategy was designed to find papers with a primary focus on the application of SD in EE. The publications identified during the scoping review were used to identify key words to develop terminology for the search, to include key words and acronyms, such as “SD”, “EA”, “Enterprise model”, “Modelling” and “Enterprise analysis”. The keywords were used in conjunction with search tools, like proximity operators, truncations and exact phrases. Also, to ensure that the papers being studied were primarily focused on the application of SD in EE, only the title and the abstract of the publications were considered in the search instead of the full text.

The final search string used on the platforms was as follows: [(enterprise (engineering OR architecture OR modelling OR modeling)] AND(“system dynamics” OR “systems modelling” OR “systems modeling” OR “business dynamics”). The search string was adapted for each of the databases depending on its algorithm. This reduced the raw results in platforms, such as JSTOR, to 59 results. To test the effectiveness of the search string, an additional validation was performed, i.e. ensuring that the papers found in the scoping review were also present in the result set of the new search.

2.2.2 Step 3: practical screening.

Practical screening was performed on the primary search results. Okoli (2015) provides a list of criteria for practical screening, from which the following exclusion and inclusion criteria were applied:

• Papers duplicated on the databases were removed from the SLR.

• Only publications that had a bearing on the research objectives, and primarily focused on the application of SD in EE were included.

• Only publications from the year 2000 onwards up until 01 September 2022 were considered.

• Only articles written in English were included.

Table 1 below summarises the raw and final set of publications after the practical screening. The database with the most relevant publications for the SLR was Compendex (n = 4) and ProQuest (n = 4) followed by Web of Science (n = 3), SpringerLink (n = 3), ScienceDirect (n = 2) and Scopus (n = 2). From the remaining databases, no relevant publications were found for this SLR.

2.2.3 Step 4: snowballing.

For an extensive search, snowballing was also performed on the 18 practical-screened sets of papers selected for the SLR. This resulted in an additional 12 publications being found on the application of SD in EE. For forward snowballing, Google Scholar was used to identify the papers that cited the practical-screened selection of papers. For backward snowballing, a search was performed in the reference list of the practical-screened plus forward snowballed set of publications. This process was repeated until data saturation, i.e. where no additional relevant publications could be identified.

2.2.4 Step 5: quality appraisal.

A quality appraisal was performed on the final paper set according to the SLR guidelines proposed by Kitchenham and Charters (2007). This was done to provide an overall measure of the quality of the papers selected for the study. In accordance with the scope of the study, the following questions were selected from Kitchenham and Charters’ (2007) quality assessment checklist for qualitative publications:

• Is the aim of the publication clearly defined?

• Is the research design clearly specified?

• Is the link between the data, its interpretation and the conclusion clear?

• Is there enough evidence to support the claims made?

• Has the research process been documented adequately?

The questions were answered using the following scale: YES (1 point), PARTIALLY (0.5 point) or NO (0 point). A final quality score was provided by summing the scores from all five questions. Table 2, which has been adapted from Hussein et al. (2016), indicates the overall quality scores for the 30 papers selected for the SLR. The quality appraisal indicates that ten publications (33%) are of good quality, whereas nine publications (30%) are rated as very good quality. In addition, nine publications (28%) publications are rated as satisfactory, and two publications (7%) are of poor quality. No publications were found to be of very poor quality, and therefore all 30 papers were included for data extraction.

Figure 1 shows the PRISMA reporting standards, as defined by Page et al. (2021), used to graphically present the order in which the publications were selected for the study.

2.3 Method for codebook development and inductive analysis

The codebook shown in the following list was developed to help identify thematic elements, and the similarity, and dissimilarity, between them within the large data set extracted from the final paper set.

Codebook list (created by authors) is as follows :

Code label: Enterprise type (ET)

Brief definition: A type of entity engaged in an economic activity.

Full definition: The enterprise type answers the question: Which enterprise is being designed/re-designed through SD application? Enterprise type refers to certain classes or categories of enterprises, such as manufacturing, and telecommunication.

Code label: Enterprise performance area analysis (EAn)

Condensed definition: The area/factors within the enterprise requiring improvement.

Full definition: The enterprise analysis answers the question: Which performance area of the enterprise is being improved/understood through the use of SD application?

Code label: Enterprise engineering phase (EEP)

Condensed definition: The current phase in the process of the evaluation and improvement of the enterprise.

Definition: The enterprise engineering phase answers the question: Which phase of the enterprise is being explored through SD application? Enterprise engineering phase refers to the EE phase the research is focused on.

Code label: SD tools (SD)

Condensed definition: Specific modelling tools applied from the general principles of SD.

Full definition: The SD tools code answers the question: Which modelling tools are being used in the application of SD? SD tools code refers to any modelling technique that makes use of a causal loop diagram or a stock and flow diagram.

Code label: Data collection (DC)

Condensed definition: The technique used for DC.

Full definition: The DC answers the question: What techniques were used to collect data? DC can refer to techniques such as interviews(s), secondary data, case study or literature review.

Code label: Data analysis (DA)

Condensed definition: The technique used for DA.

Full definition: The DA answers the question: What techniques were used to analyse the data collected? DA can refer to techniques such as modelling, simulation, sensitivity analysis and or qualitative analysis.

Code label: Managerial implications (MI)

Condensed definition: The improvement to the enterprise from the application of SD models.

Full definition: The managerial implications answer the question: What benefits of SD are observed in the enterprise? Managerial implications refer to the benefits observed in the enterprise from the implementation of the SD model by industry professionals.

Source: Created by Authors

Coding is a form of analysis that allows the reader to interpret the data as it is read, reflected upon and coded. The guidelines stated by Guest et al. (2012) were used to develop codes in terms of a code label, brief definition, full definition, elaboration, cues, examples and where clarity is needed, what it is not. For brevity, we only included the code label and brief definition and full definition in this article. The codes developed in this study helped define the types of enterprises and their stages, the techniques and tools used in the application of SD in EE and their managerial implications (MI) in industry. Figure 2 illustrates the codes and graphically links the codes and the sub-codes to the main Research Objectives 1 and 3, by answering the following questions, related to the research objectives:

Q1.

How has the literature regarding SD applications in EE evolved over time?

Q2.

What is the current state of the SD–EE research area?

Q3.

What are the research gaps and opportunities for future research within the SD–EE research area?

The 30 papers were uploaded to the qualitative data analysis software ATLAS.ti, where segments of text were manually highlighted and assigned a code using a codebook that emerged incrementally and iteratively. During the coding process, the codes were revisited and refined as new elements emerged and potentially significant segments of text were encountered as themes. Figure 2 shows the codes that were defined up-front, prior to coding, to better understand the maturity and development of the research area.

A codebook serves as a consistent means of analysing the text, where multiple people should be able to analyse the same set of text and reproduce similar results. To validate the codebook, an acceptable inter-coder agreement was obtained, prior to coding. The inter-coder agreement of 83.3% was obtained, along with examples of the coding performed, available via this Google Drive link. According to Guest et al. (2012), an agreement of 80% or more is considered good. Based on the reliability of the codes defined in the codebook, the coding of the final paper set, presented in the aforementioned list, could proceed.

The results that were obtained from the SLR using the codebook (as presented in the aforementioned list) are presented and discussed in Section 5.

2.4 Method for concept identification

The approach summarised by Novak and Cañas (2008) was used to construct a concept map. To identify the key concepts, a software tool was used to extract concepts from qualitative data based on the co-occurrence of those concepts. The papers-for-SLR result set was uploaded onto the Leximancer software tool database (Smith, 2000), to address the second research objective:

Objective 2: Identify the co-occurrence of concepts within SD–EE publications.

The results are presented in Section 4.

3. Results on publication trends and authorship

Applying the protocol for Steps 6–8 of the SLR method presented in Section 2.1, this section presents the results extracted from the 30 selected publications.

One of the objectives of the search was to find relevant publications that show the development of this research area, i.e. addressing:

Objective 1: Identify how the literature regarding SD applications in EE has evolved over time, and identify the prominent researchers within the combined use of SD and EE.

An evaluation of the papers selected indicates that from 2005 onwards, a surge in publications on this topic is observed, with the most papers being published in 2014 and 2019. The publication trend presented in Figure 3 suggests that there is some increase in interest in the SD–EE research area. However, there are fewer publications on this topic in the year 2022 as this study only considered papers published before 1 September 2022.

The Compendex and ProQuest database had the most publications as indicated in Table 1. In the result set most publications were from peer-reviewed academic journals (57%), followed by peer-reviewed academic conference papers (33%) such as the IFIP Working Conference on The Practice of Enterprise Modelling and book sections (10%). Seventeen journals have been identified from the publications, with a large majority of the journals only having a single publication on this topic. The evidence suggests that this is a growing field where a gap may exist for more research.

To understand the current state of this research area, and to identify the prominent researchers, the impact of the publications is assessed by analysing the average yearly citations. This approach is similar to the one used by Oladimeji et al.’s (2020). Google Scholar was used to find the number of citations of publications from the result set, i.e. from the papers-for-SLR result set. Followed by a calculation of the average citations from the year of publication up until 9 November 2022. Figure 4 presents the results, with 60% of the papers from the result set having one or more citations per year.

The paper with the highest yearly citations is a conference paper titled Scenario-Based Prediction of Business Processes Using System Dynamics by Pourbafrani et al. (2019) with 9.7 citations every year. According to Google Scholar, the paper has been cited 29 times since its publishing in 2019. The paper proposes an approach to improve daily operations for business owners by predicting future behaviour, whereby SD tools, such as SFDs are used for what-if analysis. Similarly, the focus of the most cited papers in Figure 4 is to help improve the decision-making process in enterprises through the application of SD tools. The results show that SD tools are most used in the decision-making process of the enterprise.

A co-authorship analysis was also performed on the final paper set using network analysis to understand the relationship between the authors and to identify the key authors in the SD–EE research area. The network analysis was performed using the VOSviewer version 1.6.18 software tool developed by Leiden University’s Centre for Science and Technology Publications (Van Eck and Waltman, 2017).

The co-authorship network in Figure 5 shows the authors with their names and a circle. The size of the circle is determined by the number of publications by the respective author, and the lines between the circles represent their collaboration on publications. The colour of the circle is determined by the cluster to which the circle belongs. In the visualization, the distance between two clusters approximately indicates the relatedness of the two authors in terms of co-citation links. The results from the network visualization show that there were 83 authors and co-authors in this result set, with 154 interactions between the authors. Most co-citations in the result set were authored by Vinay Kulkarni (n = 4) as shown in Table 3. Only authors that have authored a paper in the result set, twice or more, are mentioned in Table 3.

Vinay Kulkarni emerged as an active author in the SD–EE literature, followed by Suman Roychoudhury and Sagar Sunkle. Vinay has co-authored the following papers with Suman, and Sagar: (Multi-) Modeling Enterprises for Better Decisions (Sunkle et al., 2013a), Using Intentional and System Dynamics Modeling to Address WHYs in Enterprise architecture (Sunkle et al., 2013b), Models to aid Decision Making in Enterprises (Roychoudhury et al., 2014a) and Toward Structured Simulation of Enterprise Models (Roychoudhury et al., 2014b). However, Figure 4 suggests that although Vinay has been an active author, his work may not have had the same level of impact as that of Pourbafrani et al. (2019). The higher number of citations of the work by Pourbafrani et al. (2019) may be attributed to its open-access nature and novelty and relevance in the field. The paper is significant because it is the first to combine process mining and SD; hence, other researchers referenced and used its methodologies and findings.

The graph density is also calculated to understand the level of interaction in the network. A higher level of interaction would suggest that the research area has significant overall expertise and collaboration. Equation (1), developed by Otte and Rousseau (2002), was used to calculate a graph density of 2.2%. According to Scott (1988) a graph density of 60% or more is indicative of a research area with prominent expertise and a good collaboration. The very low graph density observed in the current network suggests that there is little interaction among the authors in the network. The research area has little expertise, and an effort needs to be made to increase collaboration:

(1) Graph density (%) = number of lines in the graphmaximum number of lines×100

4. Results on main concepts for EE and SD

This section addresses one of the main objectives of the study by presenting an indication of the affinity of concepts used together in SD–EE work published from the year 2000 to September 2022, i.e.:

Objective 2: Identify the co-occurrence of concepts within SD–EE publications.

The concept map as an analysis method was discussed in Section 2.4. As shown in Table 4, Leximancer detected three major themes that emerged from the publications.

The concepts identified by Leximancer were used in combination with the keywords listed in the papers for the nodes representing the concepts (Chen et al., 2008). A hierarchical organisation was built using linking statements. The linking statements were restricted to propositions to specify a clear relationship between the concepts. The IHMC CmapTools (Cañas et al., 2004) computer software program is used to construct the final concept map shown in Figure 6.

The figure shows that current publications focus on the integration and overlap of SD modelling methodology with the following key concepts: Business Ecosystem Modelling, Systemic Enterprise Architecture Methodology, Enterprise Modelling, Reference Model, Executable Architecture, Business Intelligence Model, Enterprise Engineering Process, Enterprise Architecture Model, Zachman Framework and Intentional Modelling. Figure 6 further details the focus of these concepts. For example, SD modelling has been integrated with enterprise modelling to help support business decision-making in production systems (Agyapong-Kodua et al., 2012).

5. Results of inductive analysis

This section presents the content analyses results, structured according to the seven main codes (ET, EEP, SD, EAn, DC, DA and MI) that are linked to Research Objectives 1 and 3, as presented in Figure 2.

5.1 Enterprise type (code: ET)

The industries studied in the final paper set were coded to understand the industry context of the publications. A significant number of the papers (n = 16) did not specify a specific enterprise for which SD applications were being considered. Rather, the publications focused on conceptual frameworks applicable to any enterprise for addressing common challenges. Some examples of the phrases used by the publications to suggest general enterprises are as follows: “industrial world”, “an organization”, “today’s businesses” and “modern enterprises” (Roychoudhury et al., 2014a; Gozali et al., 2022; Pourbafrani et al., 2019; Stadtländer et al., 2021). For example, Gozali et al. (2022) reviewed several models and frameworks including Multi-Perspective Enterprise Modeling, Capability Oriented Enterprise Modeling, Model-Based Enterprise Engineering, and SD, to examine how each model measures performance management, strategic planning, critical success factors and enterprise architecture and components.

Figure 7 shows the publications with SD applications in specific enterprises. The most mentioned enterprises were the service industry which included the “education sector”, the “healthcare sector” and “commercial and public administrations” (Fayoumi and Loucopoulos, 2016; Medvedeva et al., 2019; Dantu, 2011). This is followed by the telecom industry and manufacturing enterprises. Fayoumi and Loucopoulos (2016) contributed to the field of enterprise architecture modelling by presenting an SD application through an e-learning service case study. They demonstrated how to design intelligent information systems using SD and conceptual modelling to adapt to the evolving requirements of an enterprise. In Figure 7, the remaining industries mentioned are only found in one publication.

It is notable that there is a broad variety of the types of industries being explored in this research area including examples of governance and virtual enterprises. The results show that there is a growing collaboration between the design of SD applications in EE and its practical application, but future publications should focus on the development and re-application of SD techniques within specific enterprises that are underexplored. Fayoumi and Loucopoulos (2016) identified the need for the development of a single meta-modelling framework for SD modelling by implementing SD modelling approaches in “different companies from different industries and under different circumstances”. Furthermore, in the result set, there was no multi-industry or cross-industry analysis observed. According to Oladimeji et al. (2020), multi-industry or cross-industry analyses would be indicative of the progress in a research area, and therefore it is suggested that such publications are needed in future.

5.2 Enterprise engineering phases (code: EEP)

The five phases of enterprise design or re-design include identifying the need to design the enterprise, the design of the enterprise, implementation of the design, operating the new enterprise and monitoring it for the need for a new design (Sousa et al., 2002). Based on these phases, the papers published in this research area were coded and categorized to be in one of the four phases, namely, design, implementation, use and review.

In the design phase, SD is used for planning, conceptual design, detail design and modelling during the enterprise design phase to enhance the enterprise. The design phase is the most common EE phase in this research area, as shown in Figure 8. Sixteen papers stated the use of SD tools in their design approach for a “to-be” state of the enterprise. Tools, such as CLDs and SFDs, help specify the factors influencing the enterprise within a specific context, resulting in useful models that can be used for “what-if” analysis (Agyapong-Kodua et al., 2009). The results from the EAn code show that while the publications share a common focus on using SD for modelling and decision-making, there are notable differences in their applications and objectives. In the result set, a multi-view modelling approach is observed that integrates enterprise architecture frameworks, enterprise modelling techniques, qualitative and qualitative reasoning and SD modelling to align information systems with business goals. Bigdeli et al. (2019) modelled the causal relationships that contribute to organizational agility with a focus on the telecom industry in Iran, Dantu (2011) discussed the use of SD techniques in integration with the Zachman Framework for healthcare systems, and Fayoumi and Loucopoulos (2016) introduced a paradigm for designing intelligent information systems using conceptual modelling and SD techniques for simulation purposes.

The second most common EE phase explored in literature was the implementation phase. Fourteen papers were found to apply SD tools to model the execution and deployment of a design in the enterprise. Achieving maturity in the implementation phase enables the creation and positioning of “fit-for-purpose” multi-level models that can effectively address the enterprise’s unique context and environmental influences. For example, Agyapong-Kodua et al. (2012) developed an integrated enterprise and SD modelling methodology and applied it to transform a bearing manufacturing company’s CIMOSA enterprise model into a structured causal loop model (SCLM). The SCLMs allow for the quantitative analysis of the influences and interactions between variables in the manufacturing enterprise. The results show that SD tools are more commonly used for the design of the enterprise rather than used in the implementation of the design. Roychoudhury et al. (2014a) provided another example within the implementation phase where an SD tool, namely, an SFD, is used to capture the goals, tasks and behavioural aspects of the enterprise by analysing peak season load.

In the use phase, SD is used for assessing different scenarios. Eight papers were found to extend their enterprise implementation approach and perform “what-if” analyses or consider different scenarios. The results show that although the purpose of most of the papers in this research area is to use ICT capabilities in the workplace for improved IT-business alignment, and better decision-making, there are very few papers that validate these models within industry. Pourbafrani et al. (2019) highlighted the limitation of process mining techniques being primarily “backward-looking” rather than “forward-looking”. The authors propose a scenario-based predictive approach using SD to address this gap, enabling the exploration of “What-if” scenarios. The research area is still in its early phase, and there is a clear need for practical applications of the design approaches proposed in these publications.

The least common Enterprise engineering phase is the review phase. The review phase refers to the use of SD for monitoring, measuring and evaluation of the new “As-Is” state of the enterprise. The papers in the review phase used SFDs to ensure that the outputs correspond to actual system performance.

Few publications focused on more than one phase, where the design proposed for the “to-be” state was followed by an implementation in an example or real-life application. A significant portion of the literature in this research area focuses on the design phase, particularly on the integration of SD techniques with enterprise modelling to support business decision-making, improve organisational performance and address the challenges faced by enterprises in dynamic and complex environments. Non-modellers in the industry can also benefit from the insights generated by SD models through accessible visualizations and using the insight to inform high-level decision-making. The design phase publications suggest that these multi-models help in localising decision-making in the enterprise, however, there is still a need for these models to be implemented and explored in actual enterprises.

5.3 Extent of system dynamics application for performance areas (codes: SD and EAn)

In the paper set, the extent of SD application was investigated. The coding reveals that the papers reported the application of CLDs and their translation into SFDs, as well as only SFDs, and only CLDs. More than one SD tool was commonly associated with a single application. SFDs are used to model complex decision-making in enterprises. They were largely used to help capture the relationship between the chosen variables, and through calculations provide the future values of these variables at different points in time (Pourbafrani et al., 2019).

SD tools were not directly applied in 24% of the papers, rather these papers suggested the use of the general principles of SD modelling in their approach to enterprise design. Figure 9 shows that almost half of the paper set used simulations, with only 17% of the papers testing and reporting scenarios concerned with decision-making in the enterprise. The findings reveal that few papers in the final paper set reported the testing of alternate scenarios even if they were performed. Further research may add value beyond the design phase, applying SD models to the current state of the enterprise. A very small percentage of the papers (3%) also investigated the adaptation of existing SD tools. For example, Tulinayo et al. (2013) applied object-role modelling to SD concepts to translate object-role models to SFDs.

The EAn code identified which performance area of the enterprise is largely being improved or understood using the SD application. In this SLR, the performance area of an enterprise refers to the key aspects that are critical to the functioning and success of the enterprise. As shown in Table 5, the following eight performance areas were adapted from Folan and Browne (2005) and identified in the final paper set: Enterprise Model (n = 6), IT Performance (n = 6), Performance Management (n = 4), Operational Performance (n = 4), Organizational Performance (n = 4), Strategic Performance (n = 2), Financial Performance (n = 2) and Knowledge Management (n = 2).

The findings of the study indicate a shift in the enterprise being understood or developed through SD modelling. Previous studies primarily concentrated on enhancing the strategic performance, financial performance, organizational performance, operational performance and performance management areas. However, from 2020 onwards, there has been a greater emphasis on enterprise models and the integration of IT tools to support enterprise capabilities. The more comprehensive approach to SD–EE research is reflective of the growing recognition of the interconnectedness of various factors that impact alignment within the enterprise.

5.4 Methodologies (codes: DC and DA)

Coding was used to identify the techniques used for the collection of data, and its analysis in the final paper set. In the result set, a significant percentage of the publications were from peer-reviewed academic conference papers and peer-reviewed academic journals (90%), followed by book sections (10%). These also included practice-focused publications (40%). The practice-focused publications focused less on methodologies and more on application. Examples of these include applying SD tools to create and evaluate a virtual enterprise that constructs wireless payments, and the use of SD tools for decision-making at an e-learning services company based in Saudi Arabia (Assimakopoulos and Riggas, 2006; Fayoumi and Loucopoulos, 2016).

Figure 10 shows that the most common data collection (DC) technique observed is case studies, largely due to the applications recorded from the practice-focused publications. Case applications are implementations of SD techniques in real-life cases or demonstrative cases. The real-life case studies support the integrity of the research work by directly corroborating the design with evidence of its implementation. Other prevalent DC techniques include literature reviews and interviews. Literature reviews and interviews were the most used DC technique in academic publications. For example, Stadtländer et al. (2021) performed a literature survey using articles that applied SD tools to businesses. The data extracted was used to motivate the use of reference modelling to eliminate the shortcomings that arise from the use of SD tools in business development. The results show that the academic publications in the SD–EE area are more focused on the design of SD applications in EE, with some papers exploring the usefulness of applying SD techniques in enterprises.

Figure 11 shows the techniques that were used to analyse the data. Mathematical modelling followed by simulation is the most used DA technique. The results suggest that more research work needs to be done focusing on the testing of data, as there are only five papers that performed scenario analyses. There is also a need for other DA techniques to be introduced and used, such as statistical analysis, and research publications at a much larger scale. The practice-focused publications do not make use of DA techniques, as the papers are not answering a research question, but rather focusing on a general approach for the implementation of SD tools in EE. An evaluation of the results reveals that after 2006, papers in the SD–EE research area shifted from being completely conceptual to introducing empirical publications to support exploratory research. This shows that there has been some development in the application of SD tools in EE.

5.5 Managerial implications (code: MI)

The publications in the result set collectively provide insight into the role of managers in various aspects of enterprise management including decision-making, complexity management and performance management. The practice-focused publications in this research set received feedback from managers recognising that applying SD tools to the enterprise improves understanding of the information flow within the enterprise (Agyapong-Kodua et al., 2012; Fayoumi and Loucopoulos, 2016, 2022). The managers describe the tools as a guide for the enterprise’s thinking and design (Fayoumi and Loucopoulos, 2016; Medvedeva et al., 2019; Nakamura, 2016). This is because CLDs illustrate the cause-and-effect relationship between variables, and present a new perspective to the decision-making process (Schneider et al., 2015).

Sousa et al. (2005) highlighted the role of managers in interpreting and using performance measures effectively and explored the conceptual design of performance measurement and management systems using SD techniques. The publications collectively demonstrate a clear need for future research work that includes direct involvement of managers in mapping CLDs. Current work lacks collaboration between industry professionals and academics which can facilitate modellers to have a holistic understanding of the organisation’s goals, leverage appropriate modelling techniques, align IT with business strategies and assist managers in making informed decisions to improve organisational performance. Also, it is recognised that the usability of SD tools by managers in the workplace needs to be improved for better execution.

The academic-focused publications reinforce the need for collaboration between industry professionals and academics. The models proposed can be implemented in any modern enterprise making them more applicable in industry. However, future research work in this area can improve these models for application within a specific industry. The academic publications identify that SD applications can create a shared understanding of enterprise systems, enhancing the decision-making process in a dynamic environment. Also, the exploratory research work suggests that there is a need for validation of the SD models proposed (Pourbafrani et al., 2019; Rengkung, 2018; Stadtländer et al., 2021). Future work should focus on field experiments that can improve the validity of SD application in EE and allow work to be done on the limitations and shortcomings that are encountered through the application of the models. For example, Rengkung (2018) constructed a basic model of a CLD as a guideline for enterprises to model and analyse dynamic variables. The model can provide a comprehensive understanding of organisational behaviour and can also be extended to an SFD. However, as Rengkung (2018) himself identified, there was a need for the model to be applied and tested.

6. Discussion of SD–EE research maturity

The results from the SLR are used to assess the maturity of this research area. Oladimeji et al. (2020) proposed a maturity model adapted from the works of Pöppelbuß and Röglinger (2011), Bititci et al. (2015), Van Aken et al. (2005), Wettstein and Kueng (2002) and Keathley-Herring et al. (2016). The framework assesses the maturity of the research area based on the following measures: publication characteristics, content characteristics, authorship characteristics, research design characteristics and impact characteristics. We adopted this framework because it consolidates maturity levels from multiple authors, and it was applied successfully by Oladimeji et al. (2020) to assess the maturity of SD applications in performance management research.

Each of the measures are further characterised as emerging, developing or mature. Table 6 elaborates on assessing the maturity of a research area as proposed by Keathley-Herring et al. (2016). The authors proposed a criterion for assessing maturity within a research field based on a bibliometric analysis of 123 papers, several iterations of independent reviews and group discussions.

Applying the maturity framework, the subsequent sections present elaborate on the maturity of SD–EE research, also summarised in Figure 12.

The first criteria, publication characteristics, is appraised as emerging. Publications in this research area include peer-reviewed academic conference papers and peer-reviewed academic journals (90%), followed by book sections (10%). Despite the low average publication frequency [< 4 papers per year (Shvartsman and Zeldina, 2019)], the publication trends show that there is a growing interest in the development of the combined use of SD and EE. The result set includes a broad variety of industries such as virtual, governance and manufacturing enterprises. However, the papers are mostly published in Engineering and IT-related journals and conferences, and not many papers are published from the same outlet indicating that there is no journal dedicated to this area. For a developing maturity level, future research work should aim for more visibility and variety in publication types by including practice-focused magazines, as this will encourage development in the research area, and collaboration between industry professionals and academics.

The second criteria, content characteristics, is also appraised as emerging. The coding revealed that most of the work in SD–EE is conceptual and focused on the design phase. The SD models being proposed still need to be validated and redefined, with most models only being tested for feasibility on fictitious demonstration case studies (Fayoumi and Loucopoulos, 2016; Golnam et al., 2010; Nalchigar et al., 2014; Roychoudhury et al., 2014a). Although consistency in the content of the research work is observed, there is scope for further development through implementation in the industry. In a similar study, Oladimeji et al. (2020) identified the content characteristics of the SD and performance management research area as developing as there is an agreement between the authors on fundamental aspects such as terminology and keywords. However, while SD and EE have created their own respective languages, the present undertaking remains focused on an effort to bridge the gap between concepts within the two research areas. For instance, Bueno et al. (2014) translated the Department of Defense Architecture Framework’s Operational View concepts into SD-related concepts to develop executable architecture. Similar research is needed to translate EE concepts into SD concepts.

The next criteria for the maturity assessment are authorship characteristics which include the number of authors, number of citations and co-citations. The co-authorship analysis from the SLR identified 83 authors with minimal links between them, indicating a need for more collaboration in this research area. Although new authors emerge, very few of them collaborate on publications, with only nine authors having published more than once in this research area. In a similar study, Keathley-Herring et al. (2016) appraised the authorship characteristics aspect as developed in the field of Engineering Management (EM). This is because 451 authors are identified in the bibliometric analysis, with a stable set of “experts” that regularly publish in the EM field. An active researcher identified in the SD–EE area is Vinay Kulkarni, who co-authored four papers (Roychoudhury et al., 2014a; Roychoudhury et al., 2014b; Sunkle et al., 2013a; Sunkle et al., 2013b). However, Pourbafrani et al. (2019) were identified as the most impactful authors with the highest number of citations. The authorship characteristics aspect is still in its early stages and is emerging. Future research should not only focus on promoting collaboration among authors but also on including authors from different disciplines, such as business management, as this can introduce a new perspective and enhance the practical applications of the studies.

The next criteria, research design characteristics, assesses the research methods used in the literature, and is appraised as emerging. Prevalent methodologies included case publications, literature reviews, followed by gathering data through interviews. For EE research, action design research (ADR) is suggested as an appropriate research methodology (De Vries and Berger, 2017), when problem analyses precede the suggestion and development of a solution artefact or initiative, implemented within the real-world context. Within the ADR paradigm, we believe that SD tools could be useful during the problem analysis phase to understand the complex behaviour of an enterprise system, contributing towards problematic behaviours. A better understanding of the enterprise system should inform decision-making when suggesting an appropriate solution artefact or intervention to address a problem. The results showed that only 20% of the papers performed a scenario-analysis for prediction-based decision-making. Within the EE context, SD scenario-analysis results may contribute to better decision-making in deciding “what to do” when suggesting a solution artefact or intervention.

The analysis of the impact characteristics indicates that 62% of the papers are cited at least once a year from their date of publication. The most impactful publication, authored by Pourbafrani et al. (2019), proposed an approach to improve daily operations for business owners by predicting future behaviour, whereby SD tools, such as SFDs, are used for what-if analysis. Although some interest exists in the use of SD tools in EE, especially in the use of scenario-analysis, the impact characteristics aspect is still emerging and needs more development to increase the number and impact of SD–EE publications.

7. Conclusions, limitations and future work

Overall, the SLR results show that the combined use of SD and EE is an emerging research area that still needs to develop its maturity in authorship, content, impact, publication and research design characteristics. While all phases, related to the evolution of enterprises, and related to the discipline of EE, have received attention, the current research work is more focused on the enterprise design phase. The publications focus on the development of a multi-view modelling approach that integrates enterprise architecture frameworks, enterprise modelling techniques, qualitative and qualitative reasoning and SD modelling to align information systems with business goals. The other phases need to be explored with the combined use of SD to understand the dynamic behaviour of complex enterprise systems to guide, implement and evaluate the incremental evolution of enterprise design domains and to develop “fit-for-purpose” multi-level models that can effectively address the enterprise’s unique context and environmental influences. As shown in the network analysis, a collaboration among the authors publishing in this SD–EE area, and a collaboration amongst industry professionals and academics will benefit the development of the area.

The SLR conducted for this research was limited to publications from the 10 platforms identified in the research methodology and publications in the English language with available text. To ensure a more comprehensive analysis, future research can expand the research methodology to include additional platforms. In addition, the concepts identified were mainly extracted from abstracts, keywords and the Leximancer software tool. Future work can adopt a more rigorous approach to clarify the concepts that are shared between SD and EE. This could involve using additional analytical tools or seeking expert opinions to ensure a more accurate representation and understanding of the identified concepts.

Furthermore, the analysis of the application of SD in EE was conducted in a general manner. Future research should aim to examine context-specific applications. This could include analysing specific geographical areas, conducting cross-industry or multi-disciplinary analyses or examining different phases of enterprise, such as design, implementation, use and review. In addition, future research could use SD tools to inform decision-making on “what to do” when suggesting a solution artefact or intervention in ADR studies.

The study investigated the current research themes of using SD and EE in combination. Future work has to explore potential research gaps and novel themes, such as understanding the dynamics of enterprises with simulations using digital twins, prior to adapting enterprise designs, exemplified by Barat et al. (2022). Another research theme that already received attention within the EE discipline, is to enable enterprises via artificial intelligence (Kulkarni et al., 2023). Predictive analysis and simulation, already offered via SD. The role of artificial intelligence in predictive analysis and simulation within this, could be further explored and how the analysis results may be used to guide enterprise design.

The results from the SLR provide valuable insight into the lack of industry application of the proposed conceptual models and designs in the publications. Therefore, testing the validity of these proposed models and designs should be the objective for future research. In addition, the thematic analysis of the literature can be extended to identify trends in this research area, with an emphasis on the development and re-application of the SD tools used. Future research can also develop a vision for extending the reach of the SD–EE domain by proposing a research agenda to guide prospective SD–EE researchers towards research within the SD–EE field. It may also be useful for future research to compare a well-established discipline, such as EM, with the emerging SD–EE research area to further validate the maturity levels presented in this article.

Figures

Figure 1.

PRISMA standard of reporting to show publications selection heuristic for the SLR

Figure 2.

Graphical representation of the link between questions related to objectives 1 and 3, codes and sub-codes

Figure 3.

Publications per year (n = 30)

Figure 4.

Average yearly citations for the most cited papers

Figure 5.

Co-authorship network visualization

Figure 6.

Concept map of SD modelling applications in the Papers-for-SLR set

Figure 7.

Enterprises observed in the literature

Figure 8.

EE phases explored in the research area

Figure 9.

Extent of SD application in literature

Figure 10.

Data collection techniques used in the final paper set

Figure 11.

Data analysis techniques used in the final paper set

Figure 12.

Maturity assessment framework of this research area

Table 1.

Results after practical screening

Platforms	Results	Excluded publications	Practical-screened results (N)
Web of Science	7	4	3
Emerald journals	12	12	0
SpringerLink	83	80	3
IEEE Xplore	11	11	0
ScienceDirect	6	4	2
Compendex	17	13	4
Scopus	19	17	2
JSTOR	59	59	0
EBSCO	76	76	0
ProQuest	57	53	4

Source: Created by authors

Table 2.

Quality scores

Total quality score	Very poor (> = 1)	Poor (> = 2)	Satisfactory (> = 3)	Good (> = 4)	Very good (= 5)
No. of publications	0	2	9	10	9
Percentage (%)	0	7	30	33	30

Source: Created by author’s

Table 3.

Number of co-citations occurring by authors in the results set

Author	Publications
Kulkarni, Vinay	4
Roychoudhury, Suman	3
Sunkle, Sagar	3
Rathod, Hemant	2
Fayoumi, Amjad	2
Loucopoulos, Pericles	2
Rajbhoj, Asha	2
Matthes, Florian	2
Yu, Eric	2

Source: Created by author’s

Table 4.

Main concepts identified by Leximancer

Theme	Concepts	Hits
Model	Model, system, business, enterprise, process, models, modeling, systems, approach, dynamics, design, modelling, used, using, dynamic, processes, different, analysis, simulation, based, use, support, case, paper, development, study, SD, architecture, decision, work, elements, complex, need, making, order and specific	4,384
Performance	Performance, time, information, level, data, organization, factors, goals, change, example, related, value, number and services	2,485
Capabilities	Capabilities, management, framework, strategic, important and research	989

Sources: Created by author’s; Smith (2000)

Table 5.

Performance areas identified in the papers-for-SLR set

Performance area	Definition	Author(s)
Enterprise model	The development and maintenance of models that represent the structure, activities and relationships of an enterprise	Buckl et al. (2008); Yu et al. (2012); Sunkle et al. (2013a); Schneider et al. (2015); Stadtländer et al. (2021); Gozali et al. (2022)
IT performance	The efficiency and effectiveness of an enterprise’s IT system and services in supporting enterprise operations and goals	Huang et al. (2008); Roychoudhury et al. (2014a); Fayoumi and Loucopoulos (2016); Bigdeli et al. (2019); Fayoumi and Loucopoulos (2022); Tulinayo et al. (2013)
Performance management	The establishment of clear and measurable goals, and developing performance indicators that align with those goals, and tracking and evaluating performance against those indicators to identify areas for improvement and inform decision-making	Sousa et al. (2005); Popova and Sharpanskykh (2010); Golnam et al. (2010); Pourbafrani et al. (2019)
Operational performance	The effectiveness and efficiency with which an enterprise is able to carry out its day-to-day operations, including production, delivery, customer service and other key activities	Ajaefobi and Weston (2006); Dantu (2011); Roychoudhury et al. (2014b); Bueno et al. (2014)
Organizational performance	The measuring, monitoring and continuous improvement of the performance of the enterprise as a whole and its various departments and functions	Assimakopoulos and Riggas (2006); Rashid et al. (2009); Rajbhoj and Saxena (2016); Rengkung (2018)
Strategic performance	The ability of the enterprise to achieve its long-term goals and objectives through effective planning, implementation and management of strategies	Nalchigar et al. (2014); Nakamura (2016)
Financial performance	The enterprise’s ability to generate profits and create value for its stakeholders by effectively managing its financial resources	Agyapong-Kodua et al. (2012); Sunkle et al. (2013b)
Knowledge management	The identifying, acquiring, creating, sharing and leveraging of knowledge and information assets to support the organization’s goals and performance	Ma (2019); Medvedeva et al. (2019)

Source: Created by author’s

Table 6.

Conditions for assessing a maturity level for a research area

Characteristic	Maturity level	Conditions for achieving the maturity level
Publication	Emerging	Not meeting with the conditions of developing
	Developing	Moderately meeting all the criteria and some of the sub-criteria of mature
	Mature	Strongly meeting all the following criteria and sub-criteria: • Publication quantity: including publications (scholarly output), and publication trend (no. of papers per year and percentage increase/decrease) within author, theme, institution, country/region and journal • Publication outlets: outlets represented including the no. of unique outlets, the proportion of papers by outlet type (diversity of outlets), the outlet concentration, the proportion of papers in dedicated versus non-dedicated outlets and the number of disciplines represented in dedicated journals • References: including reference quantity (average references per paper), reference age (average age of references per paper), most referenced papers (author and theme) and reference concentration (journal)
Content	Emerging	Not meeting with the conditions of developing
	Developing	Moderately meeting all the criteria and some of the sub-criteria of mature
	Mature	Strongly meeting all the following criteria and sub-criteria: • Themes: including themes represented [number of themes identified (diversity of themes), and proportion of papers by theme], connection amongst themes (co-occurrence of themes), stability of theme’s characteristics and theme-related citation consistency • Scope: including unit of study (i.e. function, sector or country/region), addressing previously identified work and orientation to practice (proportion of practitioner papers that adopt academic research findings, proportion of papers that explicitly focus on implications for practice and proportion of academic papers that address practitioner priorities) • Topics: including development of sub-fields, consistency of terminology, keywords represented and connections among keywords
Authorship	Emerging	Not meeting with the conditions of developing
	Developing	Moderately meeting all the criteria and some of the sub-criteria of mature
	Mature	Strongly meeting all the following criteria and sub-criteria: • Author quantity: including existing authors, and rate of new authors • Author diversity: including the disciplines represented, institutions represented and countries/regions represented • Collaboration: including collaborators and collaboration between authors, disciplines, institutions and countries/regions. Including multi-author papers and connections between authors using co-authorship social network analysis (SNA) metrics, concentration and research groups
Research design	Emerging	Not meeting with the conditions of developing
	Developing	Moderately meeting all the criteria and some of the sub-criteria of mature
	Mature	Strongly meeting all the following criteria and sub-criteria: • Research methods: including methods represented [proportion of papers per method type, number of methods used (diversity of methods), proportion of papers using qualitative and quantitative data, proportion of papers where industry informs method selection and proportion of papers using mixed methods], multi-method papers and level of analysis within data collection, analysis and inference; • Rigor: including approach, clarity in research objectives/questions (proportion of papers that explicitly define research questions), reliability and validity (strength of evidence for reliability/validity), statistical rigor (proportion of papers that test statistical hypotheses), thoroughness (papers that identify gaps in the research and identify the limitations/challenges) and connection to the literature (papers that identify future work); • Variables: including the type of variables that are represented (proportion of papers per variable, number of variables identified, proportion of papers using moderating/mediating variables) and variables operationally defined and measured; and • Research orientation: including orientations represented (proportion of papers with a theoretical/applied focus or inductive/deductive focus)
Impact	Emerging	Not meeting with the conditions of developing
	Developing	Moderately meeting all the criteria and some of the sub-criteria of mature
	Mature	Strongly meeting all the following criteria and sub-criteria: • Author prominence: including institution/program rank, and author productivity (number of papers per author); and • Publication prominence: including outlet prominence (journal rank and average journal impact factor at the time of publication), citations, concentration, seminal publications and forward co-citation analysis

Sources: Created by author’s; adapted from Keathley-Herring et al. (2016)