Critical success factors for Security Education, Training and Awareness (SETA) programme effectiveness: an empirical comparison of practitioner perspectives

Abstract

Purpose

Cyber security has never been more important than it is today in an ever more connected and pervasive digital world. However, frequently reported shortages of suitably skilled and trained information system (IS)/cyber security professionals elevate the importance of delivering effective Security Education,Training and Awareness (SETA) programmes within organisations. Therefore, the purpose of this study is the questionable effectiveness of SETA programmes at changing employee behaviour and an absence of empirical studies on the critical success factors (CSFs) for SETA programme effectiveness.

Design/methodology/approach

This exploratory study follows a three-stage research design to give voice to practitioners with SETA programme expertise. Data is gathered in Stage 1 using semi-structured interviews with 20 key informants (the emergence of the CSFs), in Stage 2 from 65 respondents to a short online survey (the ranking of the CSFs) and in Stage 3 using semi-structured interviews with nine IS/cyber security practitioners (the emergence of the guiding principles). Using a multi-stage research design allows the authors to propose and evaluate the 11 CSFs for SETA programme effectiveness.

Findings

This study conducted a mean score analysis to evaluate the level of importance of each CSF within two independent groups of IS/cyber security professionals. This multi-stage analysis produces a ranked list of 11 CSFs for SETA programme effectiveness, while the difference in the rankings leads to the emergence of five CSF-specific guiding principles (to increase the likelihood of delivering an effective SETA programme within an organisational context). This analysis also reveals that most of the contradictions/differences in CSF rankings between IS/cyber security practitioners are linked to the design phase of the SETA programme life cycle. While two CSFs, “maintain quarterly evaluation of employee performance” (CSF-DS6) and “build security awareness campaigns” (CSF-EV1), represent the most significant contradiction in this study.

Originality/value

The 11 CSFs for SETA programme effectiveness, along with the five CSF-specific guiding principles, provide a greater depth of knowledge contributing to both theory and practice and lays the foundation for future studies. Therefore, the outputs of this study provide valuable insights on the areas that practice needs to get right to deliver effective SETA programmes.

Citation

Alyami, A., Sammon, D., Neville, K. and Mahony, C. (2024), "Critical success factors for Security Education, Training and Awareness (SETA) programme effectiveness: an empirical comparison of practitioner perspectives", Information and Computer Security, Vol. 32 No. 1, pp. 53-73. https://doi.org/10.1108/ICS-08-2022-0133

1. Introduction

Cybersecurity and securing information system (IS) assets have never been more important than it is today in an ever more connected and pervasive digital world (Khando et al., 2021). In fact, the cybersecurity market size is expected to surpass $400bn by 2027 (fortune.com, 2022). Organisations cannot afford the disruption brought by digital attacks and are constantly seeking to protect their critical systems and sensitive data. To mitigate against major cybersecurity incidents, organisations are constantly looking for ways to ensure that their internal (e.g. employee) and external (e.g. supplier) stakeholders are aware of potential IS/cyber security threats and have the “know how” to respond. However, it is argued that “cybercrime actors” are reinventing themselves (developing new capability) at a far quicker pace than organisations are investing in cybersecurity capabilities (fortune.com, 2022). Hence, there is a need for “remarkable changes in how companies prioritise and address their cyber risks” (fortune.com, 2022).

Extant research suggests that organisations use various strategies to safeguard their information assets against IS/cyber security threats, and the Security Education, Training and Awareness (SETA) programme is one of the most prominent strategies used for controlling IS/cyber security threats and protecting information assets (Alshaikh et al., 2018; Kirova and Baumoel, 2018; Tsohou et al., 2015; D’Arcy et al., 2009). In fact, according to Global Market Estimates (2022), the market for IS/cyber security awareness training is anticipated to increase to a value of $12.1bn by 2027, representing a compound annual growth rate of 45.6% from 2022 to 2027. However, SETA programmes often produce questionable results around effectiveness and fail to build an acceptable level of awareness and know-how among employees. In fact, employees having a greater appreciation of “what to do when” is a key outcome for SETA programme effectiveness. Furthermore, frequently reported shortages of suitably skilled and trained IS/cyber security professionals further elevate the importance of delivering effective SETA programmes within organisations.

A SETA programme is viewed as an educational process designed to reduce the number of accidental security breaches that occur due to a lack of individuals’ awareness of IS security (Whitman and Mattord, 2008; D’Arcy et al., 2009; Puhakainen and Siponen, 2010; Han et al., 2017; Alshaikh et al., 2018; Barlow et al., 2018; Yoo et al., 2018; Dhillon et al., 2020). The significance of SETA programmes is widely accepted by academics and practitioners (Alshaikh et al., 2018; Tsohou et al., 2015; D’Arcy et al., 2009; Wilson and Hash, 2003). However, despite the prominence of SETA programmes for organisational IS security, “only a small portion of practitioners” claim that their SETA programmes are “very effective” (Hu et al., 2021, p. 1). It is reported that poor SETA programme effectiveness is linked to the programme’s failure to positively impact employee security-related behaviours (Alshaikh et al., 2021; Hu et al., 2021; Alshaikh et al., 2019). Therefore, the key motivation for this study is the questionable effectiveness of SETA programmes at changing employee behaviour and an absence of empirical studies on the critical success factors (CSFs) for SETA programme effectiveness. However, more than focusing on a specific research question, this paper presents the evolution of 11 CSFs for SETA programme effectiveness. Through the execution of a multi-stage research design, we theorise from the SETA programme stories of IS/cyber security professionals to produce the ranked list of CSFs. Furthermore, given the perceived difference in importance for some of the CSFs, we also present five CSF-specific guiding principles to further clarify the importance of these specific CSFs to achieving SETA programme effectiveness.

The remainder of this paper is organised as follows: Section 2 presents a background to SETA programme effectiveness and CSFs. Section 3 describes the three-stage research approach. Section 4 presents the findings and discussion, organised around the five CSF-specific guiding principles, proposed to increase the likelihood of delivering an effective SETA programme within an organisational context. Section 5 presents the conclusions and implications of the research. Lastly, Section 6 proposes some recommendations for future research.

2. Security Education, Training and Awareness programme effectiveness and critical success factors – why?

The importance of a SETA programme to protect information assets in an organisation has led many researchers to recommend establishing a SETA programme and making it part of any organisation’s overall security strategy (D’Arcy et al., 2009; Kirova and Baumoel, 2018). SETA programmes include the following functions:

• provide employees with knowledge regarding organisational information threats and IS security (D’Arcy et al., 2009; Yoo et al., 2018; Dhillon et al., 2020);

• clarify existing technical and procedural countermeasures available to employees (Pastor et al., 2010; Silic and Lowry, 2020);

• determine the possible sanctions for security policy violations in the organisation (Siponen and Vance, 2010; Karjalainen et al., 2013; Herath et al., 2018); and

• improve employees’ awareness of their roles and responsibilities in protecting the organisation’s information assets (D’Arcy et al., 2009; Lebek et al., 2014).

Where empirical studies investigating the effectiveness of SETA programmes exist, they fail to examine all phases of the SETA programme life cycle (design, development, implementation, evaluation), tending to focus more on one or two of the life cycle phases (c.f. Puhakainen and Siponen, 2010; Okenyi and Owens, 2007; Silic and Lowry, 2020; Rantos et al., 2012). Therefore, while there are several guidelines from academia available to organisations to support the introduction of SETA programmes, a question remains about the theoretical grounding and empirical evidence available, in current literature, around these guidelines when it comes to “developing an effective SETA programme to change employee behaviour” (Alshaikh et al., 2021, p. 2). A lack of a “systematic understanding” of the “nature of SETA programmes” and their impacts on “security-related beliefs” is viewed as a possible reason for this lack of effectiveness (Hu et al., 2021, p. 1). In fact, Alshaikh et al. (2021, p. 1) argue that existing SETA programmes are “suboptimal” as they “aim to improve employee knowledge acquisition rather than behavior and belief”. Therefore, more theorising and conceptual clarity is needed in investigating the effectiveness of SETA programmes (c.f. Alshaikh et al., 2021; Hu et al., 2021; Kirova and Baumoel, 2018; Puhakainen and Siponen, 2010). In particular, guidance in the form of CSFs is seen as particularly useful by helping organisations understand where to focus their efforts (c.f. reference withheld for review purposes). In fact, CSFs have been widely investigated and used in IS research and practice over the past three decades to make sense of problems by identifying the factors that could influence business activities and outcomes (c.f. Alhassan et al., 2019). Throughout this period, researchers have identified CSFs, which need more attention from managers, in areas ranging from “project-type” operational initiatives to more “mindset shift” strategic initiatives (c.f. Alhassan et al., 2019).

It is argued that CSFs are an established approach for providing guidance as a “popular simplification mechanism to assist managers” (Borman and Janssen, 2013, p. 86). Numerous studies within IS have used the CSFs lens to establish those key areas that demand favourable results to ensure a successful performance (c.f. Rockart, 1979). Several studies have also evaluated the level of importance of CSFs for various phenomena, for example, the implementation of enterprise resource planning (ERP) systems (c.f. Reitsma and Hilletofth, 2018; Ahmad and Cuenca, 2013), the introduction of public-private partnerships (Osei-Kyei and Chan, 2017; Soomro et al., 2016) and delivering shared services (Borman and Janssen, 2013). Within these studies, the research is conducted across multiple stages and uses various techniques to show the similarities and differences in the importance of the CSFs.

To date, little or no research has documented the CSFs for SETA programme effectiveness, especially because the effectiveness of SETA programmes is routinely called into question. In fact, research shows that “failure rates” for the introduction of IS initiatives still remain high. The rate of failure suggests the need to focus the attention of IS professionals and academics on addressing and developing a list of factors that will enable the successful delivery of IS initiatives (c.f. Alhassan et al., 2019). Therefore, we argue that understanding the CSFs for SETA programme effectiveness will lend itself to increasing the effectiveness of a SETA programme within an organisation.

In the next section, we present further details on the research approach.

3. Research approach

To fulfil the research objective, we adopt an exploratory research approach (following a three-stage research design). We use this exploratory approach to give “extraordinary voice to informants, who are treated as knowledgeable agents” (Gioia et al., 2012, p. 18) to investigate a little-understood phenomenon (Marshall and Rossman, 1989). In the context of this research project, this translates as IS/cyber security professionals with SETA programme expertise. Therefore, being inspired by features of the Gioia Methodology, which is positioned as a “systematic inductive approach to concept development” (Gioia et al., 2012, p. 17), we aim to conceptualise the practitioner voice and not “substitute practitioners’ understandings for theory” (Markus and Rowe, 2021, p. 273). Therefore, we embrace the Gioia Methodology because it encourages originality in our theorising, where what we already know does not limit “what we can know” (Gioia et al., 2012, p. 16).

In Stage 1, we use an inductive open coding approach to produce 11 CSFs from our analysis of 20 key informant interviews. These CSFs are ranked in a prioritised order (descending) based on the frequency count of coded excerpts across the 20 interview transcripts. Interpretive qualitative research is an appropriate research design to apply when exploring CSFs and several scholars have investigated and explored CSFs in IS by applying qualitative methods (c.f. Alhassan et al., 2019). In Stage 2, we use mean score ranking to generate a ranked list of the 11 CSFs based on our analysis of 65 responses to a short survey. The answer to each survey question involved ranking the importance of a specific CSF (high/medium/low). In Stage 2, we also compare this ranked list to the ranked list generated in Stage 1. This comparison highlights the position of each CSF in the respective lists and suggests the similarities and differences between the lists. We also use the Mann–Whitney U test to check if there is a difference in the rank sum between the two ranked lists of 11 CSFs (Stage 1 list from 20 key informants and Stage 2 list from 65 survey respondents). The Mann–Whitney U test is a non-parametric test used to study the association of ordinal (rank order) data from two independent groups where the data sets are not assumed to follow any normal distribution pattern (Osei-Kyei and Chan, 2017; Hair et al., 2007). In Stage 3, we present our hermeneutics-inspired analysis of nine follow-up probing interviews with IS/cyber security practitioners involved in Stage 1 (4) and Stage 2 (5) of this research study. All of these practitioners have expertise in organisational SETA programmes. Therefore, our hermeneutics-inspired analysis affords us the opportunity to “understand what people say and do, and why” (Myers, 2009, p. 182). This analysis adds further insights in the five key areas of difference that emerged from our comparative analysis of the ranked lists of 11 CSFs in Stage 2. Thereafter, these differences inform five CSF-specific guiding principles to increase the likelihood of delivering an effective SETA programme within an organisational context.

3.1 Stage 1: 20 key informants (the emergence of the critical success factors)

Stage 1 of this exploratory research follows a systematic inductive approach to concept development. We adopt the “key informant” approach for data gathering and engage with key informants through semi-structured interviews. The main advantage of using the key informant approach is gaining rich data in a short period of time through in-depth interviews. When using semi-structured interviews as part of the key informant technique, it is not uncommon to have a smaller number of interviewees; this can range from six interviewees (c.f Flores and Ekstedt, 2012) to 32 interviewees (Benova et al., 2019). In using the key informant technique, it is more important to have appropriately qualified (quality) individuals participating in a study over a larger quantity of individuals. Therefore, we believe that our use of 20 key informants is appropriate for this stage of the exploratory research study. Therefore, key informants were selected based on their position, experience and professional knowledge about IS/cyber security, particularly SETA programmes. Twenty individual semi-structured interviews were conducted with selected key informants from various geographic locations, including the Gulf nations (Saudi Arabia, United Arab Emirates, Qatar and Kuwait), the Middle East (Egypt and Lebanon), the USA, the UK and Ireland. All of the interviews started by introducing the objective of the research. Each interviewee was then asked to provide a brief summary of their background. Thereafter, topics relating to the factors critical to the success of SETA programmes throughout the life cycle phases (design, development, implementation, evaluation) were discussed. The following questions were asked to explore the CSFs for SETA programme effectiveness across these life cycle phases. Questions 1–4 are also asked for the development, implementation and evaluation phases:

Q1.

What are the factors that are important in the design of a SETA programme?

Q2.

Why are these factors important in the design of a SETA programme?

Q3.

How can organisations ensure that these factors exist in their design efforts?

Q4.

Who should be responsible for the design of a SETA programme?

Q5.

What makes a SETA programme succeed/fail?

All the interviews were transcribed line-by-line and checked against the voice recordings, where necessary, to ensure the accuracy of the transcription of the interviews. This research adopted an inductive open coding approach as part of our qualitative data analysis (Corbin and Strauss, 1990). When all 20 key informant interviews were transcribed, the data analysis commenced using sentence-by-sentence coding to identify relevant codes. The open coding procedure for the 20 key informant interviews resulted in 212 coded excerpts relating to the factors impacting on the effectiveness of a SETA programme. These 212 coded concepts led to the emergence of 15 categories mapped across the four SETA programme life cycle phases (design, development, implementation and evaluation). Specifically, the code/category distribution is as follows:

• design phase − 95 codes − 8 categories;

• development phase − 27 codes − 4 categories;

• implementation phase − 50 codes − 5 categories; and

• evaluation phase − 40 codes − 3 categories.

The category with the highest coding frequency across each of the SETA programme life cycle phases is as follows:

• design phase − 18 coded concepts in the “Assessment Needs” category;

• development phase − 12 coded concepts in the “Communication” category;

• implementation phase − 17 coded concepts in the “Communication Channel” category; and

• evaluation phase − 20 coded concepts in the “Periodic Assessment” category.

See Figure 1 for a sample of our inductive open coding. Thereafter, unpacking the categories with at least five key informant voices (25% coverage) led to the emergence of the 11 CSFs for SETA programme effectiveness.

Table 1 presents these 11 CSFs organised by SETA programme life cycle phase. See reference withheld for review purposes for a more detailed discussion on these 11 CSFs. Furthermore, Figure 2 (the left-hand side of the visual) presents these CSFs in a ranked prioritised order (descending) based on the frequency count of coded excerpts across the 20 interview transcripts. The mean score is also presented for each CSF based on the following formula ([coded concepts* numerical value of a CSF importance of “high”]/total number of key informants). For example, the mean score of CSF-DV1 (ranked eighth in Figure 2) is 1.8, calculated as ([12*3]/20). Therefore, having generated a ranked order list of CSFs for SETA programme effectiveness (in Stage 1), we now needed to evaluate if this ranked order list is appropriate for others in the IS/cyber security practitioner community. This evaluation is presented in the next section.

3.2 Stage 2: 65 survey respondents (the ranking of the critical success factors)

In Stage 2 of this exploratory research study, we adopted a Request-For-Comment (RFC) approach (c.f. Chen et al., 2013) to evaluate the ranked importance of the 11 CSFs. Online surveys are an effective way “to elicit feedback from a wide range of participants in a scalable way” (Lo et al., 2015, p. 416). Surveying practitioners (using short simply surveys) to determine (i) their confidence in academic outputs or (ii) their perception of research relevance is a common practice (c.f. Kitchenham and Pfleeger, 2008). However, it is important to design these surveys such that participants require “as little effort as possible to complete it” (Lo et al., 2015, p. 416). To achieve this, we designed a short 11-question survey (one question per CSF using an ordinal scale of high/medium/low) to gather practitioner perspectives on the ranked importance of each of the CSFs for SETA programme effectiveness (see Appendix 1 for a sample of the survey questions). Therefore, we limited our response type to numerical (ordinal scale) but subjective data, capturing the respondent’s preferences (Kitchenham and Pfleeger, 2008, p. 67). Therefore, we viewed the short survey as a data gathering instrument that would provide numerical descriptions of the importance of each CSF (based on the perception of the sample population). Overall, our approach is similar in style to that taken by Nah et al. (2001, p. 295) where they refer to the use of a “survey questionnaire” to evaluate the importance of their 11 CSFs for ERP implementation and “how the perceived importance of these factors may differ” among respondents. In our survey, a respondent was required to answer all 11 questions to submit a valid survey response. This RFC approach ensured that each respondent evaluated each one of the 11 CSFs. The facility also existed for respondents to provide additional comments following completion of the survey. These comments were also factored into the follow-up probing interviews conducted in Stage 3.

The survey was designed on Google Forms and was distributed electronically to practitioners in the IS/cyber security professional community (SETA programme specialists). These practitioners were initially invited by email, and if they agreed to participate were then sent the link to the survey. For example, we invited participants from the Cyber Research Conference Ireland 2022, along with members of several cyber security groups, including Women in Cyber Security Middle East, Hemaya Cyber Ladies and Information Security Association – Hemaya. Some of the cyber security group members also shared the invite with professionals within their networks. None of the 20 key informants from Stage 1 was invited to participate in Stage 2.

The survey went live on 5 May 2022 and remained open for five days (until 9 May). A total of 65 responses were gathered during this time (25 responses on Day 1, 26 on Day 2, 7 on Day 3, 5 on Day 4 and 2 on Day 5), with responses coming mainly from Ireland, the UK and the Middle East. Once the survey was closed, the data were downloaded to MS Excel. A data analysis table was generated containing the min/max, the mean, median and standard deviation for all 11 CSFs. Figure 2 (the right-hand side of the visual) presents these CSFs in a ranked, prioritised order (descending) based on the mean score of each CSF. Figure 2 is inspired by similar comparative type visual displays used in other studies [e.g. Table 3 from Wong (1998) and Figure 4 from Ahmad and Cuenca (2013)].

In this stage, we also compare both ranked lists of the 11 CSFs emerging from the two independent groups (Stage 1 list from 20 key informants and Stage 2 list from 65 survey respondents). Figure 2 presents a visual of this comparison (similarities and differences) and highlights the position of each CSF in the respective lists. We also use the Mann–Whitney U test to check if there is a difference in the rank sum between the two ranked lists. In this research, the statistical test was performed by hand, following the steps outlined in the following video (www.youtube.com/watch?v=BT1FKd1Qzjw). The workings of the Ustat for Stage 1 (n¹) and Stage 2 (n²) are available in Appendix 2. We use the mean value for each of the 11 CSFs across both groups, rank each of the CSFs and calculate the rank sum for Stage 1 (109.5 with a Ustat = 43.5) and Stage 2 (143.5 with a Ustat = 77.5). The null hypothesis is stated as follows: in the population, the rank sum (sum of the rankings) in the two groups does not differ, whereas the alternative hypothesis suggests that the sum of the rankings does differ. The critical values of the Mann–Whitney U (two-tailed testing) are available at this links (https://ocw.umb.edu/psychology/psych-270/other-materials/RelativeResourceManager.pdf). The value of the Ucrit at α = 0.05 (95% confidence interval) is 30 (where n¹ and n² are both 11). Based on our calculations, the null hypothesis was accepted (Ustat > Ucrit), suggesting that there is no significant difference in the CSFs between the two groups (Stage 1: 20 key informants and Stage 2: 65 survey respondents). For example, in our analysis, Ucrit = 30 and the lowest Ustat = 43.5.

However, as seen in Figure 2, there is a somewhat contradictory element to the CSF rankings between Stage 1 and Stage 2. For example, CSF-EV1 is the 1st ranked CSF from Stage 1 but is the 11th ranked CSF from Stage 2. Furthermore, the inverse is also true, where CSF-DS6 is the 11th ranked CSF from Stage 1 but is the 1st ranked CSF from Stage 2. In total, five critical areas of difference emerge from our comparative analysis of the ranked lists of 11 CSFs. In Figure 2, the three red lines represent a significant difference in the ranking (between both practitioner groups) of three specific CSFs (CSF-EV1, CSF-DS5 and CSF-DS6). The two orange lines represent a modest, but of interest, difference in two specific CSFs (CSF-DS2 and CSF-DS4). Finally, the six green lines represent insignificant differences in the ranking (between both practitioner groups) of six specific CSFs (CSF-DS1, CSF-IM1, CSF-DS3, CSF-EV2, CSF-DV1 and CSF-IM2). Therefore, we progress our understanding of the differences (captured by the red and orange lines) in the next section.

3.3 Stage 3: 9 follow-up probing interviews (the emergence of the guiding principles)

In Stage 3 of this exploratory research study, we use a hermeneutics-inspired approach to analyse and interpret the answers provided by nine IS/cyber security practitioners (four from Stage 1 and five from Stage 2) to the questions emerging from the differences in the ranking of the CSFs between Stage 1 (20 key informants) and Stage 2 (65 survey respondents) (see Figure 2). In this stage, we are trying to make sense of the “seemingly contradictory” (Myers, 2009, p. 170) text that has emerged around five specific CSFs (difference in their ranked importance). Furthermore, we take these differences as a sign of “confused, incomplete, cloudy, and contradictory views” (Myers, 2009, p. 171) among the IS/cyber security community (specifically those with expertise in SETA programmes). Therefore, our interpretative work aims to bring to light an underlying sense of clarity, and the following questions were asked to qualify the importance of the five specific CSFs:

1. Is building a security awareness campaign important (CSF-DS6)? Why?

2. Is maintaining a quarterly evaluation of employee performance important (CSF-EV1)? Why?

3. How can adherence to policy (organisational and legislative) be improved (CSF-DS5)?

4. Is tailored content for employees important (CSF-DS2)? Why?

5. Is the cultural context and the employee background important (CSF-DS4)? Why?

Ultimately in this stage, we discovered reasons for the importance of CSF-DS6, CSF-EV1, CSF-DS5, CSF-DS2 and CSF-DS4. Four of these contradictions (CSF-DS2, CSF-DS4, CSF-DS5, CSF-DS6) are linked to the design phase of the SETA programme life cycle, while one contradiction (CSF-EV1) highlights the challenging nature of SETA programme evaluation. Therefore, these contradictions afford us the opportunity to present five guiding principles (four in design and one in evaluation) to complement the ranked list of 11 CSFs for SETA programme effectiveness within an organisational context (see Table 2). These guiding principles are ordered by the degree of difference (contradiction) between the Stage 1 and Stage 2 rankings (see Figure 2).

In the next section, we present a discussion of our research findings (the five CSF-specific guiding principles emerging from Stage 3).

4. Discussion of findings

The outcome of this research suggests that there is no significant difference between the 20 key informants (Stage 1) and 65 survey respondents (Stage 2) in terms of their perception of the ranked importance of the 11 CSFs for SETA programme effectiveness. However, this research also highlights that there are five specific CSFs for SETA programme effectiveness that need to be examined, given the differences that emerged from our comparative analysis. These five guiding principles are now discussed to further improve the likelihood of delivering an effective SETA programme. We also reflect these findings against existing literature.

4.1 Principle 1: raise employee IS/cyber security awareness and knowledge to enhance organisational maturity

CSF-DS6: Build Security Awareness Campaigns focuses on updating the employee on how to mitigate against the potential risks associated with an IS/cyber security threat, keeping them informed on what is coming, and, most crucially, why they need to care. Figure 2 visualises the contradictory rankings between Stage 1 and Stage 2 for this CSF. We believe that this difference exists because the perspectives of the experts (involved in delivering SETA programmes) differ regarding where to position awareness building along the SETA programme lifecycle. For example, in the early stage of the design phase of the SETA programme (to clarify all security issues for their employees to achieve the SETA programme goals) or in the final stage of the evaluation phase (to assess employee knowledge of IS security and to determine whether or not the programme is effective at changing employee behaviour).

Based on our review of the story of the importance of this CSF (CSF-DS6), the IS/cyber security practitioners highlighted that security awareness campaigns simply keep employees updated about what is going on (e.g. new cyberattack methods) and how to protect themselves and the organisation. For example, one practitioner states, “[…] as technology develops, fraud and security incidents are constantly updated, and new attack techniques are developed”. The campaign also aims to enhance organisational IS/cyber security maturity, and this is especially challenging where employees have varying levels of IS/cyber security knowledge and experience (e.g. new hires vs senior leaders) and may fail to recognise an IS/cyber security issue. As one practitioner states, “the campaign can provide detailed information about phishing, social engineering, and other technical attacks”. As a result, the main goal of an IS/cyber security awareness campaign is to raise employee awareness and knowledge.

In comparing these findings with those presented in the literature, several observations can be made around the criticality of building an IS/cyber security awareness campaign as part of a SETA programme. For example, Rantos et al. (2012) discuss launching an awareness campaign across the company to cover all IS security topics as a vital element of measuring the effectiveness of the SETA programme. Several studies highlight the need to design an awareness campaign as a periodic short communication to clarify the importance of the SETA programme in terms of protecting the IS assets, personal data, enhancing IS/cyber security awareness, complying with IS/cyber security policy and reducing IS/cyber security risks (Vroom and Solms, 2002; Puhakainen and Siponen, 2010). Therefore, formal awareness campaigns are communications with employees with the specific aim of increasing the understanding of, and reducing the likelihood of, harmful IS practices within the organisation (D’Arcy et al., 2009; Hearth et al., 2018).

4.2 Principle 2: evaluate employee performance at a frequency that aligns with the organisational IS/cyber security strategy

CSF-EV1: Maintain Quarterly Evaluation of Employee Performance focuses on providing a year-end evaluation summary (e.g. metrics) to measure each employee’s performance (e.g. level of awareness, number of training sessions completed, etc.) and to provide guidance on necessary improvements. Figure 2 visualises the contradictory rankings between Stage 1 and Stage 2 for this CSF. We believe that this difference exists because the perspectives of the experts differ on whether an assessment should be conducted at the start or at the end of the year. Some experts believe in assessing employee knowledge, to determine their level of awareness and then building the SETA programme around that, while others believe that the assessment should be done at the end of the year to evaluate the effectiveness of the current programme (while also informing the design of the forthcoming year).

Based on our review of the story of the importance of this CSF (CSF-EV1) to SETA programme effectiveness, the IS/cyber security practitioners highlight the criticality of conducting employee assessments to assess security awareness and knowledge levels and motivate employees to participate in the SETA programme. However, IS/cyber security practitioners have differing views on whether the time frame for evaluating employee performance should be quarterly or annually. A significant number of practitioners prefer to conduct annual assessments. For example, one practitioner states, “if you ask employees to do evaluations every quarter, some organisations will simply fail because employees will get tired and fatigued”. However, there is a strong preference for quarterly assessment also. For example, one practitioner states, “while it is very important to evaluate the employee at least once a year, I prefer to conduct employee assessments every three months to track their progress”. Therefore, it appears as if the time frame for conducting employee assessments can be determined based on the organisational IS/cyber security strategy. Furthermore, while organisations use various tools to assess employee performance, one practitioner calls out the importance of defining the correct KPIs, for example, “we must ensure that 90% of employees, preferably 98%, have successfully completed the training courses”. In contrast, another practitioner states, “we can evaluate employee performance by using phishing simulation (e.g. did they get the security awareness message, did they report suspicious links)”. Ultimately, the evaluation results also inform the next cycle of designing an effective SETA programme within the organisation.

In comparing these findings with those presented in the literature, several studies discuss approaches to evaluate SETA programmes. For example, Rantos et al. (2012) illustrate several methods for evaluating a SETA programme. One of those methods is using a survey/questionnaire to evaluate the success of the programme overall. Other methods evaluate security awareness campaigns by highlighting some IS security issues and measuring the effectiveness of the SETA programme in addressing the existing gaps (Alshaikh et al., 2018; Johnson, 2006). However, this is an area that still requires further research.

4.3 Principle 3: secure top management support to encourage all employees to comply with IS/cyber security policy

CSF-DS5: Adhere to Organisational Security Policy and the “Law of the Land” is concerned with focusing on the guidelines and procedures needed to protect the IS assets of the organisation, to ensure that all of the organisational security policies and the “law of the land” are adhered to when designing a SETA programme. Figure 2 visualises the contradictory rankings between Stage 1 and Stage 2 for this CSF. We believe that this difference is linked to the fact that some organisations design their SETA programmes in-house and ensure that their security policies comply with localised legal requirements. In contrast, other organisations use a more generic CBT (computer-based training) design that simply informs employees of the country’s regulations (but does not link back to the organisational security policy).

Based on our review of the story of the importance of this CSF (CSF-DS5), the IS/cyber security practitioners highlight the need to obtain top management support to ensure that all organisational employees adhere to the IS/cyber security policy. The practitioners express the view that top management can improve employee security awareness and practices because their acts of policy compliance encourage other employees to follow their lead. For example, one practitioner states, “in order to get employees to commit to security policies and regulations, we need top management support”. Therefore, improving SETA programme effectiveness begins with top management demonstrating the importance of implementing security regulations and policies and then training employees on these security regulations and policies. It also appears that for some IS/cyber security professionals enforcing severe penalties for IS/cyber security policy violations can help improve SETA programme effectiveness. For example, one practitioner states, “we have a security policy, and if you fail to follow the policy three times in a row, you will be fired”. The view exists that this enhances employee commitment and adherence to the fundamentals of IS/cyber security awareness. Furthermore, implementing the appropriate IS/cyber security standards is identified as a key to reducing IS security risks and is critical to delivering an effective SETA programme (assisting organisational employees to manage cyberattacks and IS/cyber security threats). For example, one practitioner states, “once ISO 27000 certified, you will have regular audits, and the system will be audited. The audits will ensure continuous improvement […]”. Typically, organisations conduct internal or external audits every six months to motivate employees to follow their IS security policies.

In comparing these findings with those presented in the literature, several observations can be made around the criticality of top management support to IS/cyber security policy adherence as part of a SETA programme. For example, Puhakainen and Siponen (2010) conducted an empirical investigation into the significance of the role of top management in ensuring employee compliance with IS policy. Hu et al. (2012) also provided a detailed explanation of the significance of top management support to IS policy compliance and the changes to organisational culture. Active participation by top management in the development, implementation and enforcement of security policy can enhance employees’ perceptions that IS policy and procedures are legitimate and fair (Hu et al., 2012). Therefore, top management plays an important role in encouraging employees to adhere to IS/cyber security policy to deliver an effective SETA programme.

4.4 Principle 4: avoid a one size fits all approach to programme content to promote employee engagement

CSF-DS2: Know Your Audiences to Ensure Content Suitability focuses on identifying “who your audiences are” to ensure appropriate content is delivered to the various employee types. Figure 2 visualises the contradictory rankings between Stage 1 and Stage 2 for this CSF. We believe that the slightly different stories highlight how organisational size and resources play an important role in delivering appropriate security awareness materials to employees at various levels. Employee differences (e.g. culture, knowledge, age, etc.) should be considered when preparing resources, and content customisation should align with the organisation’s own strategies and IS/cyber security plans. Therefore, IS/cyber security awareness content should be designed in such a way as it is neither too technical nor too general for the target audiences.

Based on our review of the story of the importance of this CSF (CSF-DS2) to SETA programme effectiveness, the IS/cyber security practitioners highlight the criticality of tailoring the content of the IS/cyber security message to the audience level (e.g. level of education, age, role, etc.) to provide them with the appropriate training materials. The aim of this is to increase IS/cyber security policy compliance and achieve the organisational goals. For example, one practitioner states, “we will tailor the content, based on the audience targets, in order to get people to engage with it”. Therefore, customising content is essential to ensure SETA programme effectiveness. As one practitioner states, “it is absolutely essential to tailor the content of the cyber security awareness message and make it simple, direct, and attractive […]”. As a result, it is impossible to apply the concept of “one-size-fits-all”, and the same IS/cyber security awareness message cannot be delivered to everyone in the organisation.

In comparing these findings with those presented in the literature, several observations can be made. For example, Peltier (2005) discusses establishing an IS/cyber security awareness programme by classifying the audience to ensure the IS/cyber security message is communicated effectively. Accordingly, a SETA programme must comprise a plan to transmit the IS/cyber security message to the target audience (De Maeyer, 2007; Siponen, 2000). Therefore, it can be argued that identifying the target audiences in designing a SETA programme is the main step towards its success; thereby, delivering thorough IS/cyber security training to each employee with appropriately suitable material.

4.5 Principle 5: appreciate employee cultural differences to shape programme content

CSF-DS4: Design for Cultural Context and Employee Cultural Diversity focuses on understanding the diversity of employee backgrounds (e.g. language, culture, knowledge, level of education, age, gender) so that the IS/cyber security message can be interpreted by all employees. Figure 2 visualises the contradictory rankings between Stage 1 and Stage 2 for this CSF. We believe this is linked to the fact that the IS/cyber security practitioners involved in this research study come from different cultures (e.g. Saudi Arabia, Kuwait, Ireland, the US, the UK, etc.). As a result, given their differing backgrounds and experiences, what works in one cultural context might not work in another.

Based on our review of the story of the importance of this CSF (CSF-DS4) to SETA programme effectiveness, the IS/cyber security practitioners highlight that each culture has its own sense of privacy, which should be considered when designing a SETA programme. For example, one practitioner states, “it is critical to understand the culture from which they come. To build security awareness content in simple language that adheres to the security policy”. Thus, culture is an essential factor that can influence how individuals act, with differences and similarities between individualist and collectivist cultures (Parks and Vu, 1994). For example, one practitioner reveals, “in the context of country culture, Saudi Arabia is collectivist while Ireland is individualist”. Therefore, employees from collectivist cultures tend to collaborate in a more trusting fashion and will share their passwords, whereas employees from individualist cultures tend to be more conscious and will not share their passwords (Moorman and Blakely, 1995). Essentially, navigating these cultural realities within an organisational context is extremely important for SETA programme effectiveness.

In comparing these findings with those presented in the literature, a number of observations can be made. Previous studies address “culture” in the context of IS/cyber security practice. For example, Hovav and D’Arcy (2012) examine the influence of culture on IS/cyber security policies, training and monitoring. To understand culture in terms of IS/cyber security practice is to understand individual differences within each cultural context (c.f. Walsham, 2002). These cultural differences can be beliefs, norms and values in a social setting, known collectively as a country. Thus, different cultures require different IS/cyber security interventions (Kirova and Baumoel, 2018; Karjalainen et al., 2013; von Solms and von Solms, 2004). Thus, understanding the cultural context is an essential factor when designing an effective SETA programme.

5. Conclusions and implications

It is reported that importance is the most critical dimension of relevance for IS practitioners. Similar to Rosemann and Vessey (2008, p. 3), we view importance as research that “meets the needs of practice by addressing a real-world problem in a timely manner [currently significant], and in such a way that it can act as the starting point for providing an eventual solution”. Therefore, while IS/cyber security is a current hot topic and a top concern for many practitioners (both business and IT), the ability to lead an effective SETA programme, and identify the CSFs for doing so, is an area of IS research not yet well established. Therefore, this study is unique in its approach and contributes to the IS/cyber security conversation in the following three ways:

1. one of the first studies to produce a ranked list of CSFs for SETA programme effectiveness; thereby, conceptualising SETA programme effectiveness in a digestible, easy to understand, way (see Figure 2);

2. one of the first studies to provide a set of guiding principles for the CSFs that could be the most challenging to “get right” in practice (see Table 2); and

3. one of the first studies to highlight that the “design” phase of a SETA programme life cycle will be the most contentious in terms of building a shared understanding (among all organisational stakeholders) of what is critical to delivering an effective SETA programme within the organisation.

In this research, we capture the essence of the practitioners’ views on the importance of the CSFs for SETA programme effectiveness. By comparing Stage 1 and Stage 2 outputs, we are provided with a context against which a more accurate interpretation of the realities of SETA programme effectiveness can be achieved. Therefore, the outcome of Stage 3 of this research showcases that practice matters and the topic of SETA programme effectiveness (and the CSFs to achieve it) matters to practice. In conclusion, we believe that we have taken practice into consideration and delivered something of practical value in this research. For example, the difference (as to the importance of each CSF) between the two groups of IS/cyber security professionals showcases variation based on past experiences. This variation further advances our theorising and brings further clarity to the SETA programme effectiveness story (through the emergence of the five guiding principles). Furthermore, we are also aware that the research team’s effort at qualitative data analysis (in Stage 1 of this research) sets the agenda for the remaining stages. However, the similarity in perceived importance (for the majority) of the CSFs (between Stage 1 and Stage 2) also highlights the shared theoretical sensitivities of both the researchers and the practitioners in this research study.

Beyond the value of the 11 CSFs for SETA programme effectiveness, this paper also presents an approach to evaluate the outputs of a multi-stage grounded (data-to-theory) study. For example, we use an RFC approach to take the outputs from Stage 1 (the emergence of the CSFs) and generate a comparative ranked order list to generate a new output in Stage 2 (the ranking of the CSFs). Finally, the differences between the ranked order lists in Stage 1 and Stage 2 produce the output for Stage 3 (the emergence of the guiding principles). This movement through the stages showcases an innovative approach to evaluating outputs, emerging from iterative data gathering and analysis, where the views of participants are used to fuel our theorising and theory development efforts. Therefore, this evaluation approach further increases the relevance of the work (around accessibility and applicability). To note, as per Rosemann and Vessey (2008, p. 3), accessibility is understood as “the research is understandable, readable, and focuses on results”, and applicability is understood to be “whether it provides guidance and/or direction, and whether it provides concrete recommendations” that are easy to apply in practice.

Finally, prefacing the five guiding principles (presented in Table 2) with “Do We” allows each principle to serve as a pre-commencement readiness check to guide SETA programme endeavours; and/or as an in-progress reflective aid for practitioners to assess the efficacy of their existing SETA programme activities. We believe that asking and answering these five questions will help to start conversations and build a shared understanding among organisational IS/cyber security practitioners, with the aim of delivering an effective SETA programme within an organisational context. For example, the “design” phase is the preliminary phase in a SETA programme life cycle. The design phase activities are most often concerned with identifying the target audiences and their needs, outlining and budgeting for a training and awareness plan, setting up priorities and benchmarks, along with risk management and business contingency planning (Alshaikh et al., 2018; Tsohou et al., 2015; Puhakainen and Siponen, 2010; Wilson and Hash, 2003). Therefore, having meaningful conversations around these design phase activities is of the utmost importance.

6. Recommendations for future research

Using a three-stage research approach to capture IS/cyber security practitioner voices allowed their SETA programme stories to be interrogated, the outcome of which leads to the emergence of the 11 CSFs for SETA programme effectiveness (across the life cycle phases) and the five CSF-specific guiding principles to further improve the likelihood of delivering an effective SETA programme within an organisational context. However, these CSFs and guiding principles are not yet established as universal, so while these CSFs and guiding principles provide guidance to all undertaking a SETA programme, organisations need to be “mindful of the influence of their own context” (Borman and Janssen, 2013, p. 85). Furthermore, we are also conscious that while adding to the number of key informants in this study could be very beneficial and revealing for our “concept development” work on the CSFs and five CSF-specific guiding principles for SETA programme effectiveness, it is perhaps more beneficial to move to a larger population of IS/cyber security practitioners as part of a study focused on “construct elaboration” (Gioia et al., 2012, p. 16).

Finally, providing a list of CSFs is only a partial aid to success; more is needed on the implementation actions required around any list of CSFs stated (c.f. Alhassan et al., 2019). Therefore, now that we have identified a ranked list of 11 CSFs for SETA programme effectiveness, along with the five CSF-specific guiding principles, there is a need to examine the “conjunctural” (Ragin, 1987) nature of these CSFs. Such an appreciation would further improve our understanding regarding the complexity of SETA programmes. This would lend itself to the development of a process model for SETA programme effectiveness, embracing the life cycle phases (design, development, implementation, evaluation) and chaining (c.f. Hubberman and Miles, 1994) the CSFs within and across these phases.

Figures

Figure 1.

A sample of our inductive open coding (a snapshot of the highest frequency categories across the four life cycle phases)

Figure 2.

CSF ranked list comparison (Stage 1 and Stage 2)

Figure A1.

Figure A2.

Table 1.

11 CSFs for SETA programme effectiveness (presented by life cycle phase)

Life cycle phase	CSF	Category	Description
Design	CSF-DS1: Conduct an Initial Assessment of Employee Security Awareness	Assessment needs	Determining what the employee understands about the organisation’s security policy and their appreciation of the risks associated with current IS/cyber security threats
	CSF-DS2: Know Your Audiences to Ensure Content Suitability	Target audiences	Identifying “who your audiences are” to ensure appropriate content is delivered to the various employee types
	CSF-DS3: Make a Yearly Plan to Align Goals and Objectives	Goal/Objective	Knowing what is required to be delivered to the employee to ensure that the SETA programme goals meet the specific needs of the organisation
	CSF-DS4: Design for Cultural Context and Employee Cultural Diversity	Culture	Understanding the diversity of employee backgrounds (e.g. language, culture, knowledge, level of education, age, gender) so that the IS/cyber security message can be interpreted by all employees
	CSF-DS5: Adhere to Organisational Security Policy and the “Law of the Land”	Policy	Focusing on the guidelines and procedures needed to protect the IS assets of the organisation, to ensure that all of the organisational IS/cyber security policies and the “law of the land” are adhered to when designing a SETA programme
	CSF-DS6: Build Security Awareness Campaigns	Communication	Updating the employee on how to mitigate against the potential risks associated with an IS/cyber security threat, and keeping them informed on what is coming, and most crucially, why they need to care
Development	CSF-DV1: Sustained Communication of Relevant Messages	Communication	Repeating the IS/cyber security message in various ways to avoid a lapse in employee concentration
Implementation	CSF-IM1: Apply Diverse Methods to Deliver Security Awareness Messages	Communication channel	Using various approaches to deliver IS/cyber security awareness messaging (e.g. SMS, emails, online courses, face-to-face meetings, videos, quizzes, posters, screens in public corridors, etc.) so that the employee is reminded frequently of the IS/cyber security issue
	CSF-IM2: Motivate Employees to Engage in Security Awareness	Motivation	Encouraging the employee to adhere to IS/cyber security policies by earning a bonus, or other recognition (rewards), based on their practices
Evaluation	CSF-EV1: Maintain Quarterly Evaluation of Employee Performance	Periodic assessment	Providing a year-end evaluation summary to measure each employee’s performance (e.g. level of awareness, number of training sessions completed, etc.) and to provide guidance on necessary improvements
	CSF-EV2: Measure Employee Reporting of Security Incidents	Incident indication	Using phishing campaigns to simulate attacks (knowing how many employees click the suspicious links) to measure the employee awareness and knowledge regarding IS/cyber security issues

Source: Authors’ own creation/work

Table 2.

Guiding principles for five CSFs with difference/contradiction

CSF	Ranking (Stage)		Guiding principle
	One	Two
CSF-DS6: Build Security Awareness Campaigns	11	1	Raise employee IS/cyber security awareness and knowledge to enhance organisational maturity
CSF-EV1: Maintain Quarterly Evaluation of Employee Performance	1	11	Evaluate employee performance at a frequency that aligns with the organisational IS/cyber security strategy
CSF-DS5: Adhere to Organisational Security Policy and the “Law of the Land”	9	2	Secure top management support to encourage all employees to comply with IS/cyber security policy
CSF-DS2: Know Your Audiences to Ensure Content Suitability	3	6	Avoid a one size fits all approach to programme content to promote employee engagement
CSF-DS4: Design for Cultural Context and Employee Cultural Diversity	6	9	Appreciate employee cultural differences to shape programme content

Source: Authors’ own creation/work