A validated questionnaire for measuring digitalization as sociocultural change in educational contexts

Abstract

Purpose

In the so-called digital age, there is a basic assumption that digitalization entails rapid and dramatic change in schools, education and society. However, a challenge for educational research is to clarify what digitalization precisely means. This paper aims to develop, test, and validate a digital transformation scale (DTS). More specifically, the aim is to validate digitization, digitalization and digital transformation as hierarchical levels of sociocultural learning in school and education by using cultural-historical activity theory (CHAT) as a framework.

Design/methodology/approach

An exploratory factor analysis (EFA), with principal-axis factoring as an extraction method, was used to examine the number of factors underlying the data.

Findings

Results show that the three dimensions in the DTS questionnaire explain 68% of the variance and that all dimensions show high internal consistency (a >0.87). This means that the internal structure of the DTS corresponded to the internal structure of the theory.

Research limitations/implications

The results show that the internal structure of the DTS corresponded to the internal structure of the theory and may be used quantitatively to analyze digital transformation in school organizations. However, further research is needed in other contexts and larger samples with the use of confirmatory factor analysis to develop knowledge in this area and the use of DTS.

Practical implications

This tool and theoretical construction could be used to discuss digital transformation in school and education, both local and in general. Seeing digitalization from a sociocultural perspective makes possible to conceptualize and discuss this as a process ranging from small technology investments on an individual level to digitalization as strategic and organizational development.

Originality/value

This DTS can be used quantitatively to study and analyze digital transformation in educational contexts and provides educational researchers with additional tools to articulate what they mean by digitalization.

Citation

Pettersson, F., Siljebo, J., Wolming, S. and Ferry, M. (2024), "A validated questionnaire for measuring digitalization as sociocultural change in educational contexts", International Journal of Information and Learning Technology, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/IJILT-08-2023-0149

1. Introduction

In the so-called digital age, there is a basic assumption that digitalization entails rapid and dramatic change in education and society (Hanelt et al., 2020; Hinings et al., 2018; Karanasios et al., 2021). These changes entail possibilities and challenges, and perhaps it is also a necessity to rethink previous paths of learning, interaction and even human agency in schooling, education and society in general. For example, algorithms, artificial intelligence and augmented reality may support but also interfere with human learning and activity (Lindgren and Holmström, 2020; Westerman et al., 2014) and transformations have taken place in economic markets based on novel consumer-driven digital developments (McAfee and Brynjolfsson, 2017). The contextualization of this paper concerns the utilization of digital technologies in schools and educational organizations. In this research field, several “components” have been investigated, such as teaching and learning (Siljebo and Pettersson, 2022), school leadership and school development (Dexter, 2018; Siljebo, 2023), equal access to education (Billmayer et al., 2020), professional development (Nagel et al., 2023; Røkenes et al., 2022) and more. However, educational research that investigates possibilities, challenges and change via digital technologies is often discussed via notions such as digitalization and a challenge for educational research is to clarify what digitalization precisely means (Fischer et al., 2020; Pettersson, 2021; Warner and Wäger, 2019; Wessel et al., 2021). For example, is digitalization just an expression of digital technology use in schools and education? In contrast, is it an expectation of harnessing assumed opportunities that digital technologies may have inbuilt for transforming school and educational organizations profoundly (Nambisan et al., 2017; Yoo et al., 2012)?

Considering such ambiguities regarding digitalization in schools and society, there is an emergent call for theoretical understanding (Allen et al., 2013; Hinings et al., 2018; Karanasios et al., 2021). This study proposes to answer this call through a quantitative methodology built on sociocultural learning theory. The relevance of sociocultural learning theory and specifically the paper’s adaptation of cultural-historical activity theory (CHAT) (Engeström, 2015), is the understanding of digitalization as a social process rather than a simply technical one. In this article, we assume some correspondence exists between human learning and organizational change via digital technologies. Moreover, we assume that sociocultural learning theory carries the theoretical history and robustness to approach digital and organizational change in school and education (i.e. digitization, digitalization and digital transformation). Researchers who embrace sociocultural perspectives on change would claim that learning is at the heart of human activity and that technologies in different ways change (e.g. transform) culture (cf. Fischer et al., 2020; Karanasios et al., 2021; Wells and Claxton, 2002). In mediating human actions, digital technologies are understood to transform existing practices and contribute to completely new ways of thinking about learning and working in school and education (Allen et al., 2013; Karanasios et al., 2021; Ruckriem, 2009).

The paper’s contribution via the focus on quantitative methodology is that such research approaches are seldom utilized to understand digitalization in education as a social process, beyond effect studies of digital teaching methods on learning outcomes in higher education and school subjects (e.g. building on Puentedura, 2006). We specifically suggest that digitalization can be quantitatively measured as different hierarchical levels of sociocultural learning (cf. Bateson, 1972; Engeström, 2015).

With a focus on human learning, this paper aims to develop, test and validate a digital transformation scale (DTS). More specifically, the aim is to validate digitization, digitalization and digital transformation as hierarchical levels of sociocultural learning and development in school and education by using exploratory factor analysis (EFA).

2. Digital transformation as sociocultural change

Researchers that embrace sociocultural perspectives would claim that whatever digitalization is, it is about mediated human activity with cultural artifacts (i.e. digital technologies) (cf. Karanasios, 2018; Cornet et al., 2018; Woll and Bratteteig, 2018). Furthermore, the affordances of digital technologies of today are profoundly different than previous technologies, specifically regarding their ability to contain many different affordances (Karanasios et al., 2021; Ruckriem, 2009). As such, within human learning and working in the digital age, digital technologies have and do transform culture, since cultural artifacts are foundational in human learning (Kaptelinin and Nardi, 2006; Fischer et al., 2020; Säljö, 2010).

The basic components for understanding sociocultural learning and development are found in the development and use of artifacts, in the (school and educational) systems in which they are embedded and in the corresponding development of social and organizational structures (cf. Leontyev, 2009a, b; Sannino et al., 2016). In the theory’s early development, Vygotsky (1978) focused on mastering tools to develop higher mental functions. The mediated act between individuals, tasks and tools, according to (Vygotsky, 1978; Vygotsky and Rieber, 1999), had the capacity to change and transform the human mind and actions. However, at the time, tools referred to paper, pencils, wrenches and so on, which in many ways differed from today's digital tools.

Rooted in Vygotsky's (1978, 1997) sociocultural understanding of learning and development, and later in Leontyev's (2009a, b) sociocultural system thinking, Engeström (2015) modified and expanded the sociocultural thinking to be relevant to adult learning and change in organizations (cf. Engeström, 2018; Engeström and Sannino, 2016). This meant less focus on cultural psychology and more focus on how organizations change and develop as forms of cultural-historical activity systems and networks of systems (Engeström, 1998; Nummijoki and Engeström, 2013; Woll and Bratteteig, 2018). In other words, the historical development of sociocultural learning theory moves from, “the individual model of subject-mediation-object to one that accounts for the collective view of human activity” (Karanasios et al., 2021, p. 2; see also Cole and Engeström, 1993). More than tools, this encompasses additional contextual mediators of the activity system: community, rules and division of labor. CHAT has at times been criticized for not engaging with digital developments of tools and media (e.g. Rückreim); however, this critique does not entail that CHAT does not have the conceptual tools to do so (Karanasios et al., 2021).

Following is our conceptualization of digitization, digitalization and digital transformation as an interrelated process of sociocultural change in school organization.

2.1 Digitization: implementing and learning to use tools

According to Negroponte (1995), digitization means taking something analog, for example, a physical book and digitizing it into digital bits (i.e. ones and zeroes in a digital system) or conversions from analog into binary language readable for a computer. In our meaning, digitization as it pertains to educational contexts is about doing the same tasks as always but with digital instead of analog tools (Fischer et al., 2020). In Engeström's (2015) expansive learning, the characteristics of this learning are an individual’s utilization of tools in their daily teaching and learning operations. This can entail, for example, using digital tools because they are digital, typing on a keyboard instead of writing with a pen and/or that lectures are presented on a digital screen rather than on the blackboard. Regarding work tasks, there is no qualitative change in how and why they are conducted (motive; Kaptelinin and Nardi, 2006). The object is about learning to use digital tools adequately or more efficiently and increasing knowledge about the tool’s different functions more than changing how tasks and activities are performed.

2.2 Digitalization: changing tasks through tools

It is more difficult to find a straightforward definition of digitalization compared to digitization; the two are even used interchangeably (Bloomberg, 2018; Vrana and Singh, 2021). This is particularly true in reference to school and education. In general, digitalization refers to qualitative social and structural change (e.g. changes in activities; Bockshecker et al., 2018; Brennen and Kreiss, 2016) rather than doing the same thing as always but with digital tools. In our meaning, digitalization in terms of an activity system can be described as the components of the activity system having changed because of the person’s use of digital tools (cf. Pettersson, 2021). This entails that digitalization in educational contexts is doing tasks differently because of the affordances and constraints of digital technologies. One example is development and use of the digital and scientifically tested “Integrated Write to Learn” (iWTL) method, in which pupils were instructed to use digital tools to write and produce text (see Angélii Genlott and Grönlund, 2013, 2016). Writing digital texts may not be a qualitative change, but making them accessible to classmates and teachers and subsequently discussing and refining them together during the writing process might be. As such, new guidelines were produced for the learning assignment, and classmates took an active role in giving each other formative feedback during the learning process. From a CHAT perspective, digitalization as sociocultural change entails that new rules and divisions of labor have been created. However, this change does not mean that the object of activity has qualitatively changed on a collective level. In Engeström’s (2015) expansive learning, the characteristics of this learning are the restructuring of tasks on the individual level within an activity system.

2.3 Digital transformation: changing the object of activity

Finally, digital transformation in comparison to digitalization often refers to a larger scale and/or quality of social change in general (Hinings et al., 2018; Mangematin et al., 2014). For example, regarding the cultural transformation of societies (cf. Fischer et al., 2020). In our meaning, digital transformation in terms of an activity system can be described as the object of an activity system – the why of people working together (motive) – has been qualitatively developed (cf. Pettersson, 2021). This development builds upon the previous two changes: digital tools are used, which starts to change individual components of the activity system. When these changes occur to a high enough degree in the components, the collective understanding of the activity in relation to other activity systems takes on new meaning (cf. Engeström, 2015). In the case of the digitally supported iWTL, the new form of writing, discussing, giving and receiving formative feedback as a process of refining text together resulted in new collaboration between teachers and better student results on the national tests (these tests are developed and constructed at several of the country's universities on behalf of the Swedish National Agency for Education) (Angélii Genlott and Grönlund, 2013, 2016). The initiative was spread to other teachers and subjects with the same results. Subsequently, it was noticed by the school management, and it was decided that all schools in the municipality would receive time and support to use and develop their teaching in relation to the digital iWTL model. In turn, new ways of talking about teaching and learning in schools and municipalities, new paths for collaboration and social exchange and new priorities in professional development were developed in relation to the new object of teaching. Thus, the new way of teaching due to the digital iWTL led to changes (expansion) in community, division of labor and rules. As such, digital transformation as sociocultural change can be conceptualized as the object and why (motive) being subject to development and taking on transformed meanings, collectively shared rather than individual meaning (cf. Lund and Aagaard, 2020). In Engeström’s (2015) expansive learning, this learning’s characteristics are the expansion of the object of activity. As such, digital transformation is an interrelated process of digitization and digitalization that entails that the object of activity is developed along with other components of the activity system. In short, the “why” may take on new meanings in a school organization.

3. Method

3.1 Development of the digital transformation scale

To develop, test and validate a DTS, a pool of possible items for the questionnaire was initially generated according to the theoretical rationale described above. Items respond to individual and collective dimensions of human learning and development, respectively. In total, three dimensions with corresponding items are used in our DTS questionnaire.

By dividing learning and development into three dimensions and incorporating digital technologies into questionnaire items, we also suggest that learning and development through digital technologies correspond to digitization, digitalization and digital transformation as sociocultural change. Thus, sociocultural change in the questionnaire is constructed as three dimensions that correspond to change and transformation in educational contexts caused by digital technology use in human learning (see Table 1).

The items used in the DTS questionnaire consisted of 17 positively worded statements rated on a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree). The context of the measurement is in Sweden, and the items have been translated from Swedish to English (See Table 2).

3.2 Participants and procedure

A sample (n = 337) of employees from two fields participated in the study. One private organization (n = 70) represents from public school sector in one municipality (n = 267). The sample consisted of 127 males (38%), 208 females (62%) and 3 with a nonbinary gender identity. The questionnaire was administered during autumn 2020 to the respondents with the digital tool LimeSurvey. The full questionnaire for this study took approximately 15 min to complete.

Before deploying the questionnaire, respondents received information about the study’s general aim, and they were informed that their answers were strictly confidential and would be used for research purposes only. We did not include participants’ names or other personal characteristics in the data. We also assured participants that we would handle the data to protect their privacy. They participated voluntarily, and we provided no compensation.

3.3 Data analysis

To assess if the collected data met the requirements for a deeper analysis, the first step was to calculate means and standard deviation for each item. In addition, skewness and kurtosis for each item were examined. Furthermore, intercorrelation between the items was analyzed (Field, 2018).

With the goal to determine the relationship between the observed items and dimensions, EFA, with principal-axis factoring as an extraction method, was used to examine the number of factors underlying the data (Tabachnick and Fidell, 2009). The rationale for using principal-axis factoring extraction was to reduce the data and summarize variables regarding the theoretical framework. For testing if the data were suitable for factor analysis, the Kaiser–Meyer–Olkin (KMO) test was used (Field, 2018), showing KMO was 0.90, verifying the sample adequacy for analysis.

Items with pattern coefficients greater than 0.60 were retained in the model. The criteria used for factor retention were the Kaiser–Guttman rule (eigenvalues >1) and the scree-test. We chose direct oblimin rotation as the rotation method to minimize the cross-production of loadings and improve the pattern found (Tabachnick and Fidell, 2009). It was also assumed that the factors were correlated, and the pattern matrices of the analysis were reported.

The internal consistency of the distributed questionnaire for the present sample was assessed through coefficient alpha (α), with scores over 0.80 considered satisfactory. All data analyses were made in IBM SPSS Statistics Version 28.

4. Results

In the analysis of the DTS questionnaire, means and standard deviations were examined. As seen in Table 2, the examination of skewness and kurtosis for each item revealed three items in the digitization dimension that had a skewness value below −1.0 and a positive kurtosis value. These were left in the analysis; since the deviant skewness and kurtosis could be expected due to the general content of the items, i.e. since digital tools are common and used in our so-called digital age.

The results from the EFA (presented in Table 3) showed that all items had pattern coefficients greater than 0.60 on only one factor, showing that the items are related to that specific factor. After oblimin rotation, digitization explained about 11% of the variance in the data, digitalization explained about 44% of the variance of the data, and finally, digital transformation explained about 13% of the variance of the data. Taken together, the three dimensions explained about 68% of the total variance.

The internal consistency of all 17 items was acceptable (α = 0.92). The internal consistency for the three dimensions was also calculated and found to be slightly lower. For digitization, the coefficient α was 0.87. For digitalization, the coefficient α was 0.91, and for digital transformation, the coefficient α was 0.89.

The relationships between the different factors were also examined. The correlation between the different factors was low to moderate, ranging from r = 0.32 to r = 0.43. Implying that there are three different dimensions, however, related to the overall concept of digitalization.

5. Discussion

In answering the call for theoretical understanding regarding the ambiguities of digitalization, in this paper, we have conceptualized changes in educational contexts as three dimensions (i.e. digitization, digitalization and digital transformation). These dimensions have been generally utilized in previous research on digitalization (Bockshecker et al., 2018; Correani et al., 2020; Hanelt et al., 2020; Siljebo, 2020; Westerman et al., 2014). Moreover, the results suggested that a quantitative approach to empirical testing provided validity and reliability to the theoretical conceptualization and understanding.

In this study, the conceptualization of digitization regarded changes in school and education as doing standard tasks with new technologies, but the tasks do not necessarily change because of this use. In the conceptualization of digitalization, changes in tasks were a basic assumption. Consequently, an individual or group starts to change tasks (e.g. via experimentation), and the mediators consisting of rules, community and division of labor undergo qualitative change. In the conceptualization of digital transformation, the object of activity undergoes qualitative change the “why” in the organization changes.

The results in Table 2 show that the three dimensions in the DTS are hierarchical in the sense that the mean values become lower further down the hierarchy, where digitization is highest and digital transformation is lowest. We can infer that, given that organizations today generally make use of digital technologies for standard work tasks (digitization) and that it is much rarer for digital transformation to take place than for digitalization, lower values correspond with an organization that is not currently experiencing larger changes.

The EFA (Table 3) shows that digitization, digitalization and digital transformation, when conceptualized as sociocultural change via digital technologies in school and education, are both internally consistent (all α > 0.87) and measure qualitatively different dimensions of sociocultural change. The EFA further shows that the three dimensions explained 68% of the variance.

As previous research has shown, merely implementing digital tools in activities does not lead to large change, although the digital age may be characterized by large-scale changes in education and society (Fischer et al., 2020; Hanelt et al., 2020; Karanasios et al., 2021). Our conceptualization and empirical test support the salience of sociocultural perspectives in the digital age. An important avenue for future research regarding digital technologies’ role in changing school and education is to question empirically – both quantitatively and qualitatively – whether the changes are desirable (cf. Fischer et al., 2023; Karanasios et al., 2021).

6. Conclusion and implications

To conclude, with a focus on human learning, this paper aimed to develop, test and validate a DTS. More specifically, the aim was to validate digitization, digitalization and digital transformation as hierarchical dimensions of sociocultural change in school and education by using EFA. The study shows that the internal structure of the DTS corresponded to the internal structure of the theory and may be used quantitatively to analyze digital transformation in organizations. However, the instrument presented in this study needs further development in order to consolidate its psychometric properties and enable a more thorough exploration of the construct. The present study was a first exploration, and furthering the development could mean replacing malfunctioning items with items elaborating on the construct. As presented in Table 2, some of the items, although expected, show deviant characteristics (e.g. skewness and kurtosis). Further development and validation would, therefore, include applying the instrument to other samples and contexts. Other methods, quantitative as well as qualitative, should also be used in order to sample evidence about the validity of the instrument. Although factor analysis and construct validity are strongly associated, factor analysis is only part of proper construct validation. This was a first example of quantitatively measuring the three dimensions of digitalization through DTS. The contribution of this study is a holistic quantitative measurement of digitalization, built on sociocultural learning theory, as a social process rather than primarily a technical process. This has implications: (a) that the DTS is applicable to study change and transformation in educational settings such as schools beyond learning outcomes in subject teaching (cf. Puentedura, 2006); (b) bringing the CHAT closer to the calls for development in the digital age (Karanasios et al., 2021); and, not least, (c) providing educational researchers additional tools to articulate what they mean by digitalization (Fischer et al., 2020; Pettersson, 2021; Warner and Wäger, 2019; Wessel et al., 2021).

Table 1

Description of items related to dimensions

Dimension	Description
Digitization	The degree to which digital technologies (tools) are used to achieve “daily” operations (5 items)
Digitalization	The degree to which daily tasks have changed because of digital technology use (6 items)
Digital transformation	The degree to which the understanding of the object of activity has transformed because of digital technology use. Large changes in the (organization) activity system (6 items)

Source(s): Created by authors

Table 2

Mean scores, standard deviation, skewness and kurtosis for the 17 items (n = 337)

Dimension	Item	Mean	Std Dev	Skewness	Kurtosis
Digitization	I use digital technologies to plan my daily work tasks	3.86	1.19	−0.81	−0.23
	I use digital technologies to do my daily work tasks	3.99	1.09	−0.91	0.12
	I use digital technologies for documentation	4.16	0.96	−1.01	0.48
	I use digital technologies for communication	4.23	0.91	−1.15	1.13
	I use digital technologies to search for information	4.51	0.71	−1.42	1.90
Digitalization	Digital technologies have changed how I do work tasks	3.77	1.07	−0.61	−0.15
	Digital technologies have changed how I plan work tasks	3.58	1.09	−0.51	−0.19
	Digital technologies have developed new work tasks	3.81	1.09	−0.64	−0.24
	Digital technologies have changed how I think about work tasks	3.40	1.11	−0.19	−0.50
	Digital technologies have changed how I talk about work tasks	3.20	1.13	−0.07	−0.62
	Digital technologies have resulted in different collegial discussions about work tasks	3.25	1.13	−0.20	−0.55
Digital transformation	In our organization, we collaborate on questions regarding digitalization	3.37	1.03	−0.31	−0.17
	In our organization, there is a culture that encourages development	3.50	1.02	−0.39	−0.15
	Digital tools contributed to new ways of working that have spread throughout the organization	3.67	0.98	−0.48	−0.14
	Digital tools enable us to achieve goals and visions that help develop the organization	3.41	0.97	−0.30	−0.07
	Digital tools have led us to discuss the organization differently than before	3.16	1.00	−0.20	−0.34
	Digital tools have led us to organize the organization differently than before	3.28	1.05	−0.15	−0.38

Source(s): Created by authors

Table 3

Exploratory factor analysis pattern coefficients for the 17 items in the DTS questionnaire

	Dimensions
Items	Digitization	Digitalization	Digital transformation	h2
I use digital technologies to plan my daily work task	0.690	0.256	0.055	0.65
I use digital technologies to do my daily work task	0.743	0.200	−0.019	0.72
I use digital technologies for documentation	0.807	0.061	−0.012	0.70
I use digital technologies for communication	0.779	−0.005	−0.074	0.65
I use digital technologies to search for information	0.811	−0.129	−0.078	0.63
Digital technologies have changed how I do work tasks	0.146	0.754	−0.004	0.68
Digital technologies have changed how I plan work tasks	0.116	0.763	−0.027	0.69
Digital technologies have developed new work tasks	0.163	0.713	0.029	0.60
Digital technologies have changed how I think about work tasks	0.051	0.857	0.004	0.77
Digital technologies have changed how I talk about work tasks	−0.058	0.882	−0.039	0.77
Digital technologies have resulted in different collegial discussions about work tasks	−0.070	0.803	−0.102	0.68
In our organization, we collaborate on questions regarding digitalization	0.076	−0.123	−0.849	0.69
In our organization, there is a culture that encourages development	0.082	−0.146	−0.857	0.69
Digital tools contributed to new ways of working that have spread throughout the organization	0.111	0.000	−0.793	0.70
Digital tools enable us to achieve goals and visions that help develop the organization	0.063	0.106	−0.707	0.61
Digital tools have led us to talk about the organization differently than before	−0.149	0.363	−0.642	0.66
Digital tools have led us to organize the organization differently than before	−0.140	0.267	−0.724	0.69
Eigenvalues	1.85	7.57	2.15
% of variance	10.9	44.5	12.6
α	0.87	0.91	0.89

Source(s): Created by authors

Data availability statement: Data were collected, managed and analyzed in accordance with the General Data Protection Regulation (GDPR). The data sets generated and analyzed during the current study are not publicly available. Meta data are available from the corresponding author on reasonable request.