Understanding technology use through multimodal layers: a research review

Abstract

Purpose

The purpose of this study is to examine the use of digital technologies by teachers and students in teaching and learning from a multimodal layer perspective.

Design/methodology/approach

The article reviews 64 studies on technology use. A content analysis based on the theoretical concepts of “multimodal layers” was used to synthesise previous research.

Findings

The findings indicate that the use of technology in classroom practices by teachers and students is multifaceted and that transitions exist between technologies and sign-systems and are differently related to sign-making activities and thus constitute different uses. Between layers, traces can be made that connect the use of technology to differences in sign-making activities.

Practical implications

A multimodal layer perspective on technology use is fruitful to understand what happens at the intersection of technology and human activities in school practices. Moreover, more attention to multimodal layers can inform future effective technology usage and design.

Originality/value

The review offers comprehensive insights on how previous research has studied technology using multimodal layers as an analytical lens.

Citation

Schnaider, K., Gu, L. and Rantatalo, O. (2020), "Understanding technology use through multimodal layers: a research review", International Journal of Information and Learning Technology, Vol. 37 No. 5, pp. 375-387. https://doi.org/10.1108/IJILT-02-2020-0020

Introduction

Integration of digital technology into education has become a natural part of the school's everyday work, which calls for a more comprehensive understanding of the relationship between their nature, the use and their users (Bezemer and Kress, 2016). Recent studies have focussed on technologies' visual features related to sign-systems (i.e. semiotic resources/modalities), which link technologies, and technologies and individuals in use (Jewitt, 2017). However, so far, many studies have not been considering various components of the technologies, the combinations of these components in use, and the transitions between them that affect learning in different ways (Ravelli and van Leeuwen, 2018). There is also some confusion in research and school practice in terms of how meaning-making actually happens when using technology in teaching and learning, especially how to trace the connections between technologies, their properties and the outcomes (Kress et al., 2014).

Jewitt (2002) acknowledged how technologies and sign-systems interrelate and vary in usage from the notion of transitions between mediums. She found that the visual medium of the hardware screen, combined with software CD-ROM, resulted in the formerly written text being transformed into a visual multimodal representation, traced in individuals' sign-making. Sign-making is an ongoing process that, e.g. occurs when signs are prompted from the external world, interpreted by the individual and transformed either into an inward meaning or outward materialization of new representation (Kress, 2010).

Similarly, Adami (2010) emphasized transitions and traces related to the functional and semiotic properties of smartphones. Sign-systems such as labels, keys in different colours and screens of different sizes were activated in meaning-making by touch and oral representations. More recently, these tenets were brought up by Jewitt et al. (2020), who found that there is a transition between technologies that is needed for shifting from visual to other sensory modalities. Furthermore, Adami (2014) explored transitions and traces in relation to software and found that the functional properties of software, such as anchors, make an interactive site for actions of accessing and transferring texts. Moreover, Vigild Poulsen and Kvåle (2018) studied Instagram and the transitions and traces between the use of functional properties of the icons to highlight parts of pictures with semiotic resources such as colours to convey meaning in visual modes of representation.

The notions of transitions and traces appear central to understand technology use in sign-making activities. To obtain a more comprehensive image on such matters, the purpose of this research review is to map and synthesise how previous studies have investigated the use of technology in school settings from a multimodal layer perspective, with a special focus on the variations and connections in usage between technologies, technologies' properties and human's sign-making. A multimodal layer approach is developed in this paper for that purpose, based on a combination of the social semiotic multimodal approach proposed by, e.g. Bezemer and Kress (2016), Kress (2010) and Wartofsky's (1973) artefact taxonomy, consisting of five main elements: technologies, technologies' functional and semiotic properties, and modes of representations made by individuals in activities. The research question is, From a multimodal layer perspective, how has previous research explored teachers' and students' use of technologies in sign-making activities in K-12 schools?

Theoretical outline – multimodal layers

In this section, the approach of multimodal layers is presented, focussing on the five main elements.

Technologies

The use of technologies is related to artifacts, humans' perception and interpretation of representations. Wartofsky (1973) argued that perception is a highly evolved human trait that not only refers to biological senses and projections on the retina but is also mediated by means of our representation, the artifacts we create, which refer to humans activities to engrave sign-systems in physical and symbolic form. When perceived and interpreted according to our socially and culturally shaped knowledge, they will evoke new representations and actions (Kress, 2010; Wartofsky, 1973). This is relevant to understand digital technologies as artifacts in this way, especially since previous research tends to separate the physical from the symbolic, i.e. software is often seen as not being physically manipulated in use, and hardware as a physical rather than a semiotic artifact. The separation of the semiotic features from the physical hardware have neglected the intimate interrelations between hardware and software technologies without acknowledging that various combinations of hardware and software prompt sign-systems differently and affect what is visualized. This has led to an underestimation of the inherent sign-systems of different artifacts that facilitate human needs through multimodal communication (Ravelli and van Leeuwen, 2018). Stressing the semiotic features of technology use offers an analytical lens to understand how the combined use of hardware and software in relation to sign-systems may contribute to shaping meaning (Djonov and van Leeuwen, 2013).

Technologies' functional properties

Artifacts have different functions in use (Innis, 2013; Scheele, 2006). Wartofsky (1973) explained the relationships between representations engraved in artifacts and their purposes and functions in use – taxonomized into primary, secondary and tertiary levels, here with focus on sign-systems (Innis, 2013; Vigild Poulsen, 2018). The functions are not fixed but in constant flux – transitions are made between the different levels related to the users' activities with different technologies and their combinations that render affordances (Kaptelinin and Nardi, 2009).

Wartofsky's proposal of functional properties is significant to understand humans' meaning-making when using multirepresentational digital technologies. The primary functional property is related to sign-systems that convey modes of practice and are linked to production and use (Cole, 1996; Wartofsky, 1973). This kind of representations evokes actions and work but is not immediately connected to patterns of thinking, and is hard to relate to more abstract forms of meaning-making such as learning (Adami, 2014; Wartofsky, 1973). The secondary function, as it preserves or transmits the mode of action by its sign-based properties, provides models and systems to organize production and work (Wartofsky, 1973), for instance, a template for organizing documentation in a specific way (Alnervik, 2018), or a PowerPoint for arranging text by the use of bullet points (Zhao and van Leeuwen, 2014). The tertiary function is more implicitly related to modes of action (Wartofsky, 1973). Cole (2019) asserted that the tertiary function is separated from its direct representational form and has an abstract meaning. For instance, it can be related to the online activity of using the colours inherent in the function of the technology’s toolbar, while attending to the offline intention of conveying abstract meaning through the resources hue or saturation (Kress, 2010).

Technologies' semiotic properties

From the social semiotic perspective, it is important to address the relationships between technologies' functions, the semiotic resources used and humans' representations (Vigild Poulsen, 2018). The semiotic properties are something “in-between” that mediate the use of technologies on the tertiary functional level (Wartofsky, 1973) through making the semiotic resources available for semiotic production that goes beyond the designed functional properties. For example, using the semiotic properties of zooming to negotiate certain content in speech and writing. Semiotic properties are therefore the link that additionally enables observing the traces and transitions between technologies' functions and the outcome in sign-making activities (Kress et al., 2014).

Modes of representation

Sign-systems engraved in physical and symbolic artefacts are the precondition for internal sign-making and development of cognitive processing and learning (Bezemer and Kress, 2016). This is a circular process that can be referred to as the ongoing human activities of internalization and externalization (Vygotskij, 1978) – signs are prompted by technologies and humans, internalized into mental processing and externalized into new signs. This process transforms the resources available, and the transformations are viewed as signs of learning (Kress, 2010). Representations made by individuals have traces related to the external sign-systems that were considered visually prominent and of interest in the use of technologies in own sign-making activities (Kress et al., 2014), in connection with the notions of foregrounding/backgrounding of information (van Leeuwen, 2005).

Activities

Activities relate to a context consisting of actors, social action and semiotic work (Kress, 2010). Contexts have a specific set of features and means of relating entities (e.g. technologies) that are distinct to that activity site (Norris and Jones, 2005). Kress and van Leeuwen (2001) related the means to the concept of discourse, which is the knowledge and entities developed in a specific social context that is appropriate for and maintained by the actors in that environment. Activities in learning environments are enclosed by a subject as an example of a discourse (Jewitt, 2009) and instructional or learning design principles (Selander and Kress, 2017).

Method

To gain a comprehensive understanding of technology use in sign-making activities, this paper investigates how previous researchers have studied technology use in schools in relation to multimodal layers. The study design followed recommendations proposed by Gough et al. (2012): establishing the target of the research area and questions, creating a data log, formulating inclusion and exclusion criteria, conducting pilot searches, developing the search strategy and identifying relevant literature. These actions were followed by text searching, which in turn entailed the construction of refined selection criteria, as well as screening, assessment and selection of the appropriate articles. The selected research articles were coded and analysed with the multimodal layer framework.

Search strategies

The articles were retrieved using educationally oriented databases: Scopus, Eric and Academic Search Elite. The first cycle of searches was executed at the end of May 2019 through the Scopus database, rendering 352 articles, and the second cycle was carried out at the beginning of June, covering 622 articles from the Eric and Academic Search Elite databases.

The main search terms used were “digital tools” and “digital resources”, as these were regarded as the most frequently and interchangeably addressed in research on digital technologies in education. Because these terms are commonly and widely used and refer to both hardware and software, they enabled collection of a large quantity of potential articles. Moreover, because of the breadth of studies addressing various aspects of “use” of technologies in teaching and learning, a decision was taken to exclude this term in the search string. As a result, the final search string was: ((digital AND tools OR digital AND resource) AND teacher AND student AND NOT “higher education” AND NOT university).

Selection of articles

To conduct a valid selection of articles, and to reach a consensus amongst the researchers, an iterative process was undertaken to refine the selection criteria. Initially, general inclusion and exclusion criteria were defined as connected to the purpose of the systematic review on peer-reviewed empirical articles in English, published from 2010 to 2019.

From the two searches, 974 articles were added to Mendeley, and, as duplicates were found, the articles were reduced to 796. From the general selection criteria and by making judgements on the quality and relevance of the found studies, the process started with assessing the first 100 articles. The three authors independently read the abstracts and followed a two-step process, starting with the first 43 articles: (1) individually labelled them with “include”, “exclude” or “maybe” in an Excel document; (2) after discussion, a joint decision on the selection criteria was taken. The remaining 57 articles were independently assessed following the same procedure. This resulted in a joint decision on inclusion and exclusion of the articles displayed in Table 1.

The remaining 696 articles' abstracts were divided equally between the authors for inclusion or exclusion assessment based on the final selection criteria using the program Rayyan. General exclusion categories were adapted during a second round, and articles in languages other than English, duplicate articles and shorter articles from forums, columns and journals that no longer exist were removed. This process resulted in 391 articles. To enhance reliability in the coding process, one of the authors selected five articles as good and less good examples of the selection criteria. After discussing these examples, a delimitation was made on articles according to the relevance to “digital tools” and “digital resources”, which resulted in 105 articles. The three authors divided the remaining 105 articles for full-text reading with the aim to review their correspondence with all the inclusion criteria, which resulted in 64 articles being included in this paper.

Data analysis

A simple version of content analysis (Silverman, 2006) was adopted for data coding in the program NVivo 12. The first step of the coding process used the main categories from the multimodal layers and paired them for analysis (visible in Tables 2–8): technologies (divided into categories of hardware and software), technologies' functional properties (primary, secondary, tertiary), technologies' semiotic properties (modalities/semiotic resources), modes of representation (modalities/semiotic resources) and activities. Combinations of hardware and software are important examples of technologies used in meaning-making, why the initial division into different hardware was the first categorization, under which the other layers were organized. All layers where examined from the principles of foregrounding and backgrounding of information (van Leeuwen, 2005), i.e. information explicitly and more frequently described was deemed foregrounded and coded, while other information less salient, deemed more backgrounded and overlooked.

Results and analysis

This section presents findings from the multimodal layers and the numbers of examples.

Students' use

Rich data on students' use of PCs (see table 2) related to mostly visual and linguistic software, the traces of primary functions of, for example, displaying and recording, to the use of the secondary functions to organize such actions, by editing and cutting videos and sound, and navigating content by using hotkeys was observed. These actions are used with the tertiary function of adding special effects in video making and manipulating visual components using semiotic properties such as colours, background, etcetera to convey specific meaning in mainly visual and linguistic modalities (Wartofsky, 1973). The affordances of the technologies' sign-systems – spatiality and simultaneity (visual), kineikonic with combination of different modes such as sound in recording and sequencing (linguistic) – can therefore give clues to individuals' semiotic actions and production in the same modalities (Kress, 2010; Kress et al., 2014).

Unlike PCs, smartphones (see table 3) evoke other kinds of use and actualize transitions between modes of representation and technologies. Incorporated in mostly collaborative activities, visual and unnamed software such as web pages were observed in the use of primary functions, such as navigating and scrolling on the touchscreen to access information. Often, auditory modes of representation were generated in use, indicating that smartphones are not yet fully incorporated in other sign-making activities, which need to be manifested in time and space (e.g. linguistic) (Kress, 2010). Because it is mostly feasible to trace actions but not meanings from primary functions (Wartofsky, 1973), smartphones are considered in volatile involvement in speech, prior to engaging in other activities, not least shown in the absence of tertiary functional properties.

In students' collaborative use of IWBs (see table 4), yet other traces could be observed, and transitions were visible between properties and modes of representation. Unnamed software was used with the primary capabilities of the interactive board to write and draw, organized through the secondary function of hiding and revealing content. The tertiary function was used to layer typographic semiotic properties to convey meaning, such as that of graphic structure and volatile auditory modalities including speech and gestures, into multimodal compositions (Kress, 2010; Wartofsky, 1973).

The transition and traces found in the use of tablets (see table 5) are similar to those found for PCs. A combination of visual and linguistic software was used, mainly for their primary functions such as displaying, and recording, followed by the secondary function of organizing modes of action such as taking photos, inserting and zooming. However, when the visual, linguistic and auditory modes of representation come about in combined activities in which the recording function is used as a tertiary function in writing activities, the semiotic properties are used to alter fonts in text or engage with typographic elements to combine with the recordings into multimodal ensembles (Kress, 2010; Wartofsky, 1973).

Teachers' use

Teachers' use of PCs (see table 6) is mainly in instructional design, using their primary function to display and record to create and disseminate content to students (Selander and Kress, 2017). Because it is mostly feasible to trace actions but not meaning from primary functions (Adami, 2014), connections between software technologies and the modes of representation in the same modalities might provide insights into meaning created.

Teachers' use of IWBs (see table 7) is similar to students' use and demonstrated by unifying the actors in learning design activities (Selander and Kress, 2017). The chain of layers enables traces from visual software used with the primary displaying function to organizing the visual properties by the secondary function to, for example, hide and reveal and illuminate important parts. By the tertiary spotlight function, combinations were made with certain semiotic properties, such as colours, to signify visual meaning that was collaboratively negotiated in volatile auditory modes of representation such as speech (Kress, 2010; Wartofsky, 1973).

Teachers' use of tablets (see table 8) is limited in previous research. Undertaken in mainly instructional design, auditory modes of representation complemented by gaze and gestures were used in engagement with the primary and secondary functions to display content to students and organize further learning actions by using, for example, the zoom function. When using the tertiary functions of making recordings in the design of learning material, multimodal affordances of the technologies were combined with similar modes of representation into multimodal compositions (Kress, 2010; Wartofsky, 1973).

Discussion

Research on digital technologies in education that illuminates connections between digital hardware and software use is scarce. However, the research compiled in this review indicates that PCs and visual software are frequently employed, in favour of student use. Research on other technologies such as IWBs, tablets and smartphones is limited. Teachers' use of smartphones has been absent. Another finding is that different technologies are usually incorporated into the same activities, which is why differences or similarities between activities and the other layers are difficult to distinguish, important to investigate further.

In line with Jewitt (2009) on the visual affordances of technologies, the findings indicate a strong focus on visual, closely accompanied by linguistic modes of representation. An overall tendency is oriented towards multimodality. Gaze and gestures, and sensory modality of touch is depicted in relation to students' use of IWBs, tablets and smartphones combined with visual, linguistic and multimodal software (Jewitt et al., 2020).

Below, the notions of transitions and traces in relation to the multimodal layers will be stressed in discussing the findings.

Traces

According to Kress et al. (2014), if influencing factors are defined, proposed here by the framework of multimodal layers, traces between the technologies and their properties and humans' modes of representation and activities can be illuminated to understand the outcome of use. Examples of traces can be found when the links between the layers are intact. For example, when a hardware is combined with a specific software, the primary functions used in work, and secondary functions used to organize the work, provide possibilities for the tertiary functions to add meaning to the actions. Links can be made between the sign-systems used in the functional properties and human interpretation of them; actions that will be additionally enhanced by the connection to semiotic properties. The choices made amongst all sign-systems can be related to the making of abstract meaning (Cole, 2019) and are relative to the affordances of the properties and their meaning potentials and how they are ultimately drawn upon. Yet, the traces give clues to which of all the multiple features are regarded as visually salient, choices that are integrated into mental processing and renewed and communicated outwardly in sign-making activities (Kress, 2010; Kress et al., 2014). The circular process of internalization and externalization of interpreting and transforming sign-systems constitutes learning in connection with technology use (Bezemer and Kress, 2016; Vygotskij, 1978). In this study, these kinds of traces are observed. For instance, in students' use of PCs and tablets, properties are demonstrated to come about differently in the combination with visual and linguistic software, which is unlike the use of other technologies. Traces are found between the primary recording function and the secondary editing function, as well as the tertiary function of manipulating various aspects of visual display by colour and typographic elements as examples of semiotic properties, which are linked to visual or linguistic modes of representation. That entails that, first, technologies and sign-systems handled in technology use do impinge on what meaning is subsequently formed, and their affordances and constraints impact on modal choices made by the individual (Kress, 2010; Kress et al., 2014). Second, the choices made by the individual incorporating several traits from the original input will be a new form meaning made. Hence, although it is not possible here to conduct a detailed analysis, it is important to understand how functional properties are used for action and how they merge with the semiotic properties, as well as how the sign-systems therefore are selected and used to highlight specific aspects of meaning – in particular, if certain functional properties can be paired with specific semiotic properties, and how they relate to meaning created in a specific context.

When one layer is left out, the links between the layers are broken and makes traces harder to observe. It is clear that if only primary functional properties are used, the semiotic properties are left out, and difficulties in relating technology use to sign-making activities arise (Adami, 2014). Hence, the online actions associated with the use of properties cannot be related to the offline activities of meaning-making (Wartofsky, 1973). In summary, traces made between all layers, can give comprehensive clues to how technologies relate to the construction of meaning through the sign-systems, a recognition important to acknowledge in learning settings.

Transitions

The notion of transitions refers to the process of negotiation and selection of sign-systems inherent in all technologies and used in sign-making activities. Transitions provide insight on how sign-systems move between the multimodal layers in use (Kaptelinin and Nardi, 2009). Technologies and their properties are therefore backgrounded or foregrounded in use (van Leeuwen, 2005), which entails various attentions paid to their affordances and meaning potentials, which in turn affect meaning-making (Ravelli and van Leeuwen, 2018). Transitions are implicitly recognized by previous researchers as a common phenomenon but have not yet been explored in-depth.

This review shows that transitions exist in relation to both modes of representation and properties, and they move between technologies and users. Examples are students' use of visual software, where the tertiary function to manipulate the visual display by semiotic properties such as colours and layout was used by teachers to organize content on a secondary level. In relation to hardware, the affordances of the screen to display content related to the primary function in the use of PCs and the tertiary function in the use of IWBs. More specifically, this means that the offline shape and size of the screen related to the online displaying function in hardware, or the online highlighting functions in software, to convey offline meaning in the use of colours and layout (Wartofsky, 1973) were all foregrounded sign-systems and conveyers of meaning in the use of certain technologies, and overlooked in the use of others. These findings contribute to an understanding of the complexity of sign-making in the use of technologies and why technologies can support diverse actions and sign-making activities in some cases and for some users but not others.

Conclusion and implications

Mapping technology use by focussing on the multimodal layers can provide comprehensive insights into traces – how different sign-systems are engraved in combinations of technologies, drawn upon variously by individuals, and therefore lead to variations in actions and sign-making activities. The multimodal layers approach can also contribute to clues regarding what role transitions play in representational signs between technologies, levels of functions and properties, and how such shifts can be understood when traces are linked to individuals' choices of resources to make meaning. The importance of combining technologies' inherent properties to the patterns of human action and thinking will help us to understand how technology use relates to sign-making activities (Jewitt, 2017).

The shifts in functional and semiotic properties between technologies need to be addressed when selecting appropriate technologies (Vigild Poulsen, 2018). The properties are multifaceted, and people make different choices regarding foregrounding or backgrounding information (van Leeuwen, 2005), therefore contributing to variations in meaning-making by the users. This also means that hardware and software technologies are sectioned in relation to the multimodal layers and to the user's need to represent meaning.

Evidence has also been provided that the primary functional level is more commonly used in teaching and learning. Wartofsky (1973) and Adami (2014) pointed out that the solitary use of primary functions could be problematic because it promotes actions rather than meaning-making. Hence, it is important to consider the role of primary functions in a learning situation, as they have slight relations to the other layers and poor connections to sign-making activities. Thus, teaching and learning undertaken through technology use from mainly a primary function is troublesome, why further explorations are needed on how transitions between technologies' functional properties can give clues to other kinds of multiple use.

Table 1

Final selection criteria

Criteria	Include	Exclude
Digital technologies	Digital hardware and software	Analogue technologies
Teacher and/or student usage	Teachers' and in-service teachers' use in teaching activities in schools connected to their school practice courses/periods Student usage in learning activities Teachers' and students' joint use	Articles on digital literacy, computer and software engineering, competence development and general training, and technology adoption/adaptation
Education	All subjects	Libraries, leisure time, other work-related practices
Research conducted in education (K-12)	All school forms between kindergarten and school year 12	The purpose is to investigate technology use in K-12 schools with exclusion of higher education. Other contexts, such as from purely technical aspects, teacher surveys, within healthcare issues or from a context that only reflects use in private life/leisure time
Articles	Scholarly peer-reviewed, full-text empirical studies and book chapters, qualitative and quantitative	Systematic reviews, reviews, shorter articles from forums, columns and magazines

Table 2

Students' use of PCs

Multimodal layers	Data na	Examples
Technologies –hardware	Personal computers: 29	PCs: Desktop computers, laptops, unnamed computers
Technologies – software	Visual: 30	Visual: iMovie, Photoshop, TED talks, Google Hangouts, YouTube, Pinterest
	Linguistic: 23	Linguistic: Wordle, PowerPoint, Wikis, blogs
	Unnamed/other: 15	Unnamed/other: web pages
	Multiplicity of modesb: 14	Multiplicity of modes: VoiceThread, Glogster, StoryKit
	Auditory: 8	Auditory: GarageBand, Audacity, podcasts
Technologies' functional properties	Primary function: 15	Primary: record, scan, display on screen, mouse/voiceover, keyboard, insert, import, transfer
	Secondary function: 13	Secondary: paste, edit/cut, hyperlink, navigate, bookmarks, pinning, hotkeys, zoom function
	Tertiary function: 5	Tertiary: manipulate visual display (toolbar), effects (speeding up/slowing down), typographic style
Technologies' semiotic properties	25	Colour, background, font, typographic elements, graphics, space, layout, shape
Modes of representation	Visual: 30	Visual: images, moving images, animations, photographs, graphs
	Linguistic: 22	Linguistic: word clouds, writing
	Multiplicity of modesc: 18	Multiplicity of modes: combination of auditory-linguistic-visual-gestures
	Auditory: 16	Auditory: dialogue, music, sound effects
	Gestures: 4	Gestures: gesticulating, indicating with hands
Activities	Collaboration: 20	Collaboration: sharing, joint production
	STEM: 19	STEM: mathematical content and rules, science
	Literacy: 18	Literacy: writing/reading literature
	Individual work: 11	Individual work: managing tasks and assignments separately

Note(s): ^aRefers to number of mentioned examples in the articles (i.e. not number of articles)

^bSoftware where more than one modality is foregrounded in use

^cHumans' representations where people clearly place emphasis on the use of more than one modality

Table 3

Students' use of smartphones

Multimodal layers	Data n	Examples
Technologies – hardware	Smartphones: 5	iPhones
Technologies – software	Visual: 2	Visual: AR animations
	Multiplicity of modes: 1	Multiplicity of modes: snapchat
	Unnamed/other: 2	Unnamed/other: Wikipedia, digital text
Technologies' functional properties	Primary function: 10	Primary: scan, navigate/scroll, storage, camera, display, keypad, touchscreen
	Secondary: 2	Secondary: organize notes, photos
Technologies' semiotic properties	–	–
Modes of representation	Auditory: 6	Auditory: dialogue
	Linguistic: 3	Linguistic: reading
	Gaze: 3	Gaze: directing attention
	Visual: 2	Visual: animations, videos
	Touch: 1	Touch: interacting with system using fingers
Activities	Collaboration: 3	Collaboration: pair and group work discussion
	Literacy: 2	Literacy: reading
	STEM: 1	STEM: science simulations

Table 4

Students' use of interactive whiteboards (IWBs)

Multimodal layers	Data n	Examples
Technologies – hardware	IWBs: 4	IWBs: SMARTboard, Camtasia
Technologies – software	Unnamed: 7	Documents, notebooks, websites
Technologies' functional properties	Primary function: 2	Primary: writing/drawing function, recording
	Secondary function: 2	Secondary: manipulating objects, hiding and revealing content
	Tertiary function: 2	Tertiary: drawing on board, linking and annotating photos/text, displaying and communicating
Technologies' semiotic properties	3	Typographic elements
Modes of representation	Gestures: 4	Gestures: pointing, dragging objects
	Auditory: 4	Auditory: talking
	Visual: 3	Visual: drawings, diagrams, images
	Linguistic: 3	Linguistic: writing, annotating words
	Multiplicity of modes: 1	Multiplicity of modes: combination of auditory-visual-gestures
Activities	STEM: 2	STEM: mathematical and science content
	Literacy: 2	Literacy: annotating/discussing/producing text
	Collaboration: 2	Collaboration: discussions

Table 5

Students' use of tablets

Multimodal layers	Data n	Examples
Technologies – hardware	Tablets: 12	Tablets: iPads
Technologies – software	Visual: 7	Visual: Fotobabble, iMovie, GlogsterEdu, video
	Linguistic: 6	Linguistic: storybookmaker, worksheets
	Multiplicity of modes: 2	Multiplicity of modes: Poll, Padlet
	Auditory: 1	Auditory: podcasts
Technologies' functional properties	Primary function: 5	Primary: dragging pointer, visualizing, recording
	Secondary function: 4	Secondary: taking photos/inserting into organizer, editing/shaping videos, zoom
	Tertiary function: 2	Tertiary: recording videos on narratives, tracking alphabetic inscriptions on touchscreen
Technologies' semiotic properties	3	Font, colour, typographic elements
Modes of representation	Linguistic: 6	Linguistic: words, writing
	Visual: 6	Visual: moving images, photos, drawings
	Auditory: 5	Auditory: dialogues, discussions
	Multiplicity of modes: 4	Multiplicity of modes: combination of auditory-linguistic-visual-gaze-touch
	Gaze: 2	Gaze: directing attention
	Touch: 2	Touch: touching the screen
Activities	STEM: 5	STEM: science
	Collaboration: 4	Collaboration: shared production
	Literacy: 3	Literacy: creating e-books
	Individual: 3	Individual: managing tasks and assignments separately

Table 6

Teachers' use of PCs

Multimodal layers	Data n	Examples
Technologies – hardware	5	PCs: Desktop computers, laptops, unnamed computers
Technologies – software	Multiplicity of modes: 9	Multiplicity of modes: Poll, VoiceThread, blogs
	Visual: 9	Visual: iMovie, YouTube
	Linguistic: 7	Linguistic: Socrative, PowerPoint, WhatsApp
	Unnamed: 7	Unnamed: Internet
	Auditory: 1	Auditory: podcasts
Technologies' functional properties	Primary function: 3	Primary: display, drag-and-drop, cursor, record
Technologies' semiotic properties
Modes of representation	Visual: 6	Visual: images, graphs
	Linguistic: 6	Linguistic: text/communication
	Multiplicity of modes: 4	Multiplicity of modes: combination of auditory-linguistic-visual-gestures
	Auditory: 3	Auditory: speech, oral tutoring
Activities	Instructional design: 3	Instructional design: presentation/demonstration/development of teaching content
	Learning design: 1	Learning design: teacher and student shared design

Table 7

Teachers' use of interactive whiteboards (IWBs)

Multimodal layers	Data n	Examples
Technologies – hardware	IWBs: 6	IWBs: SMARTboard, Camtasia
Technologies – software	Visual: 5	Visual: digital photos, videos, YouTube, Flipchart
	Unnamed: 4	Unnamed: DOC files, notebooks, websites, polls
Technologies' functional properties	Primary function: 4	Primary: displaying content, handwriting recognition, screen captures
	Secondary function: 4	Secondary: linking to documents, flipping back to recover material, storing/embedding in files, hiding and revealing, layering to manipulate content, drag-and-drop to focus on salient features of representations
	Tertiary function: 1	Tertiary: spotlight function to highlight important parts of contents
Technologies' semiotic properties	8	Highlighting, colour, shading, size
Modes of representation	Auditory: 5	Auditory: discuss, talk
	Visual: 3	Visual: images, graphs
	Multiplicity of modes: 3	Multiplicity of modes: combination of auditory-linguistic-visual-gestures
	Linguistic: 1	Linguistic: writing on screen
	Gestures: 1	Gestures: pointing (at screen)
Activities	Learning design: 5	Learning design: teacher and student shared design
	Instructional design: 1	Instructional design: presentation/demonstration/development of teaching content

Table 8

Teachers' use of tablets

Multimodal layers	Data n	Examples
Technologies – hardware	Tablets: 6	Tablets: iPads
Technologies – software	Visual: 2	Visual: Explain Everything, SPARK, Vue
	Multiplicity of modes: 1	Multiplicity of modes: Skitch
Technologies' functional properties	Primary function: 1	Primary: displaying content
	Secondary function: 1	Secondary: using fingers to zoom in on content
	Tertiary function: 1	Tertiary: recording written work, making videos
Technologies' semiotic properties	3	Highlighting, colour, underlining
Modes of representation	Auditory: 3	Auditory: conversations, speech, talk
	Multiplicity of modes: 2	Multiplicity of modes: combination of auditory-linguistic-visual-gestures-gaze
	Gaze: 2	Gaze: looking at technologies
	Gestures: 2	Gestures: pointing, nodding
	Visual: 1	Visual: moving images
Activities	Instructional design: 4	Instructional design: presentation/demonstration/development of teaching content
	Learning design: 2	Learning design: teacher and student shared design