Lifelong learner needs for human-centered self-regulated learning analytics

Abstract

Purpose

Self-regulated learning (SRL) is crucial for successful learning and lifelong learning in today’s rapidly changing world, yet research has shown that many learners need support for SRL. Recently, learning analytics has offered exciting opportunities for better understanding and supporting SRL. However, substantial endeavors are still needed not only to detect learners’ SRL processes but also to incorporate human values, individual needs and goals into the design and development of self-regulated learning analytics (SRLA). This paper aims to examine the challenges that lifelong learners faced in SRL, their needs and desirable features for SRLA.

Design/methodology/approach

This study triangulated data collected from three groups of educational stakeholders: focus group discussions with lifelong learners (n = 27); five teacher interviews and four expert evaluations. The groups of two or three learners discussed perceived challenges, support needs and willing-to-share data contextualized in each phase of SRL.

Findings

Lifelong learners in professional development programs face challenges in managing their learning time and motivation, and support for time management and motivation can improve their SRL. This paper proposed and evaluated a set of design principles for SRLA.

Originality/value

This paper presents a novel approach for theory-driven participatory design with multistakeholders that involves integrating learners, teachers and experts’ perspectives for designing SRLA. The results of the study will answer the questions of how learners’ voices can be integrated into the design process of SRLA and offer a set the design principles for the future development of SRLA.

Citation

Nguyen, A., Lämsä, J., Dwiarie, A. and Järvelä, S. (2024), "Lifelong learner needs for human-centered self-regulated learning analytics", Information and Learning Sciences, Vol. 125 No. 1/2, pp. 68-108. https://doi.org/10.1108/ILS-07-2023-0091

1. Introduction

Self-regulated learning (SRL) refers to the cyclical process in which learners actively and reflectively plan and adapt their learning to achieve their goals (Järvelä et al., 2018; Zimmerman, 2000). Being able to regulate one’s own learning is a critical aspect of successful learning and can enhance academic performance and motivation (Theobald, 2021). Furthermore, SRL is essential for lifelong learning because learners need to be able to monitor and control their learning throughout their lives in today’s rapidly changing world (Taranto and Buchanan, 2020). Despite the importance of SRL, many learners struggle with SRL, particularly in the face of complex and demanding learning tasks (Koivuniemi et al., 2017). This can lead to frustration, disengagement and poor performance (Anthonysamy et al., 2020). Some students may need support to regulate their own learning effectively, and achieving self-regulation can be difficult for them.

However, due to the complexity of SRL processes, its measurement has been a major challenge (Järvelä and Bannert, 2021). To address this challenge, researchers have started to use various data streams and learning analytics (LA) to unobtrusively measure facets of the SRL to provide personalized support for the different phases of SRL, even in real-time (Azevedo and Gašević, 2019). At its best, personalized support promotes learners’ SRL and, as a result, helps them to become more successful learners (Lim et al., 2023). Although the advancement of LA has offered several insights and tools to better understand and promote SRL (Järvelä et al., 2023), a significant amount of effort remains to be put into designing LA systems that support learning with the consideration of individual goals, authentic educational needs and ethical principles (Sarmiento and Wise, 2022).

Human-centered LA has gained prominence as a distinct approach to the design and implementation of LA systems that prioritize the needs and perspectives of learners and teachers. Recent calls have asked for learners’ voices to be considered, along with the involvement of teachers in LA design, to increase LA’s transparency, acceptability and successful implementation across contexts (Buckingham Shum et al., 2019). Building on this foundation, participatory design methodologies have been recognized as an effective means for achieving human-centered LA (Bødker et al., 2022). These methodologies actively involve end-users, namely, learners, in the design process, thereby ensuring that the analytics solutions are tailored to their requirements and contextual realities. Nevertheless, as the majority of current human-centered LA studies focus on higher education, it is essential that human-centered LA studies also address other learning contexts, such as lifelong learning and workplace learning.

In this paper, we respond to these recent calls by examining lifelong learner voice for self-regulated learning analytics (SRLA) in lifelong learning. Moreover, the findings from lifelong learner voice are corroborated with their teacher’s voice for validation. In designing LA, it is important to define the stakeholders who can and should participate (Dollinger et al., 2019). Thus, balancing learner voice and teacher voice can be considered because each perspective offers unique insights. On the one hand, learner voice needs to be considered in the design of LA because students are the ultimate users of the system, and their perspectives on how it can benefit their SRL process are essential. On the other hand, teachers have a wealth of knowledge about teaching and learning and their perspectives can help ensure that the system is designed to meet the needs of both students and teachers. This paper examines learners’ challenges and desirable LA features in each SRL phase from both lifelong learners’ and teachers’ perspectives.

Nevertheless, when it comes to human–computer interaction, it is well-known that users do not always know or are certain about what they want, especially with advanced novel technologies that have the potential to change the way people work (Buckingham Shum et al., 2019; Dollinger et al., 2019; Martinez-Maldonado et al., 2021). This concern also applies to human-centered LA, that learners and teachers may not be fully aware of the effectiveness and undesirable consequences of the LA features. As a result, we argue that triangulating learner and teacher voices with domain expert opinions will ensure that the system meets their needs and provides a positive experience. Furthermore, by considering the pertinent learning theories, human-centered LA can be designed based on a solid understanding of the process of learning and is more likely to be effective in supporting learner success.

In this paper, we seek to address the recent calls by examining learners’ needs and aligning those with SRL theory and LA design principles to formulate design principles for SRLA. To address this aim, first, we propose a methodology that considers SRL theory and integrates both learners, teachers and LA experts’ perspectives for designing SRLA. Second, we apply the proposed methodology in a professional development context and illustrate how the methodology can be used to develop a set of design principles for SRLA. In particular, we seek to answer the following research questions (RQs):

RQ1.

How can learners’ and teachers’ voices be integrated with learning and design theories into the design process of SRLA?

RQ2.

What are the design principles for SRLA?

In the following section, we first establish the theoretical foundations for the study by reviewing the literature on SRL in lifelong learning for professional development and SRLA. Then, we introduce our theory-driven participatory design with multistakeholders (TPDM) method for integrating learner needs in establishing design principles for SRLA. We present and evaluate the voice of lifelong learners on their SRL challenges and desirable SRLA features. The design principles for SRLA were established as prescriptive statements that guide the future development of SRLA. After presenting and evaluating the set of design principles for SRLA, we discuss the implications of our study and we conclude by discussing its limitations and future research directions.

2. Theoretical foundations

2.1 Self-regulated learning in lifelong learning for professional development

SRL constitutes a dynamic framework that describes the processes by which learners personally orchestrate their cognitive, metacognitive and motivational strategies to attain their educational goals (Zimmerman, 2000, 2002; Järvelä et al., 2018). Learners who exhibit proficiency in SRL engage in a systematic sequence of planning, implementation and adaptation, which encompasses the setting of specific learning objectives, the continuous monitoring of their progress and the recalibration of their strategies when circumstances necessitate such changes. The significance of SRL transcends the immediate context of academic settings, holding implications for students’ long-term academic success, motivation and engagement (Theobald, 2021). SRL serves as a critical scaffold that not only augments students’ academic performance but also fuels their intrinsic motivation to learn, thereby offering a synergistic effect on their educational outcomes.

SRL extends its importance to the domain of lifelong learning, especially within the scope of professional development. Lifelong learning is an ongoing process extending beyond formal educational structures, often taking the form of informal or nonformal education embedded in daily activities and occupations (Milligan and Littlejohn, 2014). Within the framework of this study, professional development is defined as a type of lifelong learning aimed at enhancing career-specific skills. Contemporary society, characterized by an accelerated pace of technological advancements, mandates an ongoing learning process for individuals to remain adaptable and competent (Taranto and Buchanan, 2020).

In this regard, SRL emerges as an indispensable skill set for continuous growth and adaptability, empowering individuals to meet the challenges and opportunities presented by a digitized and dynamic global landscape. This can occur in various learning settings such as informal workplaces, nonformal educational platforms, as well as online and offline courses. Previous studies confirm the significance of SRL in empowering learners to actively acquire the requisite skills and competencies in a fast-paced, technology-driven society (Taranto and Buchanan, 2020). These findings align with well-established SRL strategies such as self-assessment, goal setting and self-reflection (Zimmerman, 2000). Accordingly, this study focuses on a nonformal online academy designed to teach digital skills over a period of three to six months to adult learners with diverse educational and professional backgrounds.

Research evidence suggests that the capability for SRL is especially pertinent for lifelong learners in professional development settings (Persico et al., 2015, 2020). Milligan and Littlejohn (2014) identify four key behaviors that are integral to SRL in these contexts as: consume, create, connect and contribute. Furthermore, the educational environment in professional development is often less structured, requiring a higher degree of self-regulation from learners (Milligan and Littlejohn, 2014). In such educational landscapes, students exercise greater agency over their learning paths, choosing subjects and pacing autonomously (Manganello et al., 2021). The design of digital skills boot camps, such as the one examined in this study, often incorporates pedagogical techniques that necessitate active participation from the learner (Moshirpour et al., 2019).

However, it should be acknowledged that not all individuals readily use SRL strategies in their learning processes. Challenges in practicing SRL can manifest in the form of poor task management or emotional regulation, ultimately leading to reduced academic performance or disengagement (Koivuniemi et al., 2017; Anthonysamy et al., 2020). Given these challenges, it is imperative that educational settings provide adequate support to facilitate learners’ SRL strategies.

In this study, Zimmerman and Moylan (2009) SRL model is adopted as the theoretical framework for investigating learners’ challenges in the different phases of SRL and the needs for SRLA. The model depicts the SRL process in three phases of SRL in which the output of the previous phase provides input to the next phase: (i) forethought including task analysis and self-motivation beliefs, (ii) performance including monitoring and controlling one’s learning and (iii) self-reflection including of self-judgment and self-reaction. In the context of professional development, learning is mostly informal and learners are required to take decisions not only about how and when to learn but also about what to learn (Manganello et al., 2021).

2.2 Self-regulated learning analytics

The advancement of the LA field has offered unique capabilities for better understanding and supporting SRL. LA refers to “the application of data analytic techniques and tools for the purposes of understanding and enhancing learning and teaching” (Nguyen et al., 2020). The literature has shown that LA delivers remarkable benefits to the learners, such as providing insight in a timely manner to improve outcomes (Knight and Buckingham Shum, 2017), identifying learning needs and providing personalized learning support (Nunn et al., 2016). For instance, Arnold and Pistilli (2012) demonstrated how real-time analytics could be used to identify at-risk students in the early stages of a course by integrating data from various sources, such as grades, course interactions and past academic performance, to generate “traffic light” indicators. Nevertheless, prior studies have called for better alignment between LA design and learning theories to maximize its impact on learning (Gašević et al., 2015). In particular, Heikkinen et al. (2022) noted in their systematic literature review that achieving the full potential of SRLA requires that the different phases of SRL are considered when designing and developing LA solutions.

When learners enact in forethought, performance or self-reflection phases, there are several affordances of LA that can promote SRL (Persico et al., 2015). First, SRLA can help learners to monitor their own learning and to make data-informed decisions on controlling their learning accordingly (Manganello et al., 2021). Second, SRLA can provide personalized scaffolds and guide learners to use specific learning materials or resources that promote SRL (Lim et al., 2023). Third, SRLA could be used to promote self-reflection by engaging learners in self-assessment of their own learning (see Domínguez et al., 2021). While alignment between LA design and learning theories is necessary, it alone is not sufficient to guarantee the design of good LA tools. Therefore, when designing such tools, it is critical to also consider the needs of nontechnical stakeholders, including students and teachers (Giacomin, 2014).

Furthermore, recent research has made significant contributions to the field of LA by offering various strategies that give students and teachers a voice (Alvarez et al., 2020; Buckingham Shum, 2022; Prieto-Alvarez et al., 2018). This has led to the development of human-centered LA in which critical stakeholders (students, teachers), learning and teaching environments and their relationships are identified and considered (Buckingham Shum et al., 2019). Human-centered LA is often achieved through participatory design that offers an innovative approach for ensuring that analytics tools are both effective and closely aligned with the specific needs of learners (Sarmiento and Wise, 2022). First, designing the LA tools with learners and teachers can inform them on the what, how and why the different kinds of data are collected, including how LA tools will acquire and interpret the data to promote SRL (Mangaroska et al., 2021). This increased transparency and acceptability of these tools can also foster their long-term use. Second, by using human-centered LA, both students and teachers may be better equipped to derive meaningful insights and make informed decisions based on SRLA. Namely, learners’ and teachers’ implementations of LA tools may differ from the purposes of designers (Wang and Hannafin, 2005), so designing the interventions with students and teachers may bridge the gap between the desired SRL processes and strategies and those actually realized (Schumacher and Ifenthaler, 2018).

Within the context of SRLA, central to SRL is the notion of learner agency, autonomy and self-reflection, elements that are naturally supported by participatory design methodologies (Zimmerman, 2000). Inviting learners to be active participants in the design of analytics systems ensures that the resultant tools better reflect learner requirements, preferences and contexts (Shum and Ferguson, 2012). Consequently, this participative and multidimensional approach not only yields SRLA tools that are more contextually apt but also reinforces the foundational principles of SRL by encouraging learner input and agency.

Even though the lack of students involvement in the design processes has been acknowledged (Buckingham Shum et al., 2019; Mangaroska et al., 2021), considering only their voice in designing SRLA is questionable because students understanding of effective SRL processes and strategies may be limited (Bjork et al., 2013). Even though the different stakeholders (students and teachers) can have different or even conflicting voices during design processes (Bødker et al., 2022), it is important to investigate both voices to validate design principles for SRLA. Accordingly, this design study seeks to formulate design principles for SRLA by synthesizing the voices of both students and teachers with solid theoretical foundations (Zimmerman and Moylan, 2009) and subsequently subjecting these principles to experts in SRL and LA.

3. Research methodology

To formulate a set of design principles for human-centered SRLA, this study used the well-known design science research methodology in information sciences research (Peffers et al., 2007) in conjunction with a participatory design approach (Bødker et al., 2022). This approach prioritizes the values and concerns of learners as end-users in the design of LA. However, it is important to note that the needs and preferences of learners may conflict with the instructional values held by teachers; such tensions must be addressed. Additionally, in extreme scenarios, the needs and preferences of learners could potentially have adverse effects on their learning outcomes. As such, integrating the perspectives of teachers is imperative for a balanced approach, and the application of learning theories is crucial for guiding the design of human-centered SRLA. Thus, drawing upon the design science research methodology and the participatory design approach, we present our TDPM approach as a methodological framework for formulating design principles for SRLA.

3.1 Design science research methodology

The design science research methodology by Peffers et al. (2007) serves as a comprehensive set of guidelines encompassing principles, practices and protocols for carrying out studies that focus on the design and development of artifacts for information systems, particularly educational technology (e.g. Nguyen et al., 2021). Its core aim is to develop pragmatic solutions to identified issues by emphasizing a multistage process that entails artifact creation, scientific contribution, design evaluation and result dissemination. The resultant artifacts can manifest in various forms, ranging from constructs and models to methods and instantiations, as well as design principles (Baskerville et al., 2018; De Leoz and Petter, 2018).

In the context of the present study, the research output took the form of design principles aimed at human-centered SRLA. The execution of the study followed the structured steps delineated by the design science research methodology:

• identifying and substantiating the problem;

• formulating solution objectives;

• engaging in design and development;

• demonstrating the artifact;

• evaluating the artifact; and

• disseminating findings (Peffers et al., 2007).

In this study, the initial step involved identifying the research problem, which centered on the necessity of supporting SRL for lifelong learners via human-centered SRLA. The study’s objectives, encapsulated in its aim and research questions, focused on investigating the specific needs of learners and crafting design principles tailored for human-centered SRLA. Following this, the stages of design, development, demonstration and evaluation were carried out using a participatory design method (Bødker et al., 2022), adapted into the proposed TPDM approach to fit the study’s objectives. Finally, the study’s results and findings served as the vehicle for communication in line with the design science research methodology.

3.2 Theory-driven participatory design with multistakeholders

In this study, we present an approach for TPDM. Figure 1 illustrates our TPDM approach for integrating learner needs in establishing design principles for SRLA.

Based on SRL theory, learners’ focus group discussions and teacher interviews are triangulated to determine learner needs. Together with design principles for generic learning analytics information systems (LAIS), the findings are incorporated into the conceptualization of SRLA design principles. Both the learner needs and conceptualized design principles are iteratively evaluated by experts based on the following three evaluation criteria (adopted from Venable et al., 2012): 1) evaluate the artefact to establish its utility and efficacy for achieving its stated purpose; 2) evaluate the artefact to identify weaknesses and areas of improvement; and 3) evaluate the artefact to identify side effects or undesirable consequences of its use.

4. Design requirements and inputs – lifelong learners’ self-regulated learning challenges and desirable learning analytics features

In this section, we focus on the essential design requirements and inputs that will inform the foundational principles of SRLA. These guiding elements are derived from the thematic analysis of Learner Focus Group Discussions and Teacher Interviews. By examining these rich narratives, we identify the specific challenges that lifelong learners face in their learning journeys. We also discern the LA features that both learners and teachers consider most desirable for enhancing lifelong learners’ SRL. This multistakeholder insight serves as the cornerstone for establishing empirically grounded and contextually relevant design principles for SRLA.

4.1 Participants and procedures – learner focus group discussion and teacher interviews

The study started with data collection from two groups of participants which are the learners and the teachers. The first group of participants was learners who joined three- to six-months digital skills courses (e.g. Full-Stack Web Development, User Interface and User Experience, Data Science, Digital Marketing) in a nonformal online professional development academy, also known as an online boot camp. Qualitative data collection through focus group discussion was applied to acquire the learners’ voices on the challenges in applying SRL during their learning process and the desirable SRLA features to support them. In total, there were 27 students (n = 27) who formed into ten focus groups consisting of two (three groups) to three learners (seven groups) in each group. All study participants are Indonesian; their demographic characteristics are summarized in Table 1. The topics and guiding questions for the focus group discussion are depicted in Appendix 1.

The second group of participants was the teachers (n = 5) who facilitated the courses and assisted the learners during their learning process in the mentioned online boot camp. The semistructured interview was conducted to gather the teachers’ voices on the observable learners’ challenges and the desirable SRLA features to support the learners in practicing SRL. Similar questions (Appendix 1) were asked to obtain teachers’ perspectives on learners’ challenges in SRL and features that may help the learners improve their SRL.

4.2 Thematic analysis

The data collected from the learners’ focus group discussions and teachers’ semistructured interviews were analyzed using a thematic analysis approach. Thematic analysis is a widely used qualitative method for identifying, analyzing and reporting patterns or themes within the data (Braun and Clarke, 2006). The aim of the thematic analysis in this study was to examine the voices of the learners and validate them with a balance with teacher voice. The integrated voices from both learners and teachers were then conceptualized with iterative expert evaluation for establishing design principles SRLA.

The process of data analysis included several stages, and the NVivo-12 software, a commonly used tool for qualitative analysis, was used in the analysis. Initially, audio recordings of focus group discussions and interviews were transcribed into written form and checked for accuracy. The transcripts were then read several times, guided by the research question and the findings from the literature review, to identify initial codes related to the challenges in SRL and associated needs for SRLA. Following the initial code grouping, the themes were refined, combined or divided in the next stage. A series of revisions and modifications were made to the themes as new information emerged during this stage. During data analysis, a conclusion is reached when it appears that no new themes emerged from the data, and the themes seem to have captured the essence of that data. In the final stage, the themes were elaborated on in detail, and evidence of the themes was provided by relevant quotes from the participants.

4.3 Lifelong learners’ self-regulated learning challenges and desirable learning analytics features

Understanding learners’ expectations of SRLA is essential for meeting their needs. This study examines the challenges of learners and their expected features and subfeatures of SRLA in the different phases of SRL, namely, forethought, performance and self-reflection phases. The results of the thematic analysis on lifelong learners’ focus group discussions and teachers’ semistructured interviews are presented in Table 2, which summarizes lifelong learners’ voices on SRL challenges and needs, and desirable features and subfeatures for SRLA. The frequency of occurrence and examples of statements for each theme are reported in Appendix 2.

Each SRL phase is reported with associated challenges and needs as well as desirable features and subfeatures. Forethought is an important part of SRL, and it involves planning and preparation of learning activities (Zimmerman, 2002; Zimmerman and Moylan, 2009). In this phase, learners anticipate features that will assist them in understanding tasks, setting goals and planning their learning journeys. It includes information and visualizations of the learning path, instructions for tasks and end goals, as well as personalized recommendations tailored to the individual’s motivations. As part of this phase, a comprehensive and engaging dashboard will be needed to set effective goals and plans:

Group 7 – Student 3: “I start making a plan and strategy regarding what I needed to learn day by day. Sometimes, I’m confused to plan the roadmap, like what to learn first. There are so many resources on the internet, but I need a guide on what I need to learn first and how far. Fortunately, after I joined the Bootcamp, I get guidance. I also need the roadmap to keep me motivated and pressured so I will not procrastinate.”

Teacher 1: “It can be applied in the learning dashboard. I imagine it can be like a timeline or milestone which can show two dots. The first dots can be the general milestone that they need to achieve. The other dots show where they are now. So, it can be exciting that we can see where we are currently. So, this is other than the reminder notifications. It can show the end-to-end.”

During the performance phase, learners need features that will help them manage their time, monitor their progress, seek information and receive help and support. This includes task and time management, real-time monitoring, help-seeking, stimulating interests and excitement and rehearsing and constructing knowledge. The challenges in this phase include managing time, tracking progress, efficiently accessing information and receiving the required support:

Group 5 – Student 2: “[…] So, they know their progress each day and accumulate to each week, each chapter, and so on. We can see the tracker, progress, and report. It can be similar to the apps that we usually use to exercise at the gym. We can see the graph for each week, each day, and month. We can see whether it decreases or increases. If it increases, we can set it to keep on that pace. If it decreases, we can reflect, maybe we need to adjust our learning method or why am I being like this, what the cause is, is it because of my condition during the study, or any other personal problem?”

Teacher 3: “[…] motivation also plays a role. For example, I have a student that didn’t come from a technology background. Her motivation is enough to join the course in the beginning, but not enough to drive her until graduation. So, the motivation needs to be updated frequently along the learning journey. For example, actions to provide encouragement and asking questions to check up on their conditions. Those are impactful for the students…So, I think motivational encouragement should not only available in the beginning but also be provided until the end of the learning journey.”

The self-reflection phase requires features that will help learners evaluate their performance and make adaptive improvements. This includes self-evaluation analytics based on various standards and recommendations for improvement. The challenges in this phase include objectively evaluating performance and making adaptive improvements:

Group 4 – Student 1: “A summary of score to show our performance progress, like what things we still need to improve and what we have been good at.”

Teacher 2: “[…] Usually, the challenge is in how they try to acquire data to help them evaluate themselves. Because to evaluate yourself, you need to be able to see from several perspectives […] […] They tend to also see only from one factor […]”

Finally, ethics and data preferences require features that ensure the information is trustworthy and protect the privacy of the learner’s data. This includes data privacy, transparency and security. The challenges in this phase include concerns about sharing personal data and the need for information on data privacy and security:

Group 5 – Student 3: “Personally, as long as, we were asked for consent in the beginning on what kind of data and for what purpose, I think I’m okay if it is for the purpose of the learning analytics.”

Teacher 2: “Based on my understanding, I think there are two keys which are permission and personal data. It is better to not include personal data and for the permission, we stated clearly what data that will be acquired and what will be used. So, this is to protect privacy. But, we need to be very clear on what will be acquired and what it will be used for.”

In conclusion, learners expect SRLA to provide a comprehensive and engaging dashboard with features that support each phase of the learning process, from understanding the task to evaluating performance and ensuring data privacy. These features should help learners manage their time, monitor progress, seek information, receive help and support, evaluate performance and make adaptive improvements while ensuring that the information they use is trustworthy and protects their privacy.

5. Conceptualization of design principles for self-regulated learning analytics for lifelong learners

The conceptualization of design principles for SRLA was informed by the evidence gathered for lifelong-learner and teacher voices (C1–C9 in Table 2) in alignment with SRL theory (Zimmerman, 2000), and guided by the established design principles for generic LAIS (Nguyen et al., 2021) (see Figure 1). The conceptualized design principles were also iteratively evaluated and refined through expert assessment. The conceptualization centers on the activities that the system should afford for addressing learners’ needs as well as promoting SRL to enhance their learning outcomes.

According to Nguyen et al.’s (2021) principle of actionable information, LAIS “should have features that allow for the reporting of actionable information about learners and their learning” (p. 555). Our findings reported that learners need features to keep track of their learning progress toward personal goals and objectives (C1–C4). Importantly, the lifelong learners in this study highlighted the need for personalized recommendations based on their own motivational factors (C2). Accordingly, we argue that SRLA should be able to provide learners with personalized information and visualizations to self-monitor their learning process and prepare for the planned outcomes; thus, the first initial design principle [Init. design principles (DP)] is as follows:

5.1 Init. DP1. Principle of personalized monitoring

SRLA should provide personalized features for learners to self-monitor, self-track and rehearse their learning process toward the planned goals.

With LA, learners can receive personalized recommendations based on their learning interests, capabilities and outcome expectations (Nguyen et al., 2018; Van Schoors et al., 2021). As a result, learners can set more meaningful and achievable learning goals, and make more informed decisions about their futures. Furthermore, aligning learners’ individual learning progress with their own learning goals could also motivate them to better engage in learning. Motivation plays a significant role in SRL as it drives individuals to take initiative and actively engage in the learning process (Zimmerman and Moylan, 2009).

As a result of our findings, learners need analytics to regularly engage them in learning with features that stimulate their interests and motivate them to maintain their learning progress (C3, C6, C8). Accordingly, our initial DP2 was defined as the principle of continuous engagement that recommends SRLA to provide personalized features to stimulate interests and excitement to boost their motivation and engagement in learning. Nevertheless, Nguyen et al. (2021)’s principle of information timeliness recommends LAIS to deliver analytics at the right time for its maximum impact, and learning sciences and decision sciences can provide insight into the best way to design the time latency between data collection and reporting.

5.2 Init. DP2. Principle of continuous engagement

SRLA should provide personalized features to stimulate interest and excitement to maintain learners’ engagement in learning.

A key component of SRL is reflection, which enables learners to assess their learning experiences and use that information to guide future learning (Zimmerman and Moylan, 2009). It is a process of introspection in which learners examine their own experiences, thoughts and emotions. Nevertheless, learners can also reflect through the peer reflection process in which learners reflect on their own experiences and then share their insights and perspectives with their peers for discussion and further reflection. In addition to self-reflection and peer reflection, reflection on feedback from other sources, including LA tools, can also be a form of reflection. In this study, lifelong learners reported challenges to self-evaluate objectively and comprehensively to remain adaptive (C8). It is suggested for SRLA to involve features for acquiring or generating feedback from several sources, including individual and collective feedback from peers and experts.

5.3 Init. DP3. Principle of critical reflection

SRLA should deliver formative and summative feedback based on different standards and from different stakeholders for adaptive improvements.

Research has shown that social interactions and support could significantly influence learners’ SRL (Kwon et al., 2014). Our findings from learners’ focus group discussions emphasized the need for SRLA to offer access to social and psychological support during the learning process (C5). This resulted in our proposed principle of social support as follows:

5.4 Init. DP4. Principle of social support

SRLA should recommend and provide access to social and psychological support.

The principle of social support posits that SRLA systems should extend beyond mere academic metrics to include recommendations and resources for social and psychological support. This extension appears well-founded, given the gaps in the current literature. Traditional SRL theories (e.g. Zimmerman, 2000) predominantly focus on cognitive elements such as planning, time management and meta-cognition. Likewise, existing SRLA systems, as described by Winne (2017), have largely been confined to these dimensions. However, this narrow focus overlooks the significance of affective and social elements, which have been emphasized in broader educational psychology literature. The inclusion of social support aligns with Vygotsky and Cole’s (1978) constructivist theories, Bandura’s (2001) social cognitive theory and Hadwin et al.’s (2018) SRL model. Empirical evidence from Chen (2022) and Järvelä et al. (2015) supports this, showing improved student engagement and well-being. However, ethical challenges related to data privacy (Nguyen et al., 2023; Slade and Prinsloo, 2013) and measure accuracy (Ifenthaler and Widanapathirana, 2014) should be carefully considered in future implementations. Accordingly, DP5 was conceptualized to address the ethics and privacy aspects of SRLA as follows:

5.5 Init. DP5. Principle of ethics and privacy

SRLA should provide anonymity and transparency for personal data and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure, or access.

The principle of ethics and privacy calls for SRLA to emphasize anonymity, transparency and robust data protection. This principle complements existing literature that critiques the lack of ethical considerations in traditional educational analytics systems (Slade and Prinsloo, 2013). The call for anonymity aligns with broader trends in information systems emphasizing user control over personal data (Nguyen et al., 2021), while transparency fosters trust, enhancing the effectiveness of SRLA (Buckingham Shum et al., 2019). Additionally, the explicit focus on data protection addresses the real-world risks of data breaches, aligning with regulations like the General Data Protection Regulation (GDPR) in the European Union. In sum, this principle provides a crucial ethical framework for SRLA but demands ongoing commitment to ethical vigilance to adapt to emerging challenges.

6. Evaluation of the design principles for self-regulated learning analytics for lifelong learners

Four experts in the domains of LA and SRL were invited for the evaluation of both learner needs and the conceptualized design principles for SRLA. Three of the experts are working in the field of learning sciences (Expert A: Female, 12 years of experience; Expert B: Female, six years of experience; and Expert C: Male, 15 years of experience), while the other one is in the field of information sciences (Expert D: Female, eight years of experience). All the experts have published scientific papers within the domains of LA and SRL. They are presented with the conceptualized design principles for SRLA and the findings from learner focus group discussions and teacher interviews. After their own reading time, the researchers first walked them through the learners’ needs and their desirable features for SRLA. The experts were asked to provide their assessment based on the provided evaluation criteria (adopted from Venable et al., 2012; see above). They were then requested to evaluate whether the set of conceptualized design principles for SRLA has captured all learner needs, and any improvements needed for design principles to serve as useful guidance for SRLA design and development. The expert evaluation sessions lasted between 40 min and 60 min and were videotaped for later examination in evaluating and refining the design principles for SRLA. All the presented learner needs and design principles were sent to the expert after the evaluation sessions for potentially additional examination and feedback via emails.

In the evaluation, all four experts agreed with the importance of this principle of personalized monitoring (DP1 evaluated). Regarding DP2, a learning sciences expert (Expert A) has raised concerns about “continuous engagement” because excessive engagement in learning can be harmful to learners. Studies have pointed to the phenomenon of “academic burnout,” where constant engagement with educational tasks may lead to stress, emotional exhaustion and reduced academic performance (Schaufeli et al., 2002). Furthermore, Kirschner and Karpinski (2010) found that higher engagement in online learning environments could also result in decreased time for other crucial activities, including physical exercise and social interactions, which are essential for holistic well-being. Therefore, while engagement is often considered a positive metric in SRLA, there is a need to balance it with considerations for the learner’s overall health and well-being. Appropriately, after carefully reviewing the issue, we revised the initial DP2 to focus on persuasive motivation for SRLA to stimulate learning interests:

6.1 Revised DP2. Principle of persuasive motivation

SRLA should provide personalized features to stimulate interest and excitement to boost their motivation and engagement in learning.

This revised DP2 finds grounding in self-determination theory, which posits that meeting individual needs for autonomy and competence can boost intrinsic motivation (Deci and Ryan, 1985). For instance, the use of “gamification” elements has demonstrated the potential to increase both motivation and engagement in learning contexts (Deterding et al., 2011). Therefore, incorporating persuasive elements into SRLA could serve as a powerful strategy to enhance learner motivation and engagement, albeit with a mindful approach to avoid potential pitfalls such as excessive engagement.

Interestingly, regarding DP3, evaluation with the experts raised a critical discourse about the role of SRLA in improving and/or promoting learners’ SRL. Expert B from learning sciences questioned about the undesirable consequences of SRLA in establishing learners’ reliance on technology for sustaining SRL activities. It is recommended that SRLA should consider a scaffolding approach in developing learners’ SRL skills. Recent research has highlighted the importance of adaptive scaffolding in supporting SRL (Song and Glazewski, 2023). On the other hand, although Expert D from information sciences agreed with these side effects of SRLA, Expert D argued for the nature of living with technology in the age of artificial intelligence (AI). The augmentation was provided for the fact that many learning activities and processes are now cooperating with the use of technology, such as Google or Microsoft 365, which extends beyond the classroom to include daily life tasks that are integral to SRL. For instance, planning and time management – key components of SRL as outlined by Zimmerman (2000) – are often facilitated through digital tools like Google Calendar or Microsoft Outlook. These platforms not only assist in organizing academic deadlines but also help manage personal commitments, thereby enabling a holistic approach to self-regulation (Koehler and Mishra, 2005). Thus, the fusion of SRL activities with ubiquitous technology platforms highlights the increasingly blurred boundaries between educational and everyday digital tools, reinforcing the need for SRLA to be adaptable and integrated across multiple contexts. Future studies on the current topic from different philosophical perspectives are, therefore, recommended.

All the experts reached a consensus on DP4 and also referred to recent literature that highlighted the importance of emotional regulation (e.g. Järvenoja et al., 2019) and its support. They also shared an agreement on the principle of ethics and privacy that advocates SRLA to anonymous and transparent handling of personal data, with protection against unauthorized access, destruction, alteration or disclosure. It is important to consider questions, such as whether the collection and use of learner data are necessary, whether the data is being used in a fair and transparent manner, and whether the benefits of the data use outweigh any potential detriment. The ethical and privacy aspects of LA remain challenging (Ladjal et al., 2022), yet we hope that the findings from this study can contribute to the discussion to push forward the design and development of trustworthy LA systems.

Table 3 summarizes the final five design principles for SRLA as the output from our TPDM. The design principles offer prescriptive knowledge about the design of SRLA systems.

7. Discussion and implications

The importance of lifelong learning along with professional development has never been greater than it is today in a world that is fast-changing. SRL has been a key skill for successful lifelong learning (Molenaar, 2022). By providing learners with insights and recommendations based on their learning activities and progress, LA tools and technologies aim to facilitate and enhance SRL (Winne, 2017). The integration of LA into SRL can help lifelong learners make informed decisions about their learning activities, monitor their progress and evaluate their performance. Although research has provided insightful understandings of SRL processes and LA support for SRL, previous studies have been mostly conducted in the context of K-12 and higher education (Heikkinen et al., 2022). Up to now, far too little attention has been paid to SRLA targeting lifelong learners. This study thus attempts to fill a noteworthy gap by examining the challenges and needs of lifelong learners in SRL (RQ1, Table 1) to inform the design and development of SRLA (RQ2, Table 2).

Human-centered LA has grown rapidly in recent years and has the potential to support learners in developing SRL skills and improving their academic performance. In spite of these benefits, it has been questioned about how to consider the voice of learners more into the design and development of human-centered LA (Buckingham Shum et al., 2019). In this study, we present an approach for TPDM (Figure 1) that addresses the well-recognized concerns about the alignment between the needs of the learners and the design of LA tools and systems. Learners’ learning preferences are not always aligned with teachers’ pedagogical intentions or successful learning theories. Our TPDM methodology offers guidance for integrating and evaluating learner needs into the design of theory-grounded human-centered LA.

TPDM approach offers a comprehensive framework that combines viewpoints from various key players in the educational process. approach distinguishes itself from existing methodologies primarily through its comprehensive integration of perspectives, ranging from frontline users like lifelong learners and teachers to theoretical frameworks and expert evaluations. Traditional approaches, such as user-centered design (Norman, 1986) or participatory design (Schuler and Namioka, 1993) often focus more narrowly on user involvement or the inclusion of specific stakeholder groups, respectively, but may lack a structured integration of theoretical constructs. TDPM not only incorporates experiential insights from these stakeholders but also synthesizes foundational concepts from learning theories and design theories to inform its methodology. The inclusion of these nascent theories provides a structured backbone, allowing for data-driven adjustments and refinements in the design process (Baskerville et al., 2018; Heinrich and Schwabe, 2014). Finally, the TDPM approach is subject to evaluation by experts in the field, adding a layer of rigor and validity to the outcomes. This multilayered integration serves to produce more contextually relevant, empirically supported and theoretically robust educational interventions or tools, therefore, enhancing its applicability and impact.

In terms of research impact, TDPM contributes to the empirical rigor by synthesizing insights across a diverse set of stakeholders and theories. It addresses gaps in existing frameworks that may be overly reliant on a single theory or stakeholder perspective. Thus, TDPM fosters a more nuanced and comprehensive understanding of learning contexts, providing fertile ground for interdisciplinary research. For practical applications, TDPM’s multifaceted approach ensures that educational interventions or tools are not only theoretically robust but also contextually relevant and practically effective. By incorporating real-world insights from lifelong learners and teachers and subjecting the design to expert evaluations, TDPM enhances the usability and effectiveness of the final product. This broadens the research artefact’s appeal and adaptability, making it more likely to be adopted in diverse learning environments.

Our findings also suggest some practical implications for promoting SRL in professional development programs. Taking control of an individual’s learning process is an essential element of SRL, which leads to improved outcomes and greater satisfaction (Theobald, 2021). However, it is challenging for lifelong learners in professional development programs to manage their learning time while learning is often not their sole or even main occupation. Accordingly, it is crucial to provide lifelong learners with support for time management as well as to keep up their learning motivation. The results of this study can also shed light on developing SRLA systems to address lifelong learner needs.

The results of this study indicate that it can be helpful to learners to stay motivated by real-time monitoring and tracking of their learning activities and progress toward their personal goals and objectives. The use of motivational notifications and gamification can also aid in stimulating interest and excitement about learning. Previous research suggests that gamified approach embedded with SRL support can leverage students’ intrinsic motivation and lead to better learning (Qiao et al., 2022). In line with prior studies, our findings imply the importance of the self-regulated gamified approach for lifelong learning.

Furthermore, LA has been the subject of controversy for years (Pargman and McGrath, 2021). LA design and use are hampered by a lack of evidence and guidelines for practical ethics from end-users’ perceptions and assessments. Triangulating data collected from both lifelong learners and teachers, this study informs learners’ ethical concerns, relevant data and the conditions in which they are willing to share for SRLA. Learners in this study reported concerns regarding the security, privacy and transparency of their data, which is consistent with the literature. Nevertheless, as long as the LA process is secure and transparent without using sensitive personal information (e.g. income, health, etc.), lifelong learners expressed a high willingness to share their data for SRLA that could support their learning effectively.

It is important to note that this study is limited to analyzing data from a single group or context. This can limit the generalizability of the findings that may not be applicable to other settings or cultural contexts. Further research may conduct multisite studies or comparative studies in different contexts to validate the generalizability of our findings. Furthermore, the limitation of thematic analysis lies in its susceptibility to researcher bias and its challenges in managing large or context-rich data sets, which can compromise the depth and validity of the study (Braun and Clarke, 2006; Guest et al., 2011). Despite these limitations, the method’s adaptability and ease of use make it an effective approach for eliciting and understanding the varied viewpoints of stakeholders. Another limitation of this study is related to the small sample size of teachers and experts involved in this study. Although teacher voices and expert opinions were only applied to evaluate the learner needs, the small sample sizes may not provide a comprehensive validity and generalizability of the findings. Notwithstanding the relatively limited sample, we gained great insights from the triangulation of results from learners, teachers and experts.

8. Conclusion and final remarks

In the age of AI, the ability to self-regulate learning for lifelong learning is increasingly becoming important for individuals who want to remain relevant and competitive in the workplace. As an example, the recent release of OpenAI’s Chat-GPT serves as a reminder of the importance of self-regulated lifelong learning as it demonstrates the potential of AI to automate job tasks previously performed by humans. Individuals must constantly learn new skills and adapt to new knowledge due to the rapid pace of technological advancement and the constantly changing job market. Lifelong learners can stay up to date with the latest developments in their field by taking responsibility for their own learning through SRL. Embracing human-centered LA can contribute significantly to supporting lifelong learners’ SRL. The proposed approach for TPDM will allow advanced integration of learning sciences and design science research into the development of effective and trustworthy human-centered SRLA systems for lifelong learners.

Figures

Figure 1.

Theory-driven participatory design with multistakeholders (TPDM) for integrating learner needs in establishing design principles for self-regulated learning analytics (SRLA)

Table 1.

Lifelong learner participant demographics

Variable	f	%
Gender
Male	22	81.48
Female	5	18.52
Age
20–25	15	55.56
25–30	10	37.04
>30	2	7.41
Highest educational degree
Bachelor	25	92.59
Master	2	7.41
Employment status
Employee	14	51.85
Unemployed	12	44.44
Student	1	3.70

Source: By authors

Table 2.

Lifelong learner voices on self-regulated learning challenges and needs

ID	Challenges and needs	Desirable features	Desirable subfeatures
Forethought phase: task analysis and self-motivation beliefs
C1	Challenges in setting goals and planning. Need for having a comprehensive and engaging visualization dashboard	Understanding the task, setting the goal and planning	Information and visualization of the: 1) general learning path and end goal; 2) task instruction and criteria
C2	Challenges in exploring their motivational factors to set motivating goals and plans. Need for analytics and personalized recommendations based on own motivational factors	Personalized recommendation based on the motivational factors analytics	1) Learning interest; 2) initial level of capabilities; 3) outcome expectancies and goal orientation for future career
Performance phase: self-control and self-observation
C3	Challenges in time management and task prioritization. Need for information dashboard, prompts and reminders	Task and time management	1) Information for time estimation, schedule and prioritization; 2) prompts to break down the goals; 3) reminder notifications for the target, task and deadline
C4	Challenges to easily record and track the learning activities and progress to motivate performance. Need for various data to be recorded, tracked and feasible	Real-time monitoring	Recording and tracking: 1) learning activities; 2) progress of learning goals; 3) enable to observe peer’s learning activities progress
C5	Challenges in getting the required social support. Need for recommendations on whom and how to access them anytime they need, and support for psychological safety	Help-seeking	1) Recommendation and access for connecting to resourceful experts or peers; 2) function to support psychological safety
C6	Challenges in regulating emotion and motivation. Need for analytics features for gamification and personalized motivational notifications	Stimulating interests and excitement during learning	1) Gamification elements; 2) personalized notification for motivational messages
C7	Challenges in constructing new knowledge. Need for prompts and reminders to rehearse information and stimulate self-learning	Strategy to rehearse knowledge	1) Prompt and direct feedback to rehearse information; 2) stimulate self-learning; 3) reminders to review and revisit materials
Self-reflection phase: self-judgment and self-reaction
C8	Challenges in self-evaluating objectively and comprehensively to keep being adaptive. Need for analytics dashboards and improvement recommendations based on multiple standards	Self-evaluation analytics based on various standards and recommendations for adaptive improvements	1) Certain targeted output criteria; 2) group performance without competition aspects; 3) own standards or track record; 4) review from the experts; 5) review from peers or other references; 6) recommendation on future improvement
Ethics and data preferences
C9	Concern about sharing their personal data. Need for information on data privacy, transparency and security	Trustworthy information	1) Data privacy; 2) data transparency; 3) data security

Source: By authors

Table 3.

Design principles for self-regulated learning analytics (SRLA) for lifelong learners

#	Design principle	Specifications of design principle for SRLA
DP1	Principle of personalized monitoring	Should generate personalized information and visualization for learners to self-monitor and rehearse their learning process toward the planned goals
DP2	Principle of persuasive motivation	Should provide personalized features to stimulate interest and excitement to boost their motivation and engagement in learning
DP3	Principle of critical reflection	Should deliver formative and summative feedback based on different standards and from different stakeholders for adaptive improvements
DP4	Principle of social support	Should recommend and provide access to social and psychological support
DP5	Principle of ethics and privacy	Should provide anonymity and transparency for personal data and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access

Source: By authors