OER impact study perceived by faculty within ODL framework

Abstract

Purpose

The purpose of this paper is, first, to scrutinize the determinants of key benefits of open educational resources (OER) to faculty. Second, it is to expose how, in which routines the variables involved, are interrelated.

Design/methodology/approach

An exploratory design is used in this study. Qualitatively, key benefits include integration, opportunity, efficiency, enrichment, and collaboration. These benefits have direct impacts on enhancing student learning, augmenting teaching practice, improving productivity, catalyzing changes in teaching practice, and supporting non-traditional learners. Quantitatively, the key benefit is moderating the variables. Integration, opportunity, efficiency, enrichment, and collaboration are independent variables. Variables like enhancing student learning, enriching teaching practice, improving productivity, catalyzing changes, and supporting non-traditional learners are the dependent variables. The study population comprised the 721 Universitas Terbuka (UT) faculty members. The respondents were chosen randomly by distributing 450 questionnaires. Only 203 questionnaires were completed. Importance performance analysis and customer satisfaction index (IPA-CSI) were used to measure the importance level of variables involved and their benefits. Structural equation model (SEM) was used to examine the ten hypotheses developed so that the author could understand the significance level and relations power among variables engaged with reference to the qualitative outcomes previously obtained.

Findings

Six hypotheses were validated by the analysis. Statistically, efficiency and integration affect key benefits. Likewise, moderating variables affect teaching practice enhancement, productivity improvement, catalyzing changes, and supporting non-traditional learners. Conversely, key benefits were neither interrelated by opportunity, enrichment, and collaboration nor learning enhancement.

Practical implications

This study highlighted that adoption, integration, and implementation of OER in the UT milieu do take place.

Originality/value

This study recognized the variation of qualitative vs quantitative outcomes. An auxiliary inquiry is needed with broader perspective by increasing the respondents sample in order to minimize the difference between qualitative and quantitative results.

Citation

Sembiring, M.G. (2016), "OER impact study perceived by faculty within ODL framework", Asian Association of Open Universities Journal, Vol. 11 No. 1, pp. 78-89. https://doi.org/10.1108/AAOUJ-07-2016-0021

Introduction

Universities play a vital role in the creation, analysis, and spreading of knowledge and other academic entities. Advances in Information Communication Technology (ICT), the rise of the internet, and large-scale digitization of information create openings to transform how teaching and learning are developed, how knowledge and information are generated and distributed, as well as how interactions amongst students, staff, and institutions are facilitated (de Hart, 2014). “Open” means sharing and generally refers to the eradication of obstacles to access. It does not always mean “free of charge”. “Open education” is described as not limited to open educational resources (OER) alone. It draws upon open technologies that might facilitate collaborative, flexible learning, and open sharing of teaching practices that empower educators to benefit from their associates. It may also grow to incorporate new approaches to assessment, accreditation, and collaborative learning.

“OER” are described as teaching, learning, and research resources which reside in the public domain. They have been published under intellectual property license that allows their free use by other people. OER include full courses, course materials, textbooks, audio and video streaming, evaluation, software, tools, or other techniques used to preserve access to knowledge (Aitkens et al., 2007). Universitas Terbuka (UT) Indonesia is also influenced, and even being forced to integrate OER by incorporating them into the operational plans (UT, 2015a, b), as up to 2011 the integration and adoption of OER into academic service was considered to be low and also slow. It is believed, however, that by integrating OER into such services they will then mend the students’ gap in mastering the subjects they undertake. Within these four years, OER in the UT milieu have even gradually become an integral part of service using the ICT-based mode.

OER adoption and integration in the academic community have been given opportunities and benefits to anybody, primarily to the faculty members. It was confirmed that the UK OER impact study, with specific reference to the benefits to institutions and academic as well as to learners, was unquestionable; apart from the minor impediments might still take place. The key benefits elaborated here are related to factors in learning environment, such as pedagogic, attitudinal, logistics, and strategic sphere of influence (Masterman et al., 2011).

This study was initiated in order to explore OER and their benefits to faculty to take up different courses of actions on the use rather than on the production of OER in order to assure students’ performance. Additionally, it aims to distinguish how and in which behaviours were all involved variables intercorrelated with one another. The following were the further queries: what benefits can OER offer to faculty? How are pedagogic, attitudinal, logistics, and strategic factors conducive to sustain practice in the use of OER? Are integration, opportunity, efficiency, enrichment, and collaboration positive factors to those benefits? Are they applicable to Indonesian context through UT tradition?

Previous study and the framework

There are verified key benefits to educators. These are: enabling resources to be seamlessly integrated into students’ environment; addressing students’ needs by providing opportunities for supplementary learning and presenting content in different ways to address preference; it also save teachers’ effort by empowering them to offer materials and activities where they lack skills to create themselves; besides, it is related to benchmarking their own practice in terms of content, approach, and quality; enabling teachers to teach topics outside their expertise; stimulating network among teachers; and finally improving new collaborations in searching common interests (Masterman et al., 2011). At the same time, enabling factors that might be effective to sustain the practice of integrating OER can be viewed from pedagogic, attitudinal, logistics, and strategic aspects.

From a pedagogic angle, there are six attributes which are valid to be viewed. These were partly inspired by Nagashima (2014) and Kawachi (2013). These are: relevance, provenance, pedagogic intent, granularity, media, and topicality. From an attitudinal aspect, there are five attributes that are pertinent to be counted. These are: conceptualization of teaching to support independent learners; acknowledgement that combining materials they have authored from other sources; confidence (subject and teaching) to share their own materials; readiness to learn by expounding professional practice; and a sense of responsibility for inspiring comparable attitudes. From a logistics attitude, there are four attributes that are meaningful to be viewed. These are: volume of resources, technical and implementation issues, discoverability, and lack of licensing. Finally, from a strategic viewpoint, impact on individual practice is most likely achieved within dimension of social practice (networks of like-minded individual who are receptive to suggestions) and ready to share their own resources. Such networks might be fostered through: relying on diffusion of bottom-up initiatives, implementing institution-wide strategy to consistency in OER use, and identifying individuals or small groups that are using OER on their own initiative into a more structured strategy for further diffusion.

OER use can be seen as a trigger for following practices. From a learner’s perspective: implementing open pedagogic model, providing learners with a repertoire of rich or diverse resources that may include reused content in open networks. From a teacher’s standpoint: sharing or collaborating on content and encouraging them to use open content. From a community standpoint: opening up content to distance learners who are not formally enroled in higher education or university courses level; and making knowledge publicly accessible for all. This is consistent with Geser (2007).

The above items that have been elaborated are all related fundamentals which underpin the conceptual framework of this study (Figure 1). The conceptual framework previously described is used as a basis to propose the operational framework of the study later. Having formulated them expansively, it will then be elaborated under a quantitative approach (Hair et al., 2009).

Please refer to Figure 1. Five unified dimensions should be noticed here, namely, the key benefits of OER to faculty. They consisted of integration, opportunity, efficiency, enrichment, and collaboration. These dimensions were determined as a result of literature review and in-depth interviews followed by focus-group discussions with selected faculty members. Group discussions then ensued that the key benefits could be assessed by perceiving related attributes included within those dimensions.

Qualitative approach suggested that integration included attributes on student situation, ways of avoiding external link, and reducing possibility of broken link as the benefits. Opportunity included attributes on supplementary materials, reinforcing skills, and they were all regarded as alternatives of presenting materials. Group discussions insisted (from efficiency view) the benefit included the role of sharing material, benchmark of content, and embedding quality assurance in creating contents. The attributes of being able to teach subject out of current expertise, enlarging horizon, and being able to relate subject were categorized as benefits and enrichment. In terms of collaboration, another dimension of benefit included the ability of appropriating resources, filling unpredictable gaps, and searching for common interests.

The qualitative inquiry also asserted that the benefits had effects on the five factors as viewed from the quantitative perspective. First, let us look at enhancing student learning. The attributes were: as a learning supplement out of classroom and preparation, practice, reinforcement, and revision of required skills. Second, let us look at enhancing teaching practice. The attributes were: enhancing what students were able to do on their own time and visualizing a complex concept and process. Third, let us look impact on productivity. The attributes were: ability to find, evaluate, and contextualize video and animation. Fourth, let us look at catalyst for changes in teaching practice. The attributes were: ability to integrate development and master materials easily as a distance learner. Fifth, let us look at supporting non-traditional learners. The attributes were: improving qualification and nurturing education to anybody, anywhere, and anytime.

Having elaborated all variables engaged related to the dimension exploration explained above, it seems that they are much easier to comprehend by summarizing them all and put in Table I.

Before launching the operational framework, it is worth noting that the key benefits were conceptually determined by five dimensions (X₁-X₅). The key benefit was to enhance student learning, enhance teaching practice, give impact on productivity, catalyze for changes in teaching practice, and support non-traditional learners (Table I).

Methodology and the designs

The next phase is to establish the operational framework of the study (Figure 2).

This is done in accordance with the structure as summarized in Figure 1. We also reflect on all the variables involved as recapped in Table I. This framework is used as a basis to determine the methodology, design, and ways of ensuing analysis that will then be done quantitatively.

This inquiry uses mixed methods; in a more precise explanation it is referred to exploratory design (Creswell and Clark, 2011). It is actually prearranged under qualitative approach first and then followed by quantitative series. Two instruments were developed. They are list of questions for the in-depth interviews and/or focus-group discussions qualitatively and the questionnaire as an instrument to accumulate data under quantitative approach.

Table I and Figure 2 listed the highlights that influencing the key benefits of OER lead to enhance student learning, enhance teaching practice, impact on productivity, catalyze for changes in teaching practice, and support non-traditional learners. The benefits have the following attributes: pedagogic (relevance, granularity, and contemporaneity), attitudinal (conceptualization, confidence, and responsibility), strategic (network, organization, and governance), and logistics (resources, technicality, and discoverability).

The key benefits (Y) were assessed by perceiving attributes of: enhance student learning (Y_5,6), enhance teaching practice (Y_7,8), impact on productivity (Y_9,10,11), catalyst for changes (Y_12,13), and support non-traditional learners (Y_14,15). The instrument for quantitative approach consists of 2×19 questions with Likert Scale 1-5 related to the benefits of OER and its importance level; plus 11 additional questions to validate the independent variable which are related to the moderating variable.

Variables are explored through questionnaire (Tjiptono and Chandra, 2011). A survey was started to collect data from respondents (Fowler, 2014). A sampling was chosen to select respondents for qualitative purpose and simple random sampling was used to determine respondents for quantitative purposes (Cochran, 1977; Sugiyono, 2012). An important performance analysis and customer satisfaction index (IPA-CSI) were emulated with intent to simultaneously measure the beneficial level along with its importance degree (stimulated by Kitcharoen, 2004; Silva and Fernandes, 2010; Wong et al., 2011). The structural equation model (SEM) is finally utilized to detect relations power among variables engaged (Wijayanto, 2008).

This inquiry then scrutinizes ten hypotheses (H1-H10, Figure 2). Explicitly, the key benefits of OER were influenced by: integration (H1), opportunity (H2), efficiency (H3), enrichment (H4), and collaboration (H5). Likewise, enhance student learning (H6), enhance teaching practice (H7), impact on productivity (H8), catalyze for changes (H9), and support non-traditional learners (H10) were all influenced by the key benefits of OER. These ten hypotheses will be examined under SEM approach to validate the relations amongst variables engaged. The validation is intended to examine the significance level of the relations. Having validated the significance relations, it is then utilized to view the power of the relations.

Results and arguments

Before discussing the research result, we want to highlight the characteristics of the respondents (Table II), as this will influence the outcomes. The results of the analyses were later explained in detail after the respondents’ characteristics were presented.

The following were the results. The first result is related to hypothesis analysis. Figure 3 displays the four hypotheses that were not validated by analysis. These were: H2=0.18 (opportunity to key benefits), H4=0.13 (enrichment to key benefits), H5=0.24 (collaboration to key benefits), and H6=0.18 (key benefits to enhance student learning), as the t _values⩽1.96 for α=5 per cent.

The other six hypotheses were positively confirmed by the analysis. These were: H1 =2.12 (integration to key benefits), H3=2.01 (efficiency to key benefits), H7=5.29 (key benefits to enhance teaching practice), H8=11.06 (key benefits to impact on productivity), H9=7.34 (key benefits to catalyst for teaching practice), H10=15.6 (key benefits to support non-traditional learner), as the t _values⩾1.96 for α=5 per cent.

At this stage, it is worth revealing the beneficial level and degree of its importance consciously imitated from IPA-CSI attitudes. The analysis engenders lots of key benefits of OER attributes related to relevant quadrants to comprehend their behaviour as illustrated in Figure 4. Figure 4 obviously has four quadrants (Q): Q₁ (concentrate here), Q₂ (maintain performance), Q₃ (low priority), and Q₄ (possible overkill); following Wong et al. (2011).

Q₁ has no attribute that should be seriously noted here. Q₁ indicates that the benefit is at a low level whereas the degree of its importance is high. This implies that the university has no problem in integrating OER in both the strategic and operational levels. It denotes that faculty has already been aware of the OER movement and it actually gives them some gains. This is a good signal in adopting and implementing OER movement in UT.

Q₂ includes ten points that should be recognized carefully. They are: X11 (enlarging horizon), X5 (reinforce skill), X4 (supplement materials), X13 (able to appropriate resources), X16 (pedagogical), X12 (able to relate subject), X1 (student situation), X7 (shared materials), X6 (alternative presentation), and X14 (filling gaps). This quadrant is a symptom that both the benefit and the degree of its importance were being concurrently placed at a high level. The university must take care of these notions cautiously as they are good examples so that more faculty members can get involved and gain more advantages from this movement. Attributes that fall into this quadrant are the strengths and pillar of adopting and promoting OER in UT. These attributes should become the pride of the university.

Q₃ has seven points which should be studied cautiously. They are: X8 (content benchmarking), X3 (reduce broken link), X15 (searching for common interest), X9 (quality assurance), X10 (teaching out of expertise), X17 (attitudinal), and X18 (strategic). This quadrant indicated that both the benefit and degree of importance are in the low category. The university should classify this notion as the “next” focus after concentrating on critical spots in Q₂. Any attribute falling into this quadrant is actually not so critical and poses no threat as well.

Finally, there are two points that are classified as members of Q₄. They are: X2 (avoid external link) and X19 (logistics). This quadrant indicates that the attributes are considered less important but faculty regarded them as high in benefit. Attention to attributes in this quadrant can be less focused as well. So, the university can save costs by redirecting them to take up vital spots by anticipating no attributes will fall into Q₁ in future and keep maintaining the fundamental spots in Q₂.

Having positioned all attributes described previously, we are then in a position to relate the loading factors to observe the power of relations of each variable involved under SEM to work out the final outcomes (Wijayanto, 2008; Hair et al., 2009), as illustrated in Figure 5.

Figure 5 above displayed the five prime final upshots resulting from the quantitative approach, as follows:

The first effect is related to the dimensions that directly influenced the key benefits. They are: efficiency (X3=0.19) and integration (X1=0.10). The other three dimensions (opportunity, enrichment, collaboration) have no influences.

The second finding is related to the rank of attributes in efficiency (X3). They are: X31 (shared materials, 0.81), X32 (content benchmarking, 0.72), and X33 (quality assurance, 0.30). The rank in integration (X1) is: X11 (student situation, 0.96), X13 (reduce broken link, 0.92), and X12 (avoid external link, 0.89).

The third outcome, on the sequence of attributes in key benefits (Y): Y2 (attitudinal, 0.24), Y1 (pedagogical, 0.19), Y3 (strategic, 0.15), and Y4 (logistics, 0.14).

The fourth effect is associated with the relations power of the moderating variable and the dependent variables. Key benefits (Y) have significant effects, as follows: impact on productivity and support non-traditional learners (1.00), enhance teaching practice (0.91), and catalyst for changes (0.86). This entailed that enhance student learning is not interrelated with the moderating variable.

The fifth corollary is on the ranks within the dimensions of: first, support non-traditional learner (1.00): improving qualification (Y14=0.92) and able to nurture education (Y15=0.83); second, impact on productivity (1.00): contextualize video and/or animation (Y11=0.76), find video and/or animation (Y9=0.73), and evaluate video and/or animation (Y10=0.58); (3) enhance teaching practice (0.91): (i) students are able to do in their own time (Y7=0.83), (ii) visualize complex concept/process (Y8=0.70); and (4) catalyst for changes (0.86): (i) mastering the material easier (Y13=0.88) and (ii) integrate the development (Y12=0.83).

Before validating the conclusion under the mixed methods, we needed to reflect whether or not the SEM result is in the “good fit” category. If so, it is then reliable to assess hypotheses and engender loading factors to confirm its power of interrelations. The analysis in fact confirmed that they were not considered in “good” category (Table III). This implied that the validated model was not highly dependable. Conceptual and/or operational model may have substantial and/or technical differences in theoretical and/or methodological intensity.

Despite the fact that “goodness of fit” is “not in good fit” category, it is still valuable to use it as point of reference. Three core evaluations were needed to explore. The first concerned was the gap obtained using exploratory design. The second was the reason adjacent to the approach used by referring to the respondents’ characteristics; as most of respondents are categorized novice in OER context). The third was about the implication of the findings for further research.

Under the qualitative procedure, the key benefits were interlinked with integration, opportunity, efficiency, enrichment, and collaboration dimensions. Likewise, the moderating variable was interconnected with the independent variable. Only two dimensions of the independent variables (efficiency and integration) were interrelated with the moderating variable. Besides, enhancing student learning and the moderating variable were also not interdependent with one another. This implied that qualitative vs quantitative results varied considerably. Nonetheless, they did not contradict one another.

The exploratory design was conducted by first collecting and analysing data qualitatively. Then a quantitative structure is built prior to interpretation (Creswell and Clark, 2011). It aims at testing and measuring the qualitative aspects of exploratory findings. Before building the operational framework, the conceptual framework should be first established as the model will be statistically scrutinized. The end results showed that four hypotheses were not accomplished in chorus. The order of the dimensions was also in disharmony. The quantitative approach was still unable to prove the qualitative exploratory discoveries.

Referring to Table II; it is obvious that most of the respondents were reasonably qualified. This can be viewed from their background, working experience, age, and qualification. Nonetheless, the vast majority of them have limited involvement and experience in OER. Only 19.69 per cent are involved in OER for five years or more; most of the respondents are novice in OER context. It is then plausible that they might not been able to foresee collaboration, enrichment, and opportunity as pivotal and give benefits to faculty in general. Besides, as the respondents are all academic staff, they might not foresee how this benefit is able to enhance student learning.

Future research work can involve students and experts from other institutions. There must be a balance between the qualitative and quantitative outcomes. Whatever we do, it is important to bear in mind that what we are doing is to prepare, promote, and adopt effective OER for our students’ learning.

Closing interpretation

This study has discovered somewhat significant differences between what was obtained from qualitative routines as compared to quantitative approach. Four out of the ten hypotheses assessed were not validated by the analysis. This implied that the established qualitative frame is imperfectly approved by quantitative analysis.

Another inquiry was needed by enlarging the scope of the study, e.g. by inviting other academic communities from other universities as respondents and/or experts, to make the consequences closer under exploratory design, as part of mixed methods. Using the IPA-CSI procedure, we were able to show up ten vital attributes as the good clues that OER movement in UT is promising.

Figures

Figure 1

The conceptual framework

Figure 2

The operational framework

Figure 3

The t _value of analysis

Figure 4

IPA chart

Figure 5

The loading factor

Table I

Variables and elaborated dimensions

No	Variables	Dimensions	Notes for the questions
1	Integration X1	X11: student situation X12: avoid external link X13: reduce broken link	Each independent variable (X) has three dimensions and three questions that should be answered by the respondents Each question within X is answered two times simultaneously by all respondents; first question is to measure applicability and second is to measure level of its importance The key benefits (Y(1-4)) is the dependent variable upon X (X1-5) While others (Y(1-4)) are determined by the key benefits of OER
2	Opportunity X2	X21: supplement materials X22: reinforce skills X23: alternative presentation
3	Efficiency X3	X31: shared materials X32: content benchmarking X33: quality assurance
4	Enrichment X4	X41: teaching out of expertise X42: enlarging the horizon X43: able to relate subjects
5	Collaboration X5	X51: appropriating resources X52: filling the gaps X53: searching for common interests
6	Key benefits of OER to faculty Y(1-4)	Y1: pedagogic (relevance, granularity, contemporaneity) Y2: attitudinal (conceptualization, confidence, responsibility) Y3: strategic (network, organization, governance) Y4: logistics (resources, technicality, discoverability)
7	Enhance student learning Y(5-6)	Y5: supplementary learning out of the classroom Y6: prepare, practice, reinforce, revise of skills
8	Enhance teaching practice Y(7-8)	Y7: enhance student are able to do in their time Y8: visualize complex concept/process
9	Impact on productivity Y(9-11)	Y9: find video/animation Y10: evaluate video/animation Y11: contextualize video/animation
10	Catalyst for changes in teaching Y(12-13)	Y12: integrated development Y13: mastering materials for ODL academically
11	Support non-traditional learners Y(14-15)	Y14: improving qualification for everybody Y15: the university is able to nurture education for all nations

Table II

Respondents’ characteristics

Population	721		Respondents Selected respondents		203 (28.15%) 9
Faculty/%	Education	47.29	Social Science	16.74	Economics	23.15
	Grad-Programme	0.49	Mathematics and Natural Sciences			12.31
Degree/%	Doctoral	8.86	Master	82.26	Bachelor	8.86
Experience (Year/%)	0-5=1.97	6-10=16.25	11-15=13.79	16-20=14.77	21++=53.20
Age (Year/%)	20-29=0.49	30-39=18.71	40-49=16.25	50-59=49.26	60++=15.27
OER Experience (Year/%)	<1=12.80	1-2=41.87	3-4=25.61	5-6=10.83	7++=8.86
Lecturer/%	Professor 0.49	Senior 26.60	Junior 55.66	Assistant 30	Candidate 3.94

Table III

Goodness of fit

Goodness of Fit	Cut-off Value	Results	Notes
RMSEA – Root Mean Square Error Approximation	⩽0.08	0.088	Marginal fit
RMSR – Root Mean Square Residual	<0.05 or <0.10	0.190	Marginal fit
GFI – Goodness of Fit	⩾0.90	0.790	Poor fit
AGFI – Adjusted Goodness of Fit Index	⩾0.90	0.690	Poor fit
CFI – Comparative Fit Index	⩾0.90	0.850	Marginal fit
NFI – Normal Fit Index	⩾0.95	0.790	Poor fit
IFI – Incremental Fit Index	⩾0.90	0.850	Marginal fit