A practical application of statistical process control to evaluate the performance rate of academic programmes: implications and suggestions

Abstract

Purpose

This study aims to properly and objectively assess the students’ study progress in bachelor programmes by applying statistical process control (SPC). Specifically, the authors focused their analysis on the variation in performance rates in business studies courses taught at a Spanish University.

Design/methodology/approach

A qualitative methodology was used, using an action-based case study developed in a public university. Previous research and theoretical issues related to quality indicators of the training programmes were discussed, followed by the application of SPC to assess these outputs.

Findings

The evaluation of the performance rate of the courses that comprised the training programs through the SPC revealed significant differences with respect to the evaluations obtained through traditional evaluation procedures. Similarly, the results show differences in the control parameters (central line and control interval), depending on the adopted approach (by programmes, by academic year and by department).

Research limitations/implications

This study has inherent limitations linked to both the methodology and selection of data sources.

Practical implications

The SPC approach provides a framework to properly and objectively assess the quality indicators involved in quality assurance processes in higher education.

Originality/value

This paper contributes to the discourse on the importance of a robust and effective assessment of quality indicators of the academic curriculum in the higher education context through the application of quality control tools such as SPC.

Citation

Gessa, A., Marin, E. and Sancha, P. (2022), "A practical application of statistical process control to evaluate the performance rate of academic programmes: implications and suggestions", Quality Assurance in Education, Vol. 30 No. 4, pp. 571-588. https://doi.org/10.1108/QAE-03-2022-0065

Introduction

The need to advance the modernisation of universities is a key element in developing a knowledge-based society and a more competitive economy. In this context, it is not surprising that quality assurance, has become one of the cornerstones for policy and decision makers in higher education institutions in the European Higher Education Area (EHEA) (Curaj et al., 2015).

Quality assurance in higher education has two primary functions. First, it establishes the legitimacy of an institution and the academic programmes it offers. Second, it informs the institutions’ stakeholders about program objectives and outcomes and the fulfilment of the expected quality standards (Kinser, 2014).

In the European Union, national accreditation and quality assurance agencies play a key role, as they are in charge of quality management through evaluation, certification and accreditation of programmes, professors and universities. They design different programmes to guarantee internal and external quality, following the Standards and Guidelines for Quality Assurance in the EHEA (ESG), adopted by the European Association for Quality Assurance in Higher Education (ENQA) (2015).

Under ENQA’s umbrella, universities apply procedures that facilitate both the improvement of the quality of their degrees and the external evaluation processes conducted by the corresponding competent institutions. In this sense, the indicators used for the evaluation of the academic curriculum performance constitute a key element for accountability and transparency.

Despite the experience accumulated over more than a decade in university degree programme evaluation processes, criticisms of approaches to definition and to operationalisation is recurrent in the literature (Hanna et al., 2012; Strang et al., 2016). The main concerns across agencies relate, among others, to the consolidation of good practices and appropriate statistical principles when evaluating and analysing performance indicators.

The current use of a general heuristic to assess academic programmes outcomes where the motto seems to be the higher the ratings are, the better (Tomas and Kelo, 2020) has led to a misinterpretation of the outcomes of educational process measured by indicators. There are four barriers that have made this traditional approach both unreliable and inaccurate when it is used to analyse the variability of indicators such as the performance rate:

1. subjectivity about what should be considered excellent, acceptable, or insufficient prevails in the analysis;

2. the lack of approaches that consider the context of universities, faculties, departments and even courses;

3. the ignorance of uncontrollable factors that underline the inherent variability in the educational processes; and

4. more importantly – the extended practice of comparing performance with averages or the use of arbitrary cut-off numbers without taking into consideration that educational processes have an inherent variability (Bi, 2018).

An alternative approach to overcome these shortcomings and remove the barriers is the application of statistical process control (SPC), something which has been widely and successfully used for decades in the manufacturing industry, and later more slowly introduced in the service sector, in general and in the educative sector in particular (MacCarthy and Wasusri, 2002; Sulek, 2004; Suman and Prajapati, 2018; Utley and May, 2009; Yang et al., 2012). Contrary to the classical statistical methods that are developed for fixed populations and not for processes, SPC can be very effective in detecting process shifts and the dynamics of the process itself. Moreover, it enables hidden problems to be revealed therefore indicating the actions necessary for continuous improvement.

In any process, a certain amount of inherent or natural variability will always exist due to the cumulative effect of unavoidable causes. These sources of variability are called “chance causes” (Montgomery, 2009). SPC helps to assess the variability of the educational processes, distinguishing between assignable (inappropriate educational resources and methodology; ineffective curriculum, etc). and random (student profile, family context, etc). causes (Daneshmandi et al., 2020; Nikolaidis and Dimitriadis, 2014).

This work aims to contribute to understanding the usefulness of the application of SPC, in the processes of accreditation and monitoring of university degrees, through the analysis of the variability of the performance rate associated with the procedures that constitute the quality assurance system (QAS). Specifically, the authors focussed their analysis on the variation in performance rates in business studies taught at a Spanish University. Based on this goal, the following research question is suggested: Are there any significant differences between the results obtained with a standard statistical analysis and the results obtained through the SPC application?

For this purpose, the next section outlines the theoretical framework of this study. Subsequently, the paper presents the methodology and results. The last section discusses the conclusions.

Theoretical framework

Quality assurance in Spanish higher education systems

Quality assurance has become one of the key issues for higher education systems (HES) around the world. The changes that have arisen in the European education area have forced university institutions to adopt new management models, with a priority being to guarantee the quality of the studies, as a contributing factor to the development of the economy and society [Red Iberoamericana para la Acreditación de la Calidad de la Educación Superior (RIACES), 2007]. The comparability and recognition of degrees from the perspective of quality management, strengthened by evaluation and assurance mechanisms and quality of higher education qualifications and certifications, has undoubtedly made the EHEA possible. The key instrument to make it possible was the ESG [European Association for Quality Assurance in Higher Education (ENQA), 2015]. Specifically:

[…] they set a common framework for quality assurance systems for learning and teaching at European, national and institutional level; they enable the assurance and improvement of the quality of higher education in the EHEA; they support mutual trust, thus facilitating recognition and mobility within and across national borders; and they provide information on quality assurance in the EHEA [European Association for Quality Assurance in Higher Education (ENQA), 2015, p. 7].

Under these purposes, the ESG are grounded in four principles for quality assurance:

1. “Higher education institutions have primary responsibility for the quality of their provision and its assurance;

2. quality assurance responds to the diversity of HES, institutions, programmes and students;

3. quality assurance supports the development of a quality of culture; and

4. quality assurance takes into account the needs and expectations of students, and other stakeholders and society” [European Association for Quality Assurance in Higher Education (ENQA), 2015, p. 8].

European agencies vary in their approaches to the implementation and adaptation of the ESG (Alzafari and Ursin, 2019; Kohoutek et al., 2018; Nascimbeni, 2015; Manatos and Huisman, 2020). In Spain, the National Agency for Quality Assessment and Accreditation (ANECA) has developed a set of programmes to conduct its activities of evaluation, certification and accreditation of teaching staff, universities and degrees (Table 1). These programmes constitute a framework that makes it easier for universities to implement their internal QAS as well as the external evaluation processes conducted by the responsible regional agencies and universities.

The MONITOR programme instrumentalizes the operational procedures of the Protocol for the Monitoring and Renewal of the Accreditation for Official University Qualifications University Commission for the Regulation of Monitoring and Accreditation (UCRMA) (2010), designed to supervise the implementation of previously established official qualifications until the moment when it is re-assessed for the purpose of renewing and confirming its suitability (accreditation) once again. The protocol includes the obligation to report information about various processes carried out within the universities. The set of indicators proposed to report the outcomes of the qualification includes the following:

• performance rate in the qualification;

• dropout rate;

• efficiency rate; and

• graduation rate.

The reports also include global data regarding the qualification and an analysis of the adequacy of the evolution of the indicators and their consistency with the target established in the qualification verification report.

The usual practice of universities and evaluation agencies when analysing these indicators is limited to the application of conventional descriptive statistics methods that involves the analysis of variation through measures such as mean, standard deviation, median etc. As a result, the analysis seems subjective, decontextualized and unsound, as it focusses on the evaluation of the outcomes of the educational processes and their comparison with outdated fixed reference values (Andreani et al., 2020; Klasik and Hutt, 2019). Therefore, this practice does not enable us to identify whether the process is under control and if it is not, to identify if this status is due to causes attributable to the processes themselves or to random causes, depending, in turn, on factors such as the branch of study, the type of centre, the university and the region (Bi, 2018; Hanna et al., 2012; Kember and Leung, 2011). An alternative practice to overcome this limitation could be the application of SPC charts, which have become a useful tool with which to achieve stability and improvement in the quality of educational service delivery by monitoring certain variables or attributes over time, such as the previously mentioned performance indicators. The application of SPC in an educational context is already a reality, and it is increasing, as shown in the next section. This will allow the progressive implementation of typical practices of quality management in the field of education, within the general framework of the continuous improvement of processes.

Statistical control of processes in education services

SPC allows the identification of different sources of variation and also enables detection of an “out of control” status (Besterfield, 1995; Shewhart, 1936). A control chart “shows the value of the quality characteristics of interest as a function of time or sample number” (Montgomery, 2009). Thus, the variability of a quality characteristic should be based on output, which involves estimating its statistical distribution and parameters (Juran and Gryna, 1988).

The standard control chart shows a central line (CL), which is the mean of the variable being monitored, the upper control limit (UCL) and the lower control limit (LCL), which are typically set at  ± 3 standard deviations of the CL and its graphic representation is shown in Figure 1.

When a plot violates a control limit, which is when some of the plot points fall outside the three-sigma limits (Zone B), it should be treated as a special cause of variation, attributable to the system. If special causes are present, it is said that the process is out of control. The zone between the UCL and the LCL (Zone A) shows the expected normal (common cause) plot point variation. Unlike special causes of variability, common causes are also called “natural” or “random variability” and are due to a large number of small sources of variation that are not easily identifiable. If assignable or special causes of variation are removed, characteristic parameters such as the mean, standard deviation and probability distribution are constant and process behaviour is predictable; the system is said to be “in a state of statistical control” or simply “under control”. However, certain abnormal patterns (trends, sudden shifts, systematic variation, cycles and mixtures) may alert one to the existence of special causes.

There are a great variety of quality control charts, depending on the type of quality characteristic to control (variable or attributes) and the number of variables to control (univariate or multivariate) (Montgomery, 2009).

The industrial concepts of quality control have been refined and adapted for quality controlling and monitoring the educational service. The literature review section of previous applications of SPC in the educational field focusses on the main objective of each of the reported studies. From the analysis of the studies, four principal objectives have been identified:

1. controlling student achievement;

2. monitoring the effectiveness of the teaching–learning process;

3. evaluating student satisfaction; and

4. identification abnormal patterns in certain educational processes.

Statistical process control charts to control students’ achievements

Objective and quantitative measures such as examination scores or grade point averages (GPAs) to monitor learning performance have been used to conduct numerous studies, under different approaches and educational levels. Thus, for example, Schafer et al. (2011), in their work, used traditional Shewhart X-R charts to follow the performance of primary and secondary students in large-scale assessment programmes. In a similar way, Bakir and McNeal (2010) and Bakir et al. (2015) designed a non-parametric control chart based on the sign statistic to detect statistically significant shifts in students’ GPAs from a desired level. Milosavljevic et al. (2018) used attribute charts from the perspective of the number of passing exams. Other studies following a similar framework include Peterson (2015), Zolkepley et al. (2018), Hrynkevych (2017), Aldowaisan and Allahverdi (2016), Mazumder (2014), Djauhari et al. (2017), Cervetti et al. (2012) and Hanna et al. (2012).

SPC charts have also been applied to monitor the consistency of scale scores or ratings over time. Lee and von Davier (2013) used cumulative sum charts (CUSUM) in a study conducted with data from 72 countries to detect changes in a measurement process of rating performance items in operational assessments. Omar (2010) proposed X-S charts to monitor the consistency of the scores in a co-assessment process. SPC charts that have also been applied in similar frameworks are Beshah (2012), Edwards et al. (2007) and Savic (2006).

Monitoring the effectiveness of the teaching–learning process

Another line of research aims to measure both teachers’ contributions to increasing student knowledge and student’s learning outcomes. The technique usually applied for this purpose consists of administering to students, prior to a lecture, a test known as the Background Knowledge Probe (BKP), which contains questions about concepts covered during the lecture, and after the lecture students have to answer the same questions. Rather than grading the students’ outcomes, the BKP should be understood as a resource to measure student’s gain. This would enable an improvement in the teaching programmes by virtue of detecting the student’s mistakes and gaps in knowledge transfer. Thus, Green et al. (2012) used traditional mean graphics (Graph X), whereas Agrawal and Khan (2008), Grygoryev and Karapetrovic (2005a, 2005b), Karapetrovic and Rajamani (1998) and Pierre and Mathios (1995) used graphics of non-conformity attributes (p graphics).

Evaluating students’ satisfaction

SPC chart techniques have also been used to monitor and evaluate the level of student satisfaction regarding the quality of faculty members’ teaching and university service operation. Thus, Jensen and Markland (1996) published one of the first studies in this field, reporting the use of a Hotelling’s T multivariate control chart to detect shifts in satisfactory and unsatisfactory perceptions of computer services at a large university institution. Debnath and Shankar (2014) proposed the use of attribute control charts (c-charts and u-charts) to evaluate the level of student satisfaction with their academic process. In their study, students were asked to provide information about various parameters, such as the grievance-handling process with respect to students’ admissions and results, facilities, practical orientation process.

On the other hand, it is worth mentioning works that, under the controversial field of Student Evaluation of Instruction, are addressed from the perspective of SPC, bringing a new approach that focusses not on educational outputs and outcomes, but on the quality of the underlying educational processes. Bi (2018) and Nikolaidis and Dimitriadis (2014) used X-S charts; Cadden et al. (2008) and Marks and O'Connel (2003) used X charts; Manguad (2006), proposed the use of X-R charts; Ding et al. (2006) and Carlucci et al. (2019) applied attributes control chart (p and u charts, respectively). Finally, Sivena and Nikolaidis (2019) through simulation, evaluate several popular types of control charts, identifying the most suitable among them.

Identification of abnormal patterns in certain educational processes

Another segment of this body of research has looked at the use of CUSUM charts to detect known items and outliers in computer adaptive testing (Meijer, 2002; Veerkamp and Glas, 2000). This instrument allows the level of acquisition of different skills (linguistic, professional accreditations, etc). to be detected in the context of item response theory.

Given the aforementioned considerations, three facts have motivated this research. First, the existence of inadequate statistical approaches that university institutions have traditionally used to assess certain outcomes of educational processes. Second, the potential of SPC to assess different quality characteristics for higher education institutions has been demonstrated in the literature review. Third, the authors of this paper are active members of the Board for the Monitoring and Accreditation of Qualifications at a business faculty at a Spanish University. Moreover, some of the authors teach courses related to quality management. They share the aforementioned concerns and for the sake of “practice as they preach”, they advocate that some successfully industrial quality management techniques such as SPC which is occasionally taught in their courses, should be incorporated to assess and analyse the variation related to educative performance indicators. This application would improve the assessment of the quality of educative processes so that root causes can be detected and corrective actions can be taken by those who supervise instruction Hanna et al. (2012).

Materials and method

Research context

The study was conducted at the Faculty of Business Sciences and Tourism (FBST) of the University of Huelva (Spain). The FBST offers three bachelor’s programmes: Bachelor in Business Management (BM), Bachelor in Finance and Accounting (F&A) and Bachelor in Tourism. Table 2 shows the main characteristics of the study context.

Design of the study

The method adopted for this study is the action-based case study approach conducted at FBST. Previous research and theoretical issues related to SPC implementation and quality indicators of the training programmes proposed in Protocol for the Monitoring and Renewal of the Accreditation for Official University Qualifications were discussed. Then the authors built a conceptual framework that provided a step-by-step approach to SPC implementation as illustrated in Figure 2.

In this first approach applying the SPC to assess the variability of progress indicators, we only focus on the performance rate.

Applied at course level and by academic year, the performance is indicated by relationship in the total number of ordinary credits passed by the students in a particular academic year and the total number of ordinary credits in which they are enrolled [University Commission for the Regulation of Monitoring and Accreditation (UCRMA), 2010].

This rate provides a snapshot of the proportion of students passing a course in an academic year. Analysis of this rate, with the help of SPC charts, helps to identify root causes such as unmotivated students to further take actions such as monitoring the enrolment process.

For this purpose, the performance rates of the courses taught at the FBST of the University of Huelva between the years 2014 and 2020 were collected from reports on the QAS of the FBST.

To avoid skewing the results, the last academic year of the degrees was excluded from the study due to the type of courses taught. It included elective courses and credits given to students for the completion of curricular internships and the final degree project.

Applying statistical process control

To apply SPC, we followed a standard set of guidelines for setting up an SPC charting scheme, illustrated in Figure 2.

As our main goal was to monitor the effectiveness of undergraduate academic programmes at faculty level, we chose an X-R chart and the quality characteristic to control was the performance rate of the courses in a given academic year for each programme taught at the FBST As already mentioned, to draw an X-R chart, it was necessary to calculate the CL, UCL and LCL (Table 3).

To achieve the proposed goal, we used Minitab 17 statistical software.

Results and discussion

Bearing in mind the inherent variability of the educational processes, we are able to identify: first, courses that fall below the LCL underperform (Zone B, Figure 1), i.e. the percentage of students who passed the course was worryingly low. Second, courses that fall above the UCL (Zone B, Figure 1) indicate that the percentage of students who passed the course was extraordinarily high. Finally, courses that fall within the three-sigma limits had more stable performances, meaning that only common causes explain the variability presented between the control limits (Zone A, Figure 1).

In this work, several sets of X-R graphs are presented in which the parameters were calculated by programme, by academic year in which the courses were taught and by department. They were completed with a comparative analysis.

Control by academic programme

In Figure 3, we provide the results of X-bar charts that plot the variability of the performance rate of the courses for the years 2014–2020, for each of the three programmes analysed, considering the courses within the programme grouped by academic year.

The graphs show similar behaviours for the BM and F&A programmes, with a similar under control performance average (60.54% and 59.49%, respectively). In addition, by academic year, the courses with a low performance rate, e.g. below the lower control limits, were concentrated in the first year and those with a high performance rate, e.g. above the LCS, were concentrated in the third year. The performance rates of the second-year courses are considered to be normal, as they were located within the control interval zone.

The performance rate of the courses within the Tourism Degree is 79.93%, which is substantially higher than the BM and F&A ratio. It is worth noting that the distance to the LCL of the out-of-control courses is much greater for the Degree in Tourism than for the BM and the F&A.

Rather than a particular high or low value for the performance rate, the wide range in the proportions observed between degrees and as shown in the graphs, evidences that while F&A show more stability in the proportion of students passing the courses, a worrying lack of stability is a common factor for B&M and Tourism.

This suggests that higher ratings are not necessarily better, and for this reason it is very important to contextualize the analysis. Because the correct application of X chart formulas requires that all performance ratings contribute to determining the three-sigma limits, the previous analysis lacks subjectivity when it categorizes performance ratings into three groups: high, normal and low.

Surprisingly, for the BM and the F&A, the means of the performance ratios calculated using the traditional approach are higher than those estimated applying SPC: 64.67 > 60.54 and 65.14 > 59.49, respectively. By contrast, the traditional mean of the Degree in Tourism (71.39), is lower than the mean calculated under statistic control (75.93) (Table 2).

The differences pointed out above highlight the need to review the procedures for evaluating the results of the degree study programs, promoting methods such as SPC that allow a contextualized, robust and unbiased analysis.

However, it is important to bear in mind that the analysis could change if control chart parameters were calculated using the programme mean, the department mean or the programme year mean. Thus, to complement the SPC study, in the following sections we present the results according to academic year and department.

Control by academic year

Figure 4 shows the results of SPC application conducted independently for each of the academic years that make up each degree, and not in an aggregated form approach as in the previous section.

The results show differences in the control parameters (CL and control interval), depending on the programme itself and the academic year in which the courses were taught. As an example, the central values for the first year and the second year of BM and F&A were very similar (50.55% and 51.84%; 58.81% and 57.98%, respectively). However, these values increased significantly for courses taught in the third year and differed considerably between programmes (BM 79.19%, F&A 67.72%) (Figure 4). Notably, the three programmes present an overall upward trend in the under control performance rate associated with the sequence of the courses within a structured curricular of the program and the wide range in the academic year to year performance ratio observed (Table 4). This finding is consistent with the results of Hanna et al. (2012) who found a similar trend in a study of the proportion of students passing the class analysed by course and using p-charts. Taken together, these findings would suggest the need to identify the root causes of this trend such as the necessity to revise admission standards. Regarding possible reasons behind the high performance ratios values of the courses located in the second and third academic year of the program we suggest three possibilities. First, it is possible that the faculty and the administration were aware of the presence of special reasons behind the variation and responded by improving teaching strategies. Second, it is also possible that those students lacking motivation and ability were filtered out and did not pass to the subsequent academic year, thereby increasing student performance for higher courses. Third, a low performance rate in the first academic year coupled with a high dropout rate would have had the effect of lowering the standards for the high level courses.

Under this independent analysis, the results also show that the courses taught in the second year of BM and F&A were under statistical control as they were in the aggregate analysis, but with a lower average performance rate (58.81 and 57.98% versus 60.54 and 59.49% of BM and F&A, respectively). Surprisingly, for the courses that were out of control in BM and F&A, those of the first year were above the UCL limit, whereas for the third year, they were located below the LCL in clear contrast to what occurred in the aggregate analysis of the programme described in the previous section. In the case of the tourism degree, the courses outside the control interval were mainly concentrated in the last year, unlike the results obtained in the joint analysis, which were concentrated in the other years. In general, a progressive increase in the value of the CL (under control) of the performance rate was observed, to the extent that the courses were taught in higher academic years. This year-by-year contextualized analysis applying SPC demonstrates the usefulness of this approach by revealing hidden problems that affect variability, such as the profile of students who enrol in a degree for the first time or how students adapt to their university studies over time.

Finally, the use of the three-sigma limits also shows important differences with the results of the traditional evaluation (mean of the academic program) (Table 4). This reinforces the argument previously made in the analysis by academic programs with SPC.

Control by department

Bearing in mind that the courses taught in the different degrees were assigned to different departments, in this section, we present the CL for the courses, grouping them by departments with the highest teaching load for each degree. Table 5 shows the results.

These new parameters imply changes with respect to those obtained in the previous applications. For example, courses which in the control graph built by the programme were under control, were out of control with this new parameterisation and vice versa. Figure 5 shows differences of under control means (CL) between the two departments with the highest teaching load allocation for each degree.

As a summary of the results, and for comparative purposes, Figure 6 shows the performance under control of the courses, grouped by program, academic year and department, comparing them in turn with the mean performance rate, which was precisely the one traditionally used to monitor the performance rate.

Conclusions

The results of this study show significant differences in the analysis of the variability of the performance rate between the scenarios described (degree, year and department), depending on the reference parameters (CL, UCL, LCL), focussing the analyses on the process itself rather than the result.

Similarly, in the analysis of the variability of the processes, the SPC approach does not only allow hidden problems to be revealed and their causes to be determined so that corrective actions can be taken to reduce oscillations and therefore improve the stability of the educational processes. Once SPC has been incorporated as an assessment tool to measure the variation relating to academic performance ratios, better information would be available to establish more suitable target goals.

The main conclusion of this study is that the traditional analysis which compares performance ratings with means of the degrees or arbitrary cut-off numbers overlooks the inherent variability of the educative processes. In addition to being useless for comparative purposes, this traditional approach lacks the objectivity and robustness necessary for its application in decision-making, in the accreditation and monitoring processes in which this work is framed.

By contrast, SPC allows a contextualized, robust and unbiased analysis of the variability of quality indicators involved in the accreditation and monitoring processes, providing valuable information for decision making to administrators, teachers and other stakeholders in HE.

This study is a first approach to SPC application for monitoring the control indicators proposed in the ESG [European Association for Quality Assurance in Higher Education (ENQA), 2015]. However, the results are focussed on the analysis of just three programmes in a Spanish University, and thus should be interpreted with caution. In future studies, we will explore the application of multivariable charts and capability analysis, the inclusion of other programmes and time horizons, thereby widening the scope of this study.

Figures

Figure 1.

Conventional control chart for monitoring the variability of a process

Figure 2.

Application of the SPC in the scope of the study

Figure 3.

Control results by academic programmes (2014–2020)

Figure 4.

Control results by academic year

Figure 5.

Results by departments (CL)

Figure 6.

Comparative analysis related to performance rate

Table 1.

Evaluation programmes of Spanish University

Evaluation activities	Programmes
Academic staff	Teacher evaluation program: evaluates the CVs of applicants to access non-civil servant academic staff bodies
	ACADEMIA: evaluates CVs of applicants to access civil-servant academic staff bodies
	CNEAI: The CNEAI is the body within ANECA responsible for the evaluation of research activity for the purposes of assigning the corresponding retribution complements, as per applicable regulations
Programme	VERIFICA: evaluates degree proposals designed according to EHEA criteria
	MONITOR: follows-up an ex-ante accredited programme to check its correct implementation and results
	ACREDITA: checks that the degree has been carried out according to the initial project
	SIC: Assessment for quality International labels
Institution	AUDIT: provides guidance for HEIs to establish their own internal quality assurance systems
	AUDIT INTERNATIONAL: certify quality assurance systems for Higher Education Institutions (HEI) located in third countries
	INSTITUTIONAL ACCREDITATION: evaluates applications for institutional accreditation from university centres
	DOCENTIA: supports Universities to create mechanisms to evaluate academic staff quality

Source: ANECA (www.aneca.es/Programas-de-evaluacion)

Table 2.

Bachelor’s programmes offered by Faculty of Business Sciences and Tourism, University of Huelva

Bachelor’s programs	Courses		No. of credits European credit transfer system	Students enrolled (no) 2014-2020		Performance rate (mean) (%)
	Compulsory	Elective		Mean	SD	2014-2020	Departments
Business management	35	14	240	464	88.88	64.67	– Management and marketing – Financial economics and accounting
Finances and accounting	35	17	240	201	75.18	65.14	– Economy – Law – Geography and history*
Tourism	37	28	240	195	23.30	71.39	– Languages* (German, English and French)

Note:

ECTS = European credit transfer system

Table 3.

X-R chart formulas

	“x” Bar chart	“R” Chart
Central Line (CL)	X¯¯=∑i=1kx¯ik	R¯=∑i=1kRik
Upper control limit (UCL)	LCSX¯=X¯¯+A2⋅R¯	LCSR=D4⋅R¯
Lower control line (LCL)	LCIX¯=X¯−A2⋅R¯	LCIR=D3⋅R¯

Notes:

n = sample size “i”, k = number of samples; X¯= mean of the sample “i”, R_i = rank of the sample “I”, X¯¯ = grand mean of the k samples; R¯ = grand rank of the k samples: A2, D4, D3 are constant values determined by the sample size

Source: Adapted from Shewhart (1936)

Table 4.

Control results by academic year versus traditional assessment

Academic year	Business management	Finance and accounting	Tourism
First	50.55	51.84	60.1
Second	58.81	57.98	77.42
Third	79.19	67.72	80.99
Program (mean)	64.67	65.14	71.39

Table 5.

Control parameters by departments

	Bachelor’s programmes
	Business management			Finance and accounting			Tourism
Department	CL	UCL	LCL	CL	UCL	LCL	CL	UCL	LCL
Management and marketing	76.09	87	65.06	60.42	72.67	48.17	51.84	74.72	28.95
Financial economics and accounting	57.37	74.3	40.45	63.31	78.43	48.18	76.7	86.21	67.2
Economy	59.95	77.78	42.12	–	–	–	66.45	77.11	55.79
Languages	–	–	–	–	–	–	76.5	87.39	66.5