Abstract
Purpose
Saudi universities have incorporated capstone projects in the final year of an undergraduate study. Although universities are following recommendations of the National Commission for National Commission for Academic Accreditation and Assessment (NCAAA) and Accreditation Board for Engineering and Technology (ABET), no detailed guidelines for management and assessment of capstone projects are provided by these accreditation bodies. Variation in the management and assessment practices of capstone project courses and analysis of the students' capabilities to align with industry demands, to realize Vision 2030, is challenging. This study investigates the current practices for structure definition, management and assessment criteria used for capstone project courses at undergraduate level for information technology (IT) programs at Saudi universities.
Design/methodology/approach
A web-based questionnaire is administered using a web service commonly used for questionnaires and polls to investigate the structure, management and assessment of capstone projects at the undergraduate level offering software engineering, computer science and information technology (SECSIT) programs. In total, 42 faculty members (with range of experience of managing/advising capstone projects from 1 to more than 10 years) from 22 Saudi universities (out of more than 30 universities offering SECSIT undergraduate programs) participated in the study.
Findings
The authors have identified that Saudi universities are facing challenges in the utilized process model, the distribution of work and marks, the knowledge sharing approach and the assessment scheme. To cope with these challenges, the authors recommend the use of an incremental development process, the utilization of a project-driven approach, the development of a national level digital archive and the implementation of homogeneous assessment scheme.
Social implications
To contribute to the national growth and to fulfill the market demand, universities are recommended to align the capstone project courses with latest technology trends. Universities must collaborate with the industry and update the structure and requirements of capstone project courses accordingly. This will further facilitate to bridge the gap between industry and academia and will develop a win–win scenario for all the stakeholders.
Originality/value
Although universities are committed to increase innovative capacities of their students for enabling them to contribute to economic and social growth, it is still hard to know the knowledge creation and sharing at national level. Variations in the management and assessment practices for capstone projects further intensify this challenge. Hence, there is a need of smart assessment and management of software capstone projects being developed in Saudi universities. Incorporating latest technologies, such unified management can facilitate discovering the trends and patterns related to the domain and complexity.
Keywords
Citation
Ahmad, E. and Alammary, A. (2022), "Towards unified management of software capstone projects in Saudi universities: a survey-based study", Arab Gulf Journal of Scientific Research, Vol. 40 No. 2, pp. 118-138. https://doi.org/10.1108/AGJSR-04-2022-0037
Publisher
:Emerald Publishing Limited
Copyright © 2022, Ehsan Ahmad and Ali Alammary
License
Published in Arab Gulf Journal of Scientific Research. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Introduction
To ensure readiness of the students for industry and real-world problem-solving, most engineering programs consider capstone projects as an important part of their curriculum at the undergraduate level. For software engineering, computer science and information technology (SECSIT) programs, students are required to actively participate in software development projects of substantial complexity spanning the entire last year of their degree. IEEE and ACM joint task groups on SECSIT curricula encourage academic institutions to establish capstone project courses for validating students' application of knowledge in practical situations (T. G. on Information Technology Curricula, 2017; Sahami et al., 2014; Ardis et al., 2015). They also recommend accreditation agencies to regard such courses as one of the major criteria to affirm quality assurance of an academic program.
Saudi Arabia's National Commission for Academic Accreditation and Assessment (NCAAA) constitutes quality assurance and accreditation standards for higher education programs and institutes (N. C. for Academic Accreditation Assessment, 2015). Along with providing extensive guidelines for incorporating quality assurance standards, NCAAA also specifies to consider capstone projects as direct assessment methods to measure students' learning outcomes. To comply with international standards, most of the universities also opt for ABET accreditation. ABET computing accreditation commission contemplates capstone projects vital to add real value to an SECSIT academic program. According to the latest version 2.0 of the criteria for accrediting computing programs, “A project requiring integration of knowledge and skills acquired in earlier course work” (C. A. Commission, 2017).
In Saudi Arabia, keeping in view importance of the capstone projects in SECSIT and other engineering programs, such project-based courses are also been exploited as evidences program assessment and evaluation. For example, Bhatti et al. have used capstone projects for direct assessment with quantitative values computed from students' performance with course learning outcomes while exploring assessment process for computer science program accreditation at Taif University (Bhatti & Ahmed, 2015). Aoudia, Marji and AlQahsi (2015) have regarded capstone projects essential for achieving outcome “C” of ABET accreditation for industrial engineering program of Northern Borders University, while the importance of feedback analysis of capstone projects and summer training for achieving program learning outcomes for ABET accreditation at Qassim University is highlighted in Alyahya & Abo El-Nasr (2012).
Structure, management and assessment of capstone projects are often very different from the normal lecture-based courses. To assess the ability to develop a software application with several integrating components to meet the desired needs within realistic economic and timing constraints demands well-defined process guidelines (Koolmanojwong & Boehm, 2009; Thompson & Edwards, 2009). During capstone projects, students are required to produce several artifacts while performing development activities and each of these deliverable is not only considered for individual/group assessment but is also been used as evidence in internal and external reviews for quality assurance. Due to inherent multi-dimensional nature of capstone projects, entanglement of generic skills like problem formulation, critical thinking, etc. and the need for continuous evaluation, management and assessment of capstone projects is widely recognized as a challenging task. Although both NCAAA and ABET recommend the use of well-defined rubrics for capstone project process and product evaluation, no detailed guidelines for management and assessment of capstone projects are provided which further intensifies the challenge of capstone project management and assessment.
This study explores the current practices for structure definition, management and assessment criteria used for capstone project courses at undergraduate level for SECSIT programs at Saudi universities. Thus, to identify areas of improvement, allow unified management of capstone projects and ensure that projects outcomes align with Vision 2030. To meet the study objectives, we have carried out a survey at different universities in Saudi Arabia. Based upon this survey, we have highlighted the commonalities, variations and the potential improvements.
The next section describes the role of the capstone projects in the knowledge-based economy as the background of this study. Section 3 presents a detailed description of the used methodology, while Section 4 presents the results of the online questionnaire. Section 5 specifies the challenges identified by analyzing the result, and our recommendations to the identified challenges are presented in Section 6. Section 7 summarizes this paper with a glimpse of important future works.
2. Background
Transformation of existing educational infrastructure for knowledge creation and distribution is core of the Vision 2030 for modernization of the Kingdom of Saudi Arabia (K. of Saudi Arabia, 2016). This has instigated the emergence of student-centered and technology-driven learning environments. Along with traditional lecture-based face-to-face learning paradigm, universities have already started incorporating information and communication technologies (ICTs) in traditional pedagogical practices to blend their courses for promoting online learning and synchronous and asynchronous discussions. Universities like King Saud University, King Khalid University, King Fahd University of Petroleum and Minerals, and Umm Al Qura University have made great efforts to apply blended learning approaches to several courses. Taking a step further, Saudi Electronic University is offering blending learning for all the programs at undergraduate and postgraduate levels. This indicates commitment of the academic sector to realize Vision 2030.
According to Saudi Ministry of Education (MoE), there are more than 30 universities offering undergraduate programs in SECSIT. Instructor-led traditional teaching methodology, with instructor deciding and controlling the contents and the flow of the course, forms the main teaching paradigm in most of universities. A bibliometric study on software engineering in Saudi Arabia indicates a healthy growth in knowledge, research and international collaboration in the SECSIT domain (Almaliki, 2021). With 0.62% of the estimated international knowledge production, Saudi Arabia is way ahead of the neighboring countries UAE, Jordan and Egypt. Several studies like Alawairdhi (2016), Al-Yahya et al. (2021) have discussed the software development methodologies, students' perception about the teaching and assessment of these methodologies, and their effects on the digital economy and the job market.
Following the guidelines provided by NCAAA and ABET for program accreditation, it is a standard practice in Saudi higher education institutes to include a capstone project in the final year of their study. Although NCAAA and ABET have provided comprehensive guidelines for curriculum development, based on their learning paradigms, different universities implement courses in variety of ways. This leads to variances in structure, execution, and management of individual courses in general and capstone project courses in particular. These variances not only have a direct impact on the quality of the capstone project, it also affects MoE's efforts to ensure that all graduate have a certain level of skills and knowledge. Variances in management and assessment of capstone projects also result in various supervision styles followed by the capstone project advisors further creating differences in assessment artifacts to be produced by the students (Perez, Elizondo, García-Izquierdo, & Larrea, 2012). Divergence in the assessment approaches due to dissimilar assessment criteria and incompleteness of a particular assessment item has also been reported in Clear (2009) and Yorke, Bridges and Woolf (2000).
Well-defined development process, predetermined design artifacts to be produced and guidelines for assessment are vital for successful execution of any project-based course. For capstone project courses, multi-facet coordination among students, advisors and other relevant stakeholders (e.g. academic management, and external supervisor in case of industry-sponsored project) further necessitates the definition of a structured methodology that must be disseminated properly (Neyem, Diaz-Mosquera, Munoz-Gama, & Navon, 2017). Systematic pursuance of such methodology equipped with latest smart technologies like blockchain and machine learning and with comprehensible practices can facilitate to cope with the difficulties and variances in the assessment (von Konsky & Ivins, 2008).
Universities offering SECSIT programs also play a vital role to cope with the challenges of the technological advancements in digitalization of the economy by effectively training the young students (Saeedi & Anna, 2021). Towards thriving knowledge-based economy, Saudi Arabia with roughly 51% young population (< 25 years) is increasingly assisting startups to promote entrepreneurship ecosystem. With programs like Monsha'at and TAQADAM, the Kingdom is planning to increase the SME's contribution to GDP from 20 to 35% to align with the Vision 2030 (Wamda research, 2016). Universities are key contributors to this growth by training the human capital and integrating with local and international partners for indigenous and global problem-solving. Reports like King Abdullah University of Science and Technology (2016), University of Waterloo (2016) and Virginia Commonwealth University (2016) are the clear indications that capstone projects have the potential to be further exploited and can be transformed into startups. Such startups force the economy to revolve by promoting technology development, opening new markets and increasing local employments. However, for such a long life and future extension, capstone projects need to follow several standards (Thompson & Edwards, 2009; Chenoweth, 2008).
Although universities are committed to increase innovative capacities of their students for enabling them to contribute to economic and social growth, it is still very hard to know the knowledge creation and sharing at national level. With variations in management and assessment of capstone projects, analysis of the students' capabilities to align with industry demands, to realize Vision 2030, is challenging. Hence, there is a need of smart assessment and management of software capstone projects being developed in Saudi universities. Incorporating latest technologies, such unified management can facilitate discovering the trends and patterns related to the domain, complexity and time factor.
Furthermore, one of authors was previously involved in a nation-wide study for developing rubrics and support for quality assessment of software capstone projects in Pakistan (Ahmad, Raza, & Feldt, 2011). To our best knowledge, there has not been any study carried out on this scale in Saudi Arabia. Therefore, this study investigates the current practices to provide the baseline for unified management and assessment of software capstone projects in Saudi universities.
3. Method
This paper presents the results of the initial phase of a longitudinal study aiming towards unified management of capstone projects in Saudi universities. This phase focuses on identifying the practices being followed to manage and assess the capstone projects. The following research question is investigated through a web-based questionnaire administered using a web service commonly used for questionnaires and polls;
What are the practices followed in Saudi universities to structure, manage and assess software capstone projects?
To answer the above research question, the questionnaires was divided into four main themes.
The first introduction theme had three questions asking about the basic information and for how long he/she is involved in managing/advising the capstone projects.
The second theme focused on structure of the capstone project course. It had seven questions and the respondents were asked about the time to start the capstone project course, title of the course, weight of the course in terms of credit hours, overall administration of the course, process information dissemination and recommended student team size for a particular capstone project.
With seven questions, the third theme focused on the management process of the capstone project course. Under this theme, the respondents were asked about frequency of student-advisor meeting, recommended software development life cycle (SDLC), assurance of the novelty of the proposed capstone project, presence of the digital repository of the capstone projects and existence of a dedicated platform to facilitate systematic management and communication among the stakeholders (students, advisors, project coordinators, etc.).
Finally, the fourth theme with seven questions focused on the assessment scheme being followed. It contained questions about the assessment items, percentage of each assessment item, roles involved in the assessment and the percentage of the each role.
In addition, each theme had an open-ended question asking about the comments and suggestions regarding the particular theme. Complete set of survey questions is presented in Appendix.
Based on the lessons learned from our previous study described in Ahmad et al. (2011), not only the formation of the questionnaire was changed but some the of individual questions were also updated (rephrased, new options were added, etc.). In addition, four new questions were added to explore current management practices in more detail.
Our target population consists of faculty members serving in SECSIT departments of the universities listed on MoE website (http://www.moe.edu.sa). The request to participate with survey link was sent through an email to both senior and junior faculty members of 30 different universities. These faculty members were identified with their profiles at respective university websites.
Prior to beginning the survey, a pilot study was conducted. The pilot survey was intended to improve the validity of the survey, assess its difficulty and get rough estimates of the time involved with conducting the actual survey (Robin & Babbie, 2016). Validity was examined in terms of face validity and content validity. According to Burton and Mazerolle (2011), the purpose of face validity is to evaluate the survey instruments for “ease of use, clarity, and readability”, while content validity is used for “establishing an instrument's credibility, accuracy, relevance, and breadth of knowledge regarding the domain”. Two expert instructors were asked to participate in the pilot study. The pilot survey was distributed in an online format. Experts' suggestions were analyzed and a number of changes to the survey were made.
4. Results
In total, 95 instructors from different Saudi universities were invited to participate in the study, 42 of them (44%) agreed to participate and complete the online survey.
4.1 Respondents information
Table 1 shows the respondents' information. As can be seen, these respondents belong to 22 different universities. More than half of these universities (55%) were founded after the year 2000 and, therefore, are classified as young universities. In addition, four of these universities (17%) are private universities, while the rest (83%) are public universities. Respondents’ experiences in supervising capstone projects vary. Sixteen of them (38%) have one to two years of experience, while nine of them (21%) have been supervising capstone projects for more than ten years.
4.2 Capstone project structure
Table 2 presents the responses related to the structure of capstone projects. The vast majority of respondents (71%) indicated that capstone projects in their universities are undertaken during the last two semesters of study (in the 9th and 10th semesters for universities that apply 10-semester plan and, in the 7th and 8th semester for universities that apply 8-semester plan). These projects are given different names. In the majority of universities, they are either called graduation projects (noted by 36% of respondents) or senior projects (stated by 29% of respondents). The credit hours allocated to capstone projects vary. Around 31% of respondents reported that four credit hours are allocated to these projects, 21% indicated that five credit hours are assigned to these projects, while 19% reported that capstone projects are only allocated three credit hours. The vast majority of respondents (88%) reported that the process, deliverables and content of capstone projects are documented and well known to supervisors.
Regarding how this essential information is explained to students, 55% of respondents indicated that it is explained via introductory seminars but a greater percentage (88%) noted that this information is conveyed to students via regular meetings with their supervisors. These meetings, according to 38 respondents (91%), are conducted on a weekly basis. In regard to the size of project groups, around two-third of respondents (65%) indicated that only two to three students are allowed in each group.
4.3 Management of capstone projects
Data related to this theme are presented in Figure 1(a–f). Numerals in these charts are the number of respondents. As depicted in Figure 1(a), when asked about managing the capstone project, majority of the respondents (35 respondent, 83%) indicated that the project supervisor plays a vital role in managing the capstone projects. 20 respondents (48%) specify that a project coordinator is also involved in capstone project management while only 3 respondents indicated that a special interest group is mainly responsible for capstone project management. Figure 1(b) shows that most widely used SDLC model for managing these projects is waterfall model as stated by more than half of the respondents (23 respondents, 55%). Iterative SDLC is the second most widely used SDLC as stated by 9 respondents, 22% while only Agile as SDLC is indicated by 6 respondents (22%). Some of the institutes are also using custom-based SDLC for capstone project management.
More than two-third of the respondents (29) indicated that their universities do not have digital repositories of capstone projects that can be accessed locally or publicly. When asking them about how they ensure the novelty of a proposed capstone, some (11 respondents, 26%) indicated that it is the supervisors' responsibility, others (9 respondents, 21%) noted that an approving committee checks the novelty of all proposals, while (9 respondents, 17%) pointed out that certain software application are used for that purpose. Regarding the availability of a dedicated platform to manage capstone projects, 26 (62%) of the respondents acknowledged the existence of such platform; however, of these 62%, only 17 respondents (65%) stated that the functionality of the platform satisfies their requirements.
4.4 Assessment scheme
In order to identify the assessment practices being followed at Saudi universities for capstone project course, the participants were inquired about the key assessment deliverables and the assessment authorities. As can be seen in Figure 2, the key deliverables of capstones projects vary between universities. The most common deliverable is project report, noted by 33 (79%) of the respondents, followed by both project proposals and code and executable files reported by 31 (74%) of the respondents.
Software requirement specification and software design document is identified by 29 (69%) and 28 (67%) of the respondents, while software test document and project poster is noted by 21 (50%), and 18 (43%) of the respondents, respectively.
To understand the importance of different assessment items, respondents were asked to provide grade weight (out of a total of 100) for key deliverables; project proposal, project poster, intermediate presentations, process/approach, final presentation, final demonstration and final report. As depicted in Figure 3, regarding the grade weight that is given to each deliverable, majority of respondents (17 respondents, 41%) indicated that project proposal is given 10 to 20 points.
In total, 20 of the respondents (45%) also noted that process/approach is given 10 to 20 points. For intermediate presentations, 15 (36%) of respondents reported that no grade is given to this deliverable, while 13 (31%) noted that it is given between 10 to 20 points. Around half of respondents (20 respondents, 48%) noted that final presentation and final demonstration are given 10 to 20 points. The highest weight is given to final report. According to 27 (65%) of respondents, final report is given 20 points or more.
Depicted in sub-figures of Figure 4 are responses of the participants to the questions asked about the individual and group evaluation, evaluators, weightage of evaluators and the ratio of the research projects. Students of capstone projects, according to the majority of respondents (23 respondents, 55%), are assessed both individually and as a group (see Figure 4a). As can be seen in Figure 4(b), assessment is performed by faculty members from the department (noted by 35 (83%) of respondents) and supervisors (noted by 32 (76%) of respondents). Of the total, 5 (12%) of the respondents noted involvement of an external examiner, while only 2 (5%) of the respondents reported the involvement of the external supervisor. Such less involvement of external examiner/supervisor indicates that very few capstone projects are based on industry demands.
According to half of the respondents, the grade is divided between supervisors and faculty members either fifty–fifty or sixty–forty (see Figure 4c). Research thesis seems to be neither popular nor encouraged in IT colleges in Saudi Arabia. Around 60% (25) of respondents noted that the ratio of research thesis in ongoing capstone projects at their universities is less than 20% (see Figure 4d).
5. Challenges
This study was successful in recruiting instructors from 74% of Saudi universities that offer undergraduate programs in SECSIT. Therefore, it can be inferred that its findings reflect the current practices of assessing and managing capstone projects in Saudi universities.
The excellent number of universities that respondents belong to as well as the variety of experiences that respondents possess were helpful in gathering valuable opinions and suggestions for the continuous improvement of capstone projects.
The structure of capstone projects in Saudi universities is no different than that followed by many universities across the world (Ahmad et al., 2011; Lesko, 2009; Greenberg Davis & Zilora, 2016). Students are given two courses to complete their project. These courses are completed in two different semesters. They are normally titled Project 1 and Project 2, respectively, with words like Graduation, Senior or Capstone come as prefix. Project 1 is normally focused on identifying the requirements, analyzing them and designing the proposed software. Project 2 is dedicated to writing the code and testing the produced software. In this structure, universities are allocating more credit hours (double credits in most universities) to the work performed in Project 2. Students are mostly organized in small groups of a maximum of three students in each one. The current structure and management of capstone projects are raising several challenges for both students and their advisors.
5.1 Plan-driven process
Supervisors are restricted regarding the process model that they can recommend their students to use for developing their software. Although some of the participants noted the use of agile methodologies in their universities, the current structure only favors plan-driven waterfall model and make the application of other process models very difficult. Waterfall model requires gathering clear requirements at the beginning of the project and has difficulty accommodating the natural uncertainty that exists in many software projects (Pressman & Maxim, 2019).
Any changes in requirements can cause confusion as the project team proceeds. As described in studies like Coupal and Boechler (2005 and also complemented by our own experience, due to time constraints and experience level of the students, requirements are poorly defined in the first phase of the course and cannot be changed or updated. Underestimated scope and complexity of the project in the early stages results in incompleted capstone projects. This also restricts students' learning of modern techniques and methods used in the industry for software development.
5.2 Inflexibility
Another major challenge facing students and supervisors of capstone projects is related to the inflexibility in distributing the work and marks between the two phases of the capstone project courses as well as in deciding the size of the student groups. While the first phase (also project 1) is allocated less credit hours than second phase (also called phase 2), sometimes the nature of the work requires students to do equal or even more work in project 1. This could occur when applying other process models than waterfall or when a research project is undertaken. An instructor noted “Different projects might have different grading policies depending on the nature of the project and the agreed plan”. Stein (2002) also reported that capstone students prefer to work on small group. Their reasons, according to Stein, are that they could select the students they want to work with, and that it is easier for them to coordinate a small group. However, not all capstone projects are small. There are big and complex projects that can only be developed by large group of students (Mann, 2005). These projects often lead to better outcomes. Students who are involved in these projects can learn more and gain experience similar to that in a real team environment (Ikonen & Kurhila, 2009; Mann, 2005).
5.3 Limited support for knowledge sharing
The impact of successful capstone projects is long lasting. These projects not only provide evidences of the intellect and technical knowledge of the students involved in a particular project but can also provide important insights for upcoming students. As highlighted in Miller (2013), finding useful information at the early stages is the most challenging aspect of a research (capstone) project. Digital archives are considered important for storing and sharing of the capstone project outcomes (Lalisan & Sobejana, 2019). Currently, only one-third of the Saudi universities are maintaining capstone projects and these repositories are only accessible to the students of particular universities. Only one of the respondent indicated that they are developing a repository to be accessible for students from other universities within the Kingdom. Although some of the universities are archiving hard copies of the final report in their libraries, in general, the dissemination of the capstone project results and research output is not well supported.
Along with facilitating knowledge sharing, digital repositories can also assist in ensuring novelty of a proposed capstone project. Technological originality of the capstone projects is foundational for knowledge creation and continuation of the interesting projects, thus transforming them into valuable startups. Assurance of such novelty in the capstone project demands an integrated effort based on advisor's knowledge of the field, and the use of plagiarism detection tools to assess the similarity ratio. In Saudi universities, there is variation in approaches for novelty assessment and similarity checking. Most of them only consider advisor's judgment for acceptance or rejection of a capstone project proposal. Due to the absence of digital repositories, it is almost impossible to compare new proposals with the already completed capstone projects at both the intra-university and inter-university levels.
5.4 Variations in assessment scheme
Similar to other project-based courses, development deliverables are the main artifacts for continuous assessment of the capstone projects. Deliverables and their contents must be well-known to all the stakeholders as they are used to define the assessment criteria. No different from our previous study in the context of Pakistani universities, reported in Ahmad et al. (2011), final project report, and code and executable files are the key deliverables of capstone project courses in Saudi universities. From the SWEBOK's perspective, software testing is the least emphasized deliverable considered at Saudi universities for capstone project deliverables. Time constraint, short-term product vision (as most of the capstone projects die after graduation of the participated students), and the use of waterfall development life cycle (wherein testing comes at the end) are the main reasons of neglecting software testing and quality assurance.
Grade weight of each deliverable considered for assessment also varies among the Saudi universities. Final report and final demonstration, with maximum weight of 60 and 50 respectively, are the most considered assessment artifacts, while project poster and intermediate presentations, with maximum weight of 30 and 20 respectively, are the least considered assessment artifacts. With maximum weight of 40, development process/approach is included as assessment artifact by almost the universities in Saudi Arabia. This variation in the assessment scheme not only effects the assurance of the students readiness for industry but also hampers achievement of the academic goals at the national level, such as realization of the Vision 2030 for Saudi Arabia.
6. Recommendations
The above-mentioned challenges seem to inhibit universities, instructors and students from exploiting capstone projects to the full. Therefore, based on the findings of this study and best practices extracted from the literature, we recommend the following.
6.1 Incremental development process
Successful completion of any software project highly depends on the selected development process. Studies like Alshayeb, Mahmood and Aljasser (2018), Broman, Sandahl and Abu Baker (2012), Coupal and Boechler (2005), Strode ana Clark (2007) have already presented the usefulness of iterative and incremental development methods. To cope with the challenges of incompleteness, and to improve students' learning and readiness for real-world problem-solving, it is recommended that the Saudi universities must encourage application of the modern incremental and iterative software development processes.
Modern iterative and agile models would require changing the structure and the types of deliverables required in each semester. When an agile approach is used, software needs to be developed in small increments and short iterations (usually 2-4 weeks). Requirements are fully defined around each iteration, rather than at the start of the project as in the waterfall model (Pressman & Maxim, 2019). Therefore, students will not be able to complete the whole requirement gathering, analysis and design of their software by the end of first phase. They are also not required to wait until the second phase to start writing the code.
To contribute to the national growth and to fulfill the market demand, universities are also recommended to align the capstone project courses with latest technology trends. Universities must collaborate with the industry and update the structure and requirements of capstone project courses accordingly. This will further facilitate to bridge the gap between industry and academia and will develop a win–win scenario for all the stakeholders.
6.2 Project-driven approach
Teamwork is not only essential for capstone project development, but it also facilitates personal development of the students. However, team size has a significant impact on students' learning and capabilities needed for successful capstone project completion (Chou & Chang, 2018; Paretti, Layton, Laguette, & Speegle, 2011). Currently student selection and the work distribution approaches followed for capstone project courses are quite rigid. Number of students involved in a particular project does not fully depend on project complexity but the number of registered students, and faculty load also affect the students’ selection.
To ensure successful project completion and to further enhance students' learning experience, we recommend that a project-driven approach must be followed for work distribution and the marks allocation for different phases of the capstone project courses. This approach leverages evidence-based course improvement by collecting evidences throughout the project life cycle. Evidences like students’ knowledge, their career orientation and project complexity can be collected before the start of the capstone project course, while work progression and mid-project reviews can be assembled during the project execution. Post-mortem analysis, at the end of each capstone project, can be conducted to collect feedback of the faculty, assessors and the students involved to provide the baseline for improving process and the departmental culture.
For team formation, the applied project-driven approach must favor the student-centered perspective as the students' interest and preferences have strong influence on team performance and outcome (Steiner & Stresau, 2017).
6.3 Digital archives for future cohorts
Like other international universities, most of the Saudi universities are only focused on archiving the post-graduate thesis and reports. Managing capstone projects outcome and making it available with open access is something that is not properly managed (Lalisan & Sobejana, 2019). Currently very few Saudi universities keep record of capstone projects in their libraries either online or as hard copies, but students from other universities are not able to access those reports to benefit from them in their projects.
We recommend developing a national level digital repository of the capstone projects linking all the Saudi universities. As highlighted by one of the respondent “It will be great to have a database for all final year projects in KSA and shared with all universities”. Realized as a cloud-based web application, such repository can enable the students and faculty members to search and download thousands of capstone project reports and university essays. Such a platform can assist in monitoring and controlling the plagiarism in capstone projects, synchronizing the systematic evaluation, facilitating unified assessment and management and enabling the continuation of interesting projects thus transforming them into valuable startups. Before starting their capstone projects, students can review previously developed projects through an extensive searching module. This will enable them to envision the types of project being developed in their area of interest (e.g. AI, image processing, IoT, etc.). In order to cope with the challenge of identifying and monitoring the replicated work, potential project advisors and the assessors can check the similarity ratio to approve or disapprove a proposed project. Such a national level repository can provide an extensive environment for university students to promote novelty and to solve indigenous problems.
As part of smart unified management, mining such national level repositories can reveal important patterns of students' abilities, areas of interest, team structure and work distribution, and time taken in relation with successful completion of the projects. It can further assist in aligning capstone course with industry needs thus improving the employability of the students.
Despite of the SDLC used for capstone project development, extensive communication among the stakeholders (students, advisors, etc.) is necessary for planning and feedback. It is encouraging to note that most of the Saudi universities are using dedicated platforms for managing structure, assessment and communication among the stakeholders of capstone project courses. On the other hand, almost half of the respondents are not satisfied with effectiveness of these platforms. Maintaining the coordination among the stakeholders with time constraints is still challenging, thus effecting the overall completion of the capstone projects. The recommended national level repository can further be extended to improve communication and coordination among the stakeholders through latest cloud infrastructure.
6.4 Homogeneous assessment scheme
Assessment of capstone courses is different from the regular courses. Intra-team dependence, judgment of individual and group work progression, and evaluation of the final outcome make assessment of the capstone project quite a challenging task. Current assessment practices at Saudi universities are mostly focused on the end product while the process and progression of the learning, suggested by Clear (2009), are less considered comparatively. Furthermore, the artifacts examined for the assessment also vary with different weights among the universities. To minimize this variation and to promote identical assessment of the students' readiness, we recommend a homogeneous assessment scheme based on continuous evaluation of all three aspects of a capstone project; product, process and the progression of learning. Such homogeneous assessment scheme will provide a common baseline for the quality assessment. Due to variations in the nature and scope of the projects, weights of the assessment items under each aspect may vary but they must coequal as much as possible. For example, for a research project more weightage is given to the process/methodology, while for a development project the code/executable files, and test documents more grade weightage will be given. Criteria defined for each assess item must also be evaluated on a common scale.
Aligned with recommendations in our previous study with Pakistani universities and as suggested by the studies like Clear (2009, 2010), Figure 5 depicts the recommended assessment scheme for continuous evaluation. Each evaluation aspect is further provided with a minimum set of evaluation artifacts for providing a baseline for unified assessment and management.
Product: At the end of capstone project course, students must deliver a product with the features promised in the project proposal. The product demonstration (see Figure 5) can be assessed for task completion, originality, usability and coding standards. Final project report to be submitted along with the executable code is an important assessment item and must be assessed for contents, structure, and completeness and accuracy. In case of a research project, students must also submit the publishable research work. Task distribution can also be added for individual assessment.
Process: Process used for software development can be assessed, its activities and the artifacts produced at the end of each activity (as shown in Figure 5). As the universities are recommended to use iterative and incremental software development process, each iteration and increment must be assessed for analysis, design, implementation and testing against the selected requirements. The assessment scheme, explained at the start of the course, must encourage the students to follow certain industry standards and best practices for describing the required artifacts, thus facilitating continuation of projects for future cohorts.
Progression in learning: One of the important goals of capstone project courses is to improve the progression of learning. Advancement in learning can be assessed with interim presentation and reviews (see Figure 5). Project proposal presentation at the start, mid-project presentation at the end of the first phase and final project presentation are the main assessment items in this regard. Reviews from the assessors involved in these presentations and feedback from the advisor for regularity and independence must be considered for the assessment of progression.
7. Conclusion
This study explored the current practices for structure definition, management and assessment criteria used for the capstone project courses in Saudi universities. The main aim was to identify areas of improvement and help universities move towards smart unified management of capstone project courses. The findings indicate that there are several challenges that seem to inhibit universities, instructors and students from exploiting capstone projects course to the full. These challenges are related to the utilized process model, the distribution of work and marks, the knowledge sharing approach and the assessment scheme. To address these challenges, we have proposed the use of an incremental development process, the utilization of a project-driven approach, the development of digital archive and the implementation of homogeneous assessment scheme.
One of the important future work is to develop an online repository of the capstone projects with essence of enabling the students and faculty members of Saudi universities to get access to thousands of capstone project reports and university essays. With the use of latest data science technologies, this proposed national level repository can provide important insights about the readiness of the graduates for indigenous and global problem-solving thus facilitating universities in aligning themselves for realization of the Vision 2030.
Figures
Respondents involved in the study
| Respondents | Respondents | ||
|---|---|---|---|
| Universities | |||
| Yanbu University College | 7 | University of Bisha* | 1 |
| Majmaah University+ | 5 | Prince Mohammad University+* | 1 |
| Qassim University+ | 3 | Jazan University+ | 1 |
| Najran University+ | 3 | Umm Al Qura University | 1 |
| Saudi Electronic University+ | 2 | Islamic University of Medina | 1 |
| Al Jawf University+ | 2 | Taibah University+ | 1 |
| King Abdulaziz University | 2 | King Faisal University | 1 |
| Imam Muhammad bin Saud Islamic University | 2 | King Fahd University for Petroleum and Minerals | 1 |
| Kind Saud University | 2 | Alfaisal University + * | 1 |
| Fahd bin Sultan University+ | 2 | Effat University* | 1 |
| Prince Sattam Bin Abdulaziz University+* | 1 | Imam Abdulrahman Bin Faisal University | 1 |
| Experience in supervising capstone projects | |||
| 1–2 years | 16 | ||
| 3–5 years | 05 | ||
| 5–10 years | 12 | ||
| >10 years | 09 | ||
Note(s): *Private University
+Young University
Structure of the capstone project
| Respondents | Percentage | |
|---|---|---|
| Semester in which students start their capstone project | ||
| 7th or 9th (second last semester) | 30 | 71% |
| 8th or 10th (last semester) | 10 | 24% |
| 6th | 02 | 05% |
| Title of the capstone project | ||
| Graduation project | 15 | 36% |
| Senior project | 12 | 29% |
| Project | 04 | 10% |
| Capstone project | 03 | 07% |
| Other | 08 | 18% |
| Credit hours allocated to the capstone project | ||
| 04 | 13 | 31% |
| 05 | 09 | 21% |
| 03 | 08 | 19% |
| Other | 12 | 29% |
| Deliverables and content of capstone projects are documented and well known to supervisors | ||
| Documented and well known | 36 | 86% |
| Documented but not well known | 05 | 12% |
| Not documented and not well known | 01 | 02% |
| How process deliverables and contents are explained to students? | ||
| Introductory seminar | 23 | 55% |
| Through handbook | 14 | 33% |
| Regular meetings with supervisor | 37 | 88% |
| Other | 01 | 02% |
| Frequency of meetings with supervisors | ||
| Weekly | 38 | 91% |
| Once in a month | 01 | 02% |
| Twice in a month | 03 | 07% |
| Recommended group size for a capstone project | ||
| 1 student | 02 | 05% |
| 1–2 students | 05 | 12% |
| 2–3 students | 27 | 65% |
| >3 students | 08 | 18% |
Following are the questions asked to the participants, under each theme, in the form of an online questionnaire using a web service.
References
Ahmad, E., Raza, B., & Feldt, R. (2011). Assessment and support for software capstone projects at the undergraduate level: A survey and rubrics. In 2011 Frontiers of Information Technology (pp. 25–32).
AlYahya, S. A., & Abo El-Nasr, A. A. (2012). Outcomes based assessment of the engineering programs at Qassim University for ABET Accreditation. In Proceedings of 2012 International Conference on Interactive Mobile and Computer Aided Learning (IMCL), 2012, 22-31. doi: 10.1109/IMCL.2012.6396445.
Al-Yahya, M., Alkadhi, R., & Alrasheed, H. (2021). Agile transformation for capstone projects: Preparing graduates for the job market. In 2021 International Conference on Computational Science and Computational Intelligence (CSCI) (pp. 1121–1124), doi: 10.1109/CSCI54926.2021.00236.
Alawairdhi, M. (2016). Assessment of Student's perception and teaching methodology of software engineering course (undergraduate) at universities in kingdom of Saudi arabia. Research Journal of Applied Sciences, Engineering and Technology, 13(2), 98–106, ISSN: 2040-7459.
Almaliki, M. (2021). Software engineering in Saudi arabia: A bibliometric assessment. IEEE Access, 9, 17245–17255, doi: 10.1109/ACCESS.2021.3053611.
Alshayeb, M., Mahmood, S., & Aljasser, K. (2018). Moving from waterfall to agile process in software engineering capstone projects. Computer Science Information Technology, 8(8), 107–114.
Aoudia, M., Marji, K., & AlQahsi, D. A. -D. (2015). Assessment of higher education quality by using cohort of first-year in university. Procedia - Social and Behavioral Sciences, 191, 330–335, The Proceedings of 6th World Conference on educational Sciences, available from: http://www.sciencedirect.com/science/article/pii/S1877042815025707.
Ardis, M., Budgen, D., Hislop, G. W., Offutt, J., Sebern, M. & Visser, W. (2015). Se 2014: Curriculum guidelines for undergraduate degree programs in software engineering. Computer, 48(11), 106–109.
Bhatti, A., & Ahmed, I. (2015). Academic diversity and assessment process for cs program accreditation. Creative Education, 6, 773–784.
Broman, D., Sandahl, K., & Abu Baker, M. (2012). The company approach to software engineering project courses. IEEE Transactions on Education, 55(4), 445–452.
Burton, L., & Mazerolle, S. M. (2011). Survey instrument validity part i: Principles of survey instrument development and validation in athletic training education research. Athletic Training Education Journal, 6, 27–35.
C. A. Commission. (2017). Criteria for accrediting computing programs. Baltimore, MD: Accreditation Board for Engineering and Technology.
Chenoweth, S. (2008). Undergraduate software engineering students in startup businesses. In 2008 21st Conference on Software Engineering Education and Training (pp. 118–125).
Chou, P., & Chang, C. (2018). Small or large? The effect of group size on engineering studentsaˆ CTM learning satisfaction in project design courses. Eurasia Journal of Mathematics, Science and Technology Education, 14(10), 1–9.
Clear, T. (2009). Thinking issues: The three p's of capstone project performance. SIGCSE Bull, 41(2), 69–70, doi: 10.1145/1595453.1595468.
Clear, T. (2010). Thinking issues managing mid-project progress reviews: A model for formative group assessment in capstone projects. ACM Inroads, 1(1), 14–15.
Coupal, C., & Boechler, K. (2005). Introducing agile into a software development capstone project. Agile Development Conference (ADC'05) (pp. 289–297).
Greenberg Davis, H., & Zilora, S. J. (2016). A tale of two capstones. In Proceedings of the 17th Annual Conference on Information Technology Education, ser. SIGITE aˆ C™16 (pp. 130–135). New York, NY: Association for Computing Machinery, doi: 10.1145/2978192.2978234.
Ikonen, M., & Kurhila, J. (2009). Discovering high-impact success factors in capstone software projects. Proceedings of the 10th ACM Conference on SIG-Information Technology Education (pp. 235–244). ser. SIGITE aˆ C™09. New York, NY, USA: Association for Computing Machinery.
King Abdullah University of Science and Technology (2016). Saudi Arabia's innovation landscape. available from: https://innovation.kaust.edu.sa/saudi-arabias-innovation-landscape/ (accessed 2 December 2019).
Koolmanojwong, S., & Boehm, B. (2009). Using software project courses to integrate education and research: An experience report. Software Engineering Education and Training, Conference on (pp. 26–33).
Lalisan, S., & Sobejana, N. (2019). Research and capstone project electronic repository. Current Journal of Applied Science and Technology, 38(4), 1–12, doi: 10.9734/cjast/2019/v38i430378.
Lesko, C.J. (2009). Building a framework for the senior capstone experience in an information computer technology program. Proceedings of the 10th ACM Conference on SIG-Information Technology Education, ser. SIGITE aˆ C™09 (pp. 245–251). New York, NY: Association for Computing Machinery, doi: 10.1145/1631728.1631804.
Mann, L.S.S. (2005). Technical complexity of projects in software engineering. Proceedings 18th Annual NACCQ.
Miller, R. E. (2013). The almost experts: Capstone students and the research process. (pp. 16–22). Baltimore, MD, American Library Association, available from: http://www.ala.org/acrl/acrl/conferences/2013/papers (accessed 2 December 2019).
N. C. for Academic Accreditation Assessment. (2015). Handbook for quality assurance and accreditation in Saudi Arabia. Riyadh.
Neyem, A., Diaz-Mosquera, J., Munoz-Gama, J., & Navon, J. (2017). Understanding studentinteractions in capstone courses to improve learning experiences. Proceedings ofthe 2017 ACM SIGCSE Technical Symposium on Computer Science Education, ser.SIGCSE '17 (pp. 423–428), New York, NY: ACM, doi: 10.1145/3017680.3017716.
K. of Saudi Arabia (2016). Vision 2030. available from: https://vision2030.gov.sa/en/vision/themes (accessed 2 December 2019).
Paretti, M.C., Layton, R.A., Laguette, S.W., & Speegle, G. (2011). Managing and mentoring capstone design teams: Considerations and practices for faculty*. International Journal of Engineering Education, 27(6), 1192–1205.
Perez, C.D., Elizondo, A.J., García-Izquierdo, F.J., & Larrea, J.J.O. (2012). Supervision typology in computer science engineering capstone projects. Journal of Engineering Education, 101, 679–697.
Pressman, R., & Maxim, B. (2019). Software engineering: A practitioner's approach (9th ed.). New York: McGraw-Hill Education.
Robin, A., & Babbie, E.R. (2016). Essential research methods for social work. In ser. Brooks/Cole empowerment series. Boston, MA: Cengage Learning.
Saeedi, K., & Anna, V. (2021). Software development methodologies, HEIs, and the digital economy. Education Sciences, 11(2), 73, doi: 10.3390/educsci11020073.
Sahami, M., Roach, S., Cuadros-Vargas, E., Hawthorne, E.K., Kumar, A., LeBlanc, R., Reed, D., & Seker, R. (2014). Acm/ieee-cs computer science curricula 2013: Implementing the final report. Proceedings of the 45th ACM Technical Symposium on Computer Science Education, ser. SIGCSE '14 (pp. 175–176). New York, NY: ACM, doi: 10.1145/2538862.2538863.
Stein, M.V. (2002). Using large vs. small group projects in capstone and software engineering courses. Journal of Computing Sciences in Colleges, 17(4), 1–6.
Steiner, M.W., & Stresau, K.S. (2017). A case study approach for understanding the impact of team selection on the effectiveness of multidisciplinary capstone teams.
Strode, D., & Clark, J. (2007). Methodology in software development capstone projects.
T. G. on Information Technology Curricula (2017). Information technology curricula 2017: Curriculum guidelines for baccalaureate degree programs in information technology. New York, NY: ACM.
Thompson, J.B., & Edwards, H.M. (2009). A scalable approach to graduate student projects: Hundreds with industry every year. Software Engineering Education and Training, Conference (pp. 158–161).
University of Waterloo (2016). How a fourth-year design project became a startup: The story of palette. available from: http://velocity.uwaterloo.ca/2017/04/how-a-fourth-year-design-project-became-astartup-the-story-of-palette/ (accessed 2 December 2019).
Virginia Commonwealth University (2016). New capstone collaboration: Engineering and business students join forces to build startups. available from: https://news.vcu.edu/article/New capstone collaboration Engineering and business students (accessed 2 December 2019).
von Konsky, B., & Ivins, J. (2008). Assessing the capability and maturity of capstone software engineering projects. Conferences in Research and Practice in Information Technology Series (Vol. 78, pp. 171–180).
Wamda research (2016). The kingdom of Saudi Arabia: Status of the entrepreneurship ecosystem. available from: https://backend.wamda.com/api/v1/downloads/publications/kingdom-saudi-arabia-statusentrepreneurship-ecosystem (accessed 2 December 2019.
Yorke, M., Bridges, P., & Woolf, H. (2000). Mark distributions and marking practices in UK higher education: Some challenging issues. Active Learning in Higher Education, 1(1), 7–27, doi: 10.1177/1469787400001001002.