Selecting and using faculty data management software systems

Abstract

Purpose

The purpose of this paper is to determine which software systems business school administrators use to support accreditation efforts and how administrators select and use these systems. This study also provides best practice suggestions from institutions using faculty data management systems to support accreditation efforts.

Design/methodology/approach

This study used a sequential explanatory design using an internet-based survey for business school administrators involved with accreditation reporting with follow-up interviews with survey respondents.

Findings

There are four major software vendors that most respondents use for managing reporting of faculty research activity and sufficiency. The location of the school appears to influence the system selected. For assurance of learning reporting, most schools used an in-house or manual system. Respondents highlighted the importance of doing a thorough needs analysis before selecting a system.

Research limitations/implications

Although respondents were geographically diverse, having a larger sample with schools in developing regions would provide greater generalizability of results.

Practical implications

This study gives business school leaders a comprehensive overview of the business schools’ data management systems, criteria used in system selection and best practices for system selection and implementation, faculty engagement and ongoing maintenance.

Originality/value

This study addresses the limited attention given to resources and best practices for selecting and implementing faculty data management software for accreditation in the academic and industry literature despite the significant investment of resources for schools and the importance such systems play in a successful accreditation effort.

Citation

Shea-Van Fossen, R.J., Taormina, R.D.V. and LaCasse, J. (2021), "Selecting and using faculty data management software systems", Organization Management Journal , Vol. 18 No. 5, pp. 187-198. https://doi.org/10.1108/OMJ-04-2021-1210

Introduction

Many business schools recognize the inadequacy of their reporting systems and associated policies and procedures to meet the multi-year reporting requirements necessary to demonstrate compliance with accreditors’ standards once a school’s leadership begins drafting the initial AACSB accreditation self-evaluation and other pre-accreditation reports. Business schools find the faculty sufficiency and qualifications reporting requirements to be particularly challenging. To solve these challenges and meet ongoing reporting requirements, many schools recognize early in the accreditation process the need for a software solution.

Literature review

The academic and industry literature has given limited attention to the system business schools use and best practices for selecting data management software for accreditation purposes. This is despite the significant investment in time and money and the importance such systems play in the accreditation effort. Even with repeated questions about software for faculty data management at AACSB and other accreditors’ annual conferences, there is no centralized source of information on the systems schools use nor a repository of best practices in data management systems selection and implementation.

Top academic journals, including the Academy of Management Learning and Education and Organizational Management Journal, have had special issues focused on accreditation and accreditation issues, however, there’s a dearth of studies addressing data management systems. Bisoux’s (2013) Biz Ed article offered suggestions for selecting and implementing management software solutions based primarily upon suggestions from software vendors and provided information on the available systems at that time. Finally, Foshee Holmes et al. (2017) presented a case study of how one university compiled data needed for accreditation reports from individual faculty. In the eight years since Bisoux published her 2013 article, several major software providers have merged and the products available today are different than those available in 2013. This article expands on Bisoux’s work by providing the survey results of business school administrators exploring which software solutions business schools use and how they selected these systems. This article also reports business school administrators’ interview results exploring in greater depth how business schools use these software solutions to support their accreditation efforts and identified best practices in selecting and implementing a system from schools experienced in the process.

Data requirements of the new 2020 AACSB international business standards

AACSB’s revised 2020 business standards consolidated from 15 to 9, but the data reporting requirements remain similar to the 2013 standards. For schools seeking initial AACSB accreditation or reaccreditation under the new 2020 business standards, there are three quantitative reports required (AACSB International, 2020). Standard 3, faculty and professional staff resources, requires schools to complete two tables: Table 3–1 summarizes faculty sufficiency and qualifications and Table 3–2 summarizes deployment of faculty by qualification and program. These tables are similar to Tables 15–1 and 15–2 in the 2013 business standards (AACSB International, 2013). Standard 5 assurance of learning’s (AOLs) new required table, Table 5–1, summarizes the school’s assessment plan and results. Standard 8 impact of scholarship, requires schools to complete Table 8–1 summarizing faculty’s intellectual contributions. Table 8–1 is similar to Table 2–1 from the 2013 business standards. Schools pursuing separate AACSB accounting accreditation need to complete the new Table A6 summarizing the accounting faculty qualifications, current or emerging technology deployed in accounting courses and faculty assigned to teach accounting courses.

The three areas represented by these tables, namely, faculty research, faculty sufficiency and AOL – are three of the top four areas (the fourth being mission-related issues) where schools struggle to achieve alignment with AACSB standards and are denied initial or reaccreditation by AACSB (Hollenbeck & Martin, 2017). These three areas also require the compilation of multiple years of quantitative data to successfully achieve accreditation. Having well-designed processes and system support for faculty and AOL data management allow schools to not only report to accrediting bodies but also provide information to school leadership to actively manage the accreditation process; to faculty promotion, review and tenure committees for personnel decisions; and to university administrators for internal and external reporting. To better understand how business schools use software systems for their accreditation and faculty management needs and which software systems are used, a survey was conducted and business school leaders were interviewed.

Methods

A mixed methods sequential explanatory approach (Creswell, 2009) was used starting with an internet-based survey. The survey results were analyzed and used as a basis for follow-up interviews to better understand how business schools select, implement and use software systems for accreditation reporting and faculty management needs.

Survey of accreditation administrators

An internet-based survey was developed and distributed to understand which data systems schools used for faculty research, faculty sufficiency and AOL data management, what information business schools used to make the decision on which system to purchase and school demographics (accreditation status, size and country of school and years using selected system). In addition, survey respondents were asked to rank criteria used in selecting a software system by their business school on a five-point scale from not important at all to extremely important. The authors developed survey response options from a review of AACSB listserv discussions on software systems from 2018 to 2020. Additionally, the survey asked respondents to provide advice to schools looking to select and implement a data management system using open-ended questions. The survey consisted of 18 questions and took approximately 12 min to complete. Data collection instruments are available upon request.

Participants were recruited by posting solicitations on three accreditors’ listservs: AACSB’s Assessment and Accreditation Exchange affinity group site, AACSB’s Women Administrators in Management Education affinity group site and EFMD Global’s Informational Exchange site. Membership of these sites includes individuals responsible for accreditation reporting who would be familiar with the processes and software systems used by their business school to obtain information needed to complete accreditation reports. Solicitation postings included a link to complete the survey. Two requests were made three weeks apart and the survey was open for responses for two months with multiple submissions prevented by the data collection software. Of the 125 responses received, only 88 responses provided sufficient information on the school’s faculty research reporting system. The system used for faculty research reporting variable was used to develop the profile of respondents by the system used.

Two researchers independently analyzed the survey’s qualitative comments using Creswell’s (2009) steps for qualitative data analysis and initially coding data noting comments frequency using the following topics: initial selection, implementation, faculty acceptance issues, using the system and other ongoing issues. The data were re-reviewed to identify recurring themes under each topic area. Any discrepancies in coding were resolved collaboratively with the results used to develop questions for the interviews.

Accreditation administrators interviews

Interviews with accreditation administrators were conducted to better understand schools’ experiences with selecting, implementing and using data management software for accreditation purposes. These interviews were used to clarify survey responses and learn more about how business schools selected and used data management systems. Representative questions included, “What advice would you give a school that is looking at a new data management system?”, “How accepting was your faculty in adopting the new system?” And “Is there anything you did to help faculty adjust to the new system?”

To recruit interviewees, participants were taken to a second separate survey after submission of the internet survey and asked if they would be willing to participate in a Zoom interview. The 22 prospective respondents were sent an email asking for consent and to set up a mutually convenient interview time. In total, 13 different institution personnel responded, and interviews were scheduled, conducted and recorded by Zoom to produce a written interview transcript. To maintain consistency, one author conducted all approximately 20 minute interviews using a standard interview protocol. Follow-up questions were asked to clarify interviewee responses.

Two authors independently reviewed the Zoom-generated transcripts for common themes and suggestions, clarified any accuracy issues with the Zoom-generated transcripts, discussed and resolved any discrepancies in coding, and then compiled the best practice listing included in the results section. Any identifying information (i.e. person, institution or software vendor) in the interview transcript was removed before analysis.

Results

Systems used by business schools and user profiles

Table 1 summarizes survey respondent demographics and respondent schools’ software systems to support accreditation reporting needs. The four vendors representing 74.7% of the systems used were Academ by RimaOne (https://rimaone.com/), Watermark’s Faculty Success (formerly Digital Measures’ Activity Insight; https://www.watermarkinsights.com/our-approach/faculty-activity-reporting/), Interfolio (https://www.interfolio.com/) and Sedona (https://sedonaweb.com/). Most schools reported using the same system for both faculty research and sufficiency reporting. However, 76.1% of respondents reported using an in-house or manual (not a commercially available) system for AOL.

Survey respondent titles were 31.8% Assistant/Associate Dean, 29.6% administrative roles but not a dean and 14.8% were Deans. All interviewees knew how the school was using the faculty data management system and nearly all interviewees assisted in the initial system selection. In total, 51.2% of the survey respondents were from United States (US) business schools, which parallels the 58.8% of AACSB-accredited schools based in the USA.

Table 2 details school profiles using each software system to better understand the types of schools that used each major software system or designed their own systems. The profile includes the number of years using the system, school size (as measured by the number of business school and university full-time equivalent (FTE) students and business school faculty), AACSB accreditation status and percentage of respondent schools in the USA.

The results indicated that US business schools primarily used Watermark’s Faculty Success and Sedona using every 7.6 and 6.1 average years, respectively. Faculty Success and Sedona schools had the highest percentage of schools AACSB-accredited and the longest AACSB-accreditation tenure. Interfolio also had a US-centric user base, but its users had a two-year average tenure. Schools outside of the US market tended to use Academ, a newer provider of higher education software, with an average three-year tenure. School size did not appear to be a major factor in system selection as both smaller and larger schools used each system. Notably, Sedona had schools with the smallest average school size.

System selection criteria and information used

Survey respondents ranked reporting capabilities, ease of use, company responsiveness and initial system cost as the most important criteria in selecting a faculty data management system. The ability to use a system University-wide ranked as the least important selection factor noted by 27.53% of respondents. The top three information sources noted by respondents in selecting their current system were a demonstration by the vendor (33.33%), a referral from a colleague (27.64%) and prior experience with the system (21.14%). Under 10% of respondents used online reviews or sought a recommendation from an accreditor representative.

Respondents appeared committed to the selected system. When asked if they had to select a new software system, 76.0% of survey respondents indicated they were likely to select the same software platform.

Table 3 provides the complete criteria list in order of importance, system selection resources and system satisfaction information.

Implementing accreditation software systems best practices

Best practices were identified from survey respondents’ most frequently mentioned items and interview themes identified in interviews categorized by the steps of the process. The best practices coincide with the new system implementation process steps in installing a new system: system selection, system implementation, faculty engagement and ongoing maintenance. Interview insights on why schools used manual systems for AOL are also discussed. This section concludes with how schools use faculty data beyond accreditation.

System selection.

Survey respondents noted the paramount importance of clarifying the school’s system’s needs and integration of the new system with other university systems. One survey respondent noted:

Take your time to map out everything you want to capture before buying anything and prioritize the most important things you wish to track. You will never get 100% of what you want in any system, so priorities really help. Make sure it is enough but not burdensome.

Schools should identify both short- and long-term needs. Most interviewees indicated that the main driver for initially buying a system was accreditation reporting needs. Respondents consistently noted that if the system was used for accreditation reporting only, it was not worth the investment. An interviewee noted:

Having a proper understanding of the data […] needed and how we could best use [the selected system] for more than just the accreditation, because […] once we got it up and going it can be really useful. You can report on a whole variety of data across the whole Business School.

Among interviewees, it was clear that schools using a system to support faculty personnel decision-making, (e.g. to improve workflow for tenure and promotion processes) felt the system was a better investment.

In selecting a system, survey respondents and interviewees both stressed the importance of “doing your homework” and talking to various software systems users. In total, 53% of interviewees noted they were disappointed with the customer service received as compared to provider company service assurances. An interviewee suggested, “Make sure what a company is promising you is actually what they deliver.”

Interviewees also suggested benchmarking the system’s performance to comparable size schools and similar national and accreditor reporting requirements. A European Union (EU) interviewee noted:

Things to consider […] especially for triple crown schools are the availability of multiple faculty reporting mechanisms for different accreditation bodies (including your national one), different languages, and a potential U.S.-based peer review team preference.

Finally, interviewees noted the importance of cost-benefit analysis in system selection to ensure the right system for the business school long-term. One US interviewee stated:

Look at the cost benefit ratios because some [faculty data management systems] may be fairly cheap. The question is, do they deliver what you need? And then, some [systems] you will pay a big price for may not be everything they're cracked up [to be].

Two of the smaller schools who purchased a system for their initial accreditation effort were contemplating returning to a more manual process. One US interviewee noted, “If you have a small school, relatively speaking, it might be worth considering whether you can develop something in house and costing that out.”

System implementation.

When implementing a system, both survey respondents and interviewees stressed the importance of having data in a consistent format, free from input errors and clear data mapping into the new system. In total, 39% of participants mentioned implementation problems related to data, including: “In my opinion, the biggest struggle is to have ‘clean’ data (control of data). This is the most time-consuming task” (survey respondent) and “When data is migrated, there are always problems in terms of mapping” (US interviewee).

Schools that used graduate assistants or administrative staff to enter data initially into the system noted similar issues of data accuracy. Particular concern was when the person inputting had limited intellectual contributions knowledge or had unclear instructions on how to code required fields. One interviewer noted:

It is astonishing to me how nuanced this information is […]. there’s some domain knowledge that you just can't get [in] a regular clerk. There is a domain knowledge and don't underestimate the complexity of that.

Survey and interview respondents highlighted the labor intensity and time overrun considerations of a successful implementation process. One interviewee noted, “Just know there’s a huge upfront cost, but it’s got tremendous payoff and be prepared for that upfront cost of time because it is garbage in garbage out.”

Having the institution’s Information Technology (IT) department’s support was another key implementation issue. Most software providers can install their system without an institution’s IT department’s support. However, 30% of interviewees noted that system installation would have been easier with the institution’s IT department’s support. Comments included, “IT staff is critical to help with the technical interface between the company and the school” and “You need someone who understands large, relational databases.” Interviewees also noted the importance of a supportive team for a successful implementation. Comments included – “The biggest thing is to have the backing of your dean and your IT department for a successful implementation” and “It’s important to have a team around you and you don’t try to solve all the issues yourself.”

Faculty engagement.

The three main faculty engagement themes that emerged included: choose a system that aids faculty in their work, implement initial and recurring training that positively engages faculty and create incentives to motivate faculty to use the system properly. Three interviewees discussed setting up custom reports that faculty had to prepare annually. One interviewee noted:

We [asked the vendor] to produce THIS [emphasis added] report that faculty did manually. Once they [faculty] entered the data, the report just came out of the system[…]and just needed annual updates […] saving time.

Interviewees highlighted the importance of creating a positive training environment for faculty to learn the system. A US interviewee offered:

I would put all of our faculty in a [computer] lab and have them all log in [to the system.] and I would have an instructor; just a fun time of like, “hey this is how it works.”

Interviewees also recommended having an individual whom faculty members trust and are comfortable learning from available for one-on-one faculty training. Alternatively, one school put a faculty member from each department on the installation team to advocate for the system and assist with training colleagues.

Finally, interviewees highly recommended building incentives into the system for ongoing faculty compliance. Interviewee suggestions included:

“If [you] use data management system in annual review process, [it’s] likely to get traction with faculty” and “Include faculty activity report as part of a portfolio of things that faculty have to do before their annual appraisal. So, they couldn’t do an appraisal without their data record being updated […]”.

Incentive suggestions included using reports from the new system for awards, endowed chairs, travel funding and other incentives faculty members value. Using only system-generated reports for faculty annual, promotion and tenure reviews also led to faster acceptance and use. One school noted that an annual stipend was provided to faculty who updated their data in the system, but over time this payment could become burdensome to the school and ending the incentive could create animosity among faculty.

Ongoing maintenance.

After a school implements a system, survey respondents and interviewees recommended having multiple employees trained to administer the system. A survey respondent noted, “Have at least two persons that know and work in the system, so it is not so vulnerable.”

Interviewees also noted the importance of having a process for ongoing training to help correct regularly occurring errors and to train new faculty. An EU interviewee offered, “[You] may need to do training more than once a year; may need to do regularly for new faculty members and new functions.”

Another ongoing issue interviewees noted was having proper controls and regular reviews of data and data fields post-installation to ensure data accuracy. One interviewee relayed, “[Schools] should have an administrator in charge of maintaining the software […] and assure data accuracy […] tracking changes and […] maintaining data integrity […] for certain fields, such as faculty qualification status.” Regularly reviewing data is important as one interviewee highlighted, “Faculty might enter certain data, but be careful as they can wreak havoc in the system. It’s like air conditioning. If one person opens a window the whole system can go down.”

Finally, interviewees stressed considering the continuing cost of long-term system use. It is important for schools to have adequate resources to maintain a system beyond the initial accreditation needs as survey results indicated that once a school selects a system, they tended to stay with that system. One interviewee noted, “It’s not just the cost of the system and the implementation, [it’s the] training, the maintenance stuff, and server costs […].”

Use of AOL manual systems.

The interviews provided the opportunity to explore deeper the relationships seen in the quantitative survey data. Interviewees noted national system requirements (“There’s already a lot of regulatory body structures in place around QA in the UK. It’s kind of non-negotiable”), the idiosyncratic nature of AOL (i.e. “[AOL reporting is] Often a mix of university, centrally-provided solutions, and ad hoc business school solutions including old spreadsheets kicking around”) or a manual system that works (i.e. “We have a faculty member who that's [AOL] his deal and works with all the other faculty and they have a reporting system that’s pretty good. We’re small enough and we can manage that whole process”) as reasons for maintaining in-house AOL systems.

System use beyond accreditation.

Interviewees also highlighted how after schools implemented a system reporting requirements become easier to manage and schools were able to use the data for other purposes. Additional uses of the system included “The relational database is a very rich data set of all the things that faculty do right. So, the Dean has the information to talk with the university President and boast about the school” and “You can also mine […] faculty community outreach and student projects that engage with local industry and businesses.”

In conclusion, one interviewee at a school using the same data management system for over a decade noted, “To have all faculty data in one place which can be pulled in various ways, then the system becomes a management tool.”

Discussion

This study fills a gap in the academic and industry literature by providing information on the software systems business schools use for faculty and AOL data management, how the systems are used and guidance on how to select and implement data management software for accreditation purposes. This study’s results provide valuable information in a single location for schools exploring faculty data management systems. Study respondents confirmed the need for guidance to aid schools in the selection and implementation process. Finally, with major software providers’ ownership changes over the past few years, this study provides a needed update to Bisoux’s (2013) work.

The survey results showed that most business schools used data management systems to manage faculty data but used manual or home-grown systems for their AOL processes. Interviewees indicated the investment in setting up software to automate a process that was either highly regulated by a national agency or idiosyncratic was not worth the investment of time and resources.

It is important that schools choose a faculty data management system carefully. The survey results indicated that once a school selects a system, they tended to stay with that system for years with 70% of respondents indicating they do not plan on changing systems in the next three years. This study did not exam the reasons why a school stays with a system, but interviewees discussed the significant time and resource investment to move data to a new system and train faculty and staff. These investments create high procedural switching costs (Burnham et al., 2003) that may influence a school’s decision to stay with their current system.

Finally, this study provided best practice suggestions from respondents who helped select, implement and maintain the faculty management system at their business school. Suggestions for each of the four functions explored in this study are:

1. In choosing a system, understand the school’s system needs and how the system will integrate with other institution-wide systems, vet the companies and systems being considered to ensure they deliver on promises and conduct a cost-benefit analysis to ensure selection of the right system at a cost the school can afford long-term.

2. When implementing a system, ensure properly mapped data to the new system and clean data, have university-level IT support for installation and a good implementation team including Dean’s office support.

3. To engage faculty, choose a system that aids faculty with their work (e.g. creation of custom reports as needed), create a positive training environment for faculty to learn the system and require faculty to use system-generated reports for faculty incentives programs and human resource processes once the system’s fully implemented.

4. For ongoing maintenance, have multiple employees trained in system administration, plan for regular recurring training and perform regular reviews of the data inputted.

There are many items schools need to consider when selecting a system for faculty data management. The survey results and best practice results offered provide a starting point of items to consider, but the business school’s environment, financial position, national requirements and culture are also important considerations when deciding which data management system is most appropriate. This study should help schools be more effective in evaluating systems and more efficient in the resource investments to implement a system.

Limitations

Limitations to this study’s findings include the smaller, homogeneous sample composition from established markets and the dynamic educational technology industry. First, the respondents represent only a small sample of business schools accredited by AACSB or seeking AACSB accreditation, although respondents represented both small and large business schools from 16 different countries. A larger respondent group would provide greater generalizability of results. Second, the study respondents came primarily from countries in developed markets for business schools with strong technological and financial resources and English fluency. Having more business schools from developing regions (e.g. Latin America or Africa), non-English speaking and less well-resourced respondents and recruiting from non-AACSB listservs may offer different results. Finally, the educational technology industry is undergoing consolidation and change as seen in ownership changes among three of the four leading software system providers. Some respondents expressed displeasure with the ownership changes, which may have biased results.

Future research

Future research in this area should involve a larger sample size across a more geographically-diverse group of schools. Future research might explore why a business school stays with a faculty data management platform and why schools change platforms. In addition, future research might explore why schools use a single system versus multiple systems for faculty data management across institutions and integrate multiple systems.

Conclusion

Survey and interview respondents offered advice that business school leaders should consider in the purchase of a faculty data management system. However, there is no magic bullet. A successful AACSB initial or reaccreditation effort requires years of work, careful planning, well-functioning systems and faculty, staff, university leadership and student support. Software is only one small part of the system needed for a successful initial accreditation or reaccreditation effort.

Table 1.

Software system usage and respondent demographics (n = 88)

System used by process
System	Data management process
	Faculty research (%)	Faculty sufficiency (%)	Assurance of learning (AOL) (%)
Faculty success	34.5	32.5	0.0
Academ	16.1	19.3	6.0
Sedona	14.9	15.7	1.5
Interfolio	9.2	10.8	1.5
In-house system	12.6	4.8	37.3
Manual	3.4	15.7	38.8
Other	9.2	1.2	14.9
Totals	100	100	100

Respodent demographics
Respondent role	# of respondents	(%)
Dean/Associate/Assistant Dean	41	46.6
Other administrative	26	29.6
No response	12	13.6
Faculty	8	9.1
Other	1	1.1
Total	88	100.0
Respondent institutional location
USA	45	51.2
UK	10	11.4
France	6	6.8
Canada	3	3.4
Germany	3	3.4
Ireland	3	3.4
Other	9	10.2
No response	9	10.2
Total	88	100.0

Table 2.

Profile of respondents by type of system^a used (n = 88)

System	No. of respondents	Average years use	% US institutions in sample (%)	AACSB accreditation status			Institution profile
				Accredited	Pursuing	No response	FTE business students (average)	FTE business faculty (average)	FTE institution students (average)
Academ	15	3.0	0.0	7	6	2	2,252.9	90.3	16,959.2
Faculty success	30	7.6	85.0	26	3	1	2,737.4	79.9	24,054.1
Interfolio	6	2.1	50.0	3	1	2	3,262.5	60.0	20,500.5
Sedona	15	6.1	73.3	12	3	0	1,952.3	57.8	10,202.2
In-house and manual systems	16	7.3	50.0	8	6	2	2,847.0	110.3	12,315.4
Otherb	6	7.0	0.0	3	2	1	2,337.5	104.8	20,561.5
All	88	6.0	51.1	59	21	8	2,550.2	83.7	17,979.0

Notes:

^a System used by institution for faculty research was used to develop respondent profiles. ^b Other included four different systems: ORCID, Pure, SPOL and Symplectic

Table 3.

Criteria and information used in selecting a software system (n = 88) and satisfaction with existing system

Category	Criteriaa,b			Extremely or very important (%)
Functionality (8)	Reporting capabilities			90.28
	Ease of use			86.12
	Ability for faculty to input own information			75.35
	Customizable			75.00
	Single system for all accreditation reporting			66.66
	Speed to implement system			60.56
	Other college/university systems compatibility			46.48
	Ability to use university-wide			27.53
Financial (2)	Initial system cost			80.56
	Ongoing (Annual) cost			77.78
Vendor (1)	Responsiveness of company			81.95
Peer (1)	System used by other institutions			36.11
Information used in systemb selection
Vendor demonstration				33.33
Colleague(s) referral				27.64
Ability to use university-wide				27.53
Prior data management system experience				21.14
Accreditor’s representative recommendation				9.76
Online reviews				8.13
Software system satisfaction
Would select same platform (n = 75)			Likely to change platform (n = 77)
Faculty research system		76.0	Faculty research system	27.3
Faculty sufficiency system		74.7	Faculty sufficiency system	23.4
AOL system		44.3	AOL system	29.7

Notes:

^a Five-point Likert scale “extremely important” to “not important at all.” ^b Respondents could select multiple items