Abstract
Purpose
The goal of this study is to test the real-world use of an emotion recognition system.
Design/methodology/approach
The researchers chose an existing algorithm that displayed high accuracy and speed. Four emotions: happy, sadness, anger and surprise, are used from six of the universal emotions, associated by their own mood markers. The mood-matrix interface is then coded as a web application. Four guidance counselors and 10 students participated in the testing of the mood-matrix. Guidance counselors answered the technology acceptance model (TAM) to assess its usefulness, and the students answered the general comfort questionnaire (GCQ) to assess their comfort levels.
Findings
Results from TAM found that the mood-matrix has significant use for the guidance counselors and the GCQ finds that the students were comfortable during testing.
Originality/value
No study yet has tested an emotion recognition system applied to counseling or any mental health or psychological transactions.
Keywords
Citation
Gom-os, D.F.K. and Yong, K.Y. (2022), "An empirical study on the use of a facial emotion recognition system in guidance counseling utilizing the technology acceptance model and the general comfort questionnaire", Applied Computing and Informatics, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/ACI-06-2022-0154
Publisher
:Emerald Publishing Limited
Copyright © 2022, Dhong Fhel K. Gom-os and Kelvin Y. Yong
License
Published in Applied Computing and Informatics. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Introduction
Counseling sessions and similar tasks can be challenging to practitioners. On one hand, they need to deal with uncooperative patients [1] and on the other hand, they need to give their patients sound advice. Understanding their patients' current state is essential to give them good advice, which can be daunting. Often, patients do not display their true feelings [1], and this can be hard for practitioners who want no less than understand their patients' situation. They exhaust every possible avenue to extract as much information from their patients as possible by looking at body language, behavior of the eyes and tone of voice or facial expression [2]. One counselor must really be good at juggling these to be effective in his or her task.
To support the task of counseling and similar tasks, facial emotion recognition (FER) system is being proposed. FER models have recently been studied utilizing state-of-the-art machine learning and deep learning approaches [3]. These models were proven to be accurate at recognizing emotions. Some real-world applications with FER models [4–7] have been tested and were proven to be successful and well-received by their users. Since patients tend to display facial cues of their true state and feeling, an FER system will be a suitable and a useful system in accomplishing the task of counseling.
Moreover, two types of facial expressions make the job of a practitioner more difficult, namely, macro-expressions and micro-expressions [8]. Macro-expressions are the obvious and normal facial expressions that humans convey. This type of expression lasts for about half a second or less. It usually coincides with the other ways of expressing emotions such as the content of the conversation and tone of the person’s voice. This type of expression is not difficult to pick up. Micro-expressions, on the other hand, is hard to notice as they occur between one-fifteenth (1/15) or one-fourth (1/4) of a second. However, a camera with a frame rate of as low as 15 frames per second can easily pick up a micro-expression to which counselors may have a difficulty. This makes camera-based emotion recognition an ideal solution.
The objective of this study is to develop an FER system interface as a support mechanism for practitioners in counseling sessions and similar tasks. This system is expected to recognize and display both macro-and-micro-expressions expressed by patients. As a first of its kind, the system will be evaluated by one of its target users – the guidance counselors – using the technology acceptance model (TAM). The researchers believe that it is also important to consider the experience of the patients when subjected to the system. Therefore, the patients' comfort levels are also assessed by letting them answer the general comfort questionnaire (GCQ).
Four of the six basic emotions [9] are considered in the system, namely, joy, anger, surprise and sadness.
2. Literature review
Emotions are exhibited by various identifiable features in the human body. These features include physiological signals (e.g. changes in blood pressure and heart rate), speech patterns (e.g. changes in tone and pitch), electroencephalogram or EEG signals, poses, behavior, habits and voluntary and involuntary actions. These features become the input to emotion recognition systems. They help recognize and classify emotions. Emotion recognition systems are classified into two: unimodal and multimodal systems. Unimodal systems classify emotions using only one identifiable feature. Examples of these systems can be found in Refs. [10–13] which feature FER systems, and in Refs. [14–16] which feature speech, EEG and behavior and posture emotion recognition systems, respectively. Multi-modal systems, on the other hand, classify emotions using two or more identifiable features. In other words, multi-modal systems use more than one unimodal system in a combined effort to classify emotions. One example of a multi-modal system can be found in Ref. [17] which features a lie detector system using facial and voice features.
FER systems are arguably the most popular among emotion recognition systems. It is believed to be the simplest and cheapest [12]. This is maybe because high-end cameras are more ubiquitous and becoming less expensive compared with EEG sensors or even microphones. An FER system takes on an image or video stream input from a camera and processes each image frame to detect face, extract features and recognize the emotion exhibited in the face. Some of the very recent FERs developed are mostly using machine learning techniques such as SVM, CNN and MLP. The study [18] used deep Convolutional Neural Network which gained an average accuracy of 92.81% in classifying the eight basic emotions. This is an improvement to previous studies as it can reduce overfitting by employing a regularization method in fully connected layers. A more recent developed model ConvNet [19] which uses additional three layers, namely, Local Binary Pattern (LBP), Region Based Oriented FAST and rotated BRIEF (ORB), on top of CNN gained a higher accuracy of 98.13% using the same data set. This is the highest that was achieved using the said dataset even higher to a similar system which uses Extreme Learning Machine (ELM) universal approximation characteristic along with the Improved Black Hole algorithm which only achieved 90% [20]. Such model was also tested in real-time and was found to be suitable for a real-time application. While the two previous models [18, 20] achieved good performance, it did not have a publicly available source code that can be used for the purpose of this study. Although [19] has a publicly available code in Python, it was not ready for integration. The model [21] was found to be publicly available in which a ready-for-integration JavaScript API [22] was also available. This model was based on CNN and uses Tiny Detector model [23] for facial detection and SSD MobileNet v1 [24] for emotion classification. Both models achieved an average precision of 82% overall and an accuracy of 85.4% using the FERPlus data set, respectively. This model [21] makes a good candidate for this study.
Major breakthroughs have been made to significantly improve the performance of FER models. Although some of these models [19] were tested in real-time, there is very little study of their application in a real-world setting such as counseling – an aspect in psychology which is heavily “invested” in emotions. Some research was done to investigate the use of an FER in entertainment [4], e-learning [5], politics [6] and even robotics [7], among others. One study [25] investigated the use of an FER where CCTV surveillance is used to recognize and monitor students’ facial expression. Students showing signs of depression and anxiety are flagged by the system for counseling later. This study proves the usefulness of an FER but is limited to only surveillance instead of being used in an actual counseling session. Besides, an issue on privacy may arise out of such a surveillance system. The study also did not consider how the student-subjects feel about being surveilled which this study is trying to address. A more recent empirical study [26] explored the use of emotion visualization techniques for synchronous, in-person medical encounters between a physician and a patient. Thirty-seven physicians and 31 patients evaluated the developed system using the TAM. Findings showed that both physicians and patients are generally accepting of the system. This is a promising result by which this study takes inspiration upon. A similar system employed in counseling may also gain positive acceptance. However, what this study failed to perform is to measure the level of comfort of the patients. The TAM does not measure patient comfort levels, the GCQ does.
3. Materials and methods
To achieve the objectives of this study, the researchers performed the following: (1) survey different FER models and compare their performance as well as the availability of their source codes; (2) determine the functional and design features of the novel interface; (3) design and implement the user interface (UI); (4) integrate the selected FER model and UI into a fully working system; (5) test the system; and (6) ask the participants to answer a survey questionnaire on TAM and GCQ, respectively. Figure 1 shows a flowchart of the activities done throughout the study.
3.1 Survey and comparison of FER models
The researchers dug into the literature to look for an FER model that can be used in the proposed system. Accounting for timeliness, only models published from the past five years were considered. At least seven studies [10, 11, 18–21, 27] were initially considered as candidate models. However, in future studies, the researchers recommend the use of Scopus, the most comprehensive abstract database, to expand the list of candidate models. Working on the seven studies, four criteria were set when choosing for the model to use. First, the model must support the four basic emotions joy, sadness, surprise and anger. Second, the model must have a high classification rate. Classification rate refers to the success rate or correctness of the system's outputs based on the total number of correct classifications the FER has successfully registered. Third, it must be fast enough for real-time use. Lastly, it must have a publicly available source code which can be integrated easily into a web application. The researchers believe that the system would be best developed as a web application. This is considering compatibility and accessibility. In this platform, however, security must be given attention as an online platform is found to be more susceptible to security issues. This is given that the system processes very sensitive information.
All seven models supported the four basic emotions. Also, the seven models have a classification rate of at least 80% except for [27] which has a rate of at most 72.65%. The model [19] has the highest classification rate at 92.81%. In terms of speed, all models have no reported slow classification. Among the seven models, only two [19, 21] have publicly available source codes. Moreover, only [21] has a publicly available source code in the form of a JavaScript API [22] which can be easily integrated into a web application. It is currently under license by MIT. This makes [21] the best candidate to use. The model [21] features automatic extraction of facial features using deep CNN to find Action Units and uses the Extended Cohn Kanade (CK+) data set which has already been validated by experts. The study uses the definition of Ekman and Friesen’s six emotions [9]: happy, sad, surprise, fear, disgust and anger with the addition of two emotions, neutral and contempt. In addition, the model [21] uses the Tiny Face Detector model [23] for face detection which has an average precision of 82% overall. It uses the SSD MobileNet v1 model [24] for emotion classification with an accuracy of 85.4% using the FERPlus dataset.
The model [21] used the following algorithms for facial detection, feature extraction and emotion classification. For facial detection, the system has three pretrained models, namely, Single Shot Multibox Detector (SSD), Tiny Face Detector and Multi-task Cascaded Convolutional Neural Network (MTCNN). SSD has the highest accuracy, Tiny Face Detector has the fastest speed, and MTCNN is for experimental purposes. For this study, Tiny Face Detector was selected for its speed as the system should be running in real-time. For feature extraction, a CNN algorithm trained from a dataset of about 35k facial images was used. The algorithm returns 68 facial landmarks. For emotion classification, the system was trained based on the extracted features of facial shapes. A confidence threshold of 0.8 and above was set for the emotion recognition. This threshold was manually determined by the researchers after trial and error. It is recommended, however, that for future work, a calibration mechanism be employed as different people portray different levels and characteristics of emotions.
3.2 Identification of functional and design features
After choosing the FER model to use, the system functional and design features were then identified. A functional feature or requirement, in software engineering and systems engineering, defines a function of a system or its component, where a function is described as a specification of behavior between inputs and outputs [28]. In essence, it is what the system is supposed to accomplish. A design feature or requirement, on the other hand, focuses on the style or look of the system. This is not the design of the user interface per se, but it addresses the question as to what feature would help the counselors or users better read patients’ emotions on the screen. This is especially that emotions may show on a patient’s face within split seconds and counselors must be quick enough to read them. Only the researchers determined such features through assumptions and hypotheses. However, a better and more efficient way of gathering these features is to involve the users themselves during the process as what [26] did. In this way, the users have a say on what they think are the important functional and design features to include. This strategy will most likely increase the acceptability of the system.
There are two identified functional features. First, a camera takes an image of a patient’s face and displays to the counselor the type of emotion found on the face. Second, two sets of intensities (active or inactive) are displayed on the screen corresponding to the emotion detected by the system. Though currently limited, the researchers believe that these two basic features would help the counselors in their counseling sessions. It allows them to check and verify the emotion currently being felt by their patients. The researchers recommend, however, extending these features to show the fluctuation of a patient’s emotions throughout the session. This can provide more useful information to counselors during sessions. In addition, display of emotion intensities may be expanded to include different levels of emotions based on Plutchik’s wheel of emotions [29].
There is only one design feature implemented in the system, i.e. the use of colors that fit and mimic the feel and vibe of each individual's emotions. It was shown that two common stimulants directly affect emotion, namely, art and music. The mood of a person can be altered by music [30]. On the other hand, the mood and feel of a painting, artwork or environment can be established by the colors used by artists [31]. For example, if artists want to convey feelings of sadness, the color and shades of blue are often used. In this study, the green color is used to associate with “joy”, red with “anger”, blue with “sad” and yellow with “surprise.” Mathematically, this is expressed as
3.3 UI design and implementation
The design of the UI was carried out using pen and paper. Currently, only one UI was designed and implemented, i.e. the main display. The elements found in the display include the four visual cues representing the four basic emotions supported by the system, each located at the four corners of the display – starting with “joy” at the top left corner, then “angry,” “surprise,” and “sad” in a clockwise order as shown in Figure 2. The layout resembles a 2 × 2 matrix which is why the researchers call the system the mood-matrix. The four visual cues, which the researchers call mood markers, are also incorporated with emoticon symbols [33] corresponding to the emotions they represent. The colors, in hex format, used for the eyes, mouth and outer section of the mood markers are
3.4 Integration
The designed UI and the selected FER model were then integrated into a working web application developed using HTML, CSS and JavaScript. The system is solely a front-end system as there is no need for a back-end component yet. Integrating the UI and the model was straightforward as the FER model used a JavaScript API [22] that was readily available. For future work, a back-end component may be introduced to store historical information such as session recordings and emotion detections over time in a database.
3.5 Testing
Initial testing was done by the researchers to test the functionality of the system. The system was tested and was able to successfully run in Google Chrome (version 103.0.5060.134) and Microsoft Edge (version 104.0.1293.47) web browsers. It is not known if whether the system will run in a mobile application or any other browser as the API [22] also did not report any such limitations or otherwise. During the initial testing, it was found that the API implements the single face detection algorithm by default. It was observed that this algorithm slows down the detection speed. This is maybe because it performs an exhaustive scan of all possible faces in the image input and returns the detected face with the highest score. Therefore, the researchers opted to use the multiple face detection algorithm, which was found to be faster and smoother to operate. This algorithm scans for multiple faces in the image. But the researchers re-programmed it to only process facial features from the first face detected making the operation faster. Using this updated algorithm, even if somebody passes by behind the subject, the system will still focus on the subject rather than the passerby. This is if the subject’s face was detected first. Nevertheless, this can be improved to ensure that only the subject’s face is being focused by evaluating the subject’s proximity from the camera. This will eliminate the presumption that the subject’s face must be detected first. The initial testing went smoothly and only the modification described previously was made. Since it was a very simple web application, there was not much to work on after it was confirmed that the system worked fine.
The system was tested by four guidance counselors and 10 student-patients in a university library. The testing setup is shown in Figure 3. The guidance counselor and student-patient sit in front of each other with the camera facing the student-patient and the laptop computer facing the guidance counselor. The camera used is a Foscomax Motion Tracking USB Webcam with 1080p video capture resolution at 30fps and has a USB plug and play feature. In addition, it has a special low-light feature capable of auto-dimming and color corrections under bad lighting conditions. It also has smooth video quality which optimizes the images and videos automatically giving the best performances in video situations with a 10 – 300 cm high-precision autofocus. The computer used is a Huawei MateBook D14 laptop which has a 14-inch 1920 × 1080 pixels display, an AMD Ryzen 7 3700U processor and an 8GB RAM with 500GB storage. During pandemic, the university library has no constant visitors, so this made the testing run smoothly with no issues as if the counseling session was held in a closed-door private room.
The testing was done in a one-on-one counseling session with four guidance counselors paired with student-patients. Two guidance counselors were paired with three student-patients at a time while the other two were paired with two student patients at a time. The testing lasted for two weeks, and each counseling session lasted for an average of 45 minutes to one hour. A consent form was signed by each participant in compliance with the norms and ethical standards of conducting research with human participation.
3.6 Survey
After each session, the guidance counselors and student-patients answered the TAM and custom GCQ, respectively. Given that two guidance counselors were paired with three student-patients while the other two were paired with two student patients, there were a total of 20 answered questionnaires. Guidance counselors answered TAM multiple times for a total of 10 sets of responses. In this way, guidance counselors will have a chance to change their responses as using a new system may require time to familiarize. TAM was developed by Davis in 1989 [35] and is a reliable model for assessing the perceived acceptability of a system through a Likert scale [36]. It assesses the system’s perceived ease of use (PEU), perceived usefulness (PU), attitude toward usage (ATU) and behavioral intention to use (BIU). Kolcaba’s GCQ [37] was used to measure the student-patients’ level of comfort during the session. GCQ was built as an instrument to measure the comfort of patients and to identify positive and negative aspects of health care provided to patients regardless of health condition. Its original version contains 48 items covering physical, spiritual, environmental and social dimensions. A custom GCQ was used containing only 14 relevant questions.
4. Results and discussions
This section discusses the results of the TAM and GCQ surveys.
4.1 Guidance counselors’ acceptance of the mood-matrix based on the TAM
The TAM is a descriptive survey; thus, the method of descriptive statistics applies here. A total of 10 responses were gathered but only the final four responses of the four guidance counselors are assessed. With the small number of guidance counselors joining the study, raw numbers will be assessed, analyzed, and interpreted. Each sub-section below presents in a table the responses of the counselors to each TAM categories.
4.1.1 Perceived usefulness (PU)
PU is described as the “prospective user’s subjective probability that using a specific application system will increase his or her job performance within an organizational context [36].” In this study, PU is defined as the degree to which the guidance counselors believe that using the mood-matrix would enhance their job performance. Table 1 below presents the responses to the PU category.
4.1.2 Perceived ease of use (PEU)
PEU is defined as the “the degree to which the prospective user expects the target system to be free of effort [35].” Here, PEU is the degree to which the guidance counselors believe that using the mood-matrix requires relatively less effort. Table 2 below presents the guidance counselors’ responses to the PEU category.
4.1.3 Attitude toward usage (ATU)
ATU represents the “individual’s evaluative feelings (positive or negative) when performing a particular behavior [36].” This determines the intention of the guidance counselors to use the mood-matrix. Table 3 below presents the guidance counselors’ responses to the ATU category.
4.1.4 Behavioral intention to use (BIU)
BIU is defined as “the actual use of a given program and therefore determines technology acceptance [36].” This criterion simply means that the guidance counselors have a drive to apply or use the mood-matrix. Table 4 below presents the guidance counselors’ responses to the BIU category.
4.1.5 Discussion on the results of the TAM
Table 1, which shows the PU category results, showed that the guidance counselors have mixed reactions to the ease of using the mood-matrix, especially on items 1 and 2. Item 1 refers to the difficulty of which a user can work without the use of the mood-matrix, while item 2 refers to how much control they have in their work when using the mood-matrix. Two guidance counselors disagreed to item 1 and one guidance counselor disagreed to item 2. This implies that it would be difficult to perform and to take control of their jobs when using the system. The reason is that they are simply too skilled and professional to read the emotions of their patients even without the help of the system. One guidance counselor disagreed with item 6 expressing that the mood-matrix makes their job slower to finish. This is mainly due to the constant reminder to glance at the screen, to which the researchers do agree. Apart from these, the mood-matrix has garnered positive feedback overall from the guidance counselors and thus has a good score in the PU category.
As shown in Table 2 for the PEU, it is shown that the mood-matrix interface is easy to use. It has a good score in this category although there are some minor setbacks. Item 6 shows that one guidance counselor finds it hard to recover from the system’s mistakes. Item 8 finds that two guidance counselors had a hard time controlling the system on what they want it to do. Item 9 also finds that half of the guidance counselors found that it sometimes behaves in unexpected ways, although it was not a serious issue, commenting that “there is a ghost face” when it tries to scan for faces with no plain background on the screen. This can be easily addressed by having the patient sit behind a plain wall. Nonetheless, the mood-matrix garnered positive results overall in the PEU category.
Shown in Table 3 are the responses for the ATU category. The mood-matrix has a very good score in this category, although not perfect. All guidance counselors find that the system is a good program to use for their work. Additionally, they like the idea of using it as a support mechanism for reading their patients’ emotions and that it is a positive idea to use it in their everyday work.
Lastly, the mood-matrix has a good score in the BIU criteria as shown in Table 4. All guidance counselors find that they would want to use the mood-matrix in the future. Additionally, if they have access to the system, they will want to use it in their future sessions.
In conclusion, the mood-matrix generally has a positive score in most of the criteria in the TAM and therefore can be a good system to use in the future in guidance counseling and similar tasks. Additional comments to the mood-matrix include the following: (1) the outputs of the system are consistent with the context of the topics that are being discussed by the patients during the session; (2) the color and icon used may be more conspicuous/visible, and the placement of the mood markers should be closer to the center; (3) the client/patient seems to fixate more on the camera instead of the guidance counselor but is overall helpful in detecting mood changes and variety; and (4) the mood-matrix is a recommended and practical program to use.
4.2 Patients’ comfort level when subjected to the mood-matrix based on the GCQ
The patients’ comfort levels may determine how well they displayed and emitted true emotions to the guidance counselors. Thus, their comfortability was assessed. In addition, since guidance counseling and similar tasks deal with sensitive information such as emotions, the researchers believe that it is important to gather patient’s feedback to the system. Table 5 shows the responses of the student-patients to the GCQ.
Normally, before a custom GCQ is used, the calculation of the Cronbach's alpha is performed after a pre-testing to determine its internal consistency [38]. Equation (1) below shows the computation of Cronbach's alpha.
4.2.1 Discussion on the results of GCQ
Despite the poor reliability of the GCQ, the researchers still proceeded to describe its results the same way the TAM was examined. This is given the nature of the GCQ which is descriptive. From Table 5, it is found that most of the patients were comfortable during the sessions although some exceptions were noted. For example, item 2 finds that one patient did not find the library to have enough privacy. In addition, one patient was constipated during the study which indicates discomfort. Two patients did not feel healthy at the time of testing and one patient was feeling out of control. There are also other variables such as room temperature that were not collected in the GCQ which may have affected the comfortability of the patients. However, overall, the patients are generally comfortable when subjected to the system. It is undeniable though that the computed Cronbach alpha score was not good. This may be reconsidered, and the system may be retested and reevaluated in the future to improve the reliability of the evaluation instrument.
5. Conclusion
The researchers performed five steps throughout this investigation. First is choosing an FER system that yields good accuracy and speed with a source code that is publicly available and easily integrable in a web platform. CNN was chosen as it was the model that obtained a good accuracy and had a model that was open-source JavaScript API. Second, the researchers determined and implemented functional and design features for the FER system. These features were then implemented as a web interface. Third, the FER interface is integrated with the open-source code. Fourth, the mood-matrix was tested by guidance counselors and student-patients. The mood-matrix is then evaluated by the guidance counselors for its usability through the TAM survey. Finally, the comfortability of patients is assessed using the custom GCQ to determine if patients are fine when subjected to the mood-matrix. This study provides insight to future researchers on the use of an FER in guidance counseling and similar tasks. Overall, the results are promising as it was shown that an FER system is an acceptable system in the field of guidance counseling in particular and maybe psychology in general.
6. Future works
Future researchers could investigate the use of multimodal systems which was shown to improve classification rate by 20% [39]. The two emotional intensities supported in this study may be expanded further. Sonification [40] may also be studied to incorporate sound effects or music as a way of relaying emotional information to the users. A calibration mechanism may be introduced to automatically adjust the threshold of detections. Screen recording feature as well as a graph showing the fluctuations of emotions over time may be integrated as recommended by the guidance counselors.
Figures
Perceived usefulness (PU)
| No. | Item | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Total |
|---|---|---|---|---|---|---|---|
| 1 | My job would be difficult to perform without the mood-matrix | 0 | 2 | 1 | 1 | 0 | 4 |
| 2 | Using the mood-matrix gives me greater control over my work | 0 | 1 | 1 | 1 | 1 | 4 |
| 3 | Using the mood-matrix improves my job performance | 0 | 0 | 0 | 3 | 1 | 4 |
| 4 | The mood-matrix system addresses my job-related needs | 0 | 0 | 1 | 3 | 0 | 4 |
| 5 | Using the mood-matrix saves me time | 0 | 0 | 2 | 2 | 0 | 4 |
| 6 | The mood-matrix enables me to accomplish tasks more quickly | 0 | 1 | 0 | 2 | 1 | 4 |
| 7 | The mood-matrix supports critical aspects of my job | 0 | 0 | 0 | 2 | 2 | 4 |
| 8 | Using the mood-matrix allows me to accomplish more work than would otherwise be possible | 0 | 0 | 2 | 1 | 1 | 4 |
| 9 | Using the mood-matrix reduces the time I spend on unproductive activities | 0 | 0 | 2 | 1 | 1 | 4 |
| 10 | Using the mood-matrix enhances my effectiveness on the job | 0 | 0 | 0 | 2 | 2 | 4 |
| 11 | Using the mood-matrix improves the quality of the work I do | 0 | 0 | 0 | 2 | 2 | 4 |
| 12 | Using the mood-matrix increases my productivity | 0 | 0 | 0 | 3 | 1 | 4 |
| 13 | Using the mood-matrix makes it easier to do my job | 0 | 0 | 1 | 2 | 1 | 4 |
| 14 | Overall, I find the mood-matrix system useful in my job | 0 | 0 | 0 | 3 | 1 | 4 |
Perceived ease of use (PEU)
| No. | Item | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Total |
|---|---|---|---|---|---|---|---|
| 1 | I often become confused when I use the mood-matrix | 0 | 3 | 1 | 0 | 0 | 4 |
| 2 | I make errors frequently when using the mood-matrix | 0 | 3 | 1 | 0 | 0 | 4 |
| 3 | Interacting with the mood-matrix system is often frustrating | 0 | 4 | 0 | 0 | 0 | 4 |
| 4 | I need to consult the user manual often when using the mood-matrix | 0 | 4 | 0 | 0 | 0 | 4 |
| 5 | Interacting with the mood-matrix system requires a lot of my mental effort | 0 | 3 | 1 | 0 | 0 | 4 |
| 6 | I find it easy to recover from errors encountered while using the mood-matrix | 0 | 1 | 1 | 2 | 0 | 4 |
| 7 | The mood-matrix system is rigid and inflexible to interact with | 0 | 3 | 1 | 0 | 0 | 4 |
| 8 | I find it easy to get the mood-matrix system to do what I want it to do | 0 | 2 | 1 | 1 | 0 | 4 |
| 9 | The mood-matrix system often behaves in unexpected ways | 0 | 2 | 2 | 0 | 0 | 4 |
| 10 | I find it cumbersome (slow/complicated/inefficient) to use the mood-matrix | 0 | 4 | 0 | 0 | 0 | 4 |
| 11 | My interaction with the mood-matrix system is easy for me to understand | 0 | 0 | 0 | 4 | 0 | 4 |
| 12 | It is easy for me to remember how to perform tasks using the mood-matrix system | 0 | 0 | 0 | 4 | 0 | 4 |
| 13 | The Mood-matrix system provides helpful guidance in performing tasks | 0 | 0 | 0 | 4 | 0 | 4 |
| 14 | Overall, I find the mood-matrix system easy to use | 0 | 0 | 0 | 4 | 0 | 4 |
Attitude towards usage (ATU)
| No. | Item | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Total |
|---|---|---|---|---|---|---|---|
| 1 | I believe it is a good idea to use the mood-matrix | 0 | 0 | 0 | 2 | 2 | 4 |
| 2 | I like the idea of using the mood-matrix | 0 | 0 | 0 | 2 | 2 | 4 |
| 3 | Using the mood-matrix is a positive idea | 0 | 0 | 0 | 2 | 2 | 4 |
Behavioral intention to use (BIU)
| No. | Item | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Total |
|---|---|---|---|---|---|---|---|
| 1 | I plan to use the mood-matrix in the future | 0 | 0 | 0 | 3 | 1 | 4 |
| 2 | Assuming that I have access to the mood-matrix, I intend to use it | 0 | 0 | 0 | 3 | 1 | 4 |
Custom general comfort questionnaire (GCQ)
| No. | Item | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Total |
|---|---|---|---|---|---|---|---|
| 1 | My body is relaxed right now | 0 | 0 | 1 | 4 | 5 | 10 |
| 2 | I have enough privacy | 0 | 1 | 0 | 5 | 4 | 10 |
| 3 | I feel confident | 0 | 0 | 0 | 6 | 4 | 10 |
| 4 | The current surroundings are pleasant | 0 | 0 | 0 | 4 | 6 | 10 |
| 5 | I do not like it here | 7 | 3 | 0 | 0 | 0 | 10 |
| 6 | I am constipated right now | 5 | 4 | 0 | 0 | 1 | 10 |
| 7 | I do not feel healthy right now | 3 | 4 | 1 | 2 | 0 | 10 |
| 8 | This room makes me feel scared (uneasy) | 6 | 4 | 0 | 0 | 0 | 10 |
| 9 | I would like to see my counselor more often | 0 | 0 | 0 | 8 | 2 | 10 |
| 10 | The mood around here uplifts me | 0 | 0 | 0 | 8 | 2 | 10 |
| 11 | This view inspires me | 0 | 0 | 2 | 5 | 3 | 10 |
| 12 | I feel out of place here | 7 | 2 | 1 | 0 | 0 | 10 |
| 13 | I feel out of control | 6 | 3 | 0 | 1 | 0 | 10 |
| 14 | I feel peaceful | 0 | 0 | 1 | 3 | 6 | 10 |
References
1Boitt ML. Evaluation of the challenges in the implementation of the guidance and counselling programme in Baringo county Secondary Schools, Kenya. IISTE J Educ Pract. 2016; 7(30): 24-6. Available from: https://files.eric.ed.gov/fulltext/EJ1118929.pdf.
2Anake P, Asor L. Gestures in guidance and counselling and their pedagogical/androgogical implications. Glob J Educ Res. 2013; 11(2). doi: 10.4314/gjedr.v11i2.1.
3Khan AR. Facial emotion recognition using conventional machine learning and deep learning methods: current achievements, analysis and remaining challenges. Information. 2022; 13(6): 268. doi: 10.3390/info13060268.
4Sana SK, Sruthi G, Suresh D, Rajesh G, Subba Reddy GV. Facial emotion recognition-based music system using convolutional neural networks. Mater Today Proc. 2022; 62: 4699-706. doi: 10.1016/j.matpr.2022.03.131.
5Pise A, Vadapalli H, Sanders I. Facial emotion recognition using temporal relational network: an application to E-learning. Multimedia Tools Appl. 2020; 81(19): 26633-53. doi: 10.1007/s11042-020-10133-y.
6Rodríguez-Fuertes A, Alard-Josemaría J, Sandubete JE. Measuring the candidates' emotions in political debates based on facial expression recognition techniques. Front Psychol. 2022; 13. doi: 10.3389/fpsyg.2022.785453.
7Devaram RR., Beraldo G, de Benedictis R, Mongiovì M, Cesta A. LEMON: a lightweight facial emotion recognition system for assistive robotics based on dilated residual convolutional neural networks. Sensors. 2022; 22(9): 3366. doi: 10.3390/s22093366.
8Micro Expressions | Facial Expressions 2020; Paul Ekman Group [Cited 2021 Oct 11]. Available from: https://www.paulekman.com/resources/micro-expressions/.
9Kowalska M, Wróbel M. Basic emotions. In: Encyclopedia Personal Individual Differences. 2017: 1-6. doi: 10.1007/978-3-319-28099-8_495-1.
10Tarnowski P, Kołodziej M, Majkowski A, Rak RJ. Emotion recognition using facial expressions. Proced Comput Sci. 2017; 108: 1175-84. doi: 10.1016/j.procs.2017.05.025.
11Deshmukh RS, Jagtap V, Paygude S. Facial emotion recognition system through machine learning approach. In: International Conference on Intelligent Computing and Control Systems (ICICCS); 2017. doi: 10.1109/iccons.2017.8250725.
12Jens SA. Can you see it? Facial expression leakage in response to emotional intensity. Undergraduate Honors Theses. 2017. Paper 1124. Available from: https://scholarworks.wm.edu/honorstheses/1124.
13JonathanLim AP, PaolineKusuma GP, Zahra A. Facial emotion recognition using computer vision. In: 2018 Indonesian Association for Pattern Recognition International Conference (INAPR); 2018. p. 46-50. doi: 10.1109/INAPR.2018.8626999.
14Kerkeni L, Serrestou Y, Mbarki M, Raoof K, Mahjoub M. Speech emotion recognition: methods and cases study; 2018: 175182. doi: 10.5220/0006611601750182.
15Garg D, Verma GK. Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet-based deep learning framework. ScienceDirect. 2020. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1877050920310644.
16Calbi M, Langiulli N, Siri F, Umiltà MA, Gallese V. Visual exploration of emotional body language: a behavioural and eye-tracking study. Psychol Res. 2021; 85(6): 2326-39. doi: 10.1007/s00426-020-01416-y.
17Soumya BC. Lie detection based on facial micro expression, body language, and speech analysis. Int J Eng Res. 2016; V5(02). doi: 10.17577/ijertv5is020336.
18Liliana DY. Emotion recognition from facial expression using deep convolutional neural network. J Phys Conf Ser. 2019; 1193: 012004. doi: 10.1088/1742-6596/1193/1/012004.
19Debnath T, Reza MM, Rahman A, Beheshti A, Band SS, Alinejad-Rokny H. Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity. Sci Rep. 2022; 12(1). doi: 10.1038/s41598-022-11173-0.
20Deeb H, Sarangi A, Mishra D, Sarangi SK. Human facial emotion recognition using improved black hole based extreme learning machine. Multimedia Tools Appl. 2022; 81(17): 24529-52. doi: 10.1007/s11042-022-12498-8.
21Espinosa Sandoval C. Multiple face detection and recognition system design applying deep learning in web browsers using JavaScript. Computer Science and Computer Engineering Undergraduate Honors Theses. 2019. Available from: https://scholarworks.uark.edu/csceuht/74.
22Face Api.js. Awesomeopensource.Com; n.d. [Cited 2022 May 31]. Available from: https://awesomeopensource.com/project/justadudewhohacks/face-api.js#face-api.js-forthe-browse.
23Hu P, Ramanan D. Finding Tiny faces. arXiv. 2016. Available from: https://arxiv.org/abs/1612.04402.
24Nan Y, Ju J, Huo Q, Zhang H, Wang B. A-MobileNet: an approach of facial expression recognition. Alexandria Eng J. 2022; 61(6): 4435-44.
25Sinha S, Mishra SK, Bilgaiyan S. Emotion analysis to provide counseling to students fighting from depression and anxiety by using CCTV surveillance. In: Swain D, Pattnaik P, Gupta P (Eds). Machine learning and information processing. Advances in intelligent systems and computing, Vol. 1101. Singapore: Springer; 2020. doi: 10.1007/978-981-15-1884-3_8.
26Ma H, Sun X, Lawson G, Wang Q, Zhang Y. Visualising emotion in support of patient-physician communication: an empirical study. Behav Inf Technol. 2022: 1-19. doi: 10.1080/0144929x.2022.2097954.
27Ko BC. A brief review of facial emotion recognition based on visual information. Sensors (Basel, Switzerland). 2018; 18(2): E401. doi: 10.3390/s18020401.
28Fulton R, Vandermolen R. Chapter 4: requirements - writing requirements. Airborne Electron Hardware Des Assur A Practitioner's Guide RTCA/DO-254, CRC Press. 2017: 89-93. ISBN: 9781351831420 [Cited 2018 June 15].
29Abbasi M, Beltiukov A. Summarizing emotions from text using Plutchik's wheel of emotions; 2019; 166: 291-94. doi: 10.2991/itids-19.2019.52.
30Strongman KT. Emotional experience — a review. Curr Psychol Rev. 1981; 1(1): 17-33. doi: 10.1007/bf02979252.
31Kurt S, Osueke KK. The effects of color on the moods of college students. SAGE Open. 2014; 4(1): 2158244014525423. doi: 10.1177/2158244014525423.
32Jordan S. Vivaldi and the four seasons: interpretations of the music and sonnets compared to societal views by Sarah Jordan; 2014. Available from: https://www.academia.edu/23852477/Vivaldi_and_The_Four_Seasons_Interpretations_of_t he_Music_and_Sonnets_Compared_to_Societal_Views_by_Sarah_Jordan.
33Font Awesome [Cited 2022 Aug 14]. Available from: https://fontawesome.com.
34CSS linear-gradient() function n.d. W3schools. [Cited 2022 Aug 14]. Available from: https://www.w3schools.com/cssref/func_linear-gradient.asp.
35Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989; 13(3): 319-40. doi: 10.2307/249008.
36Bayrameepoor A, Leyaaghatdaar MJ, Mostafa SS. An opinion survey on the usefulness of the learning, teaching, and evaluating processes employed in the high school English language curriculum. Journal of New thoughts on Education. 2009; 4(4): 73-94.
37Ajzen I, Fishbein M. Attitudes and the attitude-behavior relation: reasoned and automatic processes. In: Stroebe W, Hewstone M (Eds). European review of social psychology. John Wiley & Sons; 2000. p. 1-33. doi: 10.1080/14792779943000116.
38Melo GAA, Silva RA, Pereira FGF, Caetano JÁ. Cultural adaptation and reliability of the general comfort questionnaire for chronic renal patients in Brazil. Revista Latino-Americana de Enfermagem. 2017; 25. doi: 10.1590/1518-8345.2280.2963.
39Meftah IT., Le Thanh N, Ben Amar C. Multimodal approach for emotion recognition using a formal computational model. Int J Appl Evol Comput. 2013; 4(3): 11-25. doi: 10.4018/jaec.2013070102.
40Burdick KJ, Jorgensen SK, Combs TN, Holmberg MO, Kultgen SP, Schlesinger JJ. Savior ICU: sonification and vibrotactile interface for the operating room and intensive care unit. J Clin Monit Comput. 2020; 34(4): 787-96. doi: 10.1007/s10877-019-00381-1.