Abstract
Purpose
This study is to establish the validity and reliability of Malay version of Post-traumatic Check List-5 (MPCL-5) among the fire and rescue officers in the state of Selangor, Malaysia.
Design/methodology/approach
A cross-sectional study was conducted, which involved 100 firefighters from the state of Selangor, Malaysia. Construct validity, internal consistency, and concurrent validity were performed and analyzed using Statistical Package for Social Sciences (SPSS) version 20. Concurrent validity was tested with validated Malay version of Trauma Screen Questionnaire (TSQ-M).
Findings
Overall internal consistency reliability was a 0.960 and individual construct Cronbach's alpha ranged from 0.827 to 0.926. The model, which consists of four constructs with 20 items, demonstrated the presence of acceptable loading factors. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) are 0.81, 0.65, 0.31 and 0.95 respectively at an optimum cut-off score of 35.
Research limitations/implications
The Post Traumatic Check List 5 (PCL-5) is the latest tool based on DSM-5 developed recently and still having limited studies on the psychometric properties of the tool in local population and the findings produced are comparable with the results from validation from previous studies. The study limitations are population samples used are considering the minimum numbers of sample for each item for factor analysis and the concurrent validation was tested with the TSQ-M instead of the Clinician Administered PTSD Scale for DSM-5 (CAPS-5).
Practical implications
The study suggested that MPCL-5 is acceptable to be used to measure post-traumatic stress disorder in local populations.
Originality/value
There are limited known validation studies for PCL-5 in local populations and this is the first study done among fire and rescue officers in Malaysia. The results are comparable with findings from previous studies and therefore MPCL-5 are valid and reliable for PTSD screening.
Keywords
Citation
Mat Salleh, M.N., Ismail, H. and Mohd Yusoff, H. (2021), "Reliability and validity of a post-traumatic checklist-5 (PCL-5) among fire and rescue officers in Selangor, Malaysia", Journal of Health Research, Vol. 35 No. 4, pp. 379-387. https://doi.org/10.1108/JHR-11-2019-0243
Publisher
:Emerald Publishing Limited
Copyright © 2020, Muhammad Naim Mat Salleh, Halim Ismail and Hanizah Mohd Yusoff
License
Published in Journal of Health Research. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
Introduction
Firefighters are emergency workers who are repeatedly exposed to traumatic events during the course of their services. Their roles are characterized by high levels of work demands and routine exposure to both physical and psychological stressors [1]. A recently published meta-analysis of the worldwide current prevalence of post-traumatic stress disorder (PTSD) in rescue workers yielded a prevalence of approximately 10% for PTSD, suggesting about 150,310 first responders may meet criteria for current PTSD, and of that, 7% of current firefighters suffer from PTSD [2]. A different study undertaken in Australia among current firefighters and retired firefighters showed a PTSD prevalence of 8% and 18% respectively [3].
In the Malaysian setting, firefighters are involved in various tasks, some of which may include traumatic events or critical incidents. They are at the front line for emergency cases and various rescue activities such as Emergency Medical Rescue Services (EMRS), Hazardous Material Unit Teams (HAZMAT), road traffics accidents, high-rise rescues, cliff rescues and water rescues. These high-risk tasks, for the most part, involve experiencing or witnessing critical incidents or traumatic events, which will carry the risk of developing psychiatric illnesses, particularly PTSD.
The Clinician-Administered PTSD Scale for DSM-V (CAPS-5) is currently the gold standard in PTSD assessment by psychiatrists; however, it is labor-intensive and the assessor requires rigorous training. Nevertheless, there are many self-reported tools available for screening purposes, which adopt fewer items and take up less time. Recently, the American Psychiatric Association (APA) developed PCL-5, which was adapted from the previous PCL-C and -M (DSM-IV) in 2013. There is a growing number in validation studies for this tool, yet, to date, there are limited reliability and validity studies across different populations. Therefore, this study aimed to establish reliability, construct validity, and concurrent validity for the Malay version of the PCL-5 (MPCL-5) among fire and rescue officers in the state of Selangor, Malaysia.
Method
Participants
Five observations were needed for each item to be applied before the sample size could be determined [4]. Therefore, 100 fire and rescue officers in Selangor, who were from operational teams with at least 1-month duration of service, were conveniently selected.
Measures
Sociodemographic
All demographic variables were collected, including age, gender, ethnicity, marital status and household income. Additional questions regarding the duration of service were also surveyed.
Malay version of post-traumatic stress checklist-5 (MPCL-5)
The self-administered Malay language version of the PCL-5 (MPCL-5) was used to measure the current status of probable PTSD, which was translated and validated by a previous researcher in a different study [5]. The PCL-5 consists of 20 items corresponding to the DSM-V for post-traumatic stress disorder (PTSD) symptoms. Each item has a five-point rating scale ranging from 0 (not at all) to 4 (extremely). Thus, the scale yields a cumulative score of 0–80. A score of 33 and above is taken as the cut-off point for positive signs of PTSD [6]. The PCL-5 consists of four clusters of symptoms, represented by item questions 1–5 (A: Intrusion), item questions 6–7 (B: Avoidance), item questions 8–14 (C: Negative states), and item questions 15–20 (D: Arousal).
Trauma Screen Questionnaire Malay version (TSQ-M)
The Trauma Screen Questionnaires (TSQ) is a 10-item symptom screen that was designed for screening PTSD, particularly among survivors of all types of traumatic stress. The TSQ is derived from the PTSD Symptom Scale–Self Report (PSS-SR), based on DSM-IV [7], and has five re-experiencing items and five arousal items. The cut-off score of TSQ-M used is any 5 out of 10 symptoms is taken as positive for PTSD, as it has optimum sensitivity and specificity [8].
Ethical consideration
This study was conducted under the ethical approval of the Research and Ethics Committee, Medical Faculty of National University of Malaysia (FF-495-2017).
Analysis
Reliability was evaluated by testing the overall Cronbach's alpha and the individual cluster of the four clusters of PCL-5. Construct validity was examined through principal component factor analysis with varimax rotation. Kaiser-Meyer-Olkin (KMO) and Bartlett's test of sphericity were used to test for the appropriateness of factor analysis. In this study, the sample was adequate, as the KMO value was 0.91. The eigenvalue greater than 1 was used to determine the number of factors to be extracted while loading factor values of 0.4 and below were suppressed. Concurrent validity was established by analyzing the PCL-5 Malay version with a validated TSQ-M to produce the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). In this study, a translated Malay version of TSQ (TSQ-M) was used as the gold standard, where it had been validated in a local sample population and other studies with good reliability and validity [9–13] and yielded high levels of sensitivity and specificity compared to those of the CAPS diagnostic interview [14]. Convergent validity was tested using the Pearson correlation between the total scores of the two instruments. All data were analyzed using the Statistical Package for Social Sciences (SPSS) version 20.
Results
Table 1 shows the characteristics of participants. The mean age of participants was 33.3 (±7.9) and 10.8 (±7.8) for years of service. The majority of them were male 96 (96%), of Malay ethnicity 93 (93%), who were single 74 (74%) and with a household income between 2001 and 3000 RM, or Ringgit Malaysia, the national currency.
Comparison between MPCL-5 and TSQ-M
The frequency of PTSD diagnosis made using the two instruments was lower in TSQ-M. It was found that the diagnosis of PTSD was 42% (n = 42) of the sample when using MPCL-5 compared to only about 16% (n = 16) when using TSQ-M.
Reliability
Internal consistency
Generally, MPCL-5 showed a strong internal consistency, with a Cronbach's alpha value of 0.96. Itemized analysis of the different clusters' symptoms also revealed that the Cronbach's Alpha values in all domains ranged from 0.83 to 0.93 (Table 2).
Table 3 showed total items statistics. Overall, the items indicated a satisfactory correlation with the corrected items' total score. Lowest values were seen in items A5 and D16. Cronbach's alpha values of deleted items range from 0.956 to 0.959.
Construct validity
Factor analysis showed four-factor components as shown in Table 4. The loading factors were acceptable, ranging from 0.462 to 0.875, and 76.9% of the variance was explained. The scree plot showed that four components had eigenvalue of at least one or more (Figure 1).
Concurrent validity
Sensitivity and specificity
Receiver operating characteristic (ROC) (Figure 2) analyses were applied to compare the screening performance of the MPCL-5 by using TSQ-M as the validated test for PTSD. The cut-off score of 5 was used for TSQ-M for positive PTSD symptoms. The area under the curve was 0.84, p < 0.001 with a 95% confidence interval between 0.74 and 0.95, substantially above the random ROC.
Convergent validity
Convergent validity was assessed using Pearson's correlation where it had indicated a relatively strong positive correlation between MPCL-5 and TSQ-M scores (r = 0.635).
Table 5 shows that the cut-off score of 33 was found to be sound with sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), which were 0.81, 0.65, 0.31, and 0.94 respectively. However, it was observed that for the cut-off score of 35, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) slightly improved with values of 0.81, 0.71, 0.35 and 0.95 respectively. A chi-square test was performed to assess the association between individual items of PCL5 and PTSD diagnosis. All items were statistically significant.
Discussion
In this study, the Malay version of post-traumatic stress checklist-5 (MPCL-5) generally showed good reliability with the evidence of high internal consistency for items and its components respectively. The construct validity effectually illustrated four-factor components with acceptable loading factors. However, concurrent validity reported an acceptable value of high sensitivity and relatively low specificity. Correlation between the total score between the two instruments showed good convergent validity.
The MPCL-5 showed a high internal consistency as reported with previous studies [6, 15, 16]. A recent validation study for post-traumatic stress checklist-5 (PCL-5) among adults was completed in Malaysia showing a high internal consistency. The study involved adult patients in orthopedic wards and adolescents, with Cronbach's Alpha values of 0.889 and 0.91, respectively [5, 17]. The Exploratory Factor Analysis (EFA) showed four factors with the original PCL-5, which correspond to the 4 clusters of symptoms. Several validation studies in different populations have also reported the four-factors model [5, 6, 16–18].
The MPCL-5 showed a relatively high value of sensitivity (81.25) with a lower value for specificity (65.48) which makes it appropriate for use as a screening tool for those who have post-traumatic stress disorder (PTSD) symptoms. Further testing should be carried out amongst those with probable PTSD by using the Clinician-Administered PTSD Scale (CAPS) via a structured interview by a trained clinician or psychiatrist to properly diagnose PTSD. A highly sensitive test is most helpful to the clinician as it will have fewer false negatives results. When the test result is negative with relatively low specificity, which means it will be falsely positive for a number of those who don't have PTSD symptoms. Conversely, a highly specific test will have fewer false positives and will be most helpful to the clinician when the test result is positive as the low sensitivity might have led to a higher false-negative result.
The sensitivity and specificity for the MPCL-5 obtained from this study are acceptable with a cut-off score of 33. However, with a higher cut-off score of 35, the sensitivity and specificity would have been slightly improved and is the optimum objective. This is very close to a study completed among veterans in the UK, where the optimum cut-off score was 34 [19]. There have been many studies that showed good cut-off points of 33 [5, 6, 15] but there are also studies with different cut-off points. For example, a study that evaluated the psychometric properties of the PCL-5 among undergraduate students who had experienced a very stressful life event found that the PCL-5 achieved a sound sensitivity with a cut-off score of 37 [6]. Several studies also recommended a wide range of cut-off scores from 30 to 60 [20, 21]. Recently, a cut-off score of 31 to 39 was recommended to predict DSM-5 PTSD diagnosis among veterans [15, 16].
Low positive predicted value (PPV) was noted possibly as a result of the relatively low prevalence [11] of PTSD by the TSQ-M. A similar trend was reported in other concurrent validations using CAPS-5, where the prevalence for PTSD was lower than that of the MPCL-5 [17]. Concurrent validation from this study is limited by using validated TSQ-M, and not the Clinician-Administered PTSD Scale-5 (CAPS-5), which is regarded as the gold standard. However, previous studies have shown an acceptable level of validity and ratability, almost as good as that of CAPS-5 [14].
Conclusion
This is the first validation study of the Malay version of a post-traumatic stress checklist-5 (MPCL-5) for Malaysian fire and rescue officers. This study demonstrates that MPCL-5 is a valid and reliable scale for screening probable PTSD diagnosis among those who experienced traumatic events.
Figures
Sociodemographic characteristic of participants
| Demographics | Mean (±SD) | n (%) |
|---|---|---|
| Age | 33.3 (7.9) | |
| Years of service | 10.8 (7.8) | |
| Gender | ||
| Male | 96(96) | |
| Female | 4 (4) | |
| Ethnicity | ||
| Malay | 93(93) | |
| Indian | 1(1) | |
| Chinese | 0(0) | |
| Others | 6(6) | |
| Marital status | ||
| Single | 74(74) | |
| Married | 26(26) | |
| Household incomea | ||
| 1001–2000 | 27(27) | |
| 2001–3000 | 58(58) | |
| 3001–4000 | 14(14) | |
| 4000> | 1(1) | |
Note(s): aIn Ringgit Malaysia (RM)
Internal consistency of each cluster component
| No | Cluster of symptoms | Cronbach's alpha | No. of item |
|---|---|---|---|
| 1 | Cluster symptom A | 0.90 | 5 |
| 2 | Cluster symptom B | 0.83 | 2 |
| 3 | Cluster symptom C | 0.91 | 7 |
| 4 | Cluster symptom D | 0.93 | 6 |
Total items statistics
| Item | Corrected item-total correlation | Cronbach's alpha if item deleted |
|---|---|---|
| A1 | 0.709 | 0.958 |
| A2 | 0.695 | 0.958 |
| A3 | 0.729 | 0.957 |
| A4 | 0.714 | 0.958 |
| A5 | 0.574 | 0.959 |
| B6 | 0.705 | 0.958 |
| B7 | 0.689 | 0.958 |
| C8 | 0.740 | 0.957 |
| C9 | 0.794 | 0.957 |
| C10 | 0.687 | 0.958 |
| C11 | 0.831 | 0.956 |
| C12 | 0.692 | 0.958 |
| C13 | 0.705 | 0.958 |
| C14 | 0.707 | 0.958 |
| D15 | 0.726 | 0.957 |
| D16 | 0.634 | 0.959 |
| D17 | 0.684 | 0.958 |
| D18 | 0.840 | 0.956 |
| D19 | 0.830 | 0.956 |
| D20 | 0.766 | 0.957 |
Factor analysis
| Rotated component matrixb | ||||
|---|---|---|---|---|
| Component factors | ||||
| Items no. | 1 | 2 | 3 | 4 |
| Item1 | 0.650 | |||
| Item2 | 0.752 | |||
| Item3 | 0.748 | |||
| Item4 | 0.595 | |||
| Item5 | 0.868 | |||
| Item6 | 0.670 | |||
| Item7 | 0.762 | |||
| Item8 | 0.462 | |||
| Item9 | 0.610 | |||
| Item10 | 0.856 | |||
| Item11 | 0.602 | |||
| Item12 | 0.770 | |||
| Item13 | 0.676 | |||
| Item14 | 0.743 | |||
| Item15 | 0.718 | |||
| Item16 | 0.875 | |||
| Item17 | 0.572 | |||
| Item18 | 0.509 | |||
| Item19 | 0.611 | |||
| Item20 | 0.747 | |||
Note(s): bExtraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 7 iterations
Validity characteristics of the MPCL-5 at different cutoffs (N = 100)
| Cutoff score | Sensitivity | Specificity | Positive predictive value (PPV) | Negative predictive value (NPV) |
|---|---|---|---|---|
| 25 | 93.75 | 42.86 | 23.8 | 97.3 |
| 26 | 93.75 | 46.43 | 25.0 | 97.5 |
| 27 | 87.50 | 48.81 | 24.6 | 95.3 |
| 28 | 87.50 | 55.95 | 27.5 | 95.9 |
| 29 | 87.50 | 58.33 | 28.6 | 96.1 |
| 30 | 81.25 | 61.90 | 28.9 | 94.5 |
| 31 | 81.25 | 63.10 | 29.5 | 94.6 |
| 33 | 81.25 | 65.48 | 31.0 | 94.8 |
| 35 | 81.25 | 71.43 | 35.1 | 95.2 |
| 36 | 75.00 | 71.43 | 33.3 | 93.7 |
| 37 | 68.75 | 76.19 | 35.5 | 92.8 |
| 38 | 68.75 | 78.57 | 37.9 | 93.0 |
| 39 | 68.75 | 80.95 | 40.7 | 93.2 |
| 40 | 68.75 | 83.33 | 44.0 | 93.3 |
| 41 | 62.50 | 85.71 | 45.5 | 92.3 |
References
1.Benedek DM, Fullerton C, Ursano RJ. First responders: mental health consequences of natural and human-made disasters for public health and public safety workers. Annu Rev Public Health. 2007; 28(1): 55-68. doi: 10.1146/annurev.publhealth.28.021406.144037.
2.Berger W, Coutinho ESF, Figueira I, Marques-Portella C, Luz MP, Neylan TC, Marmar CR, Mendlowicz MV. Rescuers at risk: a systematic review and meta-regression analysis of the worldwide current prevalence and correlates of PTSD in rescue workers. Soc Psychiatry Psychiatr Epidemiol. 2012 Jun; 47(6): 1001-11. doi: 10.1007/s00127-011-0408-2.
3.McFarlane AC, Bryant RA. Post-traumatic stress disorder in occupational settings: anticipating and managing the risk. Occup Med (Lond). 2007 Sep; 57(6): 404-10. doi: 10.1093/occmed/kqm070.
4.Gorsuch RL. Factor analysis. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; 1983.
5.Ghazali SR, Chen YY. Reliability, concurrent validity, and cutoff score of PTSD Checklist (PCL-5) for the diagnostic and statistical manual of mental disorders, Fifth Edition among Malaysian adolescents. Traumatology. 2018; 24(4): 280-7. doi: 10.1037/trm0000156.
6.Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The posttraumatic stress disorder checklist for DSM-5 (PCL-5): development and initial psychometric evaluation. J Trauma Stress. 2015 Dec; 28(6): 489-98. doi: 10.1002/jts.22059.
7.Foa EB, Riggs DS, Dancu CV, Rothbaum BO. Reliability and validity of a brief instrument for assessing post-traumatic stress disorder. J Trauma Stress. 1993; 6(4): 459-73. doi: 10.1007/BF00974317.
8.Jaapar SZS, Abidin ZZ, Othman Z. Validation of Malay trauma screening questionnaire. Inter Medical J. 2014 Dec; 21(6): 536-8.
9.Brewin CR, Rose S, Andrews B, Green J, Tata P, McEvedy C, Turner S, Foa EB. Brief screening instrument for post-traumatic stress disorder. Br J Psychiatry. 2002 Aug; 181(2): 158-62. doi: 10.1017/s0007125000161896.
10.Dekkers AMM, Olff M, Näring GWB. Identifying persons at risk for PTSD after trauma with TSQ in The Netherlands. Community Ment Health J. 2010 Feb; 46(1): 20-5. doi: 10.1007/s10597-009-9195-6.
11.Walters JT, Bisson JI, Shepherd JP. Predicting post-traumatic stress disorder: validation of the Trauma Screening Questionnaire in victims of assault. Psychol Med. 2007 Jan; 37(1): 143-50. doi: 10.1017/S0033291706008658.
12.Hustache S, Moro MR, Roptin J, Souza R, Gansou GM, Mbemba A, Roederer T, Grais RF, Gaboulaud V, Baubet T. Evaluation of psychological support for victims of sexual violence in a conflict setting: results from Brazzaville, Congo. Int J Ment Health Syst. 2009 Apr; 3(1): 7. doi: 10.1186/1752-4458-3-7.
13.Mouthaan J, Sijbrandij M, Reitsma JB, Gersons BP, Olff M. Comparing screening instruments to predict posttraumatic stress disorder. PLoS ONE. 2014; 9(5): e97183. doi: 10.1371/journal.pone.0097183.
14.Brewin CR. Systematic review of screening instruments for adults at risk of PTSD. J Trauma Stress. 2005 Feb; 18(1): 53-62. doi: 10.1002/jts.20007.
15.Wortmann JH, Jordan AH, Weathers FW, Resick PA, Dondanville KA, Hall-Clark B, Foa EB, Young-McCaughan S, Yarvis JS, Hembree EA, Mintz J. Psychometric analysis of the PTSD Checklist-5 (PCL-5) among treatment-seeking military service members. Psychol Assess. 2016 Nov; 28(11): 1392-403. doi: 10.1037/pas0000260.
16.Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, Keane TM. Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders-fifth edition (PCL-5) in veterans. Psychol Assess. 2016 Nov; 28(11): 1379-91. doi: 10.1037/pas0000254.
17.Bahari R, Alwi MNM, Ahmad MR. Translation and validation of the Malay posttraumatic stress disorder checklist for DSM-5 (MPLC-5). Malaysian Journal of Medicine and Health Sciences. 2019 April; 15(Suppl. 1): 15-20.
18.Liu P, Wang L, Cao C, Wang R, Zhang J, Zhang B, Wu Q, Zhang H, Zhao Z, Fan G, Elhai JD. The underlying dimensions of DSM-5 posttraumatic stress disorder symptoms in an epidemiological sample of Chinese earthquake survivors. J Anxiety Disord. 2014 May; 28(4): 345-51. doi: 10.1016/j.janxdis.2014.03.008.
19.Murphy D, Ross J, Ashwick R, Armour C, Busuttil W. Exploring optimum cut-off scores to screen for probable posttraumatic stress disorder within a sample of UK treatment-seeking veterans. Eur J Psychotraumatol. 2017 Nov; 8(1): 1398001. doi: 10.1080/20008198.2017.1398001.
20.McDonald SD, Calhoun PS. The diagnostic accuracy of the PTSD checklist: a critical review. Clin Psychol Rev. 2010 Dec; 30(8): 976-87. doi: 10.1016/j.cpr.2010.06.012.
21.Demirchyan A, Goenjian AK, Khachadourian V. Factor structure and psychometric properties of the posttraumatic stress disorder (PTSD) checklist and DSM-5 PTSD symptom set in a long-term postearthquake cohort in Armenia. Assessment. 2015 Oct; 22(5): 594-606. doi: 10.1177/1073191114555523.
Acknowledgements
The authors are grateful to the Fire and Rescue Department of Selangor, Malaysia for their assistance and support.Conflicts of Interest: None declared.