Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels

Abstract

Purpose

This study aims to identify the perception of people in a Portuguese company regarding the main barriers to the effectiveness of the existing performance measurement system (PMS) and whether those perceptions are dependent of people’s hierarchical levels, education levels, work shifts, gender and department.

Design/methodology/approach

Primary data was collected through structured interviews (adapted to three levels of interviewees in the company hierarchy) and Likert scale questionnaires. Descriptive statistical analysis of the collected data was performed as well as a chi-square test.

Findings

The results provide an insight on the perception of barriers to the PMS effectiveness in the company. After performing interviews and questionnaires it was possible to identify that the main perceived barriers were: poor communication system and issues on target definition, lack of trained resources, employee involvement, indicators understanding and use for improvement.

Practical implications

This study is the starting point to develop actions aiming to eliminate, or at least mitigate, the impact of the barriers on the PMS effectiveness.

Originality/value

PMSs play an essential role in an organization, so it is essential to identify what hinders its effectiveness. This study opens the discussion by diagnosing the company’s perception of the barriers to PMS effectiveness.

Citation

Cunha, F., Dinis-Carvalho, J. and Sousa, R.M. (2024), "Analysis of barriers for performance measurement system effectiveness in a company: perceptions across hierarchical levels", International Journal of Lean Six Sigma, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/IJLSS-12-2023-0205

Introduction

Continuous improvement is a key principle to be taken seriously by companies if they want to stay in the existing and future competitive reality and pursue sustainability and excellence. Continuous improvement is one of the two pillars of the Toyota Way (Corporation, 2012) as well as playing a prominent role in other models of excellence such as Lean Thinking (Womack and Jones, 1996) and Shingo Model (Rizzo, 2011; Shingo Institute, 2020). Although often referred to in the literature, both the true meaning of “improvement” and the true meaning of “continuous improvement” are unclear in most publications. For the purposes of this paper, it is useful to define these two designations in the best possible way. First, in order for a change to be considered an improvement, it must be in accordance with the two conditions below:

1. it is possible to assess a positive change in performance through an indicator; and

2. the positive change in that performance indicator is aligned with the strategic goal and vision (true north) of the organization (Carvalho, 2023).

Since this paper is focused on the first condition, measuring performance plays a key role in continuous improvement, excellence and sustainability. For continuous improvement to be effective and sustainable, a system must exist that continuously feeds the generation and implementation of all kinds of improvement actions by everyone on a daily basis. It is necessary to implement routines, create entities and adopt artefacts that support these dynamics and create a true culture of continuous improvement. To guide and monitor the effect of all continuous improvement efforts, it is necessary to have a good performance measurement system (PMS) and that it reflects the strategy of the organization at all levels (policy deployment). In this line of thought, Kanji (2002) states that the first condition to improve, and achieve business excellence is to develop and implement a PMS.

The objective of this study is to identify the perceptions of different people working in a specific organization regarding the main barriers, identified by Cunha et al. (2023), to the effectiveness of the existing PMS. In addition, the differences in perception from different people across different groups of the company will be investigated, namely, in terms of possible dependence on hierarchical levels, education levels, work shifts, gender and department.

This study was performed in an industrial company which specializes in the transformation of plastics foams and cellular rubbers for the automotive industry. This company has 85 full-time workers divided by several departments, with two-thirds of the workforce allocated to the production department. Although the company is multinational, the production unit studied is located in the north of Portugal. The company has an internally developed PMS comprised of several measures approved by top management for each department. This PMS is reviewed annually in a management meeting.

This paper is organized into the following sections: literature review, methods, results, discussion and conclusion. The literature review section explores the PMS concept and identifies the main barriers to the effectiveness of this type of systems. The following section describes the methods used to collect and analyze data to assess the perception of existing barriers to the PMS effectiveness in the company. The results section presents the outcomes of the analysis of the collected data. In the discussion section, the results of the different data sources are discussed and linked. The last section provides the conclusion of the present study, providing directions for future research on the subject.

Literature review

A PMS allows an organization to identify where to improve (Kanji, 2002) and to focus people and resources on those particular aspects of the business (Waggoner et al., 1999). A PMS is a set of metrics used to assess the efficiency and effectiveness of activities (Neely et al., 1995) and should be able to answer several questions, such as:

• What to measure?

• How to measure?

• Where to measure?

• When to measure? and

• Why measure? (Zairi, 1994).

Bititci (2015) defines the creation and the implementation of a PMS as the process of:

• defining the targets;

• developing a set of performance metrics; and

• collecting, analyzing, reporting, interpreting and acting on performance data.

A PMS is a complex system that comprises different dimensions such as performance measurement, strategy management, communication, influence behaviors and stimulate learning and improvement (Franco-Santos et al., 2007). This complexity makes the development and implementation of an effective PMS a challenge for organizations. Several developments and implementations result in ineffective PMS. Some studies carried out in small and medium enterprises (SMEs) show that most attempts to implement PMS fail (Langwerden, 2015; Malagueño et al., 2018; Neely, 2004; Todorut et al., 2013). Other studies show that as much of 70% of the attempts fail (McCunn, 1998). This level of failure is not shown in other studies, such as the one carried out in Slovakia with a sample of 336 SMEs. In this study, only 20% of companies in the sample implemented PMS, and only 36% of them were not satisfied with it.

An inefficient PMS is harmful to the organization because it supports incorrect decisions that lead to an incorrect allocation of scarce resources to incentives that will fail to deliver results (Van Camp and Braet, 2016). In a study carried out previously (Cunha et al., 2023), based on a systematic literature review, it was possible to identify and classify the main barriers that may hinder the effectiveness of PMS. This systematic literature review was performed according to the preferred reporting items for systematic reviews and meta-analyses methodology, and it was focused on identifying the most frequent factors referred to in the literature as barriers to PMS effectiveness. In this comprehensive study, a meticulous screening process was applied to evaluate 2,808 publications. Through a systematic series of analytical steps, 31 relevant publications were identified and included in the study. Subsequently, a thorough examination of these publications yielded a total of 175 references highlighting barriers to the effectiveness of PMS. These identified barriers were further categorized into 19 distinct types based on their shared meaning. The list is shown in the Table 1.

For an organization that strives for an effective PMS, it becomes necessary to identify and understand what are the barriers that are hindering the effectiveness of that PMS, so they can be eliminated or, at least, mitigated.

Methods

To identify what the perceptions of workers are regarding the main barriers to PMS effectiveness, primary data was collected (Saunders et al., 2019). The collection of data was done through the deployment of semistructured interviews and a short questionnaire with a Likert scale classification. The Likert scale provides a way of measuring unobservable constructs (Jebb et al., 2021), allowing the quantification of subjective preferential thinking, feeling and action (Joshi et al., 2015).

Three different interviews were created by the authors, based on a previous study about barriers to PMS effectiveness (Cunha et al., 2023), to evaluate the perspective of employees, middle management and top management, regarding the current state of the implemented PMS. The interviews for each hierarchical level were adapted to their target audience.

The employees’ interview was deployed to operational workers of the different departments. The ten questions that composed this interview are presented in the following list.

Questions of the employees’ interview are as follows:

1. Do you know the objectives of your team/section?

2. What are the measures/indicators (that you know) used to measure the performance of your team/section?

3. Are there any measures/indicators that do not make sense to you?

4. Are there any other measures/indicators that you would find important to adopt?

5. Do you trust the reliability of the measurements of the different indicators of your team/section?

6. How do your team/section goals contribute to the company’s goals?

7. When a measure/indicator is not achieving the objective, is anything done to correct and improve performance?

8. How do you feel about the use of indicators in your team?

9. Are indicators used as a way of blaming people when objectives are not achieved?

10. What are the main difficulties (the main barriers) to the effectiveness of the PMS?

The above list is the authors’ own creation.

The middle management interview was deployed to shift team leaders and to the different department supervisors. The nine questions that compose this interview are presented in the following list.

Questions of the middle management interview are as follows:

1. Do you know the company’s strategic objectives?

2. Is there information on whether the objectives are being achieved or not? Do you keep informed about the status of strategic objectives?

3. Are all measures/indicators useful and make sense?

4. Do you identify the need for other measures/indicators that currently do not exist?

5. Are metrics/indicators used as a way to identify where to improve? Are they used as the basis for improvement?

6. Do you trust the reliability of measurements of the different indicators in your department?

7. Are indicators used as a way of blaming the department head when objectives are not achieved?

8. Do you think your department’s indicator structure is appropriate?

9. What are the main difficulties (the main barriers) to the effectiveness of the PMS?

The above list is the authors’ own creation.

The top management interview was deployed to the general manager, production manager and financial manager of the organization. The seven questions that composed this interview are presented in the following list.

Questions of the top management interview are as follows:

• Are the strategic objectives defined in accordance with the organization’s vision and values?

• Are they communicated effectively? (In what way?)

• Are all indicators aligned with the organization’s strategic objectives?

• Do you identify the need for other measures/indicators that currently do not exist? (which?)

• Do you trust the reliability of measurements of the different indicators of your organization?

• Do you think performance measurement is effective? (used as a starting point for improvement)

• What are the main reasons that make it difficult for the PMS to work in this company?

The above list is the authors’ own creation.

Thirty-two interviews were performed, 23 for employees, 6 for middle management and 3 for top management.

The interviews were recorded and then transcribed. After being transcribed, the answers with similar meanings were organized into groups where the answers had the same meaning to allow a statistical analysis of the answers given in the interviews.

The short questionnaire was deployed to the same individuals who were interviewed right after the interview. Here, the questionnaire was the same independent of the role in the company. In this questionnaire, individuals were asked to classify, using a Likert scale, their perception of the presence in the organization of commonly known barriers to PMS effectiveness. A five-point Likert scale was used (Joshi et al., 2015), ranging from 1 to 5 (1 – Strongly disagree, 2 – Disagree, 3 – Indifferent, 4 – Agree and 5 – Strongly agree).

The barriers that respondents were asked to classify were: blame culture, unclear system, high complexity, lack of rewards, excess of indicators, inappropriate indicators, lack of connection to strategy, lack of top management involvement, lack of use for improvement, false expectations, lack of trained resources, lack of indicators understanding, issues on target definition, poor communication system, lack of balance of indicators, inappropriate information technology (IT) tools, lack of employee involvement, time and resources required and difficulties in collecting, analyzing and presenting data (Cunha et al., 2023).

For the interviews and the questionnaire, a descriptive statistical analysis was performed (Boone and Boone, 2012). Also, for the data from the employees interviews and questionnaires, a chi-square test (Pandis, 2016) was performed to ascertain whether the answers depend on the respondents degree of education, gender, shift or department. For the questionnaire it was also tested if the answers depend on the role in the company (employees, middle management and top management).

The hypothesis tested with the chi-square test were:

H0.

The variables are independent; there is no relationship between the categorical variables.

H1.

The variables are dependent; there is a relationship between the categorical variables.

To test these hypotheses, first, it is required to calculate the expected frequencies for each variable. Then, the difference between observed and expected frequencies is assessed through the chi-square test, where the p-value is obtained. If the p-value is greater than 0.05, the null hypothesis (H0) is accepted. This means that there is a small difference between the observed and the expected values. If the p-value is smaller than 0.05, the null hypothesis is rejected, meaning that there is a large difference between the observed and the expected values (Pandis, 2016).

The survey results and the interview findings were analyzed, establishing a connection between the survey outcomes and the responses provided during the interviews.

Results

This section presents, for each type of interview deployed, as well as for the questionnaire, the main results drawn from the data treatment and statistical analysis.

Twenty-three employees’ interviews were performed. Of those interviewed, the population is characterized by:

• Gender: 16 (70%) were female and 7 (30%) were male;

• Education: 21 (91%) had basic education and 2 (9%) had higher education;

• Department: 20 (87%) were from the production department and 3 (13%) from other departments; and

• Shift: 13 (57%) were from Shift A (from 6:00 am to 2:30 pm), 8 (35%) from Shift B (from 2:30 pm to 11:00 pm) and 2 (8%) from the normal shift (from 8:30 am to 5:30 pm).

Regarding the answers of employees’ interviews:

• 91% of the respondents did not know the objectives of their team or section. Through the chi-square test, it was possible to verify that the answers given depended on the degree of education, gender, shift and the department of the respondents;

• 30% of the respondents did not know any indicators, 61% knew one indicator and only 9% knew the indicators. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents;

• 30% of the respondents identified indicators that did not make sense to them. The answers were not dependent on the degree of education, gender, shift or department of the respondents;

• 74% of the respondents identified the need to adopt new indicators. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents;

• 65% of the respondents did not trust the reliability of the indicators. The answers were dependent on the degree of education and department of the respondents but not dependent on the gender and shift of the respondents;

• 83% of the respondents did not know how their team goals contribute to the company goals. The answers given depended on the degree of education, shift and the department of the respondents. The answers were not dependent on the gender of the respondents;

• 83% of the respondents did not feel an improvement culture. The answers were not dependent on the degree of education, gender, shift or department of the respondents;

• 87% of the respondents feel good about the use of performance indicators in their team. The answers were not dependent on the degree of education, gender, shift or department of the respondents;

• 43% of the respondents feel a blame culture when the objectives are not achieved. The answers were not dependent on the degree of education, gender, shift or department of the respondents;

• 52% of the respondents identified the communication system as a barrier to the PMS effectiveness in the organization. Also, 22% identified a lack of employee involvement as a barrier to PMS effectiveness. Other barriers referred to were: blame culture, issues on target definition, unclear system, lack of rewards and lack of use for improvement. The answers were not dependent on the degree of education, gender, shift or department of the respondents.

The chi-square test was performed to verify if the answers, given by the respondents, are independent of the variables: degree of education, gender, shift and department. The hypothesis accepted and the p-value are presented in Table 2.

Six middle management interviews were performed. Of those interviewed, the population is characterized by:

• Gender: three (50%) were female and three (50%) were male;

• Education: four (67%) had basic education and two (33%) had higher education;

• Department: three (50%) were from the production department and three (50%) were from other departments; and

• Shift: two (33%) were from Shift A and four (67%) were from the normal shift.

Regarding the middle management interview:

• None of the respondents were able to identify the company’s strategic objectives;

• 83% of the respondents consider that there is no information available on whether the objectives are being achieved or not;

• 50% of the respondents considered that there are some indicators that are not useful or do not make sense;

• 83% of the respondents identify the need for other indicators that currently do not exist;

• 50% of the respondents did not feel an improvement culture;

• 33% of the respondents did not trust the reliability of the different indicators in their department;

• 33% of the respondents felt that there was a blame culture in the organization;

• 50% felt that their department indicator structure was not appropriate; and

• 50% of the respondents identified the communication system as a barrier to the PMS effectiveness in the organization. Other barriers to the PMS effectiveness identified were inappropriate indicators, lack of employee involvement, lack of top management involvement, lack of use for improvement, time and resources required and unclear system.

For the middle management interview, the chi-square test was not performed because the size of the sample (six) was too small.

Three top management interviews were performed. Of those interviewed, the population is characterized by:

• Gender: two (67%) were female and one (33%) were male;

• Education: three (100%) had higher education;

• Department: one (33%) was from production department and two (67) were from other departments; and

• Shift: three (100%) were from the normal.

Regarding the top management interview:

• All the respondents considered that the strategic objectives were in accordance with the organization’s vision and values.

• All the respondents considered that the communication of the objectives is not yet made effectively throughout the organization.

• All the respondents considered the indicators aligned with the organization’s strategic objectives.

• 67% of the respondents identified the need for other indicators that currently do not exist.

• 67% of the respondents did not fully trust the reliability of measurement of the different indicators of the organization.

• All the respondents did not feel an improvement culture in the organization.

• 67% of the respondents identified the lack of use for improvement as a barrier to the PMS effectiveness in the organization. Other barriers to the PMS effectiveness identified were communication system, lack of indicators understanding and unclear system.

For the top management interview the chi-square test was not performed because the size of the sample (three) was too small.

Thirty-two questionnaires were performed. Those who answered the questionnaire are characterized by the following:

• Gender: 21 (66%) were female and 11 (34%) were male;

• Education: 25 (78%) had basic education and 7 (22%) had higher education;

• Department: 24 (75%) were from the production department and 8 (25%) were from other departments; and

• Shift: 15 (47%) were from Shift A, 8 (25%) were from Shift B and 9 (28%) were from the normal shift.

Participants were asked to classify if they strongly agree, agree, are indifferent, disagree or completely disagree, with the existence inside the organization, of each of the 19 individual barriers to the effectiveness of its PMS. The results of the classification are presented in Figure 1.

It is possible to verify that 91% of the respondents agreed or strongly agreed that the communication system is currently a barrier to the PMS effectiveness in the organization. The second barrier mostly agreed by the respondents was the lack of trained resources, with 81%. The third barrier mostly agreed was issues in defining targets, with 75%. Following with 72% were lack of employee involvement, lack of understanding the indicators and lack of use for improvement. The barriers least agreed were: excess of indicators (3%), lack of connection to strategy (19%), inappropriate IT tools (31%) and inappropriate indicators (31%).

The chi-square test was performed to verify if the classification, made by the respondents, is independent of the variables: degree of education, gender, shift, department or role in the company. The results are presented in Table 3.

Regarding the poor communication system barrier, the chi-square test revealed that the answers are dependent on the gender of the respondents. This happens because only two males disagreed that communication system is a barrier to the current PMS effectiveness, while all females classified it with agree or strongly agree.

Only the two respondents of the engineering department know the objectives, probably justifying the results of the chi-square test that identified that the answers given were dependent on the degree of education, gender, shift and department of the respondents. The two interviewed from the engineering department were both male, higher educated and from the normal shift. Also, 91% of the respondents either did not know any indicators or only knew one indicator, and 83% did not know how their team objectives contributed to the company objectives. The answers to both these questions were dependent on the degree of education, shift and department of the respondents. Here, again, this dependency might be due to the answers given by both members of the engineering department.

Regarding lack of training and lack of understanding of the PMS, through the chi-square test is possible to verify that the classification of these barriers is independent of the degree of education, gender, shift, department and role in the company of the respondents.

The perception of the barriers lack of employee involvement and lack of use for improvement are only dependent on the gender of the respondents.

The lack of rewards or recognition for achieving goals is dependent on the degree of education, shift and department. This barrier is mostly perceived by people with basic education, from Shifts A and B and from the production department.

The lack of top management commitment with the PMS is dependent on the degree of education, gender, shift, department and role in the company. This barrier is perceived by 74% of the employees and by 67% of the intermediate management. On the other hand, 100% of the top management disagree or strongly disagree that there is a lack of top management commitment. It is possible to verify that the perception of the top management is significantly different from the other roles in the company.

Discussion

This first part of the discussion is about statistically relevant differences in perceptions among different groups in the company as shown in Table 3. The authors believe it is a valuable contribution to the communities of managers and academics since these different perceptions identified in this case study may be also observed in other companies, even with different PMS approaches and maturity levels. The second part of this discussion is dedicated to the overall results from the questionnaire in this particular case.

Regarding the first part of this discussion, the cells with the text in italics in Table 3 are the ones showing the statistically relevant differences in the perception between different classes of people. It is possible to verify that the perception is dependent on the degree of education for the barriers “lack of rewards” and “lack of top management involvement.” For these barriers, the classification agree/strongly agree was made significantly more by people with basic education as can be observed in Table 4. It is interesting to notice that people with basic education pay more attention to rewards. This can be discussed based on the argument that people with less intellectually challenging work may need more extrinsic motivation (rewards) than people with more challenging work assuming that people with only basic education are more involved in less intellectually challenging jobs. The differences in perception regarding “Lack of top management involvement” could be explained by the fact that top management people do not consider themselves as not being involved, and they are normally more educated than most of the workers.

The classification is dependable of the variable gender for the barriers presented in Table 5. The female respondents agreed or strongly agreed with the existence of these barriers significantly more than the male respondents. The effect here can be explained by the fact that, in this company, senior and middle management are predominantly male while the workers are predominantly female. Since middle and top management are more involved in the PMS they tend to be more tolerant of its faults.

Shift is a variable that makes the classification dependable for the barriers “lack of rewards” and “lack of top management involvement.” Respondents from the normal shift did not classify the existence of these barriers with agree or strongly agree as much as did the respondents from Shifts A and B (Table 6). The possible reason for this difference in perception could be that there are many more middle and top managers on the normal shift than on the other shifts.

The classification of the barriers listed in Table 7 depends on the variable department. Respondents from the production department classified the existence of these barriers, with agree or strongly agree, more than respondents from other departments (Table 7). This difference may be due to the fact that it is in the production department where there are more workers with less training and who are less involved in developing the PMS.

The classification of the barrier “top management involvement” is dependable of the variable role in the company. No respondents from top management agreed or strongly agreed with the existence of this barrier, while the majority of respondents from middle management and employees agreed or strongly agreed with the existence of this barrier (Table 8). This effect is expected since top management is normally involved in the PMS implementation and for that reason they normally have the perception that they are involved. Middle management and workers do often feel it in a different way in companies with no real culture of continuous improvement or operational excellence.

This second part of the discussion is dedicated to the overall results from the questionnaires. The main barrier perceived by people in the organization was the ineffective communication system. The communication system is ineffective when it is not simple, clear, periodical and formal (Franco and Bourne, 2003). Not only 91% of the respondents agree or strongly agree that this is a barrier to the PMS effectiveness, but this evidence is supported by the answers given in the interviews. Although top management considers that the strategic objectives are defined accordingly with the organization’s mission and values, they point out that the communication of those objectives throughout the organization is not yet made effectively.

In the intermediate management interview, it was also possible to identify answers that point to an ineffective communication system because all the respondents were not able to identify the company’s strategic objectives, and 83% of the respondents considered that there is no information available on whether those objectives are being achieved or not. The answers given in the employees’ interview also suggest that the communication system is not effectively fulfilling its role. The objectives of their team or section were unknown by 91% of the respondents.

The second most perceived barrier was the lack of trained resources. This barrier is related with the lack of training or understanding of the PMS by the people in the organization (Franco and Bourne, 2003). This is perceived as barrier to the PMS effectiveness by 81% of the respondents.

The perception of the existence of this barrier in the organization is supported by the answers given in the interviews. In the interviews, the respondents show a lack of training and awareness of the different parts that compose the PMS. In the employees’ interview, it can be assessed that there is a lack of awareness about the team objectives, about the indicators used to measure their performance, and about how their goals contribute to the company goals. This lack of training or awareness is also supported in the middle management interview, where none of the respondents was able to identify the company’s strategic objectives. In the top management interview, one of the barriers identified was the lack of indicators understanding which also supports the fact that there is a lack of trained people to use the PMS.

The third most perceived barrier was issues in defining targets. This barrier might occur when there is a failure in the deployment of the objectives from the top level of the organization (Schneiderman, 1999). This is supported by the answers given in the three types of interviews (three hierarchical levels). In the top management interview, all respondents considered that the communication of the objectives is not made effectively throughout the organization. Also, in the intermediate management and employees’ interviews, it is possible to verify that this deployment of objectives is not effective since in the intermediate interview, none of the respondents was able to identify the organization’s strategic objectives, and in the employees’ interview, 91% of the respondents were not able to identify the objectives for their team or section.

The lack of employee involvement was pointed out by 72% of respondents. Although this can be caused by a fear of performance management (Zairi, 1994), it does not seem to be the case because 87% of the employees respondents feel good about the use of performance indicators in their team. Also, only 43% feel a blame culture when the objectives are not achieved. Regarding intermediate management only 33% of the respondents feel a blame culture when the objectives are not achieved. This barrier may be perceived as present by people due to the lack of an efficient communication system that does not allow to involve them properly. Also, people in the organization may mistrust the reliability of the indicators since 65% of the employees, 33% of the intermediate management and 67% of the top management do not fully trust the reliability of the different indicators.

The lack of understanding the indicators might result from indicators that are not relevant for people (Ghalayini and Noble, 1996). This barrier was perceived by 72% of the respondents, being supported by the answers given in the interviews, since 30% of the employees identified indicators that did not make sense to them. Also, 50% of the intermediate management considered that there are some indicators that are not useful or do not make sense to them. The lack of understanding the indicators can be caused or hyped by the failure of the communication system and the lack of employee involvement that may cause a higher difficulty in understanding both the indicators and the objectives.

When a PMS does not have an effective improvement process associated with it, it can become irrelevant to the people in the organization (Kennerley and Neely, 2002; Meekings, 1995; Neely and Bourne, 2000; Schneiderman, 1999). Of the respondents, 72% believe that there is a lack of use of the PMS for improvement in the organization. This perception is supported by the answers given in the interviews, where 83% of the employees, 50% of the intermediate management and 100% of the top management do not feel an improvement culture in the organization. If the PMS is not being used to continuously improve, it is not being used effectively because the effort of performance measurement is not being translated to performance improvement.

There are two barriers where the perception of people regarding their existence in the organization varies according to some of the variables that characterize the sample. Those barriers are lack of rewards and lack of top management commitment.

The lack of rewards or recognition for achieving goals (Franco and Bourne, 2003; Watts and McNair‐Connolly, 2012) is perceived as a barrier by 69% of the respondents.

The lack of top management commitment to the PMS (Bourne, 2004; Charan et al., 2009; McCunn, 1998; Meekings, 1995; Sousa et al., 2006; Townley et al., 2003) is perceived as a barrier by 66% of the respondents. This might be supported by the answers given in the interviews, where there is the insight that information does not flow due to an ineffective communication system and there is not an improvement culture in the organization, which people might feel should be instigated by top management. Of the respondents, 69% perceive that the system is not clear to people. This is supported by the data retrieved in the interviews, where people showed that they do not have a clear understanding of the PMS. This is supported by the fact that they do not know their objectives and some of the indicators in the case of employees. It is also supported by the fact that intermediate management is not aware of the company’s strategic objectives, so they are not able to clearly understand the PMS.

There are some barriers where the classification was divided and there is not a clear perception by the people in the organization. Those barriers are difficulty in collecting, analyzing and presenting data, false expectations, blame culture, system complexity, inappropriate indicators and inappropriate IT tools.

The two least perceived barriers were the lack of connection to the strategy and the existence of too many indicators. Only 19% of the respondents perceived the existence of a lack of connection to the strategy from the PMS. This is backed by the answers given by the top management, where they considered that the strategic objectives were in accordance with the organization’s vision and values and that the indicators are aligned with the organization’s strategic objectives. The answers given by employees cannot support the classification of this barrier because they were not able to identify their objectives and how they contribute to the organization’s strategic objectives. Also, answers given by intermediate management cannot support this classification because they were not aware of the company’s strategic objectives and considered that they do not have information about those objectives. If they are unaware about the company’s strategy, they will not be able to judge if the PMS is aligned with that strategy. Only 3% perceived the existence of an excess of indicators in the company as a barrier to the PMS. It is possible to verify that people identify the need for more indicators that currently do not exist. Also, in the case of employees, some indicators are unknown, so it might be difficult for them to perceive an excess of indicators when they do not know them all.

Conclusions

This study objective was to identify what is the perception of people working in an organization, regarding the main barriers to the effectiveness of the existing PMS. The differences in perception between different hierarchical levels were investigated. Perceptions between leaders and employees, levels of education, work shift, gender and department were also studied.

The study revealed that the main barriers to the effectiveness of the PMS in the organization are the ineffective communication system, the lack of trained resources and the issue of defining targets. The study also found that lack of employees’ involvement and understanding of the indicators were perceived as barriers to the effectiveness of the PMS by the respondents. Finally, the study found that the lack of an effective improvement process associated with the PMS can lead to it becoming irrelevant to the people in the organization.

The fact that the PMS is not being used to improve in this organization makes it ineffective. It is being used as a requirement by middle management to report to top management. The information about team objectives and indicators is confined to middle managers and does not flow to all employees. Since it is not being used to improve there is not an efficient communication system that allows the flow of information. This inefficient communication makes employees feel that they are not informed and trained about the PMS. Also, there is a perception that there are issues in defining targets because those targets are imposed or are even unknown by employees. This results in the perception of the other two barriers, the lack of employee involvement and the lack of indicators understanding. These two barriers seem to be caused by the inefficient communication system and the lack of trained resources. Withholding information and training from people makes them less involved and does not allow them to understand the indicators.

The research revealed that people’s perception of certain barriers within the organization is influenced by the categories they fall into. Clear distinctions in the perception of these barriers, particularly based on hierarchical levels, were evident. The primary variance in perception was identified between managers (at intermediate and top levels) and employees. In conclusion, the study emphasized that an ineffective communication system stood out as the primary barrier to PMS effectiveness. Other significant barriers included a lack of training and understanding of the PMS, difficulties in defining targets, insufficient employee involvement and an ineffective improvement process. Addressing these barriers through effective communication, training programs and inclusive employee involvement is crucial for a successful PMS. Regular reviews are also highlighted as essential to ensure the PMS remains relevant and effective in achieving organizational goals. The research also underscores variations in how different organizational hierarchies perceive the presence of these barriers.

For further investigation on this subject, the conclusions drawn by this study should be used by middle and top managers in the organization as a starting point to improve and transform their PMS into an effective one. This should be done by addressing the main barriers perceived and try to eliminate or at least mitigate them.

The main limitation of this study is the transversalization of its conclusions. As the study only covered one organization, the conclusions can only be applied to the organization studied. However, it is expected that from the ongoing project (on the development of effective PMS) in which this paper is integrated, knowledge will emerge that can be applied in other similar industrial contexts, and not just in the SME of this study.

Figures

Figure 1.

Likert scale classification of the existence of the barriers in the organization

Table 1.

Main barriers for PMS effectiveness

Blame culture	Lack of connection to strategy	Issues on target definition
Unclear system	Lack of top management involvement	Poor communication system
High complexity	Lack of use for improvement	Lack of balance of indicators
Lack of rewards	False expectations	Inappropriate IT tools
Excess of indicators	Lack of trained resources	Lack of employee involvement
Inappropriate indicators	Lack of indicators understanding	Time and resources required
Difficulties in collecting analyzing and presenting data

Source: Authors’ own creation

Table 2.

Employees interview chi-square hypothesis results and p-value

Questions	Degree of education	Gender	Shift	Department
Do you know the objectives of your team/section?	H1 (0,000)	H1 (0,025)	H1 (0,000)	H1 (0,008)
What are the measures/indicators (that you know) used to measure the performance of your team/section?	H1 (0,000)	H0 (0,077)	H1 (0,000)	H1 (0,000)
Are there any measures/indicators that do not make sense to you?	H0 (0,082)	H0 (0,668)	H0 (0,242)	H0 (0,241)
Are there any other measures/indicators that you would find important to adopt?	H1 (0,000)	H0 (0,062)	H1 (0,001)	H1 (0,001)
Do you trust the reliability of the measurements of the different indicators of your team/section?	H1 (0,019)	H0 (0,848)	H0 (0,056)	H1 (0,038)
How do your team/section goals contribute to the company’s goals?	H1 (0,000)	H0 (0,061)	H1 (0,000)	H1 (0,001)
When a measure/indicator is not achieving the objective, is anything done to correct and improve performance?	H0 (0,203)	H0 (0,349)	H0 (0,438)	H0 (0,435)
How do you feel about the use of indicators in your team?	H0 (0,567)	H0 (0,219)	H0 (0,833)	H0 (0,472)
Are indicators used as a way of blaming people when objectives are not achieved?	H0 (0,194)	H0 (0,340)	H0 (0,424)	H0 (0,103)
What are the main difficulties (the main barriers) to the effectiveness of the performance measurement system?	H0 (0,977)	H0 (0,582)	H0 (0,851)	H0 (0,974)

Notes:

Data is the p-value of the chi-square test. The significance level is 0,05

Source: Authors’ own creation

Table 3.

Questionnaire chi-square hypothesis results and p-value

Barriers for PMS effectiveness	Degree of education	Gender	Shift	Department	Role in the company
Poor communication system	H0 (0,541)	H1 (0,042)	H0 (0,720)	H0 (0,603)	H0 (0,269)
Lack of trained resources	H0 (0,302)	H0 (0,205)	H0 (0,054)	H0 (0,054)	H0 (0,673)
Issues on target definition	H0 (0,805)	H1 (0,005)	H0 (0,766)	H0 (0,059)	H0 (0,539)
Lack of employee involvement	H0 (0,143)	H1 (0,042)	H0 (0,487)	H0 (0,065)	H0 (0,940)
Lack of indicators understanding	H0 (0,850)	H0 (0,587)	H0 (0,563)	H0 (0,568)	H0 (0,940)
Lack of use for improvement	H0 (0,850)	H1 (0,018)	H0 (0,435)	H0 (0,156)	H0 (0,481)
Lack of rewards	H1 (0,001)	H0 (0,246)	H1 (0,001)	H1 (0,003)	H0 (0,055)
Unclear system	H0 (0,094)	H1 (0,004)	H0 (0,173)	H1 (0,028)	H0 (0,359)
Time and resources required	H0 (0,094)	H0 (0,210)	H0 (0,600)	H0 (0,660)	H0 (0,308)
Lack of top management involvement	H1 (0,015)	H1 (0,038)	H1 (0,046)	H1 (0,003)	H1 (0,046)
Lack of balance of indicators	H0 (0,509)	H0 (0,218)	H0 (0,279)	H0 (0,602)	H0 (0,486)
Difficulties in collecting, analyzing and presenting data	H0 (0,545)	H0 (0,222)	H0 (0,350)	H0 (0,068)	H0 (0,199)
False expectations	H0 (0,857)	H0 (0,349)	H0 (0,345)	H0 (0,167)	H0 (0,281)
Blame culture	H0 (0,610)	H0 (0,273)	H0 (0,753)	H0 (0,315)	H0 (0,872)
High complexity	H0 (0,809)	H0 (0,179)	H0 (0,652)	H0 (0,203)	H0 (0,538)
Inappropriate indicators	H0 (0,209)	H0 (0,819)	H0 (0,313)	H0 (0,221)	H0 (0,247)
Inappropriate IT tools	H0 (0,059)	H0 (0,102)	H0 (0,149)	H1 (0,026)	H0 (0,327)
Lack of connection to strategy	H0 (0,752)	H0 (0,073)	H0 (0,405)	H0 (0,806)	H0 (0,192)
Excess of indicators	H0 (0,125)	H0 (0,420)	H0 (0,456)	H0 (0,159)	H0 (0,269)

Notes:

Data is the p-value of the chi-square test. The significance level is 0,05

Source: Authors’ own creation

Table 4.

Percentage of people by degree of education that classified with agree or strongly agree

Barriers for PMS effectiveness	Higher education (%)	Basic education (%)
Lack of rewards	14.3	84.0
Lack of top management involvement	28.5	76.0

Source: Authors’ own creation

Table 5.

Percentage of people by gender that classified with agree or strongly agree

Barriers for PMS effectiveness	Male (%)	Female (%)
Poor communication system	72.7	100.0
Issues on target definition	45.5	90.4
Lack of employee involvement	45.5	85.7
Lack of use for improvement	45.5	85.7
Unclear system	36.4	85.7
Lack of top management involvement	36.4	81.0

Source: Authors’ own creation

Table 6.

Percentage of people that classified with agree or strongly agree by shift

Barriers for PMS effectiveness	Shift A (%)	Shift B (%)	Normal shift (%)
Lack of rewards	86.7	85.7	22.2
Lack of top management involvement	80.0	75.0	33.3

Source: Authors’ own creation

Table 7.

Percentage of people by department that classified with agree or strongly agree

Barriers for PMS effectiveness	Production department (%)	Other (%)
Lack of rewards	83.3	25.0
Unclear system	79.2	37.5
Lack of top management involvement	79.2	25.0
Inappropriate IT tools	37.5	12.5

Source: Authors’ own creation

Table 8.

Percentage of people by role in the company that classified with agree or strongly agree

Barriers for PMS effectiveness	Top management (%)	Middle management (%)	Employees (%)
Lack of top management involvement	0.0	66.7	73.9

Source: Authors’ own creation