Abstract
Purpose
The purpose of this study was to investigate the stability of the cost performance index (CPI) for environmental remediation projects as the topic is not addressed in the literature. CPI is defined as the earned value of work performed divided by the actual cost of the work, and CPI stability represents the point in time in a project after which the CPI varies by less than 20 percent (measured in different ways).
Design/methodology/approach
After collecting monthly earned value management (EVM) data for 136 environmental remediation projects from a United States federal agency in fiscal years 2012 and 2013, the authors used the nonparametric Wilcoxon signed-rank test to analyze CPI stability. The authors also used nonparametric statistical comparisons to identify any significant relationships between CPI stability and independent variables representing project and contract characteristics.
Findings
The CPI for environmental projects did not stabilize until the projects were 41 percent complete with respect to project duration. The most significant factors contributing to CPI stability were categorized into the following managerial insights: contractor qualifications, communication, stakeholder engagement, contracting strategy, competition, EVM factors, and macro project factors.
Originality/value
As CPI stability for environmental remediation projects has not been reported in the literature, this research provides new insights to help project managers understand when the CPIs of environmental remediation projects stabilize and which factors have the most impact on CPI stability.
Keywords
Citation
Clayson, D.S., Thal, Jr, A.E. and White III, E.D. (2018), "Cost performance index stability: insights from environmental remediation projects", Journal of Defense Analytics and Logistics, Vol. 2 No. 2, pp. 94-109. https://doi.org/10.1108/JDAL-11-2017-0024
Publisher
:Emerald Publishing Limited
Copyright © 2018, In accordance with section 105 of the US Copyright Act, this work has been produced by a US government employee and shall be considered a public domain work, as copyright protection is not available.
Introduction
Cost growth has historically been a major problem for the US federal government (US GAO, 2013). This is particularly true among government programs with major expenditures of public funds in an increasingly fiscally constrained environment. Additionally, the visibility of high-profile program failures has increased public pressure to be more cost effective; it has also resulted in government mandates to avert major overruns and divert resources to more promising programs (Kwak and Anbari, 2012). Consequently, the use of earned value management (EVM) as a tool for performance measurement has increased steadily.
EVM “is a widely accepted industry best practice for project management […] that coordinates the work scope, schedule, and cost goals of a program or contract, and objectively measures progress toward these goals” (PARCA, 2014). It combines cost, schedule and performance into an integrated program baseline that provides managers a roadmap to help execute projects. This roadmap can subsequently help managers predict cost growth (Defense Acquisition University, 2013). As such, there tends to be four major reasons managers use EVM:
to quantify and measure program/contract performance;
to provide an early warning system for deviation from a baseline;
to mitigate risks associated with cost and schedule overruns; and
to provide a means to forecast final cost and schedule outcomes (PARCA, 2014).
The EVM system tracks cost and schedule performance based on a contractor’s work breakdown structure. EVM indices, such as the cost performance index (CPI), thus provide a quantitative means of examining project performance (Lipke et al., 2009). CPI is defined as the earned value of work performed divided by the actual cost of the work, and CPI stability represents the point in a project after which the CPI varies by less than 20 percent (measured in different ways). For example, a stable CPI value of 0.8 indicates that a project is not expected to finish within budget as an improvement of 20 percent will only result in a CPI of 0.96. Similarly, a stable CPI value of 1.25 indicates that a project is expected to finish within budget as a decrease of 20 percent still results in a CPI of 1.0.
Therefore, the importance of a stable CPI during the execution of a project has often been reported in the literature. For example, Christensen and Payne (1992) state that a stable CPI is evidence that a contractor’s management control systems (i.e., planning, budgeting, and accounting) are functioning properly and that the contractor’s estimated final costs can be considered reliable. Furthermore, they suggest that knowing at what point during a project that the CPI tends to become stable provides insight into whether a contractor is capable of recovering from a cost overrun. Christensen and Templin (2002) subsequently found through their research that the CPI tends to stabilize at 20 percent completion regardless of the type of project and the success of the project. Conversely, Henderson and Zwikael (2008) found that CPI stabilization cannot be generalized across different types of projects.
Understanding how and at what point cost performance stabilizes during a project provides invaluable insight into the management practices being used within projects. Although most of the literature focuses on CPI stability in government acquisition projects, there are a few reports involving other industries. However, the extant literature appears to be void of any research concerning CPI stability for environmental remediation projects. Therefore, the purpose of this research is to determine when or if the CPI is stabilized in environmental remediation projects and identifies factors that significantly contribute to CPI stability.
Background
To provide sufficient background knowledge, we use this section of the paper to discuss CPI stability in more detail. We begin by presenting three methods found in the literature that can be used to determine CPI stability and use a hypothetical project to illustrate the results obtained from each method. We then provide a brief review of the literature regarding CPI stability.
Cost performance index stability methods
Table I lists the definitions for the three methods commonly used to examine CPI stability. We applied each of these definitions to calculate the CPI stability point for the hypothetical project shown in Figure 1. When using the range method, the CPI is considered stable when the project is 20 percent complete. This is the earliest point at which the difference between the subsequent CPImax and the CPImin values (1.27 and 1.10, respectively) is less than the predefined stability limit of 0.2 shown in Table I. With the interval method though, the CPI is considered stable when the project is 25 percent complete. This is the earliest point at which all subsequent CPI observations are within plus or minus 0.10 of the CPI at that point. At 25 percent complete, the CPI is 1.20 and all subsequent CPI observations fall between 1.10 and 1.30. Finally, the CPI is considered stable when the project is 20 percent complete when using the final range method. This is the earliest point at which the difference between the CPI and the final CPI is less than the predefined stability limit of 0.10 shown in Table I. For our hypothetical scenario, the CPI is 1.15 at 20 percent complete, and the final CPI is 1.10, which gives us a difference of 0.05. However, this method does not account for any of the CPI observations between 20 percent and the final CPI. If the latest percentage of completion point were used in the method, the CPI would not be considered stable until the project was 60 percent complete.
The results from applying each of the CPI stability definitions are summarized in Table II. Although we used a hypothetical scenario, the results effectively demonstrate how different methods impact the CPI stability point calculations. This is a critical point to understand as the results from each method do not always agree. With this in mind, we can now provide a brief review of some of the key literature regarding CPI stability.
Cost performance index stability literature
When managing costs, a general heuristic used by many project managers prior to 1990 was that the CPI stabilized once a project was 50 percent complete; however, there was no empirical evidence supporting the use of this heuristic. Payne (1990) subsequently examined 26 cost performance reports from seven aircraft programs in the Department of Defense (DoD) over a 23-year period (1966-1989) and found the heuristic to be supported. In his work, Payne (1990) considered the CPI to be stable if it did not change by more than plus or minus 0.1 for the remainder of the project. Using the same seven aircraft programs from Payne’s (1990) original research, Christensen and Payne (1992) examined CPI stability using two different approaches: the range method and the interval method (see Table I for the definitions). Their research found that the CPI was stable at 50 percent using the interval method; using the range method though, they found that the CPI was stable as early as 20 percent complete.
Christensen and Heise (1993) extended the CPI stability research to include a broader database of projects. Their data included cost performance reports from 155 contracts from 44 different programs over a 20-year period (1971-1991). These programs included airplanes, ammunition avionics, engines, ground electronics, helicopters, missiles, rockets, satellites, software, submarines, support equipment, and torpedoes. They also investigated different types of contract phases: demonstration/validation, full-scale development, follow-on development, low-rate initial production, full-rate production, and construction. Finally, their research also included differing types of contracts, including fixed-price-incentive-fee, cost plus, cost-plus-fixed-fee, cost-plus-incentive-fee, and cost-plus-award-fee contracts. The goal of their research was to determine if CPI stability could be considered generalizable over a variety of programs. Using the range method, they found that the cumulative CPI was stable at the 20 percent completion point with a 95 percent confidence level. Fixed-price contracts stabilized earlier than cost contracts, and production (including construction) phases stabilized earlier than development phases. They also found that the cumulative CPI tends to decrease (i.e., gets worse) beyond the 20 percent completion point, even though it remains in the stable range.
Christensen and Templin (2002) revisited CPI stability and suggested the following three “rules of thumb” based on empirical evidence. They explained that these rules, which can be more accurately described as observations, can be interpreted as statistical statements about the mean cost performance of programs. In other words, any cost performance that differs from these observations would be considered a statistical outlier and the accuracy of the program’s estimated cost at completion (EAC) should be questioned (Christensen and Templin, 2002).
The final cost variance (in dollars or as a percentage) will be worse than the cost variance at the 20 percent completion point.
The CPI will not change by more than 0.10 from its value at the 20 percent completion point, and in most cases, it only worsens.
The EAC computed using the cumulative CPI is a reasonable lower bound to the final cost of a defense contract.
Christensen and Templin (2002) sought to test the validity of these observations on more recent contracts by examining a sample of 240 contracts categorized into three periods:
those that finished before December 31, 1991;
those that started before but finished after December 31, 1991; and
those that started after December 31, 1991.
Although they found that DoD acquisition contract cost performance appeared to be improving, the three EAC evaluation observations were still valid. In fact, the 20 percent CPI stability rule has been reasserted in many other EVM resources and articles (Fleming and Koppelman, 2008).
However, Henderson and Zwikael (2008) challenged the 20 percent CPI stability rule and examined EVM data from 10 UK construction projects, 12 Israeli high-technology projects, and 4 Australian information technology projects. In their research, they determined CPI stability using Lipke’s (2007) “Stability Point Calculator,” which defines the stability point as “the first observation at which it and all subsequent observations have difference values within the defined stability limit.” If the defined stability limit is 0.1, this represents the interval method. For all three types of projects, as well as the composite sample, stability was not achieved by the time the projects were 20 percent complete. Henderson and Zwikael (2008) concluded that there is a wide variability in CPI stability and that “performance heuristics or rules of thumb intended to be generally applicable (e.g., the CPI stability rule) require an empirically established consistency of behavior across a broad range of projects.” They also concluded that in the case of their samples, stability was often not achieved until 80 percent completion or later. The different interpretations used for CPI stability, as well as the different types of projects included in their respective research, could explain why Henderson and Zwikael (2008) came to differing conclusions than Christensen and Templin (2002).
Henderson and Zwikael (2008) also cited contradictory evidence in the DoD. Popp (1996) conducted an internal study which asked, “Given a program has a CPI of X and a percent complete of Y, what is the most likely finishing CPI?” Although his research was not focused on CPI stability, Popp (1996) created charts depicting the correlation of cumulative CPI at each 10 percent completion interval to the final CPI. The correlation was very close to 1 by the time the project is 70-80 percent complete, while the greatest deviation was observed at 10-20 percent complete. Based on Popp’s (1996) research, Henderson and Zwikael (2008) observed that “CPI stability was also achieved very late in the project life-cycle, often as late as 70-80 percent completion.” Finding this to be consistent with their research in the international commercial sector, they concluded that statistical methods of cost prediction should be used instead of depending on heuristics that may not be universally applicable to all projects (Henderson and Zwikael, 2008; Lipke et al., 2009).
Given the conflicting results, Petter et al. (2015) examined 209 development and production contracts in the DoD from 1987 to 2012 and determined the CPI stability using three stability definitions: range, absolute interval and relative interval. When using the range definition, their results were similar to past research and the “20 percent CPI stability rule.” However, they found that the CPI stabilizes later when using the interval definitions. Based on the research, they concluded that “the question of stability, then, is intricately tied to the definition used.”
Methodology
Initially, we collected monthly EVM data for 178 environmental remediation projects from a United States federal agency in fiscal years 2012 and 2013. The monthly EVM data included the Budgeted Cost of Work Scheduled, the budgeted cost of work performed (BCWP), the actual cost of work performed, the monthly schedule performance index, CPI, schedule variance, cost variance, budget at completion (BAC) and the current percentage complete. Additional data used to represent contract and project characteristics were collected from the Office of Management and Budget’s USA Spending website, contract request for proposals (RFPs), and Congressional budget requests.
From the initial list of 178 projects, a number of projects were eliminated for various reasons. For example, any project that was not at least 80 percent complete was eliminated. This was based on the assertion that the cumulative CPI will not drastically change or affect the CPI stability from 80 to 100 percent completion (Christensen and Templin, 2002). This resulted in ten projects being eliminated. Similarly, for the 20 percent completion point, we used the cumulative CPI of the closest data point to 20 percent. If the closest data point was more than 25 percent or less than 15 percent, we excluded the project; this resulted in the elimination of 18 more projects.
Furthermore, 11 projects were eliminated if the EVM data were provided on an annual basis or the initial EVM data was lumped together in the first reported month. In these situations, the first reported month of data would indicate an unusually high level of completion, which can affect CPI stability calculations. Therefore, if the first data point in the project was the calculated stability point and it was greater than 20 percent, the project was eliminated. Finally, three additional projects were eliminated because large gaps in the data precluded the calculation of the stability point. This reduced the total number to 136 projects being managed under 12 different contracts.
To test whether the 20 percent CPI stability rule held true for these projects, we used the same hypotheses posited by Christensen and Templin (2002):
Christensen and Templin (2002) tested the hypotheses using both a paired t-test and a nonparametric Mann–Whitney test. Henderson and Zwikael (2008), using the same hypotheses with a much smaller sample, chose to use a nonparametric signed-rank test because it does not depend on the data being normally distributed. Similarly, we chose to use the nonparametric Wilcoxon signed-rank test as a more conservative alternative to the t-test.
To gain more insight, we also investigated whether any factors significantly contribute to CPI stability. The independent variables used during the research were based on various projects and contract characteristics categorized into six groups: requirements definition, regulation, contracting strategy, proposal evaluation, acquisition schedule, and other descriptive characteristics. We also included four descriptive variables: project start date, BCWP stability, contract number, and final CPI. We added the project start date to determine if earlier projects have a significantly different CPI stability than projects starting more recently. We also included the final CPI to determine if a project’s final budget performance was related to CPI stability. A complete list of the independent variables and their sources is provided in Table III. We performed nonparametric statistical comparisons to identify any significant relationships between CPI stability and the variables.
In instances where there were only two groups, such as projects with or without Davis–Bacon regulation of local prevailing wages on federal contracts, we used the Wilcoxon rank-sum test for large samples. This test assumes that the two samples are random and independent. Although we assumed that each project was independent, there may be some overlap between projects, particularly those under the same contract or at the same location. The hypotheses and related equations were as follows:
Some of the independent variables divide the projects into three or more groups, such as project type or contract type. In these instances, we used the Kruskal–Wallis H-test to determine if at least one group was significantly different from the other groups. If at least one group was found to be significantly different, we compared each group to every other group using the Wilcoxon rank-sum test. During these comparisons, the level of significance must be smaller than the error rate, α, of 0.05 to have an overall confidence level of 95 percent (McClave et al., 2011). The Bonferroni correction method accounts for this by defining a comparison-wise error rate (αc), that is 0.05 divided by the number of groups or levels within the variable (Hollander et al., 2014). If the p-value of the pairwise comparison is less than αc, the null hypothesis that there is no significant difference is rejected.
Some groups were formed based on a categorical characteristic, such as contract type. Other groups were formed by discretizing the independent variables at various values, such as the number of non-funding modifications. In these situations, we adjusted the number of groups until a significant difference, or no difference, was found. If any of the groups contained only one or two projects, the variable was discretized differently to lump the project(s) together with another group. If this was not possible, that group was eliminated from the test.
Data analysis and results
The data analysis and results are presented in two sections. First, we tested the applicability of the 20 percent CPI stability rule. We then analyzed the data to determine if there were any factors that significantly affected CPI stability.
Applicability of 20 percent cost performance index stability rule
We calculated the CPI stability points for each of the 136 projects using the three different methods shown in Table I. To illustrate the differences in the methods, Table IV lists the mean, median, and interquartile range (IQR) of the CPI stability points for each method. The results indicate that the interval method is more restrictive than the range method as the ±0.1 interval is based on the CPI at that the earliest completion point that was found. The final range method is less restrictive than either the range or the interval method because the CPI could vary widely after the earliest completion point that was found and prior to the final CPI without changing the stability index. Therefore, we used the final range method to test the hypotheses for the 20 percent CPI stability rule. The resulting p-value for the Wilcoxon signed-rank test was 0.0059. As the null hypothesis is thus rejected, this indicates that environmental remediation projects do not follow the 20 percent CPI stability rule.
However, we prefer the interval method for several reasons. First, the interval method appears to capture the meaning of CPI stability. It does not ignore intermediate CPI values between the stability point and the final CPI. Additionally, the interval method would probably be the most useful method for an analyst or manager to evaluate CPI stability for current projects as the other methods require knowing the final CPI or some unknown future CPI value to determine stability. Therefore, we relied on the interval method to determine if the CPI has become stable during project execution. Using the interval method, we found the median CPI stability point to be 41 percent, with an IQR of 56 percent (Table IV).
Contributing factors
During this analysis, we used the CPI stability points, calculated using the interval method, as the dependent variable. Table V summarizes the group comparisons and identifies only the variables that were found to have a significant impact on CPI stability. The variables in the table are arranged with the most significant (lowest p-values) toward the top of the table. The groups exhibiting significantly earlier CPI stability and the accompanying median CPI stability point of each group are listed on the left side of the table. The groups exhibiting significantly later CPI stability and the accompanying median CPI stability point of each group are listed on the right side of the table. Instead of discussing the results in relation to the six groups of project and contract characteristics shown in Table III, we thought it would be more helpful to discuss managerial insights gained from the results and grouped these insights into the categories discussed below.
Contractor qualifications.
A successful project can often be attributed to the qualifications of the contractor; therefore, we used the contractor’s annual revenue as a proxy variable related to the contractor’s position in the industry. The contractor’s annual revenue was thus discretized into three categories:
small (less than $100m in revenue);
medium (between $100m and $3.2bn in revenue); and
large (greater than $3.2bn in revenue).
As shown in Table V, large contractors achieved CPI stability significantly earlier than both small- and medium-sized contractors. The assumption is that larger revenues reflect contractors with more expertise, permanency, and established procedures – all of which may lead to earlier CPI stability.
While revenue is not part of the evaluation process for proposals, there are other factors that the project team often considers. These factors are discussed in the Section L and Section M portions of the Request for Proposals (RFPs). Section L contains instructions for formatting, organizing, and submitting proposals; it often also includes notices and conditions related to the requested work. Section M describes how proposals will be evaluated and includes the criteria to be considered when proposals are being evaluated.
We subsequently found five variables related to contractor qualifications that had a significant impact on CPI stability: past performance time limit, acceptance period length, length of transition period, past performance factor rank, and key personnel factor rank. Past performance is an indication of experience and demonstrated success. RFPs stating that past performance evaluation would be limited to five years led to projects achieving earlier CPI stability than RFPs stating a limit of three years or not specifying any limit. Limiting past performance to five years ensures that evaluations are based on more recent contractor performance and not outdated performance that may have relied on obsolete technology or practices. However, using a shorter time limit, or not specifying a time limit at all, may result in contractors who lack sufficient experience.
Another variable related to the contractor’s qualifications that significantly affected CPI stability was the length of the acceptance period or the length of time that a contractor’s proposal is valid. For the RFPs we analyzed, the acceptance period ranged from 180 to 380 days. As the results in Table V indicate, we found that acceptance periods greater than 200 days tend to result in projects reaching CPI stability significantly earlier than acceptance periods with less than 200 days. Requiring a contractor to develop a proposal that must be valid for a longer period of time may lead to better planning and more effective risk management efforts from the contractor, thus leading to earlier CPI stability.
A third variable is the length of the transition period from when the contract is awarded to the beginning of performance. When a transition period was specified, it was either 60 or 90 days. We found that transition lengths of 60 days led to projects experiencing CPI stability significantly earlier than transition lengths of 90 days. Shorter transition times help ensure that the selected contractor is capable of mobilizing in a timely manner. Additionally, longer transition times could potentially cause changes in scope or result in outdated requirements, either of which could lead to performance baseline changes and delayed CPI stability.
The last two variables related to the contractor’s qualifications concern the importance of the contractor’s past performance and key personnel. For the RFPs we analyzed, the past performance factor was ranked as the third, fourth, or sixth most important factor in Section M. Interestingly, RFPs in which past performance was ranked as the fourth or sixth resulted in projects experiencing earlier CPI stability than those in which past performance was ranked third. Similarly, key personnel was ranked as either the first or second most important factor; we found that RFPs in which key personnel was ranked second tended to result in projects achieving earlier CPI stability than those in which key personnel was ranked first. Although the results for these last two variables are somewhat surprising, the key is to ensure that the variables are considered as part of the evaluation process; it may also add further credence to the importance of a balanced evaluation approach.
Communication.
The RFP is a critically important document whose primary purpose is communication; by describing the requirements for a project (i.e., specify the need) and expectations for proposals to be submitted, it ensures that the organization procures competitively priced solutions. Effective RFPs provide additional information, such as the organization’s strategy and objectives, to help contractors develop a better understanding of the project and thus submit proposals that align with the stated requirements. From that perspective, we found four variables related to the instructions provided in RFPs that had a significant impact on CPI stability: the number of pages in Section L, the weighting of evaluation criteria, past performance rank, and key personnel rank.
Contracts for which there were less than 35 pages of instructions in Section L of the RFP achieved CPI stability significantly earlier than contracts with 35 pages or more. This is contrary to our initial assumption that the length of instructions represents a higher level of effort in explaining the process and managing expectations, which would lead to earlier CPI stability. Instead, it may be that the length of instructions represents more complexity in a project or “buries” critical information in excessive verbiage; in either case, it tends to delay CPI stability. They key is to effectively communicate requirements, expectations, and instructions in an organized manner while minimizing unnecessary and extraneous information.
An important aspect of the RFP is to describe the factors that will be used to evaluate proposals. This provides further context to help contractors develop their proposals accordingly. As the purpose of the evaluation criteria is to ensure consistent, objective, and unbiased scoring, the evaluation criteria should be designed to include measures and weights. The measures should consist of scales and definitions for each point on the scale, while weights identify the relative importance of each criteria to the organization. We subsequently found that RFPs that did not assign weights to evaluation criteria resulted in projects exhibiting significantly earlier CPI stability. This might indicate that sharing the weights in the RFP could affect the manner in which proposals are developed. In other words, contractors may place too much effort in “gaming” the scoring system instead of developing a sound proposal. From the organization’s perspective, it might also suggest that a balanced evaluation approach is more effective than subjectively weighting evaluation factors, especially if the evaluation team lacks experience and is struggling with determining the appropriate weights.
The two remaining factors included the ranking of past performance and key personnel. As discussed in the Contractor Qualifications section, we found that higher rankings of each factor resulted in later CPI stability. As with the weighting discussion above, higher ranks for these factors may result in contractors placing too much effort on them at the expense of other factors. It also reinforces the notion that a balanced approach, in terms of the proposal development and proposal evaluation, is more effective.
Stakeholder engagement.
We viewed the requirements definition and regulation factors to be representative of efforts to engage the primary stakeholders. It is our contention that more upfront communication with stakeholders helps organizations more effectively define their requirements, which should lead to earlier CPI stability. We subsequently discretized each of the requirements definition variables until either a significant difference or no difference was found. All three variables indicate a statistically different CPI stability point, as shown in Table V. Our initial assumption was that projects with better defined requirements, as indicated by fewer non-funding modifications, would display significantly earlier CPI stability. Contrary to this assumption, we found that contracts with more non-funding modifications tended to achieve CPI stability earlier. A possible explanation for this is that a higher number of non-funding modifications indicates that the project team is actively managing the project, proactively responding to any changes that occur, and adjusting the project as needed.
The only variable for regulation that significantly affected CPI stability was the number of regulators (or regulatory agencies). Of the contracts we analyzed, there were two to four, regulatory agencies involved. As the number of regulatory agencies increased, the assumption was that trying to satisfy their respective requirements would add a layer of complexity to a project and cause the CPI to stabilize later. However, we found that contracts with four regulatory agencies involved reached CPI stability earlier than contracts with two regulatory agencies. It could be argued that more regulatory agencies in this instance meant more stakeholder involvement, which could lead to a more well-defined RFP and performance baseline.
Contracting strategy.
We associated two variables, contract type and the number of projects within a contract, which significantly impacted CPI stability with the overall contracting strategy. The three types of contracts included in our analysis were cost-plus-incentive-fee (CPIF), cost-plus-award-fee (CPAF), and cost-plus-fixed-fee (CPFF). This order of contract types represents decreasing cost risk to the government. We initially thought that CPI stability would be achieved earlier for contracts representing less risk for the government; in other words, our assumption was that CPFF contracts would achieve CPI stability earlier and CPIF contracts would achieve CPI stability later. However, we found that CPAF contracts achieved CPI stability earlier than both CPFF and CPIF contracts. This seems to indicate that equitably sharing the risk results in improved CPI stability.
The second contracting strategy variable to have a significant difference was the number of projects within a contract. After the variable was discretized, we found that contracts with less than nine projects achieved CPI stability significantly earlier than contracts with nine or more projects. The primary insight from this result is that contracts with fewer projects provide project managers an opportunity to focus more clearly on an assigned project, thus leading to earlier CPI stability.
Competition.
The number of offers/proposals that was received was the only variable that was related to competition. For the contracts we analyzed, there were two, three, or five proposals that were received. Our assumption was that increased competition would motivate potential contractors to spend more time developing their respective proposals, thus leading to earlier CPI stability. However, we found just the opposite – contracts for which three or five proposals were received had CPIs that stabilized significantly later than contracts with two proposals. This might be attributed to the fact that environmental remediation involves quite a bit of uncertainty; it might be possible then that as the uncertainty increases, the number of interested contractors decreases. Furthermore, the contractors submitting a proposal might have more expertise and may have put more effort into the development of their proposal, thus leading to earlier CPI stability.
Earned value management factors.
EVM variables included the contract BAC, BCWP stability, and the project’s final CPI. We found that projects with BACs greater than $100m achieved CPI stability significantly earlier than projects with BACs less than $100m. At first, this seems somewhat counterintuitive, especially if higher costs are considered to represent higher levels of complexity and uncertainty. However, it may be that the contractors on the larger projects have higher levels of expertise and more established procedures; therefore, they tend to perform better and achieve CPI stability earlier.
For the BCWP stability variable, some projects had monthly EVM data with negative BCWP amounts. As this created discontinuities in the data which affected CPI calculations, these projects were labeled as unstable. We found that projects without these discontinuities, labeled as stable, reach CPI stability significantly earlier than those with discontinuities. Negative BCWP amounts could thus be considered as an indicator of potential trouble and should be investigated to minimize the impact on CPI stability.
The last EVM variable of significance was the final CPI of a project. We discretized this variable into three groups:
projects that ended over budget (with a final CPI less than 0.90);
projects that ended on budget (with a final CPI between 0.90 and 1.10); and
projects that ended under budget (with a final CPI greater than 1.10).
Our assumption was that projects with superior cost performance will generally achieve CPI stability earlier than other projects. However, we found that projects finishing under budget exhibited significantly later CPI stability. This could indicate that projects ending under budget do not follow the original baseline very closely, which could potentially lead to later CPI stability.
Macro project factors.
This final section examines the following four variables: type of project, project start date, location, and contract number. When considering the type of project, capital asset (C) and American Recovery and Reinvestment Act (R) projects displayed significantly later CPI stability than operations (O) and prior year (PY) projects. This supports our assumption that more complex projects, such as capital asset projects, generally have CPIs that stabilize later than other types of projects. Although the American Recovery and Reinvestment Act projects were grouped based on a funding source and not necessarily project type, these projects were accelerated by using funds provided by the Act; this may have led to less planning and thus later CPI stability. Projects classified as operations tend to be more routine with less uncertainty; therefore, we would expect them to achieve earlier CPI stability. Similar, PY projects represent a continuation of projects from previous years; the uncertainty associated with these projects is much lower, which results in very early CPI stability.
In terms of project start date, our assumption was that projects with later start dates would achieve earlier CPI stability. We thought that “lessons learned” from previous projects would translate into improved performance on later projects. However, we found that projects starting in 2009 or later had significantly later CPI stability points than projects that started before 2009. This trend could be a result of inaccurate EVM data; however, it could also be an indication of some underlying issues affecting the development of accurate cost estimates and performance baselines.
The last two variables were location of work and contract number. For the location of work, the CPI stability points ranged from 15 to 64 percent. Although there is a significant difference in CPI stability based on location, there was only one pair of locations for which the difference was significant (comparing 21 to 63 percent). Similarly, for the contract number, the stability points range from 11 to 67 percent, yet there were only four pairs of contracts for which the difference was significant. With both variables, it is difficult to make any generalizations because of the number of different variables that influence them.
Conclusions
This research has provided important insight into when the CPIs of environmental remediation projects typically stabilize and which factors significantly contribute to CPI stability. Contrary to the 20 percent CPI stability rule often used on projects, we found that environmental remediation projects do not typically have stable CPIs by the time they are 20 percent complete. Instead, we found that the CPI of environmental remediation projects usually stabilizes when projects are about 40 percent complete, although this may vary based on the factors we identified as having a significant impact on CPI stability. Depending on the situation, other factors have a significant impact as well. It is difficult to determine whether the environmental remediation projects are “violating” the commonly held 20 per cent CPI stability rule or if the heuristic is not applicable to these types of projects. Either way, it is important to understand when CPIs typically stabilize in one’s project portfolio, and this research provides a tool to help project managers develop a better understanding of when the CPIs for environmental remediation projects stabilize.
The point to keep in mind is that a project’s estimate at completion (EAC) is largely based on a stable CPI; therefore, the current EAC is likely inaccurate if the cumulative CPI for the project is not yet considered stable. Based on our research results, we summarize the primary insights gained from our research to help project managers focus on CPI stability.
Plan ahead regarding the contractor qualifications most appropriate for the project. We found that contractor revenue, past performance time limit, and length of the acceptance and transition periods had a significant impact on CPI stability. Other factors may be applicable depending on the situation. The key is to ensure that the desired qualifications are adequately captured, perhaps through proxy variables, in Sections L and M of the RFP.
Recognize that the RFP is a critical tool for communicating the requirements and expectations associated with the project such that contractors deliver sound proposals. We found that a balanced approach, in terms of both proposal development and proposal evaluation, is more effective than placing disproportionate levels of importance on specific areas.
Stakeholder engagement is equally important during the requirements definition and execution phases of the project.
A contracting strategy that equitably shares the cost risk associated with a project between the organization and the contractor helps ensure a win-win situation for both parties.
Placing too many projects on the same contract may result in project managers not being able to clearly focus on their assigned projects.
We recommend that policies be put in place that require examining the CPI stability throughout the life of a project and address the insights above. Additionally, the expected CPI stability point of a project could be determined based on the variables examined in our research.
Although it was not a central part of our research, we also briefly analyzed whether any variables were found to have a significant influence on both cost performance and CPI stability. Although superior cost performance is not equivalent to early CPI stability, the following variables were found to impact both metrics: number of non-funding modifications, number of offers received, contract type, number of pages in RFP instructions, and acceptance period length. Conversely, some variables were found to have opposing impacts, such as when cost performance was superior but there was later CPI stability, or vice versa. These variables were number of non-funding modifications per $1bn, percentage of total modifications that were non-funding, number of projects, and annual contractor revenue. More research would be required to further explore these preliminary observations.
This research effort also opens up many other areas for future study. In particular, the schedule performance and stability of environmental remediation projects should be examined in the future. The emergence of earned schedule metrics has increasingly proved useful for overcoming the limitations of the traditional EVM schedule performance index. The variables used to test for trends in cost performance could also be compared to schedule performance. Finally, other statistical techniques continue to develop to help predict future project performance based on EVM data. Lipke et al. (2009) developed a statistical forecasting method using EVM data that could be used; additionally, other potential change detection algorithms could be used to more proactively identify future project management problems.
Figures
Figure 1.
CPI stability for hypothetical project
Definitions of CPI stability methods
| Method | Definition | Source |
|---|---|---|
| Range method | CPI is stable at the earliest percent completion point where the difference between the maximum CPI (CPImax) and minimum CPI (CPImin) from that point to the end of the project is less than 0.20 | Christensen and Payne (1992) |
| Interval method | CPI is stable at the earliest percent completion point where all subsequent CPI observations are within ± 0.10 of the cumulative CPI at that point | Christensen and Payne (1992) |
| Final range method | CPI is stable at the earliest percent completion point where the difference between the CPI at that point and the final CPI observation (CPIfinal) is less than 0.10 | Christensen and Templin (2002) |
Note: References to CPI are interpreted as the cumulative CPI
Hypothetical CPI stability method comparisons
| Method | Earliest CPI stability point (%) |
|---|---|
| Range method | 20 |
| Interval method | 25 |
| Final range method | 20 |
Independent variables
| Factor | Variable | Source |
|---|---|---|
| Requirements definition | Number of Non-Funding Modifications | USA Spending |
| Non-Funding Modifications per $1bn | ||
| Percent Non-Funding Modifications | ||
| Regulation | Davis-Bacon Contract | USA Spending |
| Service Contract Act | ||
| Number of Regulators | 2103 Budget Request | |
| Contracting strategy | Performance Based Contract | USA Spending |
| Solicitation Procedures | ||
| Number of Offers Received | ||
| Contract Type | ||
| Total Contract Obligation ($) | ||
| Number of Projects within a Contract | Project data base | |
| Proposal evaluation | Recipient from State of Performance | USA Spending |
| Contractor Revenue | ||
| Pages in Section L | Request For Proposal Section L |
|
| Section L Attachments | ||
| Technical Volume Page Limit | ||
| Cost Volume Page Limit | ||
| Cost Estimate Required by WBS | ||
| Required WBS Level in Cost Estimate | ||
| Past Performance Number Limit | ||
| Past Performance Time Limit | ||
| Transition Length Provided | ||
| Length of Transition Period | ||
| Pages in Section M | Request For Proposal Section M |
|
| Number of Factors Ranked | ||
| Factors Assigned Percentage Values | ||
| Experience Factor Rank | ||
| Past Performance Factor Rank | ||
| Safety Factor Rank | ||
| Technical Factor Rank | ||
| Transition Factor Rank | ||
| Small Business Factor Rank | ||
| Risk Factor Rank | ||
| Program Management Factor Rank | ||
| Key Personnel Factor Rank | ||
| Organizational Factor Rank | ||
| Community Involvement Factor Rank | ||
| Acquisition schedule | Acceptance Period Required | Request For Proposal Section L |
| Acceptance Period Length | ||
| Descriptive characteristics | Project Type | Project data base |
| Location of Work | ||
| BAC | ||
| Project Start Date | ||
| BCWP Stability | ||
| Final CPI | ||
| Contract Number |
CPI Stability measures by method
| CPI stability method | Mean (%) | Median (%) | IQR (%) |
|---|---|---|---|
| Final range | 25 | 12 | 36 |
| Range | 32 | 24 | 45 |
| Interval | 44 | 41 | 56 |
Summary of variables with statistically different CPI stability
| Variable | Earlier stability | Later stability | |||
|---|---|---|---|---|---|
| Group | Median CPI Stability point (%) | Group | Median CPI Stability point (%) | p-value | |
| Project Type | PY | 10 | C | 67 | < 0.0001 |
| PY | 10 | R | 62 | < 0.0001 | |
| O | 20 | R | 62 | < 0.0001 | |
| Budget at Completion (BAC) | ≥$100m | 19 | < $100m | 60 | < 0.0001 |
| Project Start Date | Before 2009 | 18 | 2009 and After | 62 | < 0.0001 |
| Final CPI | 0.90-1.10 | 24 | >1.10 | 72 | < 0.0001 |
| Number of Offers Received | 2 Offers | 22 | 5 Offers | 60 | 0.0002 |
| Project Type | O | 20 | C | 67 | 0.0003 |
| Contract Number | Not applicable | 0.0008 | |||
| Percent Non-Funding Mods | ≥50% | 24 | <50% | 52 | 0.0009 |
| BCWP Stability | Stable | 34 | Unstable | 64 | 0.0012 |
| Contractor Revenue | Large (>$3.2bn) | 11 | Medium ($100m-3.2bn) |
42 | 0.0014 |
| Number of Offers Received | 2 Offers | 22 | 3 Offers | 41 | 0.0022 |
| Contract Type | CPAF | 30 | CPFF | 59 | 0.0025 |
| CPAF | 30 | CPIF | 61 | 0.0032 | |
| Factors Assigned Percentage Values | No | 31 | Yes | 54 | 0.0041 |
| Contractor Revenue | Large (>$3.2bn) | 11 | Small (<$100m) | 46 | 0.0053 |
| Number of Regulators | 4 | 23 | 2 | 63 | 0.0063 |
| Location | Not applicable | 0.0083 | |||
| Past Performance Time Limit | 5 years | 35 | 3 years | 59 | 0.0160 |
| Pages in Section L | <35 pages | 36 | ≥35 pages | 52 | 0.0200 |
| Length of Transition Period | 60 days | 15 | 90 days | 58 | 0.0204 |
| Non-Funding Mods per $1bn | ≥50 | 24 | <50 | 50 | 0.0212 |
| Past Performance Time Limit | 5 years | 35 | No limit | 64 | 0.0215 |
| Past Performance Rank | Rank 6 | 11 | Rank 3 | 63 | 0.0262 |
| Final CPI | <0.90 | 49 | >1.10 | 72 | 0.0300 |
| Number of Non-Funding Modifications | ≥150 | 30 | <150 | 44 | 0.0371 |
| Acceptance Period | ≥200 days | 36 | <200 days | 59 | 0.0381 |
| Number of Projects within a Contract | <9 projects | 14 | ≥9 projects | 43 | 0.0397 |
| Key Personnel Rank | Rank 2 | 34 | Rank 1 | 55 | 0.0398 |
| Past Performance Rank | Rank 4 | 43 | Rank 3 | 63 | 0.0493 |
Notes: CPFF: cost plus fixed fee; PY: prior year; R: American Recovery and Reinvestment Act; CPAF: cost plus award fee; O: operations; BCWP: budgeted cost of work performed; CPIF: cost plus incentive fee; C: capital asset; CPI: cost performance index
References
Christensen, D.S. and Heise, S.R. (1993), “Cost performance index stability”, National Contract Management Journal, No. 25, pp. 7-15.
Christensen, D.S. and Payne, K.I. (1992), “Cost performance index stability – fact or fiction? ”, Journal of Parametrics, Vol. 12 No. 1, pp. 27-40.
Christensen, D.S. and Templin, C. (2002), “EAC evaluation methods: do they still work?”, Acquisition Review Quarterly, Vol. 9, pp. 105-116.
Defense Acquisition University (2013), “DAU gold card”, Retrieved from Acquisition Community Connection, August, available at: https://acc.dau.mil/
Fleming, Q.W. and Koppelman, J.M. (2008), “The two most useful earned value metrics: the CPI and the TCPI”, CrossTalk, the Journal of Defense Software Engineering, Vol. 1, pp. 16-18.
Henderson, K. and Zwikael, O. (2008), “April Does project performance stability exist? a re-examination of CPI and evaluation of SPI(t) stability”, Cross Talk, the Journal of Defense Software Engineering, April Vol. 21 No. 4, pp. 7-13.
Hollander, M., Wolfe, D.A. and Chicken, E. (2014), Nonparametric Statistical Methods, John Wiley and Sons, Hoboken, NJ.
Kwak, Y.H. and Anbari, F.T. (2012), “History, practices, and future of earned value management in government: perspectives from NASA”, Project Management Journal, Vol. 43 No. 1, pp. 77-90.
Lipke, W. (2007), “Earned schedule: an extension to earned value management”, Retrieved from Stability Point Calculator, available at: www.earnedschedule.com/Docs/StabilityPointCalculatorCopyright2007Lipke.xls
Lipke, W., Zwikael, O., Henderson, K. and Anbari, F. (2009), “Prediction of project outcome: the application of statistical methods to earned value management and earned schedule performance indexes”, International Journal of Project Management, Vol. 27 No. 4, pp. 400-407.
McClave, J.T., Benson, P.G. and Sincich, T. (2011), Statistics for Business and Economics, 11th ed., Prentice Hall, Boston.
PARCA (2014), “What is earned value management?”, Retrieved from Earned Value Management, available at: www.acq.osd.mil/evm/faqs.shtml
Payne, K.I. (1990), “An investigation of the stability of the cost performance index”, MS thesis, Air Force Institute of Technology.
Petter, J.L., Ritschel, J.D. and White, E.D. III (2015), “Stability properties in department of defense contracts: answering the controversy”, Journal of Public Procurement, Vol. 15 No. 3, pp. 341-364.
Popp, M. (1996), “Probability distributions of CPI at complete vs CPI today”, NAVAIR Report Unpublished.
US GAO (2013), “High-Risk series: an update”, Report to Congressional Committees, available at: www.gao.gov/assets/660/652133.pdf (accessed 30 April 2013).
Further reading
Lipke, W. (2003), “Schedule is different”, The Measurable News, pp. 31-34.
Lipke, W. (2012), “Earned schedule contribution to project management”, PM World Journal, Vol. 1 No. 2.
Acknowledgements
Disclaimer: The views expressed in this thesis are those of the author and do not reflect the official policy or position of the United States Air Force, DoD, or the US government. This material is declared the work of the US Government and is not subject to copyright protection in the USA.