Practices of Approximation: Simplifying the Complex and Controlling the Future

Linear Production Models at the Core

Over the years, numerous methods and technologies have seen the light of day in development aid, all produced in an aim to simplify complexity and somehow control the uncertain future of aid project processes and effects. In keeping with results-based thinking, a typical model used has been a basic production model that depicts inputs (resources of various kinds) moving through a linear process via which they are turned into outputs. Ideally, these outputs should then in turn lead to tangible outcomes that have a positive impact on the lives of those in need. To this end, many aid bureaucrats have preferred to use rationalistic models, like the one below (Fig. 5), that rest upon the largely taken-for-granted idea that later results (output, outcome, and impact) can be clearly linked to earlier inputs and activities. Attempts to make sense of these relations have therefore long been a top priority for aid bureaucrats responsible for project decisions, along with other concerns like internal efficiency and external effectiveness of the operations (Modell & Grönlund, 2006) and hence the prudent use of taxpayer money.

Opens in a new window.

Fig. 5.

The Basic Results Production Model (Vähämäki, 2017).

The basic model for producing results (Fig. 5) has been introduced in a range of different methods and technologies. One of these is the Logical Framework Approach (LFA), a method developed in the 1960s by Fry Consultants Incorporated (1970), a consulting firm contracted by the United States Agency for International Development (USAID). The method came with a technology, in the form of a matrix, that has since often been required by decision-makers of organizations in the donor role to make the activities and projected results of organizations in the recipient role visible (Coleman, 1987; Earle, 2002; Martinez, 2013). When the LFA was introduced in USAID, independent measurement of output and progress toward ultimate project purpose were described as “objectively verifiable data” that could provide aid managers with “a common frame” for evaluating projects and help to reduce their “preoccupation with inputs” (USAID, 1965, p. 8). This suggests that the rationale at that time was that aid bureaucrats were spending too much time approximating the left side of the results production model (i.e., input and activities). Models like the LFA were therefore argued to provide certainty and be valuable means by which to turn aid bureaucrats' attention to results and the right side of the model (i.e., output, outcome, and impact), an argument that came to linger (Vähämäki & Verger, 2019). Since the 1960s, various versions of LFA have spread widely among development agencies and other aid organizations, with the hope and ambition of visualizing trustworthy results. This has not been a problem-free undertaking, however, and some of the key fundaments of managing aid and enduring the difficulty of realizing the dreams of simplifying the complex and controlling the future in practice are discussed below.

Shifting Focus to the Right Side of the Production Model

In the results production model, results are defined as output, outcome, and impact. In the 1970s to the 1990s, actors in the field of international aid engaged in a discussion about the experienced need to redirect their attention from inputs and activities, such as the number of seminars organized or the number of wells dug, to outputs and outcomes, such as the number of people trained or the number of people with access to clean water. The hope was that all aid projects could be planned using a causal logic of what led to what.

In 1971, Sweden's international aid agency, the Swedish International Development Authority (SIDA), implemented a technology called “results valuation.” ² When assessing project proposals from organizations in the recipient role, prior to any funding decisions, all SIDA aid bureaucrats were expected to produce a hypothesis that set out a project's main goals, subgoals, planned production goals, planned activities, and planned costs, along with indicators, i.e., approximations, for each level of the goal hierarchy. When following up the project, calculations were then to be carried out and a comparison made between the projections in the hypothesis and what had actually happened (Vähämäki, 2017). The calculations required numbers to be entered into a formula for the planned fulfillment of the main goals and then divided by the estimated total costs, resulting in a measure of the “planned significance.” A “planned productivity” measure was similarly calculated by dividing the planned production targets by estimated direct costs. During follow-up, the resulting numbers were then to be compared with the “actual productivity” and “actual significance,” yielding a figure for the “actual effectiveness” of the project. If we have confused you here, you are not alone, for in practice, the use of SIDA's results-valuation method proved to be very cumbersome.

Only about 10% of SIDA's staff ended up carrying out the required valuations as intended, with the vast majority claiming that doing so was not possible and, in addition, that requesting this type of data from recipients harmed recipient relations (SIDA, 1974). Discussions arose within the agency concerning whether or not it was even possible to isolate a project's effects in this way. After some years of testing out the technology, SIDA's board made the claim that it was “unrealistic to expect a quantitative assessment of effectiveness.” The National Audit Office, on the other hand, continued to argue that it should be doable (RRV 1972:43 PM8). In the end, the results valuation method was replaced by new methods, and the evolution of new methods has been a pattern observed ever since, not only in the Swedish aid context but also more broadly in international aid (Reinertsen, 2022; Vähämäki, 2017; Vähämäki et al., 2011). However, at the same time as older methods have been replaced by newer ones, most have shared the same ideological foundations: a belief in the ability of rational plans to control the future and yield proper results from complex, messy practice. ³ That is, a belief derived from an unrealistic assumption that decision-making follows a linear or scientific approach with a simple, sequential progression: from inputs to outputs, to outcomes, to impact (Brunsson, 1985).

In everyday practice, a difficulty that has occurred in each attempt to apply results technologies is that aid bureaucrats find it difficult to come up with a reasonable hypothesis. To cite an example, in 1995, the use of the logical framework method was made compulsory, to be appended to all aid projects at the Swedish International Development Agency (Sida). An assessment carried out in 2000 showed, however, that a hypothesis was not apparent in the majority of Sida's aid projects, indicating that only four of the agency's close to 5,000 projects (i.e., 0.08%) had actually met the LFA requirement to rate how well a project was progressing against the initial hypothesis. A few years later, in 2004, after intensive internal project work and an internal campaign stressing the need to comply with the mandatory requirements, compliance did increase to 25% (easily seen in the digital system), although even this number demonstrates that most projects were still not using LFA or a similar method (Vähämäki, 2017).

A recurring comment from evaluations of aid projects over the years has been that many projects managers have not used rational planning methods as intended (Burman, 2021). A primary reason given for this is the inherent difficulty in actually defining what should be considered as an input, output, outcome, or impact. The task is further complicated by the fact that organizations in the donor and recipient roles usually have different understandings of these elements (Brolin, 2017; Eyben, 2010; Vähämäki, 2017). One of our interviewees from Volvo, a private sector car company that collaborates with Sida, for example, noted how he was puzzled to learn that Sida considered its hosting of a meeting with businesses a project output. As this businessman himself put it, he and the representatives from other firms attending the meeting found this a little frustrating since: “After all, it was only a meeting. We hadn't achieved anything yet, but they [Sida] seemed to think so.” Hence, for Sida, in the donor role, the meeting was a result (an output or outcome) since cooperating with the private sector was a political priority, whereas all of the meeting's other participants, playing the roles of partner or recipient, were eager to progress “from talk to proper action on the ground” and saw the meeting as no more than a planning activity (an input).

Contrary to the hope-filled management dream of simplifying the process, and relevant to our interest in obsessive measurement disorder (OMD), we have noted that discussions on what should be counted as a result in the results production chain often lead to more, not less, information being produced and processed, and to confusion, not clarity. We have also observed that, in practice, it is more common to report on inputs or outputs (the left side of the ideal-typical model in Fig. 5) since this information is often easier to quantify and sometimes the only data available when a project decision has to be made (Binnedjikt, 2001; Mayne, 2007; Vähämäki & Verger, 2019). However, considering the example cited above, since there is no standard to clarify whether a meeting should be considered an input or whether it is already an output, rather than being easy to comprehend, the numbers often spur new questions and new numbers.

Even when there are positive results data at hand, which are to the benefit of both parties (organizations in both the donor and recipient roles), there may be confusion regarding who can justifiably take credit for those results. In the highly complex web of financial relations and dependencies, questions concerning the links between particular funding and particular impacts arise. When organizations in the recipient role have several organizations that finance their operations, as most do, it is often almost impossible to determine what money, from which donor, led to what specific results. Nevertheless, project managers are expected to try their best to reach such conclusions. As one Sida research aid manager explained:

Because coming to these conclusions isn't straightforward. This individual got a grant from Sida and now has a seat on that committee. It's not necessarily only thanks to Sida's research unit, but it's fair to assume we helped. That we can see that it relates to matters connected to the research they did with Sida funds, and they're in a stronger position and so on. That we can come to conclusions like that, a little. But it's not an easy task. And especially not with research that is so incredibly long term.

Thus, from a pragmatic standpoint, what usually happens is that all donors involved try to take at least some credit for any good results that materialize, that can in some way be linked to their funding. Since it is difficult to determine dependencies, due to the multitude of interacting partners and projects and the long-term result horizons, project administrators are constantly on the lookout for any positive result that can be reported home. An example of a similar pragmatic stance can be seen with the cholera vaccine Dukoral, which has had immense global impact, where support for the underlying research in the form of grants from Sida and many others began back in the 1980s. Our informants explain that continuous reporting of successful and easily comprehensible results is important even if a result may be seen as an instance of episodic evidence, citing Dukoral as an example. The concrete materiality of the vaccine in its vials also helps in this regard since tangible results are appealing and more easily showcased and communicated to those at a distance.

To summarize, despite trying out different methods and technologies over the years, aid bureaucrats continue to face the same unrealistic expectation of having to show results before they materialize, and it remains hard to tell with any confidence whether aid actually reaches those most in need, in an efficient, effective, and human way. The most recent wave of attempts, of the past 15 years or so, has targeted impacts. As White points out (2010, p. 153), one argument for this move toward impacts instead of outcomes stems from a realization that outcome monitoring “does not tell us about the success, or otherwise,” of government programs or the interventions supported by international development agencies. Impact evaluations, he argues, not only answer the question of what works but also why (White, 2009). However, while the method of “impactization” may be recent, the sales pitch for it follows the traditional script, promising that impact will connote “something visible, clear, objective and calculable in which relevant activity can be causally linked to a desired policy outcome” (Power, 2015, p. 45).

Hence, the underlying argument for impactization closely resembles that of the logical framework approach, as well as many other methods and technologies used in development aid, which is: we don't know much about what happens and why things happen in development aid projects. Only now, in the case of impactization, the insufficiency in focus concerns information about outputs and outcomes rather than inputs and activities, as it did in the 1960s. It is important to note, however, that despite the ongoing struggles and mishaps, followed by new suggestions and ideas, most of the aid bureaucrats we have encountered say that they view these results processes as unquestionably important as legitimizing rituals (Meyer & Rowan, 1977) and mechanisms of hope (Brunsson, 2006), if not as validation of actual results.

The Allure of Numbers

In theory, simplified information could naturally also consist of qualitative indicators or emotional episodic narratives about those helped to a better life. However, qualitative results data seem to be less valued and trusted in the field of development aid (Alexius & Vähämäki, 2020). ⁴ Here, it is evident that simplified information in the form of quantitative facts and figures that represent progress is alluring, that is to say, numbers that communicate that the funds reach the populations they are intended to support (Bowerman et al., 2001; Eyben et al., 2016; Vähämäki, 2015; Vähämäki & Verger, 2019).

Considering the broken feedback loop in development aid, i.e., that aid recipients have hardly no direct feedback loops to the decision-makers in the donor countries (Martens, 2005), it is understandable that numbers have been a particularly lauded driver of development policy, as well as an important means for aid bureaucrats to justify and communicate their decisions. The magic of numbers is that they allow us to imagine that we can freeze the world so that it can be more easily acted upon. As mentioned previously, numbers also offer the potential of taking action and communicating results from a distance (Erlingsdóttir, 1999; Hall, 2010; Robson, 1992).

Previous research has shown, somewhat paradoxically, that quantification tends to be valued most in complex fields such as development aid, where outcomes and effects are typically hard to measure (Jacobsen & Sandvik, 2018). The widespread praise of measuring results we found among aid bureaucrats is in line with the powerful ideal spelled out in Zall Kusek and Rist's Handbook for Development Practitioners, where they list the arguments regarding the “power of measuring results” (Kusek & Rist, 2004, p. 11):

However, as discussed in Chapters 6–8, while a few of the aid bureaucrats treat these arguments very seriously, almost like mantras, most of the aid bureaucrats and especially the more senior are also able and willing to discuss them critically if given the chance. The academic literature on approximations is dominated by critique that questions the validity of proxies and discusses, at length, the negative implications of invalidity. To give just one example, in the climate field, many experiments on biodiversity and ecosystem functions measure “species richness” and assume that this approximation can serve as an indicator of a broader suite of attractive ecosystem functions (Stephens et al., 2015). When evaluated, studies have however shown that species richness may be an unreliable or even invalid proxy with the potential to mislead management strategies in the field of ecosystems (Eigenbrod et al., 2010).

In previous literature, the allure of providing numbers as proxies has been described as coming with several unintended consequences such as that, over time, measurable activities have crowded out activities that are more difficult to measure, and that short-term outputs are pursued over long-term objectives (Natsios, 2010; Vähämäki & Verger, 2019). In some cases, the focus on proxies has led to staff spending an increasing amount of time collecting data and monitoring activities and less time managing and implementing activities (Diefenbach, 2009; Forssell & Ivarsson Westerberg, 2014; Johansson & Lindgren, 2013; Meyer & Gupta, 1994; Natsios, 2010).

The development whereby more time is dedicated to delivering numbers and less to understanding the local particularities of a context, including institution-building and policy reform, has been called “mission drift” – a drift from one particular focus to another (Alexius, 2021). According to Natsios (2010), mission drift represents a clear risk factor for OMD. In our analysis, we have seen instances of mission drift and pressure on aid bureaucrats to focus on proxy numbers rather than devoting time to the implementation and understanding of aid projects. In our interview with the head of research cooperation at Sida, she expressed it as follows:

We’ve seen in these Conclusions of performance that we write every year, where you need to enter numbers, that when our colleagues had filled in these numbers and got them in there, the analysis, then they were so tired of it, that they lost… You couldn’t see the forest for the trees… For me it was completely uninteresting, which should be really exciting, to read these Conclusions of performance, to go through all these numbers. But they didn't match the indicators… And I had to go and look at that instead of reading the text and actually see, what was it, that had happened in the project? So, which are the results really? And these numbers, that have been selected, they’re not just any numbers. So I made it very clear, that I didn't think [it was good].

This quote illustrates a typical ambivalence among aid bureaucrats regarding their production of numbers and approximations. In addition to the risk of proxies overshadowing the “exciting” part of projects, our interviewee also draws attention to two common apprehensions related to producing numbers:

1. The difficulty of validating the numbers. Jerven (2013) has shown that the statistical capabilities in the aid sector are extremely poor. He also argues that the numbers substantially misrepresent the actual state of affairs and warns of the risk of scarce resources being misapplied as a result of aid bureaucrats' poor understanding of statistics.

2. A fear of how numbers will be used. Vähämäki (2017) writes that the “fear of use” is a typical fear among aid bureaucrats. When they are afraid of how decision-makers may understand and use the numbers, and uncertain whether the numbers will be understood as good or bad and how this might affect a project, some aid bureaucrats may even try to avoid providing numbers.

Clearly, there are fundamental questions to be asked about proxy validation and calibration: about whether proxies are reasonable or bad approximations. But, as concluded by Stephens et al. (2015), reducing reliance on flawed proxies would require increasing large-scale, long-term monitoring practices that are seen as high-cost, unnecessary luxuries. ⁵ In our view, it is important that we also look beyond the rational validity critique and acknowledge that as a day-to-day pragmatic response to uncertainty in complex systems, proxies are attractive, despite their flaws.

The Challenge of Mismatching Temporalities

A common feature of uncertain systems is that decision-makers need to be engaged in temporal coordination (Gümüsay et al., 2022). When a multitude of actors are involved, different temporalities are likely to suit different actor groups. Due to the inherent temporality of the aid project itself, on one hand, and the time it takes for results to materialize in the field, on the other, there is a temporal mismatch between the critical and urgent need for short-term decisions and results (in line with the narrow project time frame and consideration of the yearly public spending rule) and the fact that most key results can only be seen in the longer run. One Sida manager engaged in research cooperation describes a typical case where experienced aid bureaucrats, like her, see the need to look at results from a 20-year perspective while at the same time having to cope with the fixed, 5-year project-funding time frame:

We’re now looking at going into Cambodia and our perspective is 20 years. We grant support for five years at a time; we can’t give more than that. But [20 years] that’s the perspective, and that’s when we’ll see the results. That’s when they’ll have a PhD education of international standards and they’ll be able to generate the next generation, or have a sustainable system. But that’s 20 years down the road.

In essence, the mismatch of temporalities implies that project managers are expected to provide regular reports with trustworthy results before these results have had a chance to materialize. Though this may sound like an impossible mission, it is a dilemma that aid bureaucrats deal with every day. Thus, from a pragmatic bureaucratic viewpoint, approximations often come in handy to save the day.

The case of Swedish support to Bai Bang in Vietnam demonstrates this difficulty of mismatched temporalities. The Bai Bang project – Sweden's largest, most expensive, and longest running aid project to date – ran from 1974 to 1995, a period during which Sweden committed about SEK 2.8 billion in total (approx. SEK 9.8 billion in 2022 figures). The idea behind the project was simple: Sweden would provide support to Vietnam to construct an integrated pulp and paper mill that would raise living standards in the Bai Bang region. Implementation of the project, on the other hand, proved considerably more complicated. From the outset, the project faced a range of serious challenges, many of which related to difficulties associated with cooperation between two countries with such different political and economic systems. Adding to these difficulties was the considerable adverse media coverage in Sweden, much of which focused on various controversies that arose, including the living conditions of workers at the mill and in the surrounding forest areas. In 1985, allegations of forced labor in the project arose, creating a political storm in Sweden (McGillivray et al., 2012) and outdating the project completely. Other difficulties included Sida feeling it was not receiving sufficient data to assess the wood supply from the designated timber, and the frequent revisions and extensions of the project time-frame required to overcome the various critical issues that arose. As Sida's director-general recalls:

It was the biggest project in terms of money that we had done, and it took twice as long as we anticipated, and cost twice as much. Bai Bang was such a stupendous effort for Sida so one can debate whether it was actually worth it for the aid authorities from a labor-economics standpoint.

Almost the entire Sida organization was involved in the project, and it was heavily discussed in Sweden among different actors, making it burdensome to deal with. However, an evaluation carried out in 1999 called the project “an aid project which obtained a measure of success despite the odds” and argued that the project had been extremely successful, having produced 110,000 tons of paper, twice as much as the target set in the 1980s (Jerve et al., 1999). Thus, despite the project being seen for many years as outdated, in terms of both the results and the negative effects of the media coverage and extra work it implied for the agency, 20–30 years later, it was considered a success. On June 11, 2022, the Hanoi Times published an article entitled “Bai Bang Paper Mill: Outstanding symbol of Vietnam-Sweden relations.” ⁶

As concluded previously in Chapter 2, grand societal challenges (such as climate change and socioeconomic inequality) are characterized by great uncertainty of the long-term affect. As the case of Bai Bang shows, key results tend to materialize at a rate incompatible with the short-term horizons of local political agendas and decisions (Biddle & Koontz, 2014), with many results visible only after decades of aid work. And when long-term results do finally appear, another uncertainty is actualized: will they help decision-makers to make better decisions in the future, decisions grounded in learning from past experience and assessments?

Lost Momentum for Learning?

The example of Bai Bang points to the risk of losing momentum in learning from project results when projects end long before results start coming in. Hence, despite frequent talk about the importance of aid funds contributing to long-term outcomes and impact on the ground, the tendency is that projects don't learn from long-term results. In the aid relationships we have studied, information on late-incoming results is sometimes toned down or even ignored – a scenario we found to be true in both aid projects carried out by the International Science Program (ISP) and those coordinated by Union to Union (a federation of Swedish unions that supports collaborations in aid). In the case of ISP, numerous reviews of the program (i.e., evaluations from 1977, 1994, 2001, 2002, 2011, and 2018) have found it to be highly supportive in achieving results (Edqvist et al., 1994; GHD Pty Ltd, 2011; Leide et al., 1977; Pain & Carneiro, 2018; Selin Lindgren & Wendiga, 2002; Wield, 2001). Despite the positive reviews, however, as one ISP representative explained, his perception was that providing positive evaluation results was not “what mattered the most” in assessments of future support:

Right, but that’s precisely what it said, that ISP was… that what ISP did was good, ISP is needed… And then it also said in the evaluation later, that our follow-up- and assessment systems weren’t good enough, and this is what we’ve worked on developing the whole time up until the 2018 evaluation, that said the system is still not good. But for other reasons… The results have actually been there all along. It’s just that now Sida was expected to report new types of results.

This representative, ISP's director, also concluded that, despite the positive results, the evaluators criticized ISP for its measuring methods, a critique that has led ISP to try out several new ways of measuring results, methods that have in turn also been criticized. Thus, in the end, the discussion on evaluation has centered on the methods and technologies used to measure results and neglected the activities, outcomes, and impact as such. When interviewed, the former secretary-general of Union to Union expresses similar thoughts:

I think that the results evaluations speak very much for our cause. But that seems unimportant. They [Sida] are like: “Right, well, you’ve attained results…. but is it really cost-effective to organize like this, or like that …

Here again, the secretary-general's comment exemplifies that despite evaluations having found that Union to Union attained positive results, the manner in which it organizes its operations has been questioned and, at times, the interest in cost-effectiveness has come to overshadow the assessment and recognition of results achieved (see also Chapter 5). The fact that the leaders of these organizations perceive that actual results are not what matter most in development aid relations has been discussed in previous studies (Andreoni, 1990; Easterly, 2002; Lindkvist and Bastøe, 2020; Martens, 2002; Pritchett, 2002), all of which argue that, despite altruistic intentions, there is a built-in disinterest in the actual results.

Explanations for this lack of interest are that policymakers mainly care about being re-elected and that aid organizations are mainly motivated by self-interest (in the sense of gaining more funding for their projects) and, unless they are explicitly punished for poor performance, they need not exert maximal effort (Martens, 2002). Another argument, linked to the emotion-based approach to reducing uncertainty (discussed in Chapter 2), is that aid is more about feeling good than doing good, which leads to the decision-makers having a greater interest in disbursements than in actually making a difference for the aid beneficiaries (Andreoni, 1990; Easterly, 2002). Pritchett (2002) argues moreover that knowledge of results may be avoided because it can hamper funding flows, meaning that the public spending rule gets higher priority than aid effectiveness, and that “it pays to be ignorant.” This previous literature also notes an element of fear – that results information and knowledge are avoided due to a fear of hampering the aid organization's routines or harming its reputation.

Our case studies confirm that providing organizations in the donor role with results information that is too complex, even when that information is accurate and positive, may lead to a negative outcome for organizations in the recipient role since doing so may confuse the aid bureaucrat responsible, thereby raising the level of uncertainty rather than clarifying and reducing the uncertainty perceived by the organization in the donor role. As will be discussed in more detail in upcoming chapters, we found this to be particularly common in situations where donor representatives lack sufficient context or the domain-specific knowledge necessary to assess the complex results information at hand.

In sum, we have shown that when aid bureaucrats and aid organizations implement the management dreams of simplification and future control in practice, difficulties and mismatches arise. There is seldom a straightforward response to the question: Does aid lead to results, and – if so – what results? But since the unrealistic expectation – that there is a simple answer – still exists, new practices for approximating results continue to emerge, practices with which aid bureaucrats and aid organizations become preoccupied. When analyzing current project work practices, we have identified what seems to be an increasingly prevalent type of approximating – an approximation practice that centers on what we call “proper organization proxies,” which we elaborate on in Chapter 5.