Methodological Guidelines

There are numerous textbooks that go deeper into the theoretical foundations of CBA and methods (e.g. Boardman et al., 2018) as well as guidelines that more fully lay out the steps of CBA (e.g. Robinson et al. 2019). These will not be re-explained here. Nevertheless, there are important, sometimes subtle methodological differences in how CBA is conducted by practitioners within the field. This section delves into some of the details of the Copenhagen Consensus approach to CBA.Use of evidenceOne of the main analytical challenges is discerning the appropriate evidence as it relates to the beneficiary population in question. Determining which models to adapt can be a complex judgment that should consider contextual relevance, study quality, and literature consistency. For this project, it is clear that preference should be given to high quality analyses from Sub-Saharan Africa, followed by other developing countries having similar socio-economic and demographic conditions. Use of effect sizes from carefully conducted meta-analyses is, of course, also encouraged. However, when the most contextually relevant studies have less robust experimental designs or the literature is divergent in its conclusions, then the parameter choice is less straightforward.Overall, it is important that economists build a case for the parameters used in the broad calculations, referring to the literature for assumptions used.Estimating all significant costs and benefitsMeaningful comparisons across interventions requires that all significant costs and benefits are accounted for. In some cases, this will be difficult to achieve due to imprecision or lack of data. In the case of the Prioritizing the Best Buys for Development Across the African Continent project, the challenge is in estimating costs and benefits for a number of countries. Nevertheless, we encourage commissioned experts to make an attempt to estimate these costs and benefits, rather than leaving them off altogether.We also encourage experts to include estimates of all costs and benefits that are likely to be significant in the analysis. This might entail examining benefits outside the primary aim of the intervention. For instance, when looking at climate change adaptation by planting mangroves, the benefit is not just climate protection, but also improved biodiversity and potential higher incomes to fishers. Likewise, an education intervention will not just increase earnings but will also improve nutritional profiles.BCR vs. net benefitsWhile many CBAs highlight net benefits (B minus C), our preferred metric is the benefit-cost ratio (BCR), which is benefit (B) divided by cost (C). The reason for this is that in developing country contexts the ability to raise funds (through taxation or debt), or the ability to enforce regulation is constrained. This is perhaps different to developed countries, which can typically raise money and legislate with much greater ease. Given a fixed pool of available funds (as is the case in our ‘thought experiment’ set up, as well as in actual political decision making), a strategy that begins with implementing the highest BCR intervention and continues in decreasing order until money runs out will maximize social welfare. A strategy that starts with the highest net benefits intervention continuing in decreasing order, almost certainly will not.For example, suppose a policy maker has $100 and is faced with the following options

If the policy maker chooses based on the intervention with the highest net benefits, then she would choose policy A, run out of money and generate net benefits of $200. However, if ranked on BCR, she would implement policy C, and then policy B, before running out of money. This would generate $340 in net benefits.Classification of costs and benefitsIt is important that costs and benefits are classified consistently to ensure comparable BCRs. Robinson et al. (2019), suggest that inputs into a process (such as materials and labor) should be classified as costs, while outcomes (such as mortality risk reductions or increased productivity) should be classified as benefits. This approach is intuitive, and we generally ascribe to this methodology.What should be included on the side of the cost equation? The total costs associated with the design and implementation of an intervention includes direct and indirect costs, as well as the opportunity cost. Take for example, an intervention to end child labour in India. The National Child Labour Project, has as its principal objective the decrease in the number of working children, ages 9 - 14, and particularly those engaged in hazardous occupations. One of the interventions associated with this programme is the establishment of special schools and rehabilitation centres, which provide bridge education, vocational training, mid-day meals, health care recreation etc. to the children, with the ultimate objective of preparing them to be mainstreamed into the formal education system.A cost-benefit analysis of these special schools must take into account the capital costs (e.g. building construction) and the recurrent costs of making education and health services available (e.g. teachers, nurses, pedagogic materials). There is also the opportunity cost of the intervention: the foregone income to the household from child employment. The children’s financial contributions to households being considerable, adding a small stipend to subsidize household expenditures, rather than assuming that the returns to education are sufficiently high, mitigates the next best alternative, which is sending the child to work. The National Child Labour Project does indeed include a stipend to families. In this case, an estimation of the foregone income would be added to the cost side. Neglecting the opportunity costs of an intervention can not only result in sub-optimal financing but also the mis-targeting of beneficiaries, consequently undermining its success.What should be included on the benefit side of the equation? In order to adequately capture the benefits of a proposed intervention, the beneficiary groups must be well-identified. It does not suffice to identify just women as beneficiaries: Are they living in rural or urban areas? Which age group is targeted? Which ethnic/social groups? Does marital status impact the delivery of the intervention? The more defined the beneficiary group, the more precise our estimates, the more likely planners will be able to adjust the intervention in order to maximize its effectiveness.Furthermore, an intervention, which leads to cost reductions, should be included on the benefit side of the equation. Take, for example, the adoption and integration of information technology in government operations. Generally, the benefits of such interventions include a reduction in service delivery costs including fewer public servants needed to render services (a benefit); a time savings experienced by citizens using the service (a benefit), and a reduction in the number of times a citizen has to travel to a public office (also a benefit).An additional principle we employ is that absolute benefits and costs should be considered where possible, with no netting off benefits or costs. For example if agricultural extension services cost $5 and this leads to increased farm revenue of $45, yet also increased farm costs of $10, we would estimate the benefits as $45, and the costs as $5 + $10 = $15 for a BCR of 3. We would not net off the revenue and costs (i.e. profit) for benefits of $35, costs of $5 and a BCR of 7. The reason for this is that the true resource cost of the intervention – the amount that is consumed from the fixed pool of funds available to society is $15 and not $5, and so the result from the first approach better captures the return on investment.

Treatment of transfers

Interventions involving transfers are an area where consistent classification matters greatly. Transfers tend to fall under the field of social protection and include unconditional cash transfers, conditional cash transfers, food transfers and subsidized insurance. In this case, the transfer appears as both a cost and a benefit in the BCR equation. It should not be netted out. For example, consider an unconditional cash transfer of $100. Suppose the administrative costs of delivering the transfer are $5 while the transfer delivers consumption-smoothing benefits of $10 to recipients. In this case, the benefits are $110, while the costs are $105 for a BCR of 1.04. If one were to net out the transfer (incorrectly), the intervention would appear as benefit = $10 and cost = $5 for a BCR of 2. However, as above, the real resource cost of the intervention is $105, not $5, so 1.04 is, in our estimation, more accurate reflection of the social return.

Time frame of analysis

In terms of the appropriate time frame of analysis, there is one principle: the time frame should be long enough to capture the most important future flow-on effects (typically benefits, but sometimes also costs) from a given intervention. The exact length will vary by analysis. For example, since infrastructure lasts for decades, CBAs of roads, public transport, sewage networks and other major capital works should take at least a 20 year (or more) time horizon to capture all the benefits. In contrast, the costs and benefits of say, crop insurance can be modeled as a one year steady-state intervention, since typically insurance covers only that year’s crop, with next year’s insurance covering next year’s crop and so on.  Importantly, as long as the time frame used captures all material flow-on effects, differences in time do not affect the comparability of interventions when using benefit-cost ratio as the metric of interest.

The analytical base year is 2018

For those economists not working from peer-reviewed publication and/or working on an entirely new intervention, the analytical base year for the Prioritizing the Best Buys for Development Across the African Continent project is 2018. This means that all costs and benefits should be reported in 2018 United States dollars. Costs sourced from earlier years should be inflated to the analytical base year using a GDP inflation index, though it is discouraged, when it can be avoided, to use data before 2016. Additionally, forecasts of costs and benefits only need to account for real growth and should ignore inflation. Additionally, all interventions should take the initial conditions of the year 2018 (or as recently as data allows) and assess the effects against this baseline.

Political considerations

All political costs regarding the decision to implement should be ignored, while political fall-out in actual implementation should be considered. In other words, all cost-benefit analyses should take as a starting point the hypothetical scenario where the decision is already made to implement the intervention. Costs associated with advocacy, campaigning, etc. to encourage implementation should be ignored. 
However, if the completed decision may make politicians decide to cheat or skim the process, this simply means a smaller benefit or a larger cost and should be included (along with all other risks, and challenges in implementation).

The concept of risk

BCR estimates should be revised downward to incorporate well-documented assessments of risk. For example, where it relates to microfinance, it is generally recognized that 2% of borrowers are at risk of default. This risk should be worked into the calculations; in this case, it is an additional cost to the lender.

Implementation failures

To the extent that the data allows, commissioned economists should account for implementation failures such as corruption and incompetence. The most straightforward way to account for this is to adopt parameter estimates from studies with high quality methods (e.g. randomized-controlled trial, difference-in-difference, regression discontinuity) which should theoretically embed all the vagaries of implementation into the effect size. However, recent literature around RCTs documents divergence between small-scale pilots and real-world implementation. In disciplines where these studies are not possible or uncommon, we suggest carefully considering to what extent the evidence represents ideal or non-realistic scenarios with respect to the actual local context and adjust accordingly.

Equity weights

As with most CBAs, as traditionally adopted, Copenhagen Consensus assigns an equal weighting to all costs and benefits regardless of who obtains or pays them. The one exception is for individuals who illegally obtained assets via corruption or theft, which we assign a weight of zero. So for example, in an intervention which reduces corruption, the loss of corrupted funds does not count as a cost in the societal cost-benefit calculation.

Jobs vs. output

Cost-benefit analysis, as is traditionally adopted, does not count the creation of jobs as a benefit. Instead the focus should be on the flow on effects of job creation – either output, income or consumption. The primary reason the value of jobs differs depending on the state of the labor market in question, and this is better determined by examining flow-on effects (the increase in output or the increase in incomes) rather than the monetary value of the number of jobs created.

If the intervention under analysis specifically targets job creation – such as a workfare program like India’s rural guarantee scheme – economists need to examine the broader general equilibrium effects to understand the impact in a cost-benefit framework.

Important common assumptions and approaches for Prioritizing the Best Buys for Development Across the African Continent project

Wages and wage forecasts

Wages and wage forecasts are required for estimating productivity and education benefits as well as time costs / benefits. The Center encourages the use of GNI per capita forecasts where:

Wages = GNI per capita * labor force participation * labor share of income

GNI per capita and GNI per capita growth for all Sub-Saharan African countries were distributed to experts.

Discount rates

We acknowledge there is considerable debate around the appropriate discount rate to use in economics, as well as the fact that discount rates differ with country context. Considering that we are analyzing countries at various stages of development, we would like experts to report BCRs at 5%.

Valuing mortality and morbidity

Valuations of mortality and morbidity follow recent guidelines developed under the Harvard led Guidelines for Conducting Benefit-Cost Analysis project (Robinson et al. 2019). These guidelines suggest a range of approaches. Given time constraints, we adopt one of these approaches for this project. Copenhagen Consensus’ preferred approach is to convert each death avoided into years of life lost (YLL) avoided, using the relevant life tables, and to value each YLL at 1.3x GNI per capita. YLLs should not be discounted.

This preferred approach was derived by taking a VSL value of $9.4m USD (2015 dollars) – representing approximately 160 times income as measured by income per capita PPP - transferred to the continent using an income elasticity of 1.5. In 2017, GNI per capita PPP for sub-Saharan Africa was Int$3700 while the corresponding value for the US was Int$61,120 (World Bank, 2019). Using these figures and applying the approach documented in Robinson et al. (2019) suggests a VSL to GNI per capita multiplier of approximately 39x for the continent.

Life years are valued using a constant value of statistical life year (VSLY). A VSLY is typically derived by dividing the VSL by the average life expectancy of an adult of average age, proxied by half the life expectancy at birth. In sub-Saharan Africa, life expectancy at birth is 61 (World Bank, 2019), implying 30.5-year life expectancy for an adult of average age. The value of a YLL therefore, as a function of GNI per capita is 39 / 30.5 = 1.3.

In terms of morbidity avoided, the Guidelines recommend adopting a cost-of-illness approach. However, this approach can be very data intensive. For parsimony, we suggest here estimating the Years of Life Lost to Disability (YLDs) avoided from morbidity benefits, and applying the same multiplier for YLLs i.e. 1.3xGNI per capita.

In summary all DALYs (whether YLLs or YLDs) should be valued at 1.3xGNI per capita and not discounted.

Value of time

Following Whittington and Cook (2019), we assess the value of time which can be put to use for productive purposes at 100% of wages, while time that cannot be applied to productive purposes is valued at 50% wages for the population in question. Analysts should be careful to include the cost of time required to access the services provided by interventions, particularly for health programs.

In some instances, economists will have to value time of children. While there appears to be no agreed consensus on appropriate valuation, it seems reasonable that i) the value should be lower than productive adult’s time and ii) very young children probably have a zero or even negative value of time (e.g. if children are not at school, adult caregivers are required). So we suggest applying a value of zero for the time of children less than 10 years old. This is consistent with the returns to education literature (e.g. Psacharopolous and Patrinos, 2018), which does not apply an opportunity cost of attending primary school before grade 5. For children aged 11 to 15, a value somewhere between children’s and adult’s time should be applied depending on the context, and potentially reflecting the value that children might contribute to agricultural activities or factory work. Individuals aged 16 and above should be considered adults.

Value of carbon emissions avoided

The value of carbon emissions avoided is drawn from a recent review of the social cost of carbon literature (Tol, 2018). According to this review, the marginal value of a ton of CO2-eq avoided varies by discount rate. For a 3% discount rate the value is USD 25.30 / ton while for a 5% discount rate it is USD 7.60 / ton. Both figures are denominated in 2010 USD. For much higher discount rates, the effective value of carbon emissions avoided at USD 0 / ton.

To estimate the value of carbon emissions reduction also requires a growth factor in the social cost of carbon emissions, since the social cost grows over time as more CO2-eq is released into the atmosphere. The growth factor should be set at 2% as per year (Tol, 2018). The equation for calculating the benefit of avoided carbon emissions is therefore:

where t=0 represents the year 2015, SCC is the social cost of carbon above in 2010 USD (note in Tol (2018) the emissions year and the currency year are different), g = 2%, r = discount rate.

Treatment of costs of raising funds

In some CBAs, analysts explicitly include the cost of raising funds or the cost of taxation. This is usually assessed as a fixed cost per dollar of investment. We recommend ignoring this in CBA since it affects all analyses approximately equally. The inclusion of this cost would add complexity without improving precision or our ability to identify outliers.

Download the full methodological appendix including references here.