Development Effectiveness Review of the Asian Development Bank

2006-2010

Final Report

Acknowledgments

CIDA’s Evaluation Division wishes to thank all who contributed to this review for their valued input, their constant and generous support, and their patience.

Our thanks go to the team that carried out the review. It was led by Ted Freeman of Goss Gilroy Inc. and included team members from the firm, as well as from the Department for International Development (UK) and the Swedish Agency for Development Evaluation (SADEV).

The Evaluation Division would also like to thank the management team of CIDA’s Multilateral Development Institutions Directorate (Multilateral and Global Programs Branch) at Headquarters in Gatineau for its valuable support.

Our thanks also go to the representatives of the ADB for their helpfulness and their useful, practical advice to the evaluators.

From CIDA’s Evaluation Division, we wish to thank Michel Pilote, Project Manager, for overseeing this review and Brendan Warren, Junior Evaluation Officer, for his assistance with the review. We also thank Michelle Guertin, Evaluation Manager, for guiding this review to completion and for her contribution to the report.

Caroline Leclerc
Director General

Strategic Planning, Performance and Evaluation Directorate

List of Abbreviations

Executive Summary

Background

This report presents the results of a development effectiveness review of the Asian Development Bank (ADB). The ADB was established in 1966 and serves as a major source of development financing for countries in Asia. Headquartered in Manila, the ADB had over US$21.7 billion in financing and more than 2,900 employees from 59 countries in 2011.^{Footnote 1} Its mandate is to reduce poverty and to improve the lives of the poor in the region by fostering economic growth and regional cooperation.

The ADB’s current strategic plans and priorities are stated in its long-term strategic framework for 2008 to 2020, Strategy 2020: Working for an Asia Pacific Free of Poverty.^{Footnote 2}

The strategy identifies three complementary strategic agendas that are pursued in order to achieve its overall vision of an Asia free of poverty:

• 1. Inclusive economic growth;
• 2. Environmentally sustainable green growth; and,
• 3. Regional integration.

Moreover, Strategy 2020 identifies five core areas of ADB programming:

• 1. Infrastructure, including transport and communications, energy, water supply and sanitation, and urban development;
• 2. Environment;
• 3. Regional cooperation and integration;
• 4. Finance sector development; and
• 5. Education.

Canada is a founding member of the ADB and is the sixth largest shareholder (5.25%). It shares a seat on the 12 member Board of Directors with Denmark, Finland, Ireland, Norway, Sweden and the Netherlands. This constituency represents the second largest non-regional voting bloc after the United States,^{Footnote 3} and is normally headed by a Canadian Executive Director. Canada supported the 2009 General Capital Increase that resulted in a 200% increase in capital for the Bank’s ordinary capital resources. Canada’s financial support to the ADB consists of: (i) a capital subscription of the Bank’s capital; and (ii) voluntary support to the Asian Development Fund (ADF) – the Bank’s concessionary funding window. Canada was the sixth largest donor contributing to the ninth replenishment (ADF X) (2009-2012) of the Asian Development Fund, with C$190.8 million. The ADB is an important development partner for Canada, targeting poverty reduction programming in the Asia-Pacific region.

Purpose

The review is intended to provide an independent, evidence-based assessment of the development effectiveness (hereafter referred to as effectiveness) of ADB programs to satisfy the requirements established by the Government of Canada’s Policy on Evaluation and to provide the Canadian International Development Agency (CIDA) with evidence on the effectiveness of the ADB.

Methodology

The methodological approach was developed under the guidance of the Organisation for Economic Co-operation and Development’s Development Assistance Committee Network on Development Evaluation (DAC-EVALNET). Two pilot tests of the ADB and the World Health Organization were conducted in 2010 during the development phase of the common approach and methodology. The report relies, therefore, on the pilot test analysis of evaluation reports published by the Independent Evaluation Department (IED) of the ADB, supplemented with a review of ADB and CIDA corporate documents (Annex 5), and consultation with the CIDA manager responsible for managing relations with ADB. The supplementary information provided context for the reviewers and allowed the review to take account of advances made since the pilot test was carried out in 2010.

The methodology does not rely on a particular definition of development effectiveness. The Management Group and the Task Team that were created by the DAC-EVALNET to develop the methodology had previously considered whether an explicit definition was needed. In the absence of an agreed upon definition, the methodology focuses on some of the essential characteristics of developmentally effective multilateral organization programming (see Annex 1 for criteria), as described below:

• 1. Relevance of interventions: Programming activities and outputs are relevant to the needs of the target group and its members;
• 2. Achievement of development objectives and expected results: The programming contributes to the achievement of development objectives and expected results at the national and local level in developing countries (including positive impacts for target group members);
• 3. Sustainability of results and benefits: The benefits experienced by target group members and the results achieved are sustainable;
• 4. Efficiency: The programming is delivered in a timely and cost efficient manner;
• 5. Crosscutting themes: The programming is inclusive in that it supports gender equality and is environmentally sustainable (thereby not compromising the development prospects in the future); and
• 6. Using evaluation and monitoring to improve development effectiveness: The programming enables effective development by allowing participating and supporting organizations to learn from experience and uses performance management and accountability tools, such as evaluation and monitoring to improve effectiveness over time.

The overall methodological approach^{Footnote 4} was endorsed by the members of the DAC-EVALNET as an acceptable approach for assessing the development effectiveness of multilateral organizations in June 2011.

The methodology used in the review is a structured meta-synthesis of a 45-evaluation sample of the universe of 90 IED evaluations, published between 2006 and 2010, to analyze their findings on the effectiveness of ADB operations (Annex 2). The sample of evaluations covered countries with 82% of ADB programming in 2009. They also accounted for 83% of loan approvals in 2006 and included the 10 countries with the largest ADB programs. The sampling process is described further in Annex 3.

After being screened for quality (Annex 4), each evaluation was reviewed to identify findings relating to six main criteria for assessing effectiveness using 18 sub-criteria that are considered to be essential elements of effectiveness (Annex 1).^{Footnote 5} Findings for each of these evaluations were classified, by the review team, using a four-point scale: “highly satisfactory,” “satisfactory,” “unsatisfactory” and “highly unsatisfactory.” The classification of findings was guided by a grid with specific instructions for each rating across all sub-criteria (Annex 6). The review team also identified factors contributing to or detracting from the achievement of results.

The percentages shown in this report are based on the total number of evaluations that addressed the sub-criteria. However, coverage of the different sub-criteria in the evaluations reviewed varies from strong to weak. Caveats are provided in the report when coverage warrants it.

Key Findings

ADB Programs are Relevant to Stakeholder Needs and National Priorities

Relevance was among the most highly rated of the six criteria in the evaluations reviewed, however some challenges were highlighted. For the period under review, ADB projects and programs were in close alignment with national development goals (sub-criterion 1.2) with 78% of 40 evaluations reviewed reporting positive findings, well suited to meeting the needs of target group members (68% of 38 evaluations reviewed rated sub-criterion 1.1 “satisfactory” or better). The objectives of ADB programs also remained valid at the time of the evaluations (sub-criterion 1.4) with 84% of 37 evaluations receiving positive ratings.

Nonetheless, the evaluations do indicate that there are some challenges for the ADB in the area of relevance. For sub-criterion 1.3 on the effective partnerships with government, only 55% of 38 reviewed evaluations reported findings of “satisfactory” or better. Evaluation findings of 38 evaluations reviewed were also less positive regarding sub-criterion 1.5 – the fit between program activities and outputs and overall program goals – with 44% reviewed as less than “satisfactory.” The most frequently cited factor contributing to this mismatch was ‘missing or poorly delineated causal linkages.’ Another frequently cited factor detracting from relevance was the ‘lack of detailed, micro-level analysis of needs of the target group.’

Most ADB Programs Achieve Their Development Effectiveness Objectives and Expected Results

The review of evaluations from 2006 to 2010 indicates broadly positive results for the achievement of objectives and expected results for ADB programs. Under the heading of “Objectives Achievement” (sub-criterion 2.1), over two thirds of the 44 evaluations reviewed (68%) reported findings of “satisfactory” or “highly satisfactory.” Similarly, two thirds of 26 evaluations reviewed reported that ADB programs reached substantial numbers of beneficiaries and thus contributed to national development goals (sub-criterion 2.3). Equally important, 71% of 38 evaluations reviewed reported that ADB programs contributed to positive changes for target group members (sub-criterion 2.2). Where programs did not achieve their objectives, or did not attain expected results, the contributing factors varied considerably across the evaluation reports. The most frequently cited factors included a lack of institutional capacity among host governments and the need for more focus in the planning of interventions.

Sustainability of Benefits and Results is a Challenge

Findings on sustainability reported in the reviewed IED evaluation reports indicate that sustainability is an area needing improvement. The sustainability of results from ADB operations (sub-criterion 3.1) is a concern, with over half (53%) of 38 evaluations reviewed reporting results for this criterion which were “unsatisfactory” or worse. A key problem is the institutional capacity of partners to sustain program benefits (sub-criterion 3.2) with 65% of 34 evaluations reviewed reporting negative findings. Two important factors detracted from sustainability, namely: (1) an inadequate investment in operation and maintenance of the infrastructure financed by the project; and (2) the absence of realistic and well-funded capacity development or, an inability on the part of the host government to fund the required ongoing institutional capacity.

Evaluations Report Less Positive Results in Efficiency

Some care should be taken in interpreting results under the criteria of efficiency, since both sub-criteria were covered only at a moderate level. Only 26 evaluation reports reviewed addressed sub-criterion 4.1 on cost efficiency, while 31 evaluations addressed sub-criterion 4.2 on timeliness of program implementation and objectives achievement.

Efficiency is one of the weaker areas of performance, according to the evaluations reviewed, with neither sub-criteria (4.1 cost efficiency or 4.2 timeliness) receiving “satisfactory” or better scores in even half of the evaluations reviewed. Only 46% of the evaluations reporting on cost efficiency (sub-criterion 4.1) were rated as “satisfactory” or better. Additionally, only 19% of evaluations found ADB programs and projects to be implemented in a timely manner (sub-criterion 4.2).

Evaluations focused on problematic areas of program start-up and implementation including delays in recruiting and fielding consultants, delays in procurement of program inputs, and delays in government compliance with agreed policy and regulatory changes. These delays, of course, also contribute to poor cost efficiency.

ADB Programs Contribute to Gender Equality

Considerable care should be taken in interpreting results for the effectiveness of ADB programs and projects in supporting gender equality. This was the only sub-criterion where coverage was rated as “weak” with only 16 evaluations addressing this sub-criterion.

For the evaluations that did address gender equality, results were positive. Over 80% of the evaluations that addressed sub-criterion 5.1 rated ADB performance as “satisfactory” or better. This was the third highest ranking of any of the sub-criteria. Where results for effectiveness in supporting gender equality were less than satisfactory, the most commonly cited contributing factor was a low priority given to funding necessary program components focusing on gender equality.

Most ADB Programs Report Positive Findings in Environmental Sustainability, but Improvements are Needed

Environmental sustainability was covered at the moderate level with 31 evaluations reporting relevant findings. Sixty-four percent of evaluations reported “satisfactory” or “highly satisfactory” results for this sub-criterion (5.2), with over a third rated as “unsatisfactory” or worse. In particular, evaluations pointed to opportunities for ADB to invest in measures to mitigate negative environmental impacts, such as better land use management and better on-farm practices for fertilization and pest control.

Evaluation is Effective and Well Used, but Challenges are Highlighted in Monitoring and Results-Based Management

The reviewed evaluations report that ADB has effective evaluation systems and that the results of evaluations are consistently used to improve effectiveness but there are important weaknesses in local systems for results-based management and reporting.

The system of independent evaluation at the ADB (sub-criterion 6.1) is effective (82% of 39 evaluations rated evaluations systems as “satisfactory” or better) and well used to improve effectiveness (sub-criterion 6.4). Eighty percent of 45 evaluations found that the ADB’s response to the recommendations made in the evaluation reports was either “satisfactory” or “highly satisfactory.” However, results monitoring and reporting systems at the local level (sub-criterion 6.2) did not score as well as evaluation systems in IED evaluations. Only 20% of 36 evaluations rated local monitoring and results reporting systems as “satisfactory” while only 8% of evaluations found results-based management systems “satisfactory.” This was the lowest rated of the eighteen sub-criteria. The most frequently cited factors contributing to poor findings on the effectiveness of results-based management systems, including results monitoring, involved problems with the indicators used to track results such as poor quality and coverage, and an absence of targets. Evaluations also point to weaknesses in knowledge and capacity for monitoring and evaluation among program partners. These weaknesses in results-based management, including monitoring, were reported at the local level in the independent evaluations reviewed.

Conclusions: Development Effectiveness of the ADB

Evaluations carried out by IED between 2006 and 2010 indicate that ADB programming is relevant to the needs of target group members and well aligned with the development goals of its national partners. The evaluations also reflect positive results in the achievement of objectives and expected development results in over two thirds of evaluation reports. The sustainability of program benefits remains an important challenge for the ADB and its partners, especially in terms of the capacity of partner institutions to sustain program results. Reported results in the area of efficiency indicate another important challenge for the ADB--timeliness of program implementation. While evaluations often do not address gender equality, those that do indicate that ADB programs have been effective in achieving results. Evaluations also report that most ADB programs have generally been effective in addressing environmental sustainability, although improvements are needed to ensure that ADB projects include effective measures to address environmental challenges. Finally, systems for program evaluation are effective, and are well used, but there is a continuing need to strengthen results-based management, including monitoring and reporting at the national and local level.

The annual review of evaluation by IED and the annual development effectiveness report by ADB produced in 2012, including the development effectiveness reports on private sector operations, illustrate the Bank’s commitment to using evidence gathered through the evaluation system to report openly on achievements and trends in effectiveness. Given the quality of ADB-published evaluations and the continued refinement of these reports, there is no apparent need for another external effectiveness review of the ADB in the medium-term. These two sources of information will provide donors and other shareholders with reliable reporting on development effectiveness in the future.

ADB programming contributes to all three of CIDA’s priorities: 1) increasing food security, 2) stimulating sustainable economic growth, and, 3) securing the future of children and youth. CIDA’s priorities for engaging with the ADB include responding effectively to the financial crisis, particularly for the poor and vulnerable; strengthening programming in Canada’s countries of focus; and improving institutional effectiveness. The ADB reacted in a timely and proactive way to the financial crisis by increasing and expediting approvals and disbursements. It also continues to focus operations on Afghanistan and Pakistan (priority countries for Canada). The ADB also continues to invest in efforts to improve institutional effectiveness, especially in the analysis and reporting of the development effectiveness of ADB programs.

Recommendations to CIDA

1. Due to a low level of coverage of gender equality in the evaluations reviewed, CIDA should emphasize the need to ensure that gender equality is directly addressed in future IED evaluations. Evaluation policies at both the central and local levels should pay adequate attention to gender equality as a key evaluation issue. This may also require greater attention to gender analysis skills among evaluation teams.

2. CIDA should engage with the ADB to ensure that the environmental sustainability of infrastructure and other assets financed by the Bank receives sufficient attention, and that results in this area are improved over time.

3. CIDA should raise the issue of the sustainability of the benefits of ADB investments to a strategic level in its engagement with the Bank. This review and the Bank’s own annual assessment of development effectiveness have highlighted the need to improve sustainability by making greater investment in operations, maintenance of infrastructure and improving capacity-development components of programs.

4. Improving the timeliness of ADB operations should be emphasized in CIDA’s interactions with the ADB. According to the evaluations reviewed, timeliness could be improved through changes in systems and procedures used during each phase of the project life cycle. This represents one of the most readily apparent ways to improve the efficiency of ADB operations.

5. CIDA should emphasize the need to strengthen systems for program results-based management, including: results monitoring at the local level, improving the quality and coverage of indicators and the establishment of targets to track results. This will require investments in capacity development for ADB staff and partners at the country level, and, in the case of partners, at the local level.

1.0 Introduction

1.1 Background

This report presents the results of a development effectiveness review of the Asian Development Bank (ADB). The common approach and methodology were developed under the guidance of the Organisation for Economic Cooperation and Development (OECD)’s Development Assistance Committee (DAC) Network on Development Evaluation (DAC-EVALNET). Two pilot tests, the ADB and the World Health Organization, were conducted in 2010 during the development of the common approach and methodology. The report relies, therefore, on the pilot test analysis of evaluation reports published by the Independent Evaluation Department (IED) of the ADB supplemented with a review of ADB and the Canadian International Development Agency (CIDA) corporate documents, and consultation with the CIDA manager responsible for managing relations with ADB.

The method uses a common set of assessment criteria derived from the DAC’s evaluation criteria (Annex 1). The overall methodological approach^{Footnote 6} was endorsed by the members of the DAC-EVALNET as an acceptable approach for assessing the development effectiveness of multilateral organizations in June 2011.

From its beginnings, the process of developing and implementing the reviews of development effectiveness has been coordinated with the work of the Multilateral Organization Performance Assessment Network (MOPAN). By focusing on development effectiveness and carefully selecting assessment criteria, the reviews seek to avoid duplication or overlap with the MOPAN process. Normal practice has been to conduct the development assessment review in the same year as a MOPAN survey for any given multilateral organization. A MOPAN Survey of the ADB was conducted in 2010 in parallel with this review.^{Footnote 7}

1.2 Why Conduct this Review?

The review provides Canada and other shareholders an independent, evidence-based assessment of the development effectiveness of ADB programs. The review satisfies the requirements of the Government of Canada’s Policy on Evaluation.^{Footnote 8}

The objectives of the review are:

To provide CIDA with evidence on the development effectiveness of the ADB which can be used to guide CIDA’s engagement during the present period of long-term institutional funding;^{Footnote 9} and;

To provide evidence on effectiveness which can be used in the ongoing relationship between CIDA and the ADB to ensure that Canada’s international development priorities are served by its investments.^{Footnote 10}

Although this report is intended, in part, to support CIDA’s accountability requirements within the Government of Canada, the results are expected to be useful to other bilateral shareholders as well.

1.3 The ADB: A Major Source of Development Financing for Asia

1.3.1 Background and Objectives

The ADB was established in 1966 and is a major source of development financing for countries in Asia. Headquartered in Manila, the ADB had, in 2011, over US$21.7 billion in approved financing and more than 2,900 employees from 59 countries.^{Footnote 11} Its mandate is to reduce poverty and to improve the lives of the poor in the region by fostering economic growth and regional cooperation.

1.3.2 Strategic Plan

The ADB’s current strategic plans and priorities are stated in its long-term strategic framework for 2008 to 2020, Strategy 2020: Working for an Asian Pacific Free of Poverty.^{Footnote 12} The strategy identifies drivers of change to be stressed in all ADB operations: developing the private sector, encouraging good governance, supporting gender equity, helping developing countries gain knowledge, and expanding partnerships with other development institutions, the private sector, and community-based organizations.

The strategy also identifies three complementary strategic agendas that are pursued in order to achieve the ADB’s overall vision:

• 1. Inclusive economic growth;
• 2. Environmentally sustainable green growth; and
• 3. Regional integration.

Moreover, Strategy 2020 identifies five core areas of ADB programming:

• 1. Infrastructure, including transport and communications, energy, water supply and sanitation, and urban development;
• 2. Environment;
• 3. Regional cooperation and integration;
• 4. Finance sector development; and
• 5. Education.

Since 2008, as a result of Strategy 2020, the ADB has continued to operate in health, agriculture, and disaster and emergency assistance, but on a more selective basis. Progress towards achieving the objectives of Strategy 2020 is monitored and reported on annually in the Development Effectiveness Review reports.

1.3.3 Work and Geographic Coverage

The ADB provides support to governments and the private sector in the Asia-Pacific region through financial operations and technical assistance. Financial operations include both concessional^{Footnote 13} and non-concessional loans, as well as grants,^{Footnote 14} guarantees^{Footnote 15} and equity investments.^{Footnote 16} Financing is provided to both governments (sovereign operations), as seen in Table 1,^{Footnote 17}and private sector firms (non-sovereign).

ADB non-sovereign disbursements are substantially smaller in scale than sovereign disbursements. In 2011, the Bank reported disbursements of US$8.1 billion on sovereign operations, excluding regional projects. In the same year, it reported close to US$1.5 billion in non-sovereign disbursements.^{Footnote 18}

ADB financing is provided through two “windows”, the ADB and the Asian Development Fund, which provides both grants and loans on a concessional basis.

Taken together, the ADB and Asian Development Fund disbursements to governments rose from US$10.3 billion in 2008 to US$10.5 billion in 2009, before declining to US$8.6 billion in 2010 and rising again to US$9.4 billion in 2011.^{Footnote 19} This may reflect the role played by the ADB in providing financing in response to the financial crisis of 2008-2009.

Countries eligible for ADB financing are divided into three distinct groups depending on their need for access to concessional lending and grants.^{Footnote 20} Table 2 describes the three groups of countries are divided into and their share of ADB disbursements in 2011.

1.3.4 Evaluation and Results Reporting

Evaluation

The ADB uses a two-tier approach to evaluate individual operations.^{Footnote 21} The first tier consists of self-evaluation through the preparation of ‘Project Completion Reports’ for sovereign operations and ‘Expanded Annual Review Reports’ for non-sovereign operations. These are prepared by the responsible operational departments.

The second tier of evaluating individual operations involves the IED conducting an independent validation of all ‘Project Completion Reports’ and ‘Expanded Annual Review Reports,’ as well as in-depth evaluation of selected individual programs and projects. The IED reports to the ADB Board through its Development Effectiveness Committee.

In addition, IED undertakes independent strategic and higher level evaluations. There are five main types of IED-led strategic and higher level evaluations, as seen in Figure 1.

Figure 1: Types of IED-Led Strategic and Higher Level Evaluations

The Annual Evaluation Review reports on the number and type of evaluations completed by IED each year and on annual and multi-year trends in findings. The reports also assess the status of evaluation recommendations and provide validation of the actions taken on these recommendations by management.

The review team conducted a quality review of the evaluations included in the review. The results were positive, with 86.6% of the reviewed evaluations scoring 30 points or more out of a possible 48. Furthermore, only two evaluation reports received a score of less than 24. Due to these results, all 45 IED evaluation reports were included in the analysis (Please refer to Annex 3 for details of the review methodology and Annex 4 for the evaluation quality-scoring grid).

Results Reporting

There are two reports on development effectiveness produced by the ADB each year: the annual Development Effectiveness Review and the Development Effectiveness Report: Private Sector Operations. Further to which, the IED also produces its Annual Evaluation Review. The reports improved over time and the 2011 evaluations published in 2012, provide a detailed overview of the effectiveness of ADB operations.

The 2012 Annual Evaluation Review, for example, reported that following an assessment of the 26 country assistance programs and a validation of the six-country partnership strategy, 67% of 30 country program evaluations with overall ratings were assessed as “successful” and 33% as “less than successful.” The report also mentioned factors contributing to the success, noting both the quality of ADB’s strategic positioning of its development aid globally (including a felicitous choice of sectors and themes for prioritization, and of aid harmonization) and the relevance of the program adopted (with regard to country constraints and government priorities). Effectiveness and efficiency were rated more varied, and a high proportion of ADB’s country interventions were rated, overall, less likely sustainable.^{Footnote 22}

The Bank’s annual Development Effectiveness Review examines the performance of ADB at four levels:

• Level One examines progress in achieving development objectives in Asia and the Pacific. This section examines the region’s performance in achieving the objectives of poverty reduction and human development, as well as other regional outcomes (i.e., growth, regional cooperation and integration, basic infrastructure, finance, governance and the environment);
• Level Two examines two aspects of ADB’s operations: (i) status of achievement of the results framework core sector outputs targeted for a specific period; and (ii) contribution of recently completed operations (both sovereign and non-sovereign) to their intended sector outcomes;
• Level Three examines operational effectiveness (i.e., quality of completed operations, knowledge management, partnerships, etc.) and;
• Level Four examines the ADB’s organizational effectiveness (i.e., budget adequacy, human resources, and business processes and practices).

For Level Two results (development outputs and outcomes), the ADB Development Effectiveness Review relies mainly on inputs from IED evaluation reports stating in 2011 that:

ADB improved the quality of its ongoing operations, and earned good ratings for its two previously weak performance areas: support for gender mainstreaming in operations and finance mobilization. However, the review confirmed that the quality of ADB’s recently completed operations – including their delivery of core sector outcomes – remained considerably below target despite improvements from the previous year. Furthermore, project delays and cancellations lowered the outputs to be delivered during 2009-2012 from Asian Development Fund (ADF) operations.

The same report noted that two-thirds of ADB operations in the same period had achieved their stated objectives. Similar findings can be seen in section 3.2 of this report.

2.0 Methodology

This section briefly describes key aspects of the review’s methodology. A more detailed description of the methodology is available in Annex 3.

2.1 Rationale

The term “common approach” describes the use of a standard methodology, as was implemented in this review, to consistently assess the development effectiveness of multilateral organizations. It offers a more rapid and cost effective way to assess effectiveness rather than the more costly and time consuming joint evaluation.^{Footnote 23} The approach was developed to fill an information gap regarding the development effectiveness of multilateral organizations. Although these multilateral organizations produce annual reports for their management and/or boards, bilateral shareholders were not receiving a comprehensive overview of the performance of multilateral organizations in the countries. The Multilateral Organization Performance Assessment Network (MOPAN) seeks to address this issue through organizational effectiveness assessments. This approach complements MOPAN’s assessments.

The approach suggests conducting a review based on the organization’s own evaluation reports when two specific conditions exist:

• 1. There is a need for field-tested and evidence-base information regarding the effectiveness of the multilateral organization; and
• 2. The multilateral organization under review has an evaluation function that produces an adequate body of reliable and credible evaluation information that supports a meta-evaluation synthesizing an assessment of the organization’s development effectiveness.^{Footnote 24}

The first condition was satisfied, since the ADB’s reporting mechanisms did not provide sufficient, field tested information on the organization’s development effectiveness in 2010, when the pilot test analysis was carried out. Additionally, the independent evaluation function at the ADB produced a sufficient number of robust evaluation reports to support an assessment of the development effectiveness of the ADB. Further details on the rationale are available in Annex 3.

Given the demonstrated strength of ADB annual evaluations and development effectiveness reporting in 2011, and assuming these strengths are maintained in the future, there is little utility in repeating an effectiveness review of this type in the near future.

2.2 Scope

Ninety IED evaluations were published between 2006 and 2010 (described in more detail in Annex 3) including: country assistance program evaluations in countries with both large and small ADB program portfolios; sector assistance program evaluations in a wide range of sectors (agriculture, transport, public sector reform, and microfinance); special evaluations of different forms of development assistance (capacity development, technical assistance, policy support); and evaluations of different policy initiatives and strategies within the ADB (gender equality, conforming to the 2005 Paris Declaration on Aid Effectiveness, and partnering and harmonization).

The sample of 45 IED evaluations (see Annex 2 for details on the review sample) focused first on geographic coverage and included country assistance program evaluations for the countries that received 82% of ADB assistance in 2009. The sample was supplemented by sector and thematic evaluations. Overall, the sample of 45 IED evaluations provided coverage of the breadth of activities and the critical mass of ADB investments over a four-year time frame.

This review of evaluation reports was supplemented by a review of ADB corporate documents related to evaluation and reporting on development effectiveness as well as consultations with the CIDA manager responsible for managing relations with the ADB.^{Footnote 25} This contextualized the results of the review and took account of advances since the pilot test analysis was carried out in 2010. A list of the documents consulted is provided in Annex 5.

2.3 Criteria

The methodology of this review does not rely on a particular definition of development effectiveness. The Management Group and the Task Team that were created by the DAC-EVALNET to develop the methodology had previously considered whether an explicit definition was needed. In the absence of an agreed upon definition, the methodology focuses on some of the essential characteristics of developmentally effective multilateral organization programming, as described below:

1. Relevance of interventions: Programming activities and outputs are relevant to the needs of the target group and its members;
2. The achievement of development effectiveness objectives and expected results: The programming contributes to the achievement of development objectives and expected results at the national and local level in developing countries (including positive impacts for target group members);
3. Sustainability of results and benefits: The benefits experienced by target group members and the results achieved are sustainable;
4. Efficiency: The programming is delivered in a cost efficient manner;
5. Crosscutting themes : The programming is inclusive in that it supports gender equality and is environmentally sustainable (thereby not compromising the development prospects in the future);
6. Using Evaluation and Monitoring to Improve Development Effectiveness: The programming enables effective development by allowing participating and supporting organizations to learn from experience and uses performance management and accountability tools, such as evaluation and monitoring to improve effectiveness over time.

The review methodology involves a systematic and structured meta-synthesis of the findings of a sample of IED evaluations, as they relate to six main criteria and 18 sub-criteria of that are considered to be essential elements of effective development (Annex 1). The main criteria and sub-criteria are derived from the DAC Evaluation Criteria.

2.4 Limitations

This review sought to mitigate several methodological challenges including, reducing sampling bias, ensuring the sample adequately reflected the criteria under evaluation, and in assessing the effectiveness of complex, multi-part programs.

There is no evident bias in the sample of 45 evaluations chosen since there is no reason to believe that the evaluations selected are likely to produce more or less positive results than the others left out. As noted above, the sample provides adequate coverage of the ADB’s national, regional and global programs. In addition, there was adequate coverage of the criteria since 17 of the 18 sub-criteria used to assess effectiveness were covered in the evaluations reviewed (Annex 3). For the sub-criterion that did not have adequate coverage, the detailed limitations are explained in Section 3 “Findings on the ADB’s Development Effectiveness.”

A problem, however, arises in interpreting the results of country assistance program evaluations because these evaluations usually cover very different types of programming within the country. Arriving at an overall finding for an ADB program portfolio in a given country requires an overall assessment combining results from different program areas. In most instances, the evaluation report makes an effort to merge results across the main program areas into a single finding on each of the sub-criteria. Where the country program evaluations did not make an overall judgment, the reviewers compiled evidence from all program areas and made their own assessment.

The review was also not able to distinguish differences in effectiveness of ADB programs by either the type of program or the sector of disbursements. The sample of 45 evaluations did not allow for a comparative analysis of ADB loans and grants by sector, country classification or other dimensions of programming. However, the ADB did undertake this type of analysis in its Annual Evaluation Review reports. It is able to provide a breakdown of project success rating by approval period, country type, financing type, etc. by examining results report in both project completion reports and evaluations (which allows for a much larger data set).

A final limitation is that recent advancements of the organization are not necessarily reflected in the findings of the review, particularly with regard to criterion 4 “Efficiency” and criterion 6 “Using Evaluation and Monitoring to Improve Development Effectiveness”, due to the evaluations included in the sample, which were published between 2006 and 2010. As such, he review report, was strengthened by the inclusion of supplementary information from CIDA and ADB corporate documents.

3.0 Findings on the Development Effectiveness of the ADB

This section presents the results of the development effectiveness review relating to the six main criteria and their associated sub-criteria (Figure 2 and Annex 1). In particular, Table 3 below describes:

• The number of evaluations addressing each sub-criterion (represented by the letter a);^{Footnote 26}
• The coverage level for each sub-criterion based on the number of evaluations addressing each sub-criterion; and
• The percentage of evaluations addressing each sub-criterion that have been rated by the review team as “satisfactory” (i.e., “satisfactory” and “highly satisfactory”) or “unsatisfactory” (i.e., “unsatisfactory” and “highly unsatisfactory”).

*a = number of evaluations addressing the given sub-criterion

**Coverage levels: strong: a = 35 – 45, moderate: a = 25 – 34, weak: a = under 25

*** Satisfactory ratings include “satisfactory” and “highly satisfactory”; unsatisfactory ratings include “unsatisfactory” and “highly unsatisfactory”

3.1 ADB Programs are Relevant to Stakeholder Needs and National Priorities

3.1.1. Coverage

All five sub-criteria for relevance have high levels of coverage and are addressed by most evaluations as illustrated by Figure 2.

Figure 2: Number of Evaluations Addressing Sub-Criteria for Relevance

3.1.2 Key Findings

Relevance was among the most highly rated of the six criteria in the evaluations reviewed but some challenges were highlighted as shown in Figure 3.

Figure 3: Relevance (Findings as a % of number of evaluations addressing the issue (=a), n=45)

The evaluations reviewed illustrate that ADB “Programs are suited to the needs of target group” (sub-criterion 1.1) and “Programs align with national development goals” (sub-criterion 1.2).

Twenty-six of 38 of the evaluations reviewed (68%) indicated that “Programs are suited to the needs of the target group” (sub-criterion 1.1), while 78% of evaluations reported that ADB “Programs align with national development goals” (sub-criterion 1.2), with over half of those being rated “highly satisfactory”(Highlight Box 1).

The findings for sub-criteria 1.1 and 1.2 are corroborated by the 2010 MOPAN survey of the ADB. MOPAN micro-indicator I-4 “Results developed in consultation with beneficiaries” (aligning with pilot test sub-criterion 1.1) received an “adequate” result. Similarly, micro-indicator I-4 “Expected results consistent with national development strategies” (aligning with sub-criterion 1.2) also scored “adequate”.

The reviewed evaluations consistently reported that ADB “Program objectives remained valid” at the time of the evaluation, with 84% of evaluations rated as “satisfactory” or better (sub-criterion 1.4).^{Footnote 27}

However, there are some challenges for the ADB in the area of relevance. For sub-criteria 1.3 “Effective partnerships with government” and sub-criterion 1.5 “Program activities are consistent with program goals”, only 55% and 56%, respectively, of evaluations reported findings of “satisfactory” or better, respectively. MOPAN results for the micro-indicator III-1 “Proposals developed with national government or clients” (aligning with sub-criterion 1.3) were “adequate,” with the indicator scoring 4.17.^{Footnote 28}

The evaluations reviewed also point to the requirement for better microanalysis of the needs of target group members for some evaluated programs and projects as indicated in 31% of evaluations. It was also noted that there is a need for improvements in program and project design to better link individual program components and their outputs to overall development objectives of ADB programs with 44% of evaluations indicating that this link was less than satisfactory).

3.1.3 Contributing Factors

Three factors noted as frequently contributing to the overall relevance of ADB operations were:

1. The direct effort on the ADB’s part to ensure that country assistance programs were integrated into national development programs and priorities during program development stages, which strengthened the alignment of ADB programs with national priorities (14 evaluations);^{Footnote 29}
2. The ADB’s drive in the past decade to focus on poverty alleviation contributed to the relevance of programming (9 evaluations);
3. ADB operations also focused on sectors such as energy initiatives, infrastructure or agriculture that were rated as essential to supporting national economic growth (3 evaluations).

Factors detracting from the relevance of ADB operations include;

• Review of the Development Effectiveness of the Asian Development Bank 17
• Weak and/or poorly defined causal linkages from program components to related objectives during project design (8 evaluations).
• Lack of detailed, micro-level analysis of the needs of the target group (8 evaluations);
• Weak partnerships between the ADB and local governments due to decentralization leading to diluted local government capacity (2 evaluations);
• Confusion over roles and responsibilities (1 evaluation);
• Onerous ADB rules and procedures (1 evaluation) and poor communication of procedures (1 evaluation).

3.2 The ADB is Achieving Most of its Development Effectiveness Objectives and Expected Results

3.2.1 Coverage

As illustrated by Figure 4, two of the three sub-criteria under the heading of “objectives achievement ”were rated “strong” in coverage. Sub-criterion 2.1 “Programs achieve stated objectives and attain expected results” and 2.2 “Positive benefits for target group members” were addressed in 44 and 38 evaluation reports respectively. Coverage was rated as moderate given that only 26 evaluations addressed sub-criterion 2.3 “Differences for substantial number of beneficiaries”.

Figure 4: Number of Evaluations Addressing Sub-Criteria for Objectives Achievement

3.2.2 Key Findings

The evaluations reviewed from 2006 to 2010 reported broadly positive results for the achievement of objectives and expected development results for ADB programs (Figure 5). Under the heading of “Programs achieve stated objectives and attain expected results” (sub-criterion 2.1), 30 of the 44 relevant evaluations (68%) received a rating of “satisfactory” or “highly satisfactory.” The reported evaluation results were similar for sub-criterion 2.2 “Positive benefits for target group members”. In 26 of the 38 evaluations (71%) which addressed this issue, ADB programs and projects were found to have resulted in positive changes for target group members at the “satisfactory” level (one evaluation was coded as “highly satisfactory” under this criterion).

Sub-criterion 2.3 “Differences for substantial number of beneficiaries” was also assessed positively with 17 of 26 evaluations (66%) reporting findings of “satisfactory” or better.

Figure 5: Achievement of Development Effectiveness Objectives and Expected Results (Findings as a percentage of number of evaluations addressing the issue (=a), n=45)

The small number of evaluations fulfilling this evaluation’s sample criteria only permitted a broad overview of the achievement of objectives of ADB programming. As such, analysis of the objectives achieved within each areas of ADB programming as well as a comparison of results across different types of evaluation was not possible.

On the other hand, the evaluations reviewed did point to a range of positive outcomes associated with ADB operations, often relating directly to the sector of interventions. These include:

• Improvements in national or local economic growth rates and improved socio-economic development with different types of associated benefits for the poor (11 evaluations);
• Reductions in transport costs and decreased transport times leading to economic benefits for rural communities resulting from investments in local transport infrastructure (4 evaluations);
• Improved services and reduced energy costs as a result of investments in the power sector (3 evaluations);
• Improved sector governance and restructured policy and institutional frameworks in important sectors of development at a national level (4 evaluations); and,
• More equitable and effective administration of justice through reforms to the judicial system (1 evaluation).

• Furthermore, as illustrated in Highlight Box 2, Country Assistance Program Evaluations often describe a wide range of development results within a single country resulting from ADB operations.

  Highlight Box 2

  Effective ADB Assistance in Mongolia

  “The development results of ADB’s assistance as a whole have been considerable, particularly through its support for market-oriented policy reform, social sector development, economic corridor development, capacity building for public sector governance, government systems for procurement, project management, and a number of other aspects of public sector planning and financial management.”

  CAPE for Mongolia

  3.2.3 Contributing Factors

  Factors that contributed positively to the achievement of objectives and expected results included:

• High levels of performance by government and non-government implementing partners (5 evaluations);
• High levels of national government ownership (5 evaluations); and
• Participatory design processes that incorporated the needs of target group members (2 evaluations).

• Evaluations also identified factors that detracted from the achievement of objectives and expected results including:

• Inadequate provision for operation and maintenance of the infrastructure constructed with ADB financing or technical assistance (8 evaluations);
• Lack of institutional capacity among host governments (6 evaluations);
• Lack of strategic focus and planning of program interventions (4 evaluations);
• Inadequate understanding of the local program context and the needs and capacities of both target group members and participating actors (3 evaluations); and,
• Poor economic viability for some private sector development projects (2 evaluations).

• 3.3 Sustainability of Benefits and Results is a Challenge for the ADB

  3.3.1 Coverage

  Sub-criterion 3.1 “Program benefits are likely to continue” was addressed by 38 evaluation reports, resulting in a strong level of coverage, while sub-criterion 3.2 “Program sustainability in terms of institutional capacity” was rated as moderate in coverage since 11 evaluations did not address this sub-criterion (Figure 6).

  Figure 6: Number of Evaluations Addressing Sub-Criteria for Sustainability of Results and Benefits

  

  3.3.2 Key Findings

  The evaluations reviewed reported challenges with sub-criterion 3.1 “Program benefits are likely to continue.”. Problems were noted with “Program sustainability in terms of institutional capacity” for delivering services or managing investments, including policy agencies (sub-criterion 3.2). Highlight Box 3 provides an example of these challenges. The results for both sub-criteria are presented in Figure 7. Twenty evaluations of the 38 (53%) that addressed the sustainability of benefits reported findings that were classified “unsatisfactory” or worse, including seven classified as “highly unsatisfactory.” Institutional sustainability results show similar challenges, with 22 of the 34 evaluations (65%) reporting less positive findings.

  Figure 7: Sustainability of Results and Benefits (Findings as a % of number of evaluations addressing the issue (=a), n=45)

  

  Highlight Box 3

  Sustainability Challenges in Sri Lanka

  “The sector assistance program is assessed as ‘less likely to be sustainable’. Specific risks to the sustainability of the assistance program include: (i) operation and maintenance of reforestation and coastal stabilization projects; (ii) frequent institutional changes; (iii) lack of capacity and resources among executing agencies to sustain project outcomes, particularly after project completion; (iv) resurgence of the conflict in the North and East that has affected outcomes in conflict-affected areas; (v) instances of policy reversals; and (vi) conflicting government programs in some areas. The improved relationship between extension staff and the communities has enhanced natural resources management, but the lack of funds to support operation, maintenance, and extension services will likely undermine the gains from such partnerships.”

  Sri Lanka CAPE

  3.3.3 Contributing Factors

  Factors contributing to the sustainability of the results of ADB operations included:

• Adequately funded capacity development activities as an integral component of programs (6 evaluations);
• Strong program ownership by national and local governments and implementing agencies (5 evaluations) (Highlight Box 4); and,
• Basic economic viability as tested in program design, especially for private sector investment projects (2 evaluations).

• Highlight Box 4

  A Positive Example: the Sustainability of Energy Sector Support to Bangladesh

  “The investments and policy reforms are, as a whole, “likely to be sustainable” from the financial and institutional points of view. The sense of ownership is strong and management in the utilities set up with ADB assistance is competent. The utilities are profitable and some are even listed in the stock market. The cost recovery in the energy sector is improving due to reduced transmission and distribution losses, and higher bill collection. The projects and structural reforms that ADB has supported are technically and operationally sustainable and enjoy the backing of the Government in terms of political commitment.”

  Sector Assistance Program Evaluation: Bangladesh Energy Sector

  Factors that detracted from the overall criterion of sustainability included:

• Inadequate program investment in the operation and maintenance of the infrastructure financed by the project (8 evaluations); and,
• Absence of realistic and well-funded capacity development in programs or an inability on the part of the host government to fund the required ongoing institutional capacity (6 evaluations).

• Similarly, the ADB’s Development Effectiveness Review for 2011 also identified the weak capacity of microfinance institutions to attain operational and financial viability, and the lack of market-friendly policies to encourage commercial lending.^{Footnote 30}

  Additionally, the IED’s Post-Completion Sustainability of Asian Development Bank-Assisted Projects report noted a number of important factors affecting sustainability, including: financial arrangements; profitability of beneficiaries’ enterprises; human resource issues; institutional and market conditions and incentives; government ownership and commitment; and environment and social risks.^{Footnote 31}

  3.4 Evaluations Report Less Positive Results in Efficiency

  3.4.1 Coverage

  Some care should be taken in interpreting the results of the review of IED evaluations related to efficiency. Coverage of sub-criterion 4.1 “Programs evaluated as cost efficient” was moderate, as it was addressed in only 26 of 45 evaluations. Coverage of sub-criterion 4.2 “Programs implemented, objectives achieved on time” was also moderate, and was addressed in 38 evaluations (Figure 8).

  Figure 8: Number of Evaluations Addressing Sub-Criteria for Efficiency

  

  3.4.2 Key Findings

  Even considering for the moderate level of coverage, efficiency represented one of the weaker areas of the ADB’s effectiveness as reported in the evaluations reviewed (Figure 9). The findings with respect to “Programs evaluated as cost efficient” (sub-criterion 4.1) and “Programs implemented, objectives achieved on time” (sub-criterion 4.2) received negative ratings in more than half of the evaluations. Fourteen of the 26 (54%) evaluations, which reported findings on the cost-effectiveness of program activities, rated them as “unsatisfactory” or “highly unsatisfactory.” Of the 31 evaluations that reported findings on the timeliness of program implementation, 25 (81%) were assigned ratings of “unsatisfactory” or “highly unsatisfactory.”

  Figure 9: Efficiency (Findings as a percentage of number of evaluations addressing the issue (=a), n=45)

  

  3.4.3 Contributing Factors

  The evaluations reviewed reported a wide range of factors that hindered efficiency, with most related to delays in design and implementation (Highlight Box 5). These included:

• Excessive delays in mobilizing consultants and program specialists including delays in establishing terms of reference, recruiting consultants and mobilizing them in the field (9 evaluations);
• Delays resulting from complex procurement conditions and arrangements (8 evaluations);
• Delays in government compliance with agreed upon policy and regulatory changes necessary for project program implementation (4 evaluations); and
• Failure to begin program design until after ADB loan approval with subsequent lengthy delay in program design (2 evaluations).

• Highlight Box 5

  Delays Contributing to Inefficiencies in Pakistan

  “A perennial problem in Pakistan is delayed project implementation and the need for multiple extensions to the loan closing dates. ADB and the Government need to identify and address the causes of delayed project implementation and/or start out with more realistic implementation schedules based on experience. Actions could include (a) designing less complex projects, (b) ensuring more complete understanding and acceptance of the project at all relevant levels of government where support and commitment are required (including through building country capacity to handle project design), (c) more support to executing agencies during implementation, and (d) incorporation of incentives and sanctions to encourage timely implementation.”

  CAPE Pakistan

  3.5 ADB Programs Contribute to Gender Equality and Face Challenges in Environmental Sustainability

  3.5.1 Coverage

  Considerable care is required in interpreting the review results for sub-criterion 5.1 ADB “Programs effectively address gender equality”. Coverage for this sub-criterion is weak, as it was addressed in only 16 of the 45 evaluations (Figure 10). Coverage for sub-criterion 5.2 “Changes are environmentally sustainable” is moderate, being addressed by 31 of the 45 evaluations reviewed.

  Figure 10: Number of Evaluations Addressing Sub-Criteria for Gender Equality and Environmental Sustainability

  

  3.5.2 Key Findings

  For the 16 evaluations that addressed gender equality, results were positive. Over 80% of the evaluations that addressed sub-criterion 5.1 on gender equality (Figure 11) rated ADB performance as “satisfactory” or better (Highlight Box 6). This was the third highest ranking of any of the sub-criteria.

  The CIDA assessment of gender equality rated the ADB as ‘promising’ overall, based on the Bank’s own available reporting^{Footnote 32} and achievement of gender equality results. This rating is similar to the positive results for sub-criterion 5.1 in a key instance: shifting to a gender equality focus as an element of ADB institutional performance was rated “good” because there is a clear focus on gender equality and women’s empowerment, found to be integral to its development objectives.

  For effectiveness in environmental sustainability (sub-criterion 5.2), the reported level of success was lower. Twenty of 34 valid evaluations (64%) rated results in this area as “satisfactory” or, in one case, “highly satisfactory,” and 11 evaluations (36%) were assigned a rating “below satisfactory.” Six evaluations (20%) rated ADB programs “highly unsatisfactory” in terms of environmental sustainability.

  Figure 11: Crosscutting Themes Gender Equality and Environmental Sustainability (Findings as a percentage of number of evaluations addressing the issue (=a), n=45)

  

  Highlight Box 6

  Supporting Education for Girls in Pakistan and Bangladesh

  “The promotional campaign of the Pakistan girls’ primary education project contributed to changing attitudes such that girls’ education was more readily accepted in the project communities. By the time the project was evaluated, some families not only accepted educational opportunities for girls, but also pursued them. Project schools were also observed to have had greater acceptance for co-education than non-project schools, mainly in kindergarten and the lower grades. Enrolment of girls in project schools increased almost four times during the 10 years from the time the project started; more than twice the national growth rate. The Bangladesh primary education project supported the recruitment of more female teachers by lowering their academic requirements. At the time of evaluation, gender parity in enrolment in primary classes had been achieved.”

  Sector Synthesis of Evaluation Findings: Education

  3.5.3 Contributing Factors

  In relation to sub-criterion 5.1 on gender equality, the most frequent factor contributing to “unsatisfactory” evaluation findings was the project being classified as a low priority, by either the ADB or the national governments involved, to supporting necessary program components focusing on gender equality (3 evaluations).

  The factors cited in the reviewed evaluations as contributing to low ratings in relation to sub-criterion 5.2 on environmental sustainability were:

• Low priority given to supporting necessary program components focusing on environmental sustainability (3 evaluations);
• Under investment in program components to mitigate negative environmental impacts, such as better land use management or better on-farm practices for fertilization and pest control (4 evaluations); and
• Insufficient staff time invested in the analysis of possible environmental impacts and strategies to mitigate them (3 evaluations).

• 3.6 Evaluation is Effective and Well Used, but Weaknesses are Highlighted in Monitoring and Results-Based Management

  3.6.1 Coverage

  Coverage of the sub-criteria relating to the use of evaluation and monitoring to improve effectiveness was generally “strong” (Figure 12) with sub-criteria 6.1 “Systems and processes for evaluation are effective”, 6.2 “Systems and processes for monitoring are effective” and 6.4 “Evaluation results used to improve development effectiveness” all rated as “strong.” Sub-criterion 6.4 was addressed by all 45 evaluations in the review sample. Sub-criterion 6.3 “Results-based management systems are effective” was rated “moderate” in coverage as it was addressed by 26 evaluations.

  Figure 12: Number of Evaluations Addressing Sub-Criteria for Use of Evaluation and Monitoring to Improve Development Effectiveness

  

  3.6.2 Key Findings

  The reviewed evaluations (Figure 13) concluded that ADB has effective evaluation systems; and that the results of evaluations are consistently used to improve effectiveness. However, important weaknesses remain, particularly in local systems for results-based management, including results monitoring.

  The system of independent evaluation at the ADB (sub-criterion 6.1) is effective given that 82% of evaluations rated the Bank’s evaluations systems as “satisfactory” or better. The evaluations are also well used to improve effectiveness (sub-criterion 6.4). Eighty percent of evaluations found that the ADB’s response to the recommendations made in evaluation reports was either “satisfactory” or “highly satisfactory” (sub-criterion 6.4).^{Footnote 33} MOPAN results for related indicators corroborate these results: IV-1 “Independent evaluation” was “Strong,” IV-1 “Programs subject to Independent Evaluation” was also “Strong,”; and IV-1 “Stakeholders involved in monitoring and evaluation” scored “adequate.”. With respect to sub-criterion 6.4, MOPAN reported similar results under micro-indicator II-4 “Evaluations recommendations are acted upon,” as it received a score of “adequate”.^{Footnote 34}

  Since 2008, the IED at the ADB has been responsible for a system used to track evaluation recommendations and the level of agreement to those recommendations by ADB management. As noted in the 2012 Annual Evaluation Review Report, the majority of IED’s recommendations (92% over 2008-2011) have been accepted by Management and a significant share (73% over 2009-2011) have been acted upon.^{Footnote 35}

  However, results monitoring and reporting systems at the local level (sub-criterion 6.2) did not score as well as evaluation systems in IED evaluations.^{Footnote 36} Twenty percent of evaluations rated local monitoring and results reporting systems as “satisfactory,” while only 8% of evaluations were given the same ratings for results-based management systems (sub-criterion 6.3). The latter sub-criterion was rated the weakest out of any of the eighteen sub-criteria.

  It should be noted, however, that the same evaluations pointing to deficiencies in results-based management, including results monitoring and reporting, often pointed to recent efforts to strengthen these systems and to overcome weaknesses inherited from the early years of the evaluations period covered. As such, both these areas may receive stronger ratings in future evaluation reports.

  Sub-criteria 6.2 and 6.3 were the main area of variance between the MOPAN results and the findings of this report. For these two sub-criteria, the results from the report were largely “unsatisfactory” or “highly unsatisfactory.” By contrast, results reporting was rated as “Strong” by the MOPAN survey (micro-indicator IV-2 “Reports on Results, Including Outcomes”) while results-based management was rated “adequate” (micro-indicator I-4 “Frameworks include indicators at project, program, sector and country level”).^{Footnote 37} This difference may have arisen from the micro-level focus of the evaluation reports, which tended to look at results-based management systems from the bottom up and thereby focused on the development and collection of results indicators at the project level. Another factor in the difference between the MOPAN results and the pilot test may be found in the time frames covered in the two reviews—the MOPAN evaluation results referred to a successful recent initiative at the ADB to focus on strengthening results reporting and results-based management. Since the evaluations reviewed for the pilot test were retrospective and covered multi-year country programs, these evaluations do not reflect recent efforts to strengthen both functions.

  Figure 13: Using Evaluation and Monitoring to Improve Development Effectiveness [Findings as a percentage of number of evaluations addressing the issue (=a), n=45]

  

  3.6.3 Contributing Factors

  Factors that have contributed to achieving an effective use of evaluation and monitoring to improve the effectiveness of ADB operations include:

• The commissioning of studies including project completion reports, special studies and evaluations at the project, sector and country assistance program level that, in turn, support the design, implementation and subsequent evaluation of programs (26 evaluations) (Highlight Box 7); and
• The existence of a formal Management Action Record System in place at the ADB since 2008 (as described in the Annual Evaluation Review) which tracks recommendations to management and the level of management agreement. This places a higher priority on monitoring and evaluation systems.

• The reviewed evaluations reported on factors that mitigated against the use of evaluation and monitoring to improve effectiveness, including:

• Absence of targets (7 evaluations); the poor quality and coverage of indicators (6 evaluations); or absence of indicators (3 evaluations);
• Insufficient knowledge or capacity for evaluation and monitoring on the part of partner agencies, thus undermining monitoring and reporting systems (7 evaluations); and
• Absence of baseline data as a factor limiting evaluation, monitoring and results reporting (5 evaluations).
• Failure to monitor specified results indicators (3 evaluations); and
• Lack of ownership on the part of host governments (2 evaluations).

• Highlight Box 7

  Evaluating ADB Support to Public Sector Reform in the Pacific

  “In addition to preparing program completion reports for each program loan, ADB has independently evaluated four program loans to Pacific Developing Member Countries (DMCs) through program performance evaluation reports. This study evaluates a wider set of program loans to Pacific DMCs to identify cross-country and strategic issues that have influenced the development effectiveness of program loans.”

  Special Evaluation of ADB Support to Public Sector Reform in the Pacific

  4.0 The ADB and Canada’s Priorities in International Development

  In May 2009, the Minister for International Cooperation announced Canada’s intention to focus its development assistance on three thematic priorities: increasing food security; stimulating sustainable economic growth; and securing the future of children and youth (Figure 14). This section considers the extent to which the ADB contributes to these development priorities. The section first reviewed Canada’s relationship with the ADB, including accountability and management oversight responsibility within CIDA, and then addressed the extent to which the ADB contributed towards Canada’s priorities in international development, and to the implementation of CIDA’s strategic objectives for engagement with the ADB.

  Figure 14: Canada’s Thematic Priorities^{Footnote 38}

  

  4.1 CIDA’s Support to the ADB

  The volume of CIDA’s financial support to the ADB (Figure 15) illustrates the organization’s importance as a multilateral channel for Canadian development assistance. It also demonstrates Canada’s importance to the ADB as a source of funding.

  The ADB is a major recipient of Government of Canada support (through CIDA),receiving C$330 million of financial support comprised of the capital subscription (long-term or “core” support) and non-core funding over the past five fiscal years (2006-2007 to 2010-2011).^{Footnote 39} This represents a critical component of CIDA support to the region of the Asia-Pacific.

  Figure 15: CIDA Funding to the ADB 2006-2007 to 2010-2011 (C$ million)

  

  Prepared by Statistical Analysis and Reporting Section, Chief Financial Officer Branch, CIDA (2011).

  As illustrated in Figure 15, most CIDA support to the ADB has been in the form of long- term institutional support. The portion of CIDA commitments made up of the capital subscription has varied from a high of 88% of all funding in 2007-08 to a low of 64% in 2009-2010. Non-core funding has grown from C$10.06 million dollars in 2007-08 to C$29.52 million in 2010-2011.

  Canada is a founding member of the ADB and is the sixth largest shareholder (5.25% of total shares). Canada shares a seat on 12 members Board of Directors with Denmark, Finland, Ireland, Norway, Sweden and the Netherlands. This constituency represents the second largest non-regional voting bloc after the United States, and is usually headed by a Canadian Executive Director. Canada was the sixth largest donor contributing to the ADF X replenishment (2009-2012) of the Asian Development Fund when it contributed C$190.8 million.^{Footnote 40}

  4.2 Managing CIDA’s Relationship with the ADB

  Canada’s International Financial Institutions Act names the Minister of Foreign Affairs as the Governor responsible for some of the regional development banks, including the ADB. CIDA has led the Government of Canada’s day-to-day relationships with these institutions given the regional development banks’ poverty reduction mandates. CIDA’s management of these relationships includes the concessional development funds of the regional development banks, funded and managed under the authority and accountability of the Minister of International Cooperation.

  Since 2010, CIDA, through the International Assistance Envelope (IAE), also finances general capital increases (paid-in capital) under the authority and accountability of the Minister of International Cooperation. In addition, CIDA supports various regional development bank trust funds, projects and programs.

  Foreign Affairs and International Trade Canada (DFAIT) provides guidance on foreign policy considerations and Finance Canada (FIN) advises on financial, fiduciary and risk management issues. CIDA leads DFAIT and FIN in coordinating Canadian relationships with these banks.

  Canadian positions on major policy issues and policy dialogue are developed jointly by all three departments (CIDA, FIN and DFAIT), including Canada’s participation in ADB annual meetings, discussions with bank presidents and other senior-level officials, general capital increases, concessionary fund replenishments as well as key policies and strategies.

  Canada is represented on the ADB’s Board of Directors by an Executive Director. At the Bank, the Canadian Executive Director leads a constituency that includes Denmark, Finland, Norway, Sweden, Ireland and the Netherlands.

  In recent years, Canada has focused on strengthening the independence of the evaluation function of the Independent Evaluation Department (formerly, the Operations Evaluation Department), improving results-based management, strengthening gender mainstreaming (including sharing experiences on the impact of the financial crisis on women), and strengthening safeguards for environmental protection and resettlement.

  CIDA’s Multilateral and Global Programs Branch is responsible for managing the long-term institutional support and initiative-specific funding to the ADB (see Annex 7 for CIDA funding to multilateral development organizations).^{Footnote 41} These two forms of funding account for 78% of CIDA funding to the ADB in the past four fiscal years.

  Multilateral and Global Programs Branch’s relationship with the ADB is guided by CIDA’s Institutional Strategy for Engagement for the ADB.^{Footnote 42} CIDA’s engagement with the ADB is defined by the following three strategic objectives, as per the strategy:

• Press the ADB to mobilize an effective response to the short and long-term impacts of the global financial crisis in its member countries, particularly for the poor and vulnerable;
• Work with the ADB to strengthen programming development and implementation in CIDA’s countries of focus, notably, the key fragile states of Afghanistan and Pakistan;
• Work more intensively with ADB management and the Board to tackle remaining bottlenecks preventing achievement of critical advances in institutional effectiveness, particularly, those relating to more open management, stronger delegation and human resource practices.

• 4.3 Alignment with CIDA’s Thematic Priorities

  4.3.1 Increasing Food Security

  The ADB’s contribution to increased food security is more indirect than direct, especially in light of the long-term institutional framework, Strategy 2020, decision to focus on core sectors that did not include agriculture.^{Footnote 43} In 2011, the ADB and Asian Development Fund approved a combined US$15.7 billion in new financing,^{Footnote 44} with only US$159 million or 1% allocated to agriculture.

  This contradicts a Strategy 2020 commitment to support agriculture development through the core area of support to infrastructure: “ADB will support agriculture mainly through investment in infrastructure for rural transport, irrigation and water systems.” In 2011, the ADB approved financing of US$13.9 billion or 88% of the total, to the area of infrastructure investment. These investments, along with smaller amounts in education and public sector management, should make a significant contribution to improved food security in Asia.^{Footnote 45}

  4.3.2 Stimulating Sustainable Economic Growth

  Strategy 2020 identifies five crucial drivers of change in Asia for the period from now until 2020 including: private sector development; encouraging good governance; supporting gender equity; helping developing countries gain knowledge; and expanding development partnerships. All five of these drivers represent opportunities for the ADB to support sustainable economic growth in Asia under its over-arching goal of reducing poverty and its overall vision of “an Asia-Pacific free of poverty”.

  The ADB portfolio of operations also includes direct investments in private sector development. As an example, in 2011, the Bank approved financing of US$390 million (2% of total approvals) for the development of the finance sector in member countries. Finally, the ADB allocated US$660 million to programs in public sector management, thereby contributing to strengthening the development of national policies and programs in support of sustainable economic growth.^{Footnote 46}

  4.3.3 Securing the Future of Children and Youth

  Two of the ADB’s five core areas of programming, environmental sustainability and education are directly related to securing the future of children and youth. In terms of environment, US$3.2 billion of new financing was approved in the water, sanitation and waste management sectors in 2011 (20% of the total) (Highlight Box 9). An additional US$1.1 billion was approved for financing investments in education.^{Footnote 47} The net effect of investments in education, environment and infrastructure, combined with the levels of effectiveness reported in Section 3.0, is expected to make a contribution to securing the future of children and youth in Asia.

  Highlight Box 9

  Improving Water Supply in the Punjab to Contribute to the Future of Children and Youth

  The results show that the projects had a clear and large influence on the intermediate outcome - that is, access to water supply. The projects drastically reshaped the sources of household water in project areas, raising the proportion of households with piped water in their dwellings and reducing reliance on hand pumps, tube-wells, and boreholes, which were still the major sources of water in the comparison villages.

  Evaluation of the Impact of Rural Water Supply in the Punjab, Pakistan, p. ii. (2009)

  4.4 How is the ADB Fulfilling the Strategic Objectives that CIDA Identified?

  4.4.1 Responding Effectively to the Financial Crisis

  The increase in ADB disbursements from 2008 to 2009 (which increased to US$10.5 billion) indicated an effort to respond to the need for increased financial support to governments in Asia and the Pacific during the financial crisis. Specifically, the ADB responded to the 2008-2009 economic crisis by allocating US$400 million to help the most fiscally stretched, eligible Asian Development Fund countries. It also allowed the same countries to front load up to 100% of their biennial [Asian Development Fund] allocation for 2009-2010.^{Footnote 48} The mid-term review meeting of the Ninth Cycle/Replenishment of the Asian Development Fund described the Bank’s response to the financial crisis as “proactive and timely, with significantly increased approvals and disbursements”. ^{Footnote 49} The IED’s Real-time Evaluation of Asian Development Bank’s Response to the Global Economic Crisis of 2008-2009 also found that the ADB’s assistance was “relevant, responsive and “satisfactory” in achieving program objectives and initial results”. However, it noted that inadequate financial resources slowed its response to the crisis.^{Footnote 50}

  4.4.2 Strengthening Programming in Canada’s Countries of Focus

  Five of CIDA’s countries of focus continue to be an important focus of ADB operations. In 2011, the ADB approved financing (loans, guarantees, investments and grants) to Vietnam of US$3.6 billion, making it the largest recipient of ADB financing. Pakistan and Bangladesh also received a large amount of financing with US$2.9 billion (third largest recipient) and US$2.3 billion (fourth largest recipient), respectively. Indonesia and Afghanistan received approval for US$809 million and US$300 million, respectively.^{Footnote 51}

  In October 2011, the ADB published an evaluation of the operations of the Asian Development Fund from 2001 to 2010, entitled the Asian Development Fund Operations: A Decade of Supporting Poverty Reduction in the Asia and Pacific Region. This study of a decade of operations highlighted the Asian Development Fund’s role in providing financing to poorer countries in the region. From 2001 to 2010, the ADB approved US$20 billion in Asian Development Fund loans and grants to 29 countries. The largest recipients in descending order were Bangladesh, Vietnam, Pakistan, Afghanistan, and Nepal; accounting for 64% of overall approval for loans and grants.^{Footnote 52}

  4.4.3 Improving Institutional Effectiveness

  There are also indications that the ADB is investing in improved management systems. In 2010, the Bank unveiled a new overall human resource strategy (its “People Strategy”) and followed up with the Human Resources Function Strategic Framework and Action Plan. This also marked the first full year of operation of the Board’s Human Resources Committee, which met seven times during the year and issued its first annual report to the board.

  In summary, the ADB does appear to be responding to the messages received from Canada (and other donors) as embodied by CIDA’s strategic goals but there is a continuing need to emphasize the requirement for progress. The 2010 Annual Report for example, while praising the level of economic growth in the region since the financial crisis of 2008, also points to worsening income inequalities. Fourteen of 20 developing countries saw their Gini coefficients (a measure of income inequality) increase as economic growth accelerated in the period since the crisis.^{Footnote 53}

  5.0 Conclusions

  The 45 evaluations reviewed provided reasonable coverage of ADB programming and offer adequate confidence for the findings reported. Since the period under review (2006-2010), the ADB has continued to improve its programming. Many of these changes correspond to the findings of the evaluations used in this review. A comprehensive survey of the improvements initiated by the ADB in the recent past is outside the scope of this review. However, the ADB has provided an overview of the most important of these recent changes (see Annex 8).

  Based on the key findings and related contributing factors reported, this review concludes the following

  5.1 Development Effectiveness of the ADB

  1. ADB operations are successful in achieving most of their development objectives as reported in over two thirds of the evaluations reviewed. They are also effective in contributing to positive changes for target group members and in contributing to national development goals. The one third of evaluations reporting “unsatisfactory” or “highly unsatisfactory” results for objective achievement indicate that the ADB had opportunities to increase its overall development effectiveness. An important factor contributing to results in objectives achievement has been high levels of national ownership, which, in turn, result in higher levels of performance, by government and non-government implementing partners. An important factor limiting objectives achievement for some programs has been the lack of institutional capacity on the part of host government agencies.
  2. Given the weak coverage of gender equality (16 of 45 reviewed evaluations), it is difficult to draw conclusions on results in this area. For the few evaluations addressing gender equality, the results reported are generally positive. Nonetheless, there are still challenges for ADB programs in this area. First, without regular evaluation of results in gender equality, it is unlikely that results will improve. Those evaluations that address the issue also indicate the need to assign a higher priority to budgeting for gender-focused components in some projects.
  3. IED evaluations more frequently address issues of environmental sustainability. Almost two-thirds of evaluation report results of “satisfactory” or “highly satisfactory.” However, there is an important challenge for the ADB in ensuring that its operations support environmentally sustainable investments. Over a third of the evaluations addressing this criterion rated program performance as “unsatisfactory” or “highly unsatisfactory.” Evaluation results indicate that there is an opportunity to improve by investing more in program elements aimed directly at mitigating negative environmental impacts.
  4. The sustainability of benefits resulting from ADB operations is a major concern with over half of evaluations reporting results of “unsatisfactory” or worse. A key problem is the institutional capacity of partners to sustain program benefits. Two important factors contributing to poor results for sustainability are inadequate investment in the maintenance of infrastructure and the absence of a realistic and well-funded capacity development component in some programs.
  5. ADB projects and programs meet the needs of target group members, are in line with national development goals and pursue objectives that remain valid over time. There is a need, however, to make improvements in program design to ensure that activities and outputs are more directly related to program objectives for a significant number of projects and programs. There is also an opportunity to improve the suitability of program components to the needs of target group members by investing more resources in needs-analysis during program design.
  6. Efficiency is one of the weakest areas of ADB performance according to the evaluations reviewed, with neither sub-criterion (cost efficiency and timeliness) receiving “satisfactory” or better scores in half of the evaluations reviewed. The evaluations listed a number of problematic areas of program start-up and implementation, including delays in recruiting and fielding consultants, delays in procurement of program inputs, and delays in government compliance with agreed policy and regulatory changes. These delays are also cited as a factor contributing to poor overall performance in the area of cost efficiency.
  7. The system of independent evaluation at the ADB is both effective and well used to improve effectiveness. The effectiveness of evaluation systems, coupled with the ADB’s effective use of evaluation findings indicates that the organization is committed to systematic learning from program experience and that it puts this learning to use in program development. There were many examples of program improvements resulting from earlier evaluation studies reported in the evaluation reports reviewed.
  8. Results monitoring and reporting systems at local level did not score as well as the evaluation systems in IED evaluations. This was the worst rated of any of the eighteen sub-criteria. The most frequent cause of poor findings in the effectiveness of results-based management systems, including results monitoring, involved problems with the indicators, including the absence of targets, and poor quality and coverage of the indicators that were being used. Evaluations also point to weaknesses in knowledge and capacity for monitoring and evaluation among program partners.
  9. ADB programming contributes to all three of CIDA’s development priorities: increasing food security, stimulating sustainable economic growth, and securing the future of children and youth. CIDA’s priorities for engaging with the ADB include: responding effectively to the financial crisis, particularly for the poor and vulnerable; strengthening programming in Canada’s countries of concentration; and improving institutional effectiveness. The ADB reacted in a timely and proactive way to the financial crisis with significantly faster approvals and increased disbursements. It continues to focus operations on Afghanistan and Pakistan (countries of focus for CIDA). The ADB also continues to invest in efforts to improve institutional effectiveness, especially in the analysis and reporting of the development effectiveness of ADB programs.

  5.2 Development Effectiveness Reporting at the ADB

  The annual reports on evaluation and on development effectiveness produced by the IED and ADB in 2012, including Development Effectiveness Report 2011: Private Sector Operations, illustrate the Bank’s commitment to using evidence gathered through the evaluation systems to report openly on achievements and trends in development effectiveness. The 2011 Development Effectiveness Review report, in particular, highlights the need to improve results in the achievement of outcomes and to continue to mainstream measures introduced in 2010-2011 to improve project implementation and outcomes. Given the quality of ADB-published evaluations and the continued refinement of the annual Development Effectiveness Review report and of similar reports on private sector operations, donors and other shareholders should be able to rely on these sources for reporting on development effectiveness in the future. There is no apparent need for another external review of the ADB in the medium-term.

  6.0 Recommendations for CIDA

  This section includes recommendations to CIDA, based on the findings and conclusions of the development effectiveness review of the ADB. As one of several shareholders working with the ADB, CIDA’s individual influence on the organization is limited and it may need to engage with other shareholders to implement these recommendations.

  CIDA should:

  Emphasize the need to ensure that gender equality is directly addressed in future IED evaluations. Evaluation policies at both the central and local office-level should pay adequate attention to gender equality as a key evaluation issue. This may also require greater attention to gender analysis skills (and adequate training) among evaluation teams.

  Engage with the ADB to ensure that the environmental sustainability of infrastructure and other assets financed by the Bank receives sufficient attention, and that results in this area are improved over time.

  Raise the issue of the sustainability of the benefits of ADB investments to a strategic level in CIDA’s engagement with the Bank. Two areas of promise are greater investment in operation and maintenance of the assets created (as identified in the ADB’s own annual assessment of development effectiveness) and improved planning and execution of capacity development components of programs.

  Make improving the timeliness of ADB operations a priority area for CIDA’s interaction with the ADB. According to the evaluations reviewed, timeliness could be improved through changes in systems and procedures used during each phase of the project life cycle. This represents one of the most readily apparent ways to improve the efficiency of ADB operations.

  Emphasize the need to strengthen systems for program results monitoring and reporting at the local level and results-based management, including improving the quality and coverage of indicators and the use of associated targets to track results. This will require investments in capacity development for ADB staff and partners at the country office level and more locally (in the case of partners).

  Annex 1: Criteria Used to Assess Development Effectiveness

  Relevance
  1.1 Programs are suited to the needs and/or priorities of the target group.
  1.2 Programs align with national development goals.
  1.3 Effective partnerships with government.
  1.4 Program objectives remain valid.
  1.5 Program activities are consistent with program goals and objectives achievement.

  Achieving Objectives and Expected Results
  2.1 Programs achieve stated objectives and attain expected results.
  2.2 Positive benefits for target group members
  2.3 Substantial numbers of beneficiaries

  Sustainability of Results and Benefits
  3.1 Benefits continuing or likely to continue after program completion.
  3.2 Programs are sustainable in terms of institutional capacity.

  Efficiency
  4.1 Program activities are evaluated as cost/resource efficient.
  4.2 Implementation and objectives achieved on time.

  Inclusive Development which can be Sustained (Gender Equality and Environmental Sustainability)
  5.1 Programs effectively address the crosscutting issue of gender equality.
  5.2 Extent to which changes are environmentally sustainable.

  Using Evaluation and Monitoring to Improve Development Effectiveness
  6.1 Systems and process for evaluation are effective.
  6.2 Systems and processes for monitoring and reporting on program results are effective.
  6.3 Results-based management systems are effective.
  6.4 Evaluation is used to improve development effectiveness

  Annex 2: Evaluation Sample

  #	Year	Title	Type
  1	2010	Indonesia: Has the Multi-subsector Approach Been Effective for Urban Services Assistance?	Special Evaluation
  2	2009	Country Assistance Program Evaluation for Bangladesh	Country Assistance Program Evaluation
  3	2009	Country Assistance Program Evaluation for Nepal	Country Assistance Program Evaluation
  4	2009	Country Assistance Program Evaluation for Vietnam	Country Assistance Program Evaluation
  5	2009	Country Assistance Program Evaluation for Cambodia	Country Assistance Program Evaluation
  6	2009	Impact of Rural Water Supply and Sanitation in Punjab, Pakistan	Sector Assistance Program Evaluation
  7	2009	Sector Assistance Program Evaluation Agriculture and Rural Development Sector in Cambodia	Sector Assistance Program Evaluation
  8	2009	Sector Assistance Program Evaluation Transport Sector in Cambodia	Sector Assistance Program Evaluation
  9	2009	Special Evaluation Study on ADB Assistance for Public-Private Partnerships in Infrastructure Development	Special Evaluation
  10	2009	Evaluation on ADB Technical Assistance for Justice Reform in Developing Member Countries	Special Evaluation
  11	2009	The Asian Development Bank's Support to Gender and Development Phase I Evaluation Report: Relevance, Responsiveness, and Results to Date	Special Evaluation
  12	2009	ADB Support for Public Sector Reforms in the Pacific: Enhance Results through Ownership, Capacity, and Continuity	Special Evaluation
  13	2009	Asian Development Bank's Contribution to Inclusive Development through Assistance for Rural Roads	Special Evaluation
  14	2009	SAPE Bangladesh: Energy Sector	Sector Assistance Program Evaluation
  15	2009	Agriculture and Natural Resources Sector in Nepal	Sector Assistance Program Evaluation
  16	2009	Asian Development Bank Support for the Transport Sector in Viet Nam	Sector Assistance Program Evaluation
  17	2008	Country Assistance Program Evaluation for the Philippines	Country Assistance Program Evaluation
  18	2008	Regional Cooperation Assistance Program Evaluation for the Greater Mekong Sub region	Thematic / Regional
  19	2008	Sector Synthesis of Evaluation Findings – Education	Thematic
  20	2008	Evaluation Study on Policy Implementation and Impact of Agriculture and Natural Resources Research	Special Evaluation
  21	2008	Effectiveness of the ADB's Capacity Development Assistance: How to Get Institutions Right	Special Evaluation
  22	2008	Implementing the Paris Declaration at the ADB	Special Evaluation
  23	2008	Special Evaluation Study on Private Equity Fund Operations	Special Evaluation
  24	2008	Country Assistance Program Evaluation for Mongolia	Country Assistance Program Evaluation
  25	2007	Evaluation of the ADB's Country Assistance Program for India	Country Assistance Program Evaluation
  26	2007	Country Assistance Program Evaluation: Pakistan	Country Assistance Program Evaluation
  27	2007	Evaluation of the ADB's Country Assistance Program in the People's Republic of China: Success Drives Demand for More Innovative and Responsive Services	Country Assistance Program Evaluation
  28	2007	Evaluation of the ADB's Country Assistance Program to Sri Lanka	Country Assistance Program Evaluation
  29	2007	Special Evaluation Study on the ADB's Private Sector Development and Operations	Special Evaluation
  30	2007	Special Evaluation Study on Indigenous Peoples Safeguards	Special Evaluation
  31	2007	Special Evaluation Study on Performance of Technical Assistance	Special Evaluation
  32	2007	Evaluation on the Effect of Microfinance on Poor Rural Households and the Status of Women	Special Evaluation
  33	2007	The ADB's Approaches to Partnering and Harmonization: In the Context of the Paris Declaration on Aid Effectiveness	Special Evaluation
  34	2007	Evaluation of the Asian Development Fund VIII and IX Operations	Special Evaluation
  35	2007	Environment Management Technical Assistance Projects to Selected Central Asian Republics	Thematic Regional
  36	2007	Evaluation of the ADB's Assistance for the Roads and Railways Sector in the People's Republic of China	Sector Assistance Program Evaluation
  37	2007	Transport Sector in India – Focusing on Results	Sector Assistance Program Evaluation
  38	2007	Evaluation of the Japan Special Fund	Special Evaluation
  39	2007	Evaluation of the ADB's Assistance to the Energy Sector in India — Building on Success for More Results	Sector Assistance Program Evaluation
  40	2006	Country Assistance Program Evaluation: Lao People's Democratic Republic	Country Assistance Program Evaluation
  41	2006	Country Assistance Program Evaluation for Uzbekistan	Country Assistance Program Evaluation
  42	2006	Evaluation of the Tuvalu Islands Development Program	Thematic
  43	2006	Special Evaluation Study on Lessons in Capacity Development: Sectoral Studies in Sri Lanka	Special Evaluation
  44	2006	Sector Assistance Program Evaluation for the Road Sector in Pakistan	Sector Assistance Program Evaluation
  45	2006	Technical Assistance in Support of the Pacific Financial Technical Assistance Centre in the Pacific Island Countries	Thematic

  Annex 3: Methodology

  This Annex explains more thoroughly key elements of the methodology used for the development effectiveness review of the ADB. It is structured around the sequence of tasks undertaken during the review: determining the rationale for the review; drawing the sample of evaluations; undertaking the process of review and controlling for quality during the analysis phase; and assessing the level of coverage provided by the effectiveness review. A more detailed description of the methodology can be found in the methodology guide for using the common approach to assessing development effectiveness.^{Footnote 54}

  This review of evaluation reports was supplemented by a review of ADB corporate documents related to evaluation and reporting on development effectiveness and by a consultation with the CIDA manager responsible for managing relations with the ADB.^{Footnote 55} This research was done to contextualize the results of the review and to take in account advances following the pilot test analysis carried out in 2010. A list of the documents consulted is provided in Annex 5.

  Rationale for a Development Effectiveness Review

  The common approach and methodology offer a rapid and cost-effective way to assess the development effectiveness. The approach was developed to fill an information gap regarding effectiveness of multilateral organizations. Although these multilateral organizations produce annual reports for their management and/or boards, bilateral shareholders were not receiving a comprehensive overview of the performance of multilateral organizations in the countries. MOPAN seeks to address this issue through organizational effectiveness assessments. This approach complements MOPAN’s assessments.

  The approach suggests conducting a review of effectiveness, based on the organization’s own evaluation reports when two specific conditions exist:

  1. There is a need for field-tested and evidence-base information regarding the effectiveness of the multilateral organization; and
  2. The multilateral organization under review has an evaluation function that produces an adequate body of reliable and credible evaluation information that supports a meta-evaluation synthesizing an assessment of the organization’s development effectiveness.

  The independent evaluation function at the ADB does produce enough evaluation reports of good quality to support an assessment of the development effectiveness of the organization. The first condition was also satisfied since the ADB’s existing reporting mechanisms did not provide sufficient information on the organization’s effectiveness in 2010, when the pilot-test analysis was carried out.

  The ADB’s Evaluation Function (Quantity and Quality)

  Quantity of Evaluations:

  The ADB’s Independent Evaluation Department (IED) leads five different types of strategic and higher level evaluations. Figure 1 provides an overview of the different evaluation types. A review of the ADB’s website was carried out in July 2010 to identify evaluation reports led by IED and published by the ADB between the beginning of 2006 and early 2010. This was supplemented by a request to the ADB to identify any missing documents.

  The final result was a list of 90 evaluations including: Country Assistance Program Evaluations in countries with both large and small ADB program portfolios; Sector Assistance Program Evaluations in a wide range of sectors (i.e., agriculture, transport, public sector reform, microfinance); special evaluations of different forms of development assistance (i.e., capacity development, technical assistance, policy support); and evaluations of different policy initiatives and strategies within the ADB (i.e., gender equality, conforming to the Paris Declaration, partnering and harmonization). From this group of evaluation reports, the review team was able to develop a sample of 45 evaluation reports published by the ADB in the 2007 to 2010 period that covered countries with 82% of ADB programming in 2009. They also accounted for 83% of loan approvals in 2006 and included the 10 countries with the largest ADB programs. A sufficient number of evaluations existed in the public domain to support a meaningful development effectiveness review.

  Quality of Evaluations:

  The Annual Evaluation Review reports of the IED do not assess the quality of evaluation reports, but they do catalogue the number and type of evaluations completed by IED each year. They also report annual and multi-year trends in findings.

  The review team conducted its own quality review of a sample of 45 different evaluation reports (described more fully in this annex). The review team conducted its own quality review of the evaluations included in the review. The results were positive, with 39 of the 45 evaluation reports (86.6%) of the reviewed evaluations scoring 30 points or more out of a possible 48. Further, only two evaluation reports received a score of less than 24. Given these results, all 45 IED evaluation reports were included in the analysis.

  Therefore, the second condition for carrying out a development effectiveness review using the common approach is met. There are a sufficient number of good quality evaluation reports to support the use of meta-evaluation to conduct an assessment of the ADB’s development effectiveness.

  The ADB’s Reporting on Development Effectiveness

  The ADB has a mix of instruments for analyzing and reporting its development effectiveness. There are two main reports on development effectiveness produced by the ADB each year: the Annual Evaluation Review and the annual Development Effectiveness Review. These have been consistently improved and by 2011 (when the 2010 editions of each report were produced), they provided a detailed overview of the effectiveness of ADB operations.

  There is a series of reports on development effectiveness produced by the ADB each year: the Annual Evaluation Review (by the IED), the annual Development Effectiveness Review and the Development Effectiveness Report: Private Sector Operations. These have been consistently improved^{Footnote 56} and, by 2012 when the 2011 editions of each report were produced, they provided a detailed overview of the effectiveness of ADB operations. ADB’s analysis of the 3-year average development effectiveness ratings shows that ADB performance is influenced by specific country and sector circumstances, as well as the expertise used to design and manage projects.

  The annual Development Effectiveness Review report examines the performance of ADB operations at four levels:

  1. Level one: Examines progress in achieving development objectives in Asia and the Pacific. This section examines the region’s performance in achieving the objectives of poverty reduction and human development, and other regional outcomes (i.e., growth, regional cooperation and integration, basic infrastructure, finance, governance and the environment);
  2. Level two: Examines two aspects of ADB’s operations: (i) status of achievement of the results framework core sector outputs targeted for a specific period; and (ii) contribution of recently completed operations (both sovereign and non-sovereign) to their intended sector outcomes;
  3. Level three: Examines operational effectiveness (i.e., quality of completed operations, knowledge management, partnerships, etc.) and;
  4. Level four: Examines the ADB’s organizational effectiveness (i.e., budget adequacy, human resources, and business processes and practices).

  The 2012 Annual Evaluation Review, for example, reported that following an assessment of the 26 country assistance programs and a validation of the six country partnership strategy final review, 67% of 30 country program evaluations with overall ratings were assessed successful and 33% less than successful. The report also noted factors contributing to the success, including the quality of ADB’s strategic positioning (good choice of sectors and themes, and aid harmonization) and the relevance of the program adopted (good matching with country binding constraints and government priorities). The effectiveness and efficiency were rated more varied, and a high proportion of ADB’s country interventions were rated, overall, less likely sustainable.^{Footnote 57}

  For level two results (development outputs and outcomes), the Development Effectiveness Review relies mainly on inputs from IED evaluation reports of completed operations.^{Footnote 58} It was particularly direct in its 2011 edition^{Footnote 59} when it reported that:

  ADB improved the quality of its ongoing operations, and earned good ratings for its two previously weak performance areas: support for gender mainstreaming in operations and finance mobilization. However, the review confirmed that the quality of ADB’s recently completed operations – including their delivery of core sector outcomes – remained considerably below target despite improvements from the previous year. Furthermore, project delays and cancellations lowered the outputs to be delivered during 2009-2012 from Asian Development Fund (ADF) operations.”

  The same report noted that two thirds of ADB operations in the same period had achieved their stated objectives. Similar findings can be seen in section 3.2 of this report.

  Given the quality of ADB-published evaluations and the continued refinement of the annual Development Effectiveness Review report, donors and other shareholders should be able to rely on these two sources for reporting on development effectiveness in the future. There is not apparent need for another external development effectiveness review of the ADB in the medium-term. Nonetheless, at the time this analysis was conducted (in early 2010), the first condition for undertaking a development effectiveness was met, since earlier versions of the Development Effectiveness Review report were not as comprehensive in their use of evaluation material and there was still a need to improve the availability of effectiveness information of the organization that is based on field-tested and evidence-based reporting.

  Selecting the Evaluation Sample

  Subsequent to the review team’s classification of the universe of IED evaluations published between 2006 and 2010 based on evaluation type (Figure 16) and by year of publication (Figure 17), the universe of evaluations was assessed based on the following principals:

  1. Maximize coverage of ADB loans and disbursements;
  2. Maximize geographic coverage;
  3. Include a mixture of evaluations types, especially Country and Sector Program Evaluations; and,
  4. Ensure the main areas of ADB programming were adequately represented.

  From an original sample of 50 evaluation reports, the sample list was subsequently reduced to 45 during the review process. The reduction was not done in order to eliminate evaluations with relatively low quality screening results scores, rather it occurred during the analysis phase when five of the studies were found to be focused on very different evaluation criteria than those included in the criteria being tested (relevance, objectives achievement, impact, cost effectiveness and sustainability).

  The twelve countries represented by Country Assistance Program Evaluations in the sample accounted for 82.2% of all ADB assistance (sovereign and non-sovereign) in 2009. It is reasonable to assume that the 12 country programs whose evaluations were included in the sample account for a significant proportion of ADB assistance over the period under review.

  Country Assistance Program Evaluations cover all forms of assistance to a given country so the sample also covers loans, grants, trade finance facilitation, equity investments and technical assistance grants. The review team also adopted a strategy of selecting one or more significant Sector Assistance Program Evaluations in most of the countries where a Country Assistance Program Evaluation was reviewed.

  The sample for the ADB included evaluations of regional programs such as the Regional Assistance Program Evaluation for the Greater Mekong Sub-Region as well as policy-oriented evaluations such as the Special Evaluation Study on Indigenous Peoples Safeguards.

  The sample also focused on covering key sectors of intervention in ADB programs (agriculture, transport, energy, roads). It also encompassed key thematic areas such as indigenous people’s safeguards, capacity development assistance, and private sector development. Overall, the sample of 45 IED evaluations provides coverage of the breadth of activities and the critical mass of ADB investments over a four-year timeframe.

  Figure 16: Coverage, by Type of Evaluation, as a Percentage of Evaluations in Sample

  

  Figure 17: Coverage by Year of Publication as a Percentage of Evaluations in Sample

  

  The Review Process and Quality Assurance

  The review itself was conducted by a team of four analysts and a team leader. A two-day training session was held for analysts to build a common understanding of the review criteria. This was followed by a pre-test in which the analysts and team leader independently reviewed the same two evaluations. The team compared ratings from these two evaluations and developed common agreement on the classification of results for all sub-criteria. This process helped to standardize classification decisions made by the analysts. During the review of evaluations, analysts conferred regularly over any classification issues that arose.

  Once the reviews were completed, the team leader reviewed the coded findings and examined the cited evidence and contributing factors. Based on this examination, the team leader made a small number of adjustments to the coded findings. The process of training, testing and monitoring the evaluation review process was aimed to maximize inter-analyst reliability and to control for bias on the part of any one reviewer.

  All 45 evaluations were found to be of sufficient quality to be included in the review. Of a possible maximum total quality score of 48, the mean score of all evaluations was 38 points. The distribution of total scores for all is described in Annex 4. Only one evaluation had a score of 25 or less.

  Coverage of Effectiveness Criteria

  The review team developed ranges in order to assess the level of coverage of a given sub-criterion (Table 9). Strong coverage was assigned when the number of evaluations—a-- addressing the criterion ranged between 35 and 45. Moderate coverage for a particular criterion was assigned when a ranged between 25 and 34. Weak coverage for a particular sub-criterion was assigned when a fell below 25.

  Eleven (of 18 sub-criteria) yielded valid findings in 35 or more of the evaluation reports and are rated strong in coverage. Another six received valid findings in the moderate range (25 to 34 evaluation reports). Only criterion 5.1 on the effectiveness of ADB programs in supporting gender equality received a weak rating for coverage. Only 16 evaluations addressed this sub-criterion.

  Table 4: Levels of Coverage for Each Assessment Sub-Criterion

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  1.1 Programs are suited to the needs and/or priorities of the target group.	38	Strong	68%	32%
  1.2 Programs align with national development goals.	40	Strong	78%	22%
  1.3 Effective partnerships with government.	38	Strong	55%	45%
  1.4 Program objectives remain valid.	37	Strong	84%	16%
  1.5 Program activities are consistent with program goals and objectives achievement.	38	Strong	56%	44%

  Achieving Objectives and Expected Results

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  2.1 Programs achieve stated objectives and attain expected results.	44	Strong	68%	32%
  2.2 Programs have resulted in positive benefits for target group members.	38	Strong	71%	29%
  2.3 Programs made differences for a substantial number of beneficiaries	26	Moderate	66%	34%

  Sustainability of Results and Benefits

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  3.1 Benefits continuing or likely to continue after program completion.	38	Strong	47%	53%
  3.2 Programs are sustainable in terms of institutional capacity.	34	Moderate	35%	65%

  Efficiency

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  4.1 Program activities are evaluated as cost/resource efficient.	26	Moderate	46%	54%
  4.2 Implementation and objectives achieved on time.	31	Moderate	19%	81%

  Inclusive Development which can be Sustained (Gender Equality and Environmental Sustainability)

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  5.1 Programs effectively address the crosscutting issue of gender equality.	16	Weak	81%	64%
  5.2 Extent to which changes are environmentally sustainable.	31	Moderate	64%	36%

  Using Evaluation and Monitoring to Improve Development Effectiveness

  Sub-Criteria	a*	Coverage Level**	Evaluations Rated Satisfactory (%)***	Evaluation Rated Unsatisfactory (%)***
  6.1 Systems and process for evaluation are effective.	39	Strong	82%	18%
  6.2 Systems and processes for monitoring and reporting on program results are effective.	36	Strong	20%	80%
  6.3 Results-based management systems are effective.	26	Moderate	8%	92%
  6.4 Evaluation is used to improve development effectiveness	45	Strong	80%	20%

  *a = number of evaluations addressing the given sub-criterion

  **Coverage levels: strong: a = 35 – 45, moderate: a = 25 – 34, weak: a = under 25

  *** Satisfactory ratings include “satisfactory” and “highly satisfactory”; unsatisfactory ratings include “unsatisfactory” and “highly unsatisfactory”

  Annex 4: Evaluation Quality - Scoring Guide and Results

  	Criteria to be Scored	Points	Score
  A	Purpose of the evaluation is clearly stated. The report describes why the evaluation was done, what triggered it (including timing in the project/program cycle) and how it was to be used.	4
  B	The evaluation report is organized, transparently structured, clearly presented and well written. There is a logical structure to the organization of the evaluation report. The report is well written with clear distinctions and linkages made between evidence, findings, conclusions and recommendations.	3
  C	Evaluation objectives are stated. Evaluation objectives are clearly presented and follow directly from the stated purpose of the evaluation.	3
  D	Subject evaluated is clearly described. Evaluation report describes the activity/program being evaluated, its expected achievements, how the development problem would be addressed by the activity and the implementation modalities used.	5
  E	Scope of the evaluation is clearly defined. The report defines the boundaries of the evaluation in terms of time period covered, implementation phase under review, geographic area, and dimensions of stakeholder involvement being examined.	5
  F	Evaluation criteria used to assess program effectiveness are clearly identified in the evaluation report and cover a significant number of the Common Criteria for Assessing Development Effectiveness.	5
  G	Multiple lines of evidence are used. The report indicates that more than one line of evidence (case studies, surveys, site visits, and key informant interviews) is used to address the main evaluation issues. One point per line of evidence to maximum of 5.	5
  H	Evaluations are well designed. The methods used in the evaluation are appropriate to the evaluation criteria and key issues addressed. Elements of good design include: an explicit theory of how objectives and results were to be achieved, specification of the level of results achieved (output, outcome, impact), baseline data (quantitative or qualitative) on conditions prior to program implementation, a comparison of conditions after program delivery to those before, and a qualitative or quantitative comparison of conditions among program participants and those who did not take part.	5
  I	Evaluation findings and conclusions are relevant and evidence based. The report includes evaluation findings relevant to the assessment criteria specified. Findings are supported by evidence resulting from the chosen methodologies. Conclusions are linked to the evaluation findings as reported.	5
  J	Evaluation report indicates limitations of the methodology. The report includes a section noting the limitations of the methodology. It indicates any limitations in the design as well as any problems in the implementation (low survey returns for example) and describes how their impact on the validity of results and any measures taken to address the limitations (re-surveys, follow-ups, additional case studies, etc.	3
  K	Evaluation includes recommendations. The evaluation report contains specific recommendations that follow on clearly from the findings and conclusions. Further, the recommendations are specifically directed to one or more organizations and are actionable and aimed at improving Development Effectiveness. (Objectives achievement, crosscutting themes, sustainability, cost efficiency or relevance).	5
  	Total Possible Score	48

  Evaluation Quality Scoring Results

  During the pilot test analysis, the review team grouped the total quality score results for each evaluation into groupings of six in order to present a transparent description of the distribution of quality scores.

  Table 5: Evaluation Quality Scoring Results

  

  Annex 5: References

  CIDA Corporate Documents

  CIDA Strategy for Engagement with the Asian Development Bank (ADB), CIDA, February 2011.

  Review of the Evidence of the Effectiveness of CIDA’s Grants and Contributions: 2005/06-2010/11, CIDA, 2011.

  ADB Corporate Documents

  ADB Action Plan on Managing for Development Results, ADB, October 2009 Annual Evaluation Review 2011, ADB, 2011

  Annual Report 2010, Volume 1, ADB, 2011.

  Annual Report 2011, Volume 1, ADB, 2012.

  Annual Report 2011, Volume 2, ADB, 2012.

  Asian Development Bank’s Support to Gender and Development, Phase I: Relevance, Responsiveness, and Results to Date, IED, ADB, 2009

  Asian Development Bank’s Support to Gender and Development, Phase II: Results from Country Case Studies, IED, ADB, 2010

  Asian Development Fund Operations: A Decade of Supporting Poverty Reduction in the Asia and Pacific Region, ADB, 2011

  CIDA’s Food Security Strategy, CIDA, 2010

  CIDA’s Sustainable Economic Growth Strategy, CIDA, 2010

  CIDA’s Children and Youth Strategy, CIDA, 2010

  Development Effectiveness Review 2010, ADB, 2011

  Development Effectiveness Review 2011, ADB, 2012

  Managing for Development Results: Evaluation Study, ADB, October 2011

  Operations Manual: Bank Policies – Independent Evaluation, ADB, 2011

  Operations Manual “Bank Policies: Classification and Graduation of Developing Member Countries” (2011), ADB, 2003

  Overview of ADF X at Mid-Term, ADB, 2010

  Post-Completion Sustainability of Asian Development Bank-Assisted Projects, IED, ADB, 2010

  Real-time Evaluation of Asian Development Bank’s Response to the Global Economic Crisis of 2008-2009, IED, ADB, 2011

  Strategy 2020: Working for an Asia Pacific Free of Poverty, ADB, 2008

  Other Documents

  Assessing the Development Effectiveness of Multilateral Organizations: Approach, Methodology and Guidelines, Management Group of the Task Team on Multilateral Effectiveness, DAC EVALNET, 2011

  2009 COMPAS Report, World Bank Group, 2010

  MOPAN Common Approach Institutional Report for the Asian Development Bank (ADB), MOPAN, 2010

  Annex 6: Guide for Review Team to Classify Evaluation Findings

  Part One: Common Development Evaluation Assessment Criteria

  Relevance	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  1.1 Multilateral Organization (MO) supported programs and projects are suited to the needs and/or priorities of the target group	The evaluation report indicates that substantial elements of program or project activities and outputs were unsuited to the needs and priorities of the target group.	The evaluation report indicates that no systematic analysis of target group needs and priorities took place during the design phase or the evaluation reports some evident mismatch between program and project activities and outputs and the needs and priorities of the target group.	Evaluation report finds that the MO supported activity, program or project is designed taking into account the needs of the target group as identified through a process of situation or problem analysis and that the resulting activities are designed to meet the needs of the target group.	Evaluation report identifies methods used in project development to identify target group needs and priorities (including consultations with target group members) and finds that the program and project takes those needs into account and is designed to meet those needs and priorities (whether or not it does so successfully.
  1.2 MO supported projects and programs align with national development goals:	The evaluation reports that significant elements of MO supported program and project activity run counter to national development priorities with a resulting loss of effectiveness, overlap or duplication of effort.	The evaluation reports a significant portion (1/4 or more) of the MO supported programs and projects subject to the evaluation are not aligned with national plans and priorities but there is no evidence that they run counter to those priorities or result in overlap and duplication.	Most MO supported programs and projects are reported in the evaluation to be fully aligned with national plans and priorities as expressed in national poverty eradication and sector plans and priorities. Wherever MO supported programs and projects are reported in the evaluation as not directly supportive of national plans and priorities they do not run counter to those priorities or result in overlap and duplication.	All MO supported projects and programs subject to the evaluation are reported in the evaluation to be fully aligned to national development goals as described in national and sector plans and priorities, especially including the national poverty eradication strategy and sector strategic priorities.
  1.3 MO has developed an effective partnership with governments, bilateral and multilateral development organizations and NGOs for planning, coordination and implementation of support to development	The evaluation report indicates that the MO experiences significant divergence in priorities from those of its (government, NGO or donor) partners and lacks a strategy or plan which will credibly address the divergence and which should result in strengthened partnership over time.	The evaluation reports that the MO has experienced significant difficulties in developing an effective relationship with partners and that there has been significant divergence in the priorities of the MO and its partners.	The evaluation reports that the MO has improved the effectiveness of its partnership relationship with partners over time during the evaluation period and that this partnership was effective at the time of the evaluation or was demonstrably improved.	The evaluation reports that the MO has consistently achieved a high level of partnership during the evaluation period.
  1.4 Objectives of MO supported programs remain valid	The evaluation reports that either a significant number of sub-objectives or some of the most important objectives of MO supported programs and projects are no longer valid to the needs and priorities of the target group at the time of the evaluation and that this raises important concerns regarding effectiveness.	The evaluation reports that, while the majority of the objectives of MO supported programs and projects remain valid in terms of addressing target group needs and priorities some objectives and/or sub-objectives are no longer valid. Nonetheless, the evaluation reports that the most important objectives remain valid.	The evaluation reports that, while no systematic effort has been made by MO supported programs and projects to assess and adjust program objectives in order to confirm their validity, the objectives do remain valid in terms of addressing target group needs and priorities.	The evaluation reports that the MO supported programs and projects subject to evaluation have carried out a systematic review of the continued validity of program objectives and have either confirmed validity or made appropriate adjustments to the objectives.
  1.5 Activities and outputs are consistent with program goal and with objectives achievement	The evaluation report finds that there are serious deficiencies in the causal link between the activities and outputs of MO supported projects and programs and their objectives. This can occur either because the linkages are weak or non-existent or because the scale of activities and outputs is not matched to the scale of the objectives to be achieved. Note: the evaluation should recognize that not all project and program inputs will be provided by the MO in joint and country-led projects and programs.	The evaluation report is not able to verify that the design of MO supported programs and projects includes a systematic assessment of causal linkages between program activities and outputs and objectives achievement. Nonetheless, there is no indication that these links do not exist in the program as implemented.	The evaluation report notes that the activities and outputs of MO supported programs and projects are clearly linked to a causal process that should logically contribute significantly to the achievement of stated objectives. However, the scale of the activities and outputs is either not described or is inconsistent with the contribution to achieving the stated objectives.	The evaluation report notes that the activities and outputs of MO supported programs and projects are clearly linked to a causal process that should logically contribute to the achievement of stated objectives. Further, the scale of the activities and outputs is consistent with the expected contribution to achieving the objectives as stated or the MO makes a significant contribution to overall strategy in the sector.

  2. Achievement of Development Objectives and Expected Results	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  2.1 Multilateral organization supported programs and projects achieve their stated objectives and attain expected results.	Evaluation finds that one or more very important output and outcome level objective(s) has not been achieved.	Evaluation finds that half or less than half of stated output and outcome level objectives are achieved.	Evaluation finds that multilateral organization supported programs and projects either achieve at least a majority of stated output and outcome objectives (more than 50% if stated) or that the most important of stated output and outcome objectives are achieved.	Evaluation finds that multilateral organization supported programs and projects achieve all or almost all significant development objectives at the output and outcome level.
  2.2 Multilateral organization supported programs and projects have resulted in positive changes for target group members.	Evaluation finds that problems in the design or delivery of multilateral organization supported activities mean that expected positive impacts have not occurred or are unlikely to occur.	Evaluation finds that it is not possible to make a credible assessment of program impacts because the program design did not specify intended impacts. If credible data is available and the design specifies impacts but sufficient time has not passed for expected impacts to emerge, this should be coded not addressed.	Evaluation finds that multilateral organization supported projects and programs have resulted in positive changes experienced by target group members (at the individual, household or community level).	Evaluation finds that multilateral organization supported projects and programs have resulted in widespread and significant positive changes experienced by target group members as measured using either quantitative or qualitative methods (possibly including comparison of impacts with non-program participants).
  2.3 Multilateral organization programs and projects made differences for a substantial number of beneficiaries.	Evaluation finds that multilateral organization supported projects and programs have not contributed to positive changes in the lives of beneficiaries as measured quantitatively or qualitatively.	Evaluation finds that multilateral organization supported projects and programs have contributed to positive changes in the lives of only a small number of beneficiaries (when compared to project or program targets and goals if established).	Evaluation finds that multilateral organization supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries as measured quantitatively or qualitatively.	Evaluation finds that multilateral organization supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries and accounting for most members of the target group as measured quantitatively or qualitatively.

  3. Sustainability	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  3.1 Benefits continuing or likely to continue after project or program completion	Evaluation finds that there is a very low probability that the program/project will result in continued intended benefits for the target group after project completion.	Evaluation finds that there is a low probability that the program/project will result in continued benefits for the target group after completion.	Evaluation finds it is likely that the program or project will result in continued benefits for the target group after completion.	Evaluation finds that it is highly likely that the program or project will result in continued benefits for the target group after completion.
  3.2 Extent multilateral organization supported projects and programs are reported as sustainable in terms of institutional and/or community capacity	Evaluation finds that the design of multilateral organization supported programs and projects failed to address the need to strengthen institutional and/or community capacity as required.	Evaluation finds that multilateral organization programs and projects may have failed to contributed to strengthening institutional and/or community capacity.	Evaluation finds that multilateral organization programs and projects may have contributed to strengthening institutional and/or community capacity but with limited success	Evaluation finds that either multilateral organization programs and projects have contributed to significantly strengthen institutional and/or community capacity as required or institutional partners and communities already had the required capacity to sustain program outcomes.

  4. Efficiency	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  4.1 Program activities are evaluated as cost efficient	Evaluation finds that there is credible information indicating that multilateral organization supported programs and projects are not cost efficient.	Evaluation finds that the multilateral organization supported programs and projects under evaluation do not have credible, reliable information on the costs of activities and inputs and therefore the evaluation is not able to report on cost efficiency.	Evaluation finds that the level of program outputs achieved when compared to the cost of program activities and inputs is appropriate even when the program design process did not directly consider alternative program delivery methods and their associated costs.	Evaluation finds that multilateral organization supported programs and projects are designed to include activities and inputs that produce outputs in the most cost efficient manner available at the time.
  4.2 Evaluation indicates implementation and objectives achieved on time	Evaluation finds that less than half of stated output and outcome level objectives of multilateral organization supported programs and projects are achieved on time and there is no credible plan found by the evaluation which would suggest significant improvement in on-time objectives achievement in the future.	Evaluation finds that less than half of stated output and outcome level objectives of multilateral organization supported programs and projects are achieved on time but the program or project design has been adjusted to take account of difficulties encountered and can be expected to improve the pace of objectives achievement in the future.	Evaluation finds that more than half of stated output and outcome level objectives of multilateral organization supported programs and projects are achieved on time and that this level is appropriate to the context faced by the program during implementation.	Evaluation finds that nearly all stated output and outcome level objectives of multilateral organization supported programs and projects are achieved on time.

  5. Crosscutting Themes: Gender Equality and Environmental Sustainability	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  5.1 Extent multilateral organization supported activities effectively address the Crosscutting issue of gender equality.	Evaluation finds multilateral organization supported activities are unlikely to contribute to gender equity or may in fact lead to increases in gender inequities.	Evaluation finds that multilateral organization supported activities either lack gender equality objectives or achieve less than half of their stated gender equality objectives at the outcome level. (Note: where a program or activity is clearly gender focused (maternal health programming for example) achievement of more than half its stated objectives warrants a satisfactory rating).	Evaluation finds that multilateral organization supported programs and projects achieve a majority (more than 50%) of their stated gender equality objectives at the outcome level.	Evaluation finds that multilateral organization supported programs and projects achieve all or nearly all of their stated gender equality objectives at the outcome level.
  5.2 Extent changes are environmentally sustainable.	Evaluation finds that multilateral organization supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. In addition the evaluation reports that changes resulting from multilateral organization supported programs and projects are not environmentally sustainable.	Evaluation finds that multilateral organization supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. There is, however, no direct indication that project or program results are not environmentally sustainable.	Evaluation finds that multilateral organization supported programs and projects include some planned activities and project design criteria to ensure environmental sustainability. These activities are implemented successfully and the evaluation reports that the results are environmentally sustainable	Evaluation finds that multilateral organization supported programs and projects are specifically designed to be environmentally sustainable and include substantial planned activities and project design criteria to ensure environmental sustainability. These plans are implemented successfully and the evaluation reports that the results are environmentally sustainable.

  Part Two: Use of Evaluation and Monitoring

  6. Using Evaluation and Monitoring to Improve Development Effectiveness	(1) Highly Unsatisfactory	(2) Unsatisfactory	(3) Satisfactory	(4) Highly Satisfactory
  6.1 Systems and process for evaluation effective	Evaluation finds that evaluation practices in use for programs and projects of this type are seriously deficient.	Evaluation finds no indication that the program is subject to systematic and regular evaluations.	Evaluation finds that program being evaluated is subject to systematic and regular evaluations or describes significant elements of such practice. No mention of policy and practice regarding similar programs and projects.	Evaluation finds that program being evaluated (along with similar programs and projects are subject to systematic regular evaluations or describes significant elements of such practice.
  6.2 Systems and processes for monitoring and reporting on program results are effective	Evaluation finds an absence of monitoring and reporting systems for the program.	Evaluation finds that while monitoring and reporting systems for the program exist, they either do not report on a regular basis or they are inadequate in frequency, coverage or reliability.	Evaluation finds that monitoring and reporting systems for the program are well established and report regularly.	Evaluation finds that monitoring and reporting systems for the program are well established and report regularly. The quality of regular reports is rated highly by the evaluation and results are reportedly used in the management of the program.
  6.3 Results–based management systems effective	Evaluation finds that there is no evidence of the existence of an results-based management system for the program and no system is being developed.	Evaluation finds that while an results-based management system is in place, or being developed, it is unreliable and does not produce regular reports on program performance.	Evaluation finds that results-based management system is in place and produces regular reports on program performance.	Evaluation finds that results-based management system is in place for the program and there is evidence noted in the evaluation that the system is used to make changes in the program to improve effectiveness.
  6.4 Multilateral organization makes use of evaluation to improve development effectiveness	Evaluation report does not include a management response and does not have one appended to it or associated with it. There is no indication of how the evaluation results will be used. There is no indication that similar evaluations have been used to improve effectiveness in the past.	Evaluation report includes a management response (or has one attached or associated with it) but it does not indicate which recommendations have been accepted: or there is some, non-specific indication that similar evaluations have been used to improve program effectiveness in the past.	Evaluation reports includes a management response (or has one attached or associated with it) that indicates which recommendations have been accepted. Or there is a clear indication that similar evaluations in the past have been used to make clearly identified improvements in program effectiveness.	Evaluation reports includes a management response (or has one attached or associated with it) describes a response to each major recommendation which is appropriate and likely to result in the organizational and programmatic changes needed to achieve their intent.

  Annex 7: CIDA Funding to Multilateral Development Organizations^{Footnote 60}

  Long-term Institutional Funding

  Long-term institutional funding refers to an un-earmarked funding to a Multilateral Organization in support of that organization’s mandate. According to the OECD there are currently 170 multilateral organizations active in the development agenda and are eligible to receive aid funding. As of 2010-2011, CIDA provided long-term institutional funding to 30 of these multilateral organizations. CIDA’s funding was highly concentrated, with 9 multilateral organizations^{Footnote 61} receiving 80% of its total long-term institutional funding from 2007-2008 to 2010-2011.

  Funding to Specific Multilateral and Global Initiatives

  Specific multilateral and global funding can be defined as funding to multilateral organizations in support of a key program or activity usually in a specific thematic area and often global in scope. Within this category there are two sub-types: 1) humanitarian assistance; and 2) other global initiatives programming.

  Humanitarian assistance is provided on a needs basis and usually in response to specific appeals issued by multilateral organizations with expertise in providing humanitarian assistance. The main multilateral organizations involved in providing humanitarian assistance are the World Food Programme (WFP), the United Nations High Commissioner for Refugees (UNHCR), the International Committee of the Red Cross (ICRC) and the UN Office for Coordination of Humanitarian Affairs (OCHA). The United Nations Children’s Fund (UNICEF), although not primarily a humanitarian organization, also delivers humanitarian assistance with a specific emphasis on the needs of children.

  The second sub-type of specific multilateral and global funding involves global initiatives in other sectors. These initiatives are in sectors that deal with issues which transcend borders and thus lend themselves to a multilateral approach. The main sectors CIDA supports with this type of funding are health, environment and economic growth. The health sector is the most important of these, especially in light of the challenges of infectious diseases like AIDS and tuberculosis, which do not respect international borders. Bilateral programming in a single country is unlikely to succeed in meeting the challenges of infectious diseases in the absence of regional and global programs.

  Funding to Multilateral Initiatives Delivered by other CIDA Branches

  Multilateral initiatives can also receive funding from other CIDA branches, mostly through multi-bi funding from Geographic Programs Branch. Multi-bi funding refers to earmarked funding to a specific multilateral organization initiative by a CIDA geographic program to support a specific activity in a specific country or group of countries. It is considered “bilateral” assistance because it is funded through CIDA’s Geographic Programs Branch in the context of the program’s country strategies or programming frameworks.

  Multi-bi funding accounts for a large and growing share of CIDA resources. It more than tripled in the five years from 2002-2003 to 2007-2008, mainly as a result of substantial funding to programs in fragile states. By 2007-2008, CIDA multi-bi funding had reached C$691 million, with 53% spent in fragile states including 37% of all multi-bi funding spent in Afghanistan.

  In fragile states, where United Nations (UN) organizations and the World Bank are often assigned specific roles by member governments, use of multi-bi funding by CIDA can sometimes help the Agency limit fiduciary risk, which results in a reduced administrative burden on the developing member countries very weak national institutions. The use of this type of funding is also consistent with Canada’s commitment to the 2005 Paris Declaration on Aid Effectiveness principles, which includes a call for donors to harmonize their aid and use program based approaches where they can be effective.

  It is important to note that CIDA’s Geographic Programs Branch manage multi-bi funding according to the same basic processes that govern all of the Agency’s geographic programming. For example, CIDA’s Geographic Programs are responsible for monitoring and reporting on the effectiveness of funds used in this way (with coordination and oversight by the Multilateral and Global Programs Branch). Country program evaluations that examine CIDA’s bilateral programs in a given country include in their remit programming delivered by multilateral organizations and supported by multi-bi funding.

  Annex 8: Management Response for the Asian Development Bank Effectiveness Review, 2012

  The Development Effectiveness Review of the Asian Development Bank, prepared by CIDA’s Evaluation Division, provides a clear picture of the strengths of the Bank and outlines challenges and areas for improvement going forward. As of 2011, the Bank has over US$21.7 billion in approved financing and more than 2,900 employees from 59 countries.

  Canada is a founding member of the Asian Development Bank, and is the sixth largest shareholder (5.25%). Canada supported the 2009 General Capital Increase that resulted in a 200% increase in capital for the Bank’s ordinary capital resources. Canada’s financial support to the Bank consists of: (i) core funding of the Bank’s capital; (ii) voluntary support to specific trust funds; and, (ii) voluntary support to the Asian Development Fund – the Bank’s concessionary funding window. The Review, conducted principally through a meta-synthesis of the Asian Development Bank’s Independent Evaluation Office evaluations between 2006 and 2010, identified findings relating to six main criteria for assessing development effectiveness. These criteria include: 1) Relevance of interventions; 2) Achievement of development objectives and expected results; 3) Sustainability of results and benefits; 4) Efficiency; 5) Crosscutting themes (environmental sustainability and gender equality); and, 6) Using evaluation and monitoring to improve development effectiveness.

  The overarching finding of the Independent Evaluation Department evaluations indicates that Bank programming is relevant to the needs of target group members and well aligned with the development goals of its national partners. The Independent Evaluation Department evaluations also reflect positive results in the achievement of objectives and expected development results in over two-thirds of evaluation reports. The sustainability of program benefits remains an important challenge for the Bank and its partners. Reported results in the area of efficiency indicate an important challenge particularly regarding the timeliness of program implementation. Furthermore, broader evaluations often do not address gender equality and it was noted in the Development Effectiveness Review that greater attention is required for gender analysis skills amongst the Bank’s evaluation teams. However, those Asian Development Bank/Independent Evaluation Department evaluations that do address gender equality do indicate that Bank programs have been effective in providing support to acknowledging and addressing gender equality. Evaluations also report that most Bank supported programs have generally been effective in addressing environmental sustainability, although improvements are needed in ensuring that Bank supported projects include effective measures to address environmental challenges.

  Finally, systems for program evaluation are effective and are well used but there is a continuing need to strengthen monitoring, results reporting and results-based management at both the national and local level.

  CIDA accepts all five of the Development Effectiveness Review’s recommendations for improving the Asian Development Bank’s development effectiveness, while noting that there have been some improvements since the review period. For example, the Bank has improved the quality of indicators in its design and monitoring frameworks, and is in the process of integrating additional sex-disaggregated indicators in its corporate Results Framework. The Asian Development Bank continues to be a key partner for CIDA in helping to achieve poverty reduction in the Asia-Pacific region. CIDA’s interventions to address the recommendations will focus on the following two-pronged approach: 1) Write a letter to the proper authorities within the Bank to address the issues and recommendations outlined in the evaluation; and, 2) Ask the Canadian Executive Director at the Asian Development Bank to engage with appropriate authorities within the Bank to address the specific recommendations and to strive to build a coalition around these recommendations with other like-minded donors at the Asian Development Bank.

  Recommendations	Commitments/measures	Responsible	Completion date	Status and Comments
  1. CIDA should emphasize the need to ensure that gender equality is directly addressed in future ADB (and Independent Evaluation Department) evaluations. Evaluation policies at both the central and local office-level should pay adequate attention to gender equality as a key evaluation issue. This may also require greater attention to gender analysis skills among evaluation teams.	Agreed. 1.1. As confirmed by the findings of the Draft Gender Equality Institutional Assessment (conducted in 2008 and 2011), Canada has played a leadership role in promoting gender mainstreaming and the achievement of gender equality results at the Asian Development Bank. 1.2. CIDA will write a letter to the proper authorities within the Bank to outline the results of this review. 1.3. CIDA will ask the Canadian Executive Director at the Asian Development Bank to raise these issues with key personnel at the Bank and to reach out to other like-minded donors to build a coalition around these issues.	CIDA Multilateral and Global Programs Branch (Multilateral Development Institutions Directorate)	1.1.Completed (2011) 1.2. March 2013 1.3. March 2013	Please reference EDRMS #: 5798585
  2. CIDA should engage with the ADB to ensure that the environmental sustainability of infrastructure and other assets financed by the Bank receives sufficient attention, and that results in this area are improved over time.	Agreed. 2.1. Actions referenced in Section 1.2. and Section 1.3., will also outline requests related to the issue of ensuring that the environmental sustainability of infrastructure and other assets financed by the Bank receives sufficient attention.	CIDA Multilateral and Global Programs Branch(Multilateral Development Institutions Directorate)	2.1. March 2013
  3. CIDA should raise the issue of the sustainability of the benefits of ADB investments to a strategic level in its engagement with the Bank. Two areas of promise are greater investment in operation and maintenance of the assets created (as identified in the ADB’s own annual assessment of development effectiveness) and improved planning and execution of capacity development components of programs.	Agreed. 3.1. Actions referenced in Section 1.2. and Section 1.3., will also emphasize the need for attention to the sustainability of the benefits of Asian Development Bank investments through greater investment in operation and maintenance and improved planning and execution of capacity development components of programs.	CIDA Multilateral and Global Programs Branch(Multilateral Development Institutions Directorate)	3.1. March 2013
  4. CIDA should make improving the timeliness of ADB operations a priority area for CIDA’s interactions with the ADB. According to the evaluations reviewed, timelines could be improved through changes in systems and procedures used during each phase of the project life cycle. This represents one of the most readily apparent ways to improve the efficiency of ADB operations.	Agreed. 4.1. CIDA, along with other donors, urged the Bank at the recent Asian Development Fund (ADF XI) replenishment meetings to increase efficiency in project implementation. The Asian Development Fund (ADF XI) negotiations concluded as of April 2012. 4.2. Actions referenced in Section 1.2. and Section 1.3., will also outline the necessity for the Bank to improve the timeliness and efficiency of Asian Development Bank operations.	CIDA Multilateral and Global Programs Branch(Multilateral Development Institutions Directorate)	4.1. Completed 4.2. March 2013	Please reference: http://www.adb.org/documents/adf-xi-donors-report-empowering-asias-most-vulnerable-0
  5. CIDA should emphasize the need to strengthen systems for program results monitoring and results-based management at the local level, including improving the quality and coverage of indicators and the use of associated targets to track results. This will require investments in capacity development for ADB staff and partners at the country office level and more locally (in the case of partners).	Agreed. 5.1. To maximize results, efficiency and impact, Asian Development Bank decided to be more selective and focused in its operations under Strategy 2020, based on the needs of its developing member countries and Asian Development Bank’s comparative strengths and to ensure the consistency of efforts with other development partners. The Bank has also taken a number of steps to improve the quality of country partnership strategies and programs, which further strengthens systems for program results monitoring and results-based management. 5.2. CIDA will emphasize the need to strengthen systems for program results monitoring and results-based management at all levels (including at the local level), in its upcoming comments on the Asian Development Bank Results Framework, which is in the process of being reviewed by the Bank.	CIDA Multilateral and Global Programs Branch(Multilateral Development Institutions Directorate)	5.1. Completed 5.2. February 2013	Please reference: http://www.adb.org/documents/strategy-2020-working-asia-and-pacific-free-poverty