Archived information

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

A Review of African Development Bank (AfDB) Program Evaluation Reporting - 2007-2012 - Report

July 2013

Table of Contents

List of Acronyms and Abbreviations

ADB
Asian Development Bank
ADF
African Development Fund
AfDB
African Development Bank
CIDA
Canadian International Development Agency
CIF
Climate Investment Funds
DAC-EVALNET
Development Assistance Committee Network on Development Evaluation
DFATD
Department of Foreign Affairs, Trade and Development
MDGs
Millennium Development Goals
n.d.
No date provided
OECD/DAC
Development Assistance Committee of the Organization for Economic Development and Cooperation
UA
Unit of Account
UNDP
United Nations Development Programme
WHO
World Health Organization
WFP
World Food Programme

Acknowledgments

The Development Evaluation Division wishes to thank all who have contributed to this review exercise for their valued input, their constant and generous support, and their patience. Our thanks go first to the independent team of the consulting firm Universalia Management Group, made up of Principal Evaluator Marie-Hélène Adrien, Senior Evaluator Charles Lusthaus and Consultants Halcyon Louis, Elisabetta Micaro and Emma Mason.

The Development Evaluation Division would also like to thank Vaughn Lantz, Vincent Raiche and Mariette Maillet from the Multilateral and Global Programs Branch for their valuable support. Our thanks also go to Mohamed Manai and Rakesh Nangia, representatives of the AfDB in Tunis, for their useful and practical advice to the evaluators.

From the Development Evaluation Division, we wish to thank Nancy Odeh, Evaluation Officer, for her significant contribution to refining report drafts. We also thank Andres Velez-Guerra, Team Leader, for contributing to report revisions and guiding this review to completion, and James Melanson, Evaluation Director, for overseeing the review.

Caroline Leclerc
Head of Development Evaluation

Executive Summary

Background

The AfDB was founded by signatory agreement between regional member states in 1963. It is one of three entities that together comprise the African Development Bank Group, the other two being the African Development Fund and the Nigeria Trust Fund.

As a regional multilateral development finance institution, the mission of the AfDB is to contribute to the sustainable economic development and social progress of African countries. Membership of the AfDB (as of December 2011) comprises 77 countries, of which 54 (70 percent) are regional members. The AfDB provides development assistance to its regional member countries through the public and private sectors, in the form of financial instruments, policy advice, and technical assistance. In 2011, the AfDB financed 184 operations in regional member countries at a cost of UA 5.72 billion.

Purpose and Objectives

The purpose of the review was to provide an independent, evidence-based assessment of the AfDB’s development effectiveness for use by the Government of Canada, the AfDB and other interested shareholders. However, a preliminary assessment of the AfDB’s program evaluation reporting for the period 2007- 2012 noted shortcomings in coverage and quality.

This report therefore provides forward-looking recommendations to:

  1. Support the strengthening of the AfDB’s evaluation and reporting on its development effectiveness; and
  2. Guide future engagement between DFATD’sFootnote 1 Multilateral and Global Programs Branch, and the AfDB on evaluation and development effectiveness reporting.

Methodology

A generic approach and methodology for the review of development effectiveness of multilateral organisations was developed under the guidance of the OECD-DAC Network on Development Evaluation (DAC-EVALNET). In line with this methodology, the review of the AfDB involved three distinct steps:

Following the implementation of the above-mentioned process, it was determined that the AfDB’s program evaluation reporting is not adequate and the multilateral organisation’s evaluation function does not provide an adequate information base on overall development effectiveness. Therefore, this report presents recommendations aimed at supporting improved evaluation and development effectiveness reporting, as suggested by the OECD-DAC evaluation methodology.

Key Findings of the Preliminary Review

The AfDB evaluation function has not produced an adequate number of program-level evaluations and development effectiveness reports over the period 2007-2012.

Out of the 43 documents provided by the AfDB for the formulation of the evaluation universe, only 20 documents met all the DAC-EVALNET methodology preliminary review criteria. The remaining 23 reports were not screened in during the preliminary review for three main reasons.

First, the document’s scope was outside the selected review timeframe (6 documents). Second, while fulfilling other objectives, the document in question was not a program-level evaluation or a development effectiveness report (13 documents). Third, the document received a low quality score (4 reports) for reasons that included: the lack of a clear description of the evaluation objectives, or limitations; or a poor evaluation design.

Program-level evaluations and development effectiveness reports within the review period (2007-2012) provided inadequate coverage of the results of AfDB programming in its priority areas and its strategic objectives.

The 20 documents that met all the preliminary review criteria, together addressed all but two of the AfDB’s priority areas, with one report taking a multisector focus. Importantly, however, these documents provided inadequate coverage of the results achieved. The preliminary review has noted that AfDB program-level evaluation reporting on its priority areas and strategic objectives is largely descriptive, and focuses on AfDB programming activities without covering AfDB development results. This implies a need for the AfDB to revisit its approach to program-level evaluation, with a focus on its reporting on development results.

AfDB program-level evaluations and development effectiveness reports provide insufficient coverage of AfDB investment in its crosscutting thematic priorities.

Of the 20 reports that met all the preliminary review criteria, only five reports addressed the AfDB crosscutting thematic priorities. One report included discussion of the gender theme, and three reports addressed the environment. An additional report addressed climate change. This finding indicates insufficient coverage by the AfDB’s program-level evaluations and development effectiveness reports of investment made in priority crosscutting themes.

The usefulness of AfDB development effectiveness reporting to decision-making on the Bank’s forward-looking investments is sometimes impeded by the time lag between the year an evaluation is conducted and the year it is finalized and published.

All 43 documents included in the preliminary review universe were published during the established timeframe of January 2007 to September 2012. However, 8 reports had a significant lag between their publication and the years they reviewed. For example, a report published in 2007 evaluated programming during 1996-2004, thus implying a three-year lag between implementation and publication. The lag between implementation and finalization of AfDB evaluations reduced their usefulness in informing future programming.

Some evaluation designs in the program-level evaluation reports reviewed did not follow the DAC-EVALNET quality guidelines. This is a potential area for improvement.

The preliminary review awarded points to each report using the quality screening criteria of the DAC-EVALNET (Evaluation Coverage and Quality Screening Scoring Guide) to determine evidence of two aspects of quality: the existence of basic evaluation report characteristics; and the existence of key components required for a review of development effectiveness.

The preliminary review findings indicated that all 20 program-level evaluation reports surpassed the threshold required for basic evaluation report characteristics. However, preliminary review findings also indicate that the reviewed program-level evaluations received low scores on the criterion of appropriate evaluation design. These scores echo the 2012 self-assessment of the AfDB’s independent evaluation function, which stated that the AfDB employed a narrow range of evaluation methodologies in its project-level evaluations.

The preliminary review findings indicated that all 20 program-level evaluation reports surpassed the threshold required for basic evaluation report characteristics. However, preliminary review findings also indicate that the reviewed program-level evaluations received low scores on the criterion of appropriate evaluation design. These scores echo the 2012 self-assessment of the AfDB’s independent evaluation function, which stated that the AfDB employed a narrow range of evaluation methodologies in its project-level evaluations.

Conclusions

The findings of the preliminary review of the AfDB program-level evaluations indicate that the evaluations included in the review do not provide an adequate level of coverage of the AfDB’s program-level development results.

At the same time, it should be noted that the AfDB has demonstrated a clear commitment to addressing the challenges mentioned above. In particular, the AfDB has indicated their intention to: ‘raise the bar on quality’, ensure methodological rigour, conduct more program level evaluations, focus on timing/usefulness and strategic priorities (including crosscutting issues), and improve the quality assurance process. These intentions arise from both the 2012 self-assessment of the AfDB’s independent evaluation function, and the AfDB 2013-2017 independent evaluation strategy.

Recommendations

Strengthening its independent evaluation function should be a priority for the AfDB.

A stronger evaluation function would enable assessment of progress towards the achievement of program-level development outcomes and the capture of lessons learned to improve its investments in regional member countries. The areas of AfDB evaluation activity in which the review findings have noted particular weakness were the design of evaluations and the coverage of program-level development effectiveness results. The anticipated results of strengthening the AfDB evaluation function would be increased robustness in evaluation design through the use of appropriate methodologies, and more comprehensive reporting on development effectiveness.

The AfDB should consider engaging in a peer review of its evaluation function.

As it moves toward improving its evaluation and development effectiveness reporting, the AfDB could add value to its earlier self-assessment by engaging a peer review. This could be used by the AfDB to gain further insight into the capacity and quality of its evaluation function, and to assess the relevance, coverage, and overall quality of its development effectiveness reporting. The results of this assessment could help the AfDB increase the extent to which its program evaluations and development effectiveness reports inform AfDB strategic decision-making. Importantly, the AfDB has indicated that it plans to engage in a peer review in 2014 once the independent evaluation strategy is operational.

Canada, in concert with other donors and AfDB Board members, should support the Bank’s efforts to strengthen its evaluation function.

Canada should support the AfDB’s efforts to strengthen its evaluation function and the design of its evaluations, as outlined above. One possibility, among others, would be an annual analysis of the AfDB’s evaluation coverage to be included as part of the AfDB’s evaluation work plan. Specifically, the AfDB’s rolling work plan could provide some measure of the target for, and actual implementation of, program-level evaluations, perhaps as a percentage of total expenditure. In working with other Board members and donors, Canada should signal its readiness to support the strengthening of the AfDB’s evaluation function. Important elements would include peer review; capacity development to strengthen evaluation coverage, design, and timeliness; and strengthened program-level development effectiveness reporting.

1.0 Background

1.1 Introduction

The overall approach and methodology for the preliminary review was developed by a multidonor task group under the guidance of the OECD-DAC Network on Development Evaluation (DAC-EVALNET). In 2010, the DAC-EVALNET methodological approach was piloted on reviews of the development effectiveness of the Asian Development Bank (ADB), and the World Health Organization (WHO). In 2011-2012, the assessment methodology was applied to full reviews of the development effectiveness of the United Nations Development Programme (UNDP), and the World Food Programme (WFP).

1.2 Why Conduct this Review?

The purpose of the review was to provide an independent, evidence-based assessment of the AfDB’s development effectiveness for use by the Government of Canada, the AfDB and other interested shareholders. However, a preliminary assessment of the AfDB’s program evaluation reporting for the period 2007-2012 noted shortcomings in coverage and quality. This report therefore provides forward-looking recommendations to:

  1. support the strengthening of the AfDB’s evaluation and reporting on its development effectiveness; and
  2. guide future engagement between DFATD’s Multilateral and Global Programs Branch and the AfDB on evaluation and development effectiveness reporting.

The African Development Fund provides concessional loans and grants to 41 low-income African countries, and technical assistance for studies and capacity-building activities.Footnote 3 Discussions during the midterm review of ADF-12 focused on AfDB development effectiveness, as well as on its operational priorities, institutional effectiveness, resource allocation, and the financing capacity and sustainability of the African Development Fund. This report will therefore build on the findings of the midterm review, and may inform potential discussion of the Bank’s evaluation function.

1.3 The African Development Bank

Founded by signatory agreement between member states in 1963, the African Development Bank is one of three entities that together comprise the African Development Bank Group (AfDB), the other two being the African Development Fund and the Nigeria Trust Fund. As a regional multilateral development finance institution, the mission of the AfDB is to contribute to the sustainable economic development and social progress of African countries. Membership of the AfDB comprises 77 countries, of which 53 (69 percent) are regional member countries. The remaining 25 countries (31 percent) are non-African/ non-regional member countries. The supreme governing body of the AfDB is the Board of Governors, the members of which are ministers and high-level officials from economic and financial institutions of member countries.

The AfDB provides development assistance to its regional member countries through the public and private sectors, in the form of financial instruments, policy advice and technical assistance. The main financial instruments used in AfDB interventions in its regional member countries are:

Contributions to enable the delivery of development assistance are provided by each of the three AfDB entities:

In 2011, the AfDB financed 184 operations in regional member countries at a cost of UA 5.72 billion. Table 1 provides the percentage breakdown by type of financial instrument. Figure 1 provides an overview of financial allocations to regional member countries in 2011, by area of strategic focus.

Table 1: AfDB Financing in Regional Member Countries, 2011Footnote 4
Type of FinancingNumber of Operations*2011 Financial Amount(UA)*
Note: Figures in parentheses represent percentages of total.
Loans60 operations (33)3.55 billion (62)
Grants75 operations (41)578.7 million (10)
Heavily Indebted Poor Countries Initiative7 transactions (4)1.35 billion (24)
Equity Participations7 investments (4)53.4 million (1)
Special Funds35 operations (19)188.1 million (3)
Total184 operations5.72 billion

Figure 1: AfDB Sector Financing to Regional Member Countries, 2011

AfDB Sector Financing to Regional Member Countries, 2011

1.4 Strategic Objectives and Priority Areas of the AfDB

The 2008-2012 Medium–Term Strategy of the AfDB outlined the Bank Strategic Framework, comprising five strategic objectives and related priority areas for AfDB activities over this period. The strategic objectives identified in the 2008-2012 Medium–Term Strategy are presented below:

Under the strategic objective of Operational Selectivity, the AfDB committed itself to vertical and horizontal areas of focus, as well as crosscutting priorities and selected areas of intervention. The AfDB prioritized a vertical focus on four key areas that are distinct to the AfDB’s sectors and topics: Infrastructure; Governance; Higher Education and Vocational Training; and the Private SectorFootnote 5. This was supported by a horizontal emphasis on: Regional Integration; Fragile States; and Client-responsive Engagement, with a focus on middle-income countries. In the interest of mainstreaming crosscutting themes, the thematic focusesfor the AfDB were: gender; the environment; and climate change. The AfDB further indicated that it would provide selected support to agricultural development.

In April 2013, the Bank adopted a new Ten Year Strategy (TYS) for the period 2013-2022Footnote 6. It will focus on two objectives: inclusive growth, and the transition to green growth. The TYS reaffirms the Bank’s strategic focus on five core operational areas:

In implementing its ten-year strategy, and as an integral part of the two objectives, the Bank will pay particular attention to fragile states, agriculture and food security, and gender.

1.5 Self-Assessment of the AfDB’s Independent Evaluation Function

In its capacity as the premier development-finance institution in Africa, the AfDB is constantly engaged in institutional reform to maximize its development effectiveness, as well as to increase the quality of its operations. To this end, following the commissioning of a self-assessment of its independent evaluation function in 2011, the AfDB revised its independent evaluation policy and developed a new, independent evaluation strategy.

The 2012 self-assessment of the AfDB’s evaluation function was conducted five years after the implementation of the independent evaluation policy. The primary objective of the self-assessment was to identify the strengths, as well as the limitations, of the AfDB evaluation function, and to recommend improvements. Importantly, the 2012 self-assessment preceded the current preliminary review. It is illustrative of an ongoing process of assessment and reform at the AfDB to improve on institutional, as well as development effectiveness. The findings of the current preliminary review are informed by and build upon the key findings of the self-assessment.

Findings of the 2012 self-assessment identified general support across the institution for the independent evaluation function, in light of the perceived value-added of the evaluation activity of the AfDB Operations Evaluation Department. There was, however, an equal perception of the need to realign the approach to evaluation with the strategic direction of the AfDB in order to determine what has worked well, and identify areas for improvement. It will be important for the Bank to generate evaluations with increased relevance to AfDB strategic decision-making processes and effectiveness, through the application of evaluation findings and lessons.

The findings of the self-assessment have informed the 2013-2017 strategy for independent evaluation. The revised policy and strategy provide the basis for the AfDB’s rolling work program for 2013-2015, including resource allocation, within its Operations Evaluation Department.

According to the Bank, "strengthening the evaluation function is at the heart of the 2013-17 Independent Evaluation Strategy… The (revised) strategy highlights a diverse range of actions designed to strengthen the quality of evaluations including preparation of an evaluation manual, establishing quality standards and a clearer quality assurance process. These are being implemented during the course of 2013 and beyond. It also proposes more explicit coverage of the themes outlined in the Bank’s Ten Year Strategy and has increased evaluations at country, sector and thematic level."Footnote 7

2.0 Methodology

This section provides an overview of the methodology used to conduct this preliminary review, which involved three steps:

The following section provides specific details on the implementation of the above-mentioned approach.

2.1 Rationale

The DAC-EVALNET methodology was designed to address a gap in information about the development effectiveness of multilateral organizations. It is modest in terms of time and resource requirements, and therefore minimizes the burden on multilateral organizations for its execution. The DAC-EVALNET methodological approach involved a review of program evaluations and development effectiveness reports produced by the AfDB.

The first stage of the review involved formulating the evaluation universe by identifying all program-level reports on AfDB development effectiveness and evaluations, which were produced between 2007 and 2012. Although the DAC-EVALNET approach does not specify a timeframe for the eligibility of reports for review, all reviews conducted to date have focused on reports published during the five-year period that preceded each review.

The evaluation universe was formulated in consultation with the AfDB to ensure that it included all program-level evaluations and development effectiveness reports produced by the AfDB over the period January 2007 – September 2012. Once the evaluation universe was agreed upon individual reports were reviewed using the criteria specified in the DAC-EVALNET methodology. Reports were screened in or out depending on the extent to which they conformed to these criteria. The overall result of this preliminary review then suggested which of the three scenarios outlined below was most appropriate for the AfDBFootnote 12:

Scenario A – Reporting on development effectiveness is adequate (No Further Action, Option 1).

If the preliminary review indicates that the multilateral organisation’s reporting on the development effectiveness criteria is rigorous, evidence-based and covers a significant proportion of the multilateral organisation’s investments, then the reporting may be relied on, especially if based on evaluation data. This indicates the implementation of Option 1 – Rely on the multilateral organisation reporting systems.

Scenario B –Reporting on development effectiveness is not adequate but the multilateral organization’s evaluation function provides an adequate information base on development effectiveness (Conduct a meta-synthesis of the organization’s program-level evaluations, Option 2).

If MOPAN, the Development Assistance Committee, the United Nations Evaluation Group, or the Evaluation Cooperative Group peer review of the evaluation function indicates that the evaluation function is adequate, and the preliminary review confirms that the evaluation function provides a critical mass of credible information on the development effectiveness criteria, then implement Option 2. This option requires conducting a meta-synthesis of program-level evaluations to come to conclusions about development effectiveness.

Scenario C – Multilateral Organization reporting on development effectiveness is not adequate and the multilateral organisation’s evaluation function does not provide an adequate information base on development effectiveness (Support the organization’s development effectiveness reporting, including the monitoring and evaluation function, Option 3).

If MOPAN, the Development Assistance Committee, the United Nations Evaluation Group or the Evaluation Cooperative Group peer review of the evaluation function suggest that development effectiveness reporting may be inadequate, and the preliminary review confirms that the evaluation function does not provide a critical mass of credible information on the development effectiveness criteria, implement Option 3.

Option 3 requires the implementation of actions aimed at strengthening the multilateral organization’s development results reporting. Potential actions include direct support to multilateral organization results reporting and evaluation systems (i.e. peer review of the evaluation function), and/or support to strengthen the multilateral organization’s decentralized evaluation systems.

Following the preliminary review of program-level reports on development effectiveness, and a review of the MOPAN report, the AfDB was categorized under Scenario C, suggesting implementation of Option 3. (Subsections 2.2 to 2.5 explain the rationale and present the criteria used for document selection and review, as well as the limitations of the methodological approach.)

The rationale for suggesting that the AfDB follow scenario C rests on the two criteria outlined in the DAC-OECD methodology. First, the reports included in the universe did not represent adequate coverage of programmatic expected results during the period under review. Second, the MOPAN review found “major gaps” in the implementation of evaluations, although their rating was based on planned evaluations. Both this report and the MOPAN review indicate that insufficient “actual implemented” program-level evaluations were carried out by the AfDB during the period under review.

At the same time, project completion reports and the limited number of project evaluation reports play an important role in the Bank’s management processes. However, they do not provide the breadth of evaluative evidence necessary for a rigorous meta-synthesis of development effectiveness.

2.2 Scope

Document selection for establishing the AfDB evaluation universe was guided by the DAC-EVALNET methodological guidelines for the review, and was done in consultation with the Bank.

Given that the DAC-EVALNET approach is designed for high-level assessment of development effectiveness, project evaluations are not considered because of their limitations in formulating program-level findings for use at an institutional level. Therefore, this review drew on an agreed evaluation universe of 43 reports produced by the AfDB between January 2007 and September 2012 that included the following type of documents:

  1. Country Assistance Evaluations;
  2. Country Sector Reviews and Case StudiesFootnote 13;
  3. Process Reviews; and
  4. Sector, Thematic and Policy Reviews.

Given the small size of this evaluation universe, (compared to that available in reviews of other organisations), it was decided to assess all reports in the universe, rather than selecting a more limited sample.

2.3 Criteria

Preliminary review of the agreed AfDB universe used the four DAC-EVALNET criteria for screening: type of document, scope, coverage, and quality. This was done in two stages:

  1. Determining whether a given document was an evaluation report, as opposed to a policy review or a case study; and whether it fell within the time scope established for the review.
  2. Rating the coverage and quality of evaluation reports using the DAC-EVALNET Evaluation Coverage and Quality Screening Guide (Appendix II).

The four criteria used in the preliminary review are described in Subsections 2.3.1 through 2.3.4. All four criteria were used to conduct the preliminary review.

2.3.1 Type of Document

Program evaluations and development effectiveness reports provide information on development results, and can be used to reflect the development effectiveness of AfDB programming. Documents that did not meet this criterion were screened out at this stage of the preliminary assessment.

2.3.2 Scope

Two scope criteria were used. First, a given evaluation report must be published between January 2007 and September 2012. Second, the programming period covered by a given evaluation report must end after 2005 (i.e. a report covering 1999-2004 and published in 2008 would be screened out). Documents that did not meet these criteria were screened out at this stage of the preliminary assessment.

2.3.3 Coverage

The coverage criterion was subdivided into three areas: financial; AfDB priority areas; and AfDB strategic objectives. This criterion was used to inform the review analysis as opposed to screen documents during the preliminary assessment.

The review of financial coverage was aimed at assessing the percentage of AfDB total expenditure that was covered by particular evaluations. However, this objective could not be achieved given the lack of precision in the financial data provided by several evaluations reviewed.

In reviewing coverage of AfDB priority areas and strategic objectives, the preliminary review aimed to identify AfDB priority areas and/ or strategic objectives that were identified in each document. This information was then used to determine whether there was coverage of the results of AfDB programming in the priority areas and/ or against the strategic objectives.

2.3.4 Document Quality

The preliminary review awarded points to each report using the quality screening criteria of the DAC-EVALNET Evaluation Coverage and Quality Screening Scoring Guide (Annex 2). Each report was required to achieve a minimum of 25 points, out of a maximum of 40 points, based on the statement of essential evaluation characteristics:

Three criteria are particularly relevant to reviews of development effectiveness: use of multiple lines of evidence; the evaluation design; and the development of relevant, evidence-based findings and conclusions. (Criteria G – I of the Evaluation Coverage and Quality Screening Scoring Guide – Appendix II). Documents were required to achieve a minimum of 10 points on these sub-components of quality screening.

2.4 Quality Assurance

The review adhered to established control systems in order to ensure quality, consistency and transparency during the application of screening procedure and criteria. The following controls were followed:

2.5 MOPAN Assessment of the AfDB Evaluation Function

MOPAN indicates that major areas of the AfDB portfolio are not adequately covered by evaluations, most significantly the rapidly growing area of private sector operations. Similarly, the number of Country Assistance Evaluations carried out annually by the Bank may be insufficient. In addition, the Bank has not been able to launch a programme of rigorous, well-resourced impact evaluations.

The AfDB’s Three-Year Rolling Work Program and Budget 2011-2013 states that although the evaluation function at the Bank has been strengthened with additional human resources in recent years, additional staff is required to effectively respond to the Bank’s evaluation needs. In fact, the evaluation function’s Budget and Work Program points to an “evaluation gap”.

Given that the MOPAN assessment of the AfDB suggests quality assurance mechanisms of the evaluation function of the Bank are inadequate, and that this review found that there is not a critical mass of program level evaluations, the AfDB’s evaluation function has been categorized under Scenario C.

2.6 Limitations

There was one key limitation to conducting a development effectiveness review. It pertained to the number of documents that were available to establish the AfDB evaluation universe. In contrast to previous reviews of development effectiveness conducted using this methodology, the AfDB evaluation universe of 43 documents was small.

For the review of the development effectiveness of the UNDP, a sample of 55 documents was selected from an evaluation universe of 199 documents. For the development effectiveness review of the World Food Programme, the evaluation universe comprised 52 documents. In order to mitigate the limitation of the small size of the AfDB’s evaluation universe, the preliminary review involved an assessment of all documents included in the evaluation universe.

This limitation became more acute following the selection and rating of documents using the DAC-EVALNET criteria. This process resulted in the reduction of the evaluation universe by 47 percent. Only 20 documents of the 43 included in the universe were retained following screening based on the criteria of type of document and scope.

However, 12 of the 20 documents that were retained in the final universe focused on policy and finance, and therefore provided partial coverage of the AfDB's priority areas (sectors). In addition, 5 of the 20 documents retained in the final universe provided partial coverage of the AfDB's work, as they were joint evaluations with other development agencies.

It was not possible to mitigate this limitation as documents that were not selected did not meet the DAC-EVALNET standards, notwithstanding their importance for the AfDB in terms of providing information to manage financial risks, maintaining effective financial oversight, and responding to the views of financial rating agencies.

3.0 Results of the Preliminary Review

This section presents the results of the preliminary review of program evaluations and development effectiveness reports published by the AfDB over the period January 2007 – September 2012. The review was based on an evaluation universe of 43 AfDB reports from four reporting categories, as identified in Subsection 2.2.

3.1 Key Findings

The results of the preliminary review of the AfDB’s program evaluations and development effectiveness reports have been summarized under six key findings in Subsections 3.1.1 and 3.1.2.

3.1.1. Coverage

The AfDB evaluation function has not produced an adequate number of program-level evaluations and development effectiveness reports over the period 2007-2012.

Upon implementing the DAC-EVALNET preliminary review criteria to the evaluation universe, only 20 documents out of the aggregate 43 documents comprising the evaluation universe met all the review criteria.

The remaining 23 reports were screened out by the preliminary review for three main reasons. The first pertained to the scope of individual reports. Although all 43 documents in the universe were published during the review timeframe of January 2007 – September 2012, the scope of 6 documents (14 percent of the evaluation universe) was outside the selected review timeframe. These documents focused on a previous reporting period, for example, 1996-2004.

The second reason was that thirteen documents (30 percent of the evaluation universe) were neither program evaluations nor development effectiveness reports.

The third reason pertained to the quality of the document. Four documents (9 percent of the evaluation universe) were screened out for reasons that included the lack of a description of the evaluation’s stated objectives, omission of limitations, and/or a poor evaluation design.

Figure 2 provides an overview of documents that were screened out by the preliminary review, by type of report and/ or the reason for rejection.

Figure 2: Screened Out Documents by Type of Document/ Reason (Actual values, n=23)

Screened Out Documents by Type of Document/ Reason (Actual values, n=23)

Overall, less than half of the documents listed in the evaluation universe met all the DACEVALNET screening criteria. This is illustrated in Figure 3.

Figure 3: Selected versus Screened out Reports (%, n=43)Footnote 15

Selected versus Screened out Reports (%, n=43)

Review findings show that the AfDB’s evaluation function has not produced a critical mass of information on AfDB development effectiveness over the review period of January 2007 – September 2012. Of the 20 documents that met the minimum criteria for the preliminary review, there was an overwhelming focus on policy-based, and financial management evaluations (55 percent of all documents in the evaluation universe).

Very few documents (three) examined the changes observed in regional member countries arising from AfDB financial assistance. Two reports (10 percent of all selected reports) focused on AfDB country assistance, and one report (5 percent of all selected reports) explored AfDB financial support to a given sector (see Figure 4). As a result, the documents that were selected did not provide an adequate representation of progress made by the AfDB in meeting its planned development outputs and outcomes.

Figure 4: Selected Documents by Area of Focus (%, n=20)Footnote 16

Selected Documents by Area of Focus (%, n=20)

This finding leads to categorizing the AfDB’s program-level evaluations and development effectiveness reports under Scenario C – Option 3. The review methodology suggests that the multilateral organization’s reporting on development effectiveness is not adequate, and the evaluation function does not provide an adequate information base on development effectiveness. It further suggests that the multilateral organization requires support for its development effectiveness reporting, including its evaluation function.

The review notes that the AfDB has taken steps to strengthen its evaluation function through the 2012 self-assessment of its independent evaluation function, and the resulting revision of its independent evaluation policy and strategy. The revised evaluation policy has outlined the AfDB’s commitment to improving its development effectiveness by focusing on three complementary objectives for the strengthening of its independent evaluation function, specifically: learning; accountability and the promotion of an evaluation culture across the AfDB. This review anticipates that the strengthening of the AfDB evaluation function will include the strengthening of AfDB program-level results reporting.

Program-level evaluations and development effectiveness reports within the review period (2007-2012) provided inadequate coverage of the results of AfDB programming in its priority areas and its strategic objectives.

The 20 documents that met all the preliminary review criteria together addressed all but two of the AfDB’s priority areas. In addition, while one report had a multisector focus, two reports examined issues pertaining to social and human capital, which have an implicit link to the AfDB commitment to provide selective support to health-related millennium development goals (MDGs). All 20 reports addressed the AfDB’s strategic objectives. These findings are illustrated by Figures 5 and 6.

Figure 5: AfDB Priority Areas Addressed by Reports (Actual values, n=20)Footnote 17

AfDB Priority Areas Addressed by Reports (Actual values, n=20)

Figure 6: AfDB Strategic Objectives Addressed by Reports (%, n=20)Footnote 18

AfDB Strategic Objectives Addressed by Reports (%, n=20)

The preliminary review suggests, however, that AfDB reporting on its priority areas and strategic objectives is largely descriptive, involving an overview of AfDB programmatic activity. Significantly, the reporting provides inadequate coverage of results achieved through AfDB programming in its priority areas and under its strategic objectives. Using the DAC-EVALNET guidelines, the review indicates that the AfDB’s reports do not provide outcomes-based information on AfDB programming activity. This implies a need for the AfDB to revisit its approach to program-level evaluation reporting, as well as review its capacity to report on its development results.

AfDB program-level evaluations and development effectiveness reports provide insufficient coverage of AfDB investment in its crosscutting thematic priorities.

In its Medium-Term Strategy for 2008-2012, the AfDB committed itself to mainstreaming three thematic areas: Gender; the Environment; and Climate Change. Preliminary review findings indicate, however, relatively little evidence of the exploration of these crosscutting themes in AfDB development effectiveness reporting. Of the 20 reports that met all the preliminary review criteria, five reports (20 percent) addressed the crosscutting thematic priorities. One report included discussion around the gender thematic, and three reports addressed the environment. An additional report explored the theme of climate change. This finding is a reflection of insufficient coverage by the AfDB’s development effectiveness reporting of investment made in its priority crosscutting themes. This is crucial, as crosscutting priorities are core areas of AfDB investment requiring financial support to regional member countries. There is an equivalent need for assessment through program evaluations and/or development reports to determine the extent to which planned development outcomes are achieved in these areas.

This assessment notes that the AfDB produced a major internal report on gender in 2012, and has put new rules in place for gender mainstreaming. The AfDB also plans to conduct a review of gender mainstreaming in 2014. In the meantime, the AfDB has conducted a synthesis exercise to learn lessons on gender mainstreaming as practised by other organisations. At the time of the assessment, the AfDB submitted a report on environmental mainstreaming within the transport sector and engaged in a joint evaluation of the Climate Investment Funds (CIF).

The usefulness of AfDB development effectiveness reporting to the Bank’s forward-looking investments is sometimes impeded by the time lag between the year an evaluation is conducted and the year it is finalized and published.

All 43 documents included in the preliminary review universe were published during the established timeframe of January 2007 to September 2012. However, 8 reports had a significant lag between their publication and the years they reviewed. For example, a report published in 2007 evaluated programming during 1996-2004, thus implying a three year lag between implementation and publication. Because context and priorities change, AfDB evaluations with significant time lags between publication and the years reviewed have reduced usefulness for new program planning. This finding has been corroborated by the 2012 self-assessment of the AfDB’s independent evaluation function, which raised concerns about the timeliness of many AfDB evaluation reports. That said, there is value in the feedback provided to program managers as part of an evaluation process and prior to report publication.

Section Summary:

The findings from the preliminary review of AfDB program-level development effectiveness reports and evaluations indicate insufficient evidence to illustrate the AfDB’s development effectiveness during period 2007 to 2012.

Following the formulation of the AfDB evaluation universe of 43 program evaluations and development effectiveness reports, more than 50 percent of the reports did not meet the minimum standards established by the DAC-EVALNET guidelines.

Overall, the evaluation universe did not provide a critical mass of credible information on the AfDB’s development effectiveness. It further did not provide adequate coverage of crosscutting priorities identified by the AfDB in its 2008-2009 Medium-Term Strategy. There is an additional suggestion that development reports have decreased usefulness for AfDB planning and future investments because of time lags between evaluation activity and the publication of the evaluation report. Importantly, however, the AfDB has documented a commitment to strengthening its independent evaluation function. The review anticipates that this will lead to a strengthening of AfDB program-level results reporting.

3.1.2. Quality

Some evaluation designs in the program level evaluation reports reviewed did not follow the DAC-EVALNET quality guidelines. This is a potential area for improvement.

The preliminary review awarded points to each report using the quality screening criteria of the DAC-EVALNET Evaluation Coverage and Quality Screening Scoring Guide to determine evidence of two aspects of quality: (i) basic evaluation report characteristics; and (ii) the existence of key components required for a review of development effectiveness.

Preliminary review findings indicated that all 20 program-level evaluation reports that were retained in the universe surpassed the threshold of 25 points required to demonstrate having basic evaluation report characteristics. The score range was 27 – 36 points out of a maximum of 40 points, equivalent to a percentage score of 68 – 90 percent.

In terms of the existence of report characteristics that are required for a review of development effectiveness, documents received high scores on two of the three screening criteria: ‘Multiple lines of evidence’; and ‘Evaluation findings and conclusions are relevant and evidence-based’. Documents received 3 or 4 points out of a total of 4 points for each criterion.

Points attained for ‘Evaluations are well-designed’ were significantly lower, and ranged from one to five, out of a maximum of five attainable points. Of the 20 documents reviewed, only one received the maximum number of points attainable for a well-designed evaluation. These scores echo the 2012 self-assessment of the AfDB’s independent evaluation function, which stated that the AfDB employed a narrow range of evaluation methodologies in its project-level evaluations.

This finding provides evidence of AfDB attention to the quality of its development effectiveness reporting, as illustrated by the moderate to high scores awarded to reports that were screened for quality. There is an equally strong indication, however, of the need for a strengthening of the evaluation design process.

A reference in this regard would be the "DAC Guidelines and Reference Series: Quality Standards for Development Evaluation". According to the DAC guidelines, the core elements of a good evaluation design that are to be used to produce well-articulated evaluations include:

4.0 Conclusions

The purpose of the preliminary review was to provide an independent, evidence-based assessment of the development effectiveness of the AfDB, for use by the Government of Canada, the AfDB and other interested stakeholders. It was based on a universe of 43 AfDB program-level reports, comprising program-level evaluations and development effectiveness reports that were produced over the review period of January 2007 – September 2012.

The preliminary review of the AfDB’s program-level development effectiveness and evaluation reports involved the application of the DAC-EVALNET criteria to assess quality and coverage. The findings indicate that the evaluations included in the review do not provide an adequate level of coverage of the AfDB’s development results.

This leads to the following conclusions:

  1. The application of the DAC-EVALNET criteria to the preliminary review of the AfDB evaluation universe has shown that the AfDB has not produced a critical mass of information for the assessment of development effectiveness. Through the provision of forward-looking recommendations, this review can support the AfDB in improving evaluation and development effectiveness reporting in the future.
  2. Overall, AfDB coverage of its priority areas and strategic objectives in program-level reporting is inadequate. This is seen in its descriptive accounts of programmatic activity and a lack of details on development results. As well, there is a dearth of evidence on its priority crosscutting themes in program evaluations and development effectiveness reports. Further work is required to provide adequate outcome-based reporting for the Bank’s priority areas, strategic objectives, and crosscutting themes.
  3. The utility of some of the AfDB evaluations included in the universe of this review has been decreased by the length of time that elapsed between the evaluation activity and the year of report publication. The time lag has implied a decreasing usefulness of AfDB’s evaluations, given internal institutional changes, as well as contextual developments at the level of regional member countries. This leads to the conclusion that some of the AfDB’s evaluations included in the universe of this review are limited in their relevance to AfDB strategic decision-making processes.
  4. AfDB program-level evaluations have been hindered by the narrow range of methodologies employed. The use of methodologies that are not conducive to program-level evaluations provide an unreliable overview of progress made by the AfDB towards its planned development results. While the AfDB has met the DAC-EVALNET standards of quality for the production of evaluation reports, this has not been matched by well-articulated evaluation designs. This implies that there is need to strengthen AfDB capacity for designing its program-level evaluations.

5.0 Recommendations

These recommendations have been developed to assist the AfDB in developing a critical mass of program-level development effectiveness reports and evaluations, which would in future be amenable to the application of the DAC-EVALNET methodology for meta-synthesis of evaluation results. Recommendations are also provided to support CIDA’s future engagement with the AfDB, with emphasis on the strengthening of the AfDB’s evaluation function.

The recommendations build upon the results of this review, the MOPAN assessment of the Bank, the 2012 self-assessment of the AfDB’s evaluation function, and the AfDB 2013-2017 independent evaluation strategy. The latter has outlined the AfDB’s commitment to ‘raising the bar on quality’, to ensure methodological rigour, as well as improving the quality assurance process. Similarly, the need for a periodic peer review of the AfDB’s independent evaluation function was emphasized by the 2012 self-assessment, which recommended that the AfDB should consider engaging in a peer review once every 5 years.

5.1 Recommendations for the AfDB

Strengthening its independent evaluation function should be a priority for the AfDB.

A stronger evaluation function would enable the AfDB to accurately assess progress towards the achievement of program-level development outcomes, and capture lessons learned to improve its investments in regional member countries.

Based on the findings and conclusions of the preliminary review, the AfDB’s evaluation function has not produced the critical mass of reports required to facilitate a review of AfDB development effectiveness. The areas of AfDB evaluation activity in which the review findings have noted particular weaknesses were the design of evaluations and coverage of program-level development effectiveness results. The strengthening of the evaluation function will allow the AfDB to effectively address each of these areas. The anticipated results of this process are increased robustness in evaluation design through the use of appropriate evaluation methodologies, and more comprehensive reporting on development effectiveness.

The AfDB should consider engaging in a peer review of its evaluation function.

In order to improve its evaluations and maintain the DAC-EVALNET standards for conducting evaluations and reporting on development effectiveness, the AfDB should engage in a peer review of its evaluation function. A peer review is markedly different from a self-assessment. It would be of use to the AfDB for assessing the capacity and quality of its evaluation function. Findings from the peer review can be further used by the AfDB to assess the relevance, comprehensiveness, and overall quality of its development effectiveness reporting. The results of this assessment would be of value to the AfDB in determining the extent to which its program evaluations and development effectiveness reports can be used to inform AfDB strategic decision-making.

In addition to benefiting from advice for the strengthening of its evaluation function, the AfDB can use findings from the peer review to develop best practices for dissemination and learning. This is in accordance with the AfDB’s commitment to ‘using knowledge management’Footnote 19 as outlined in its independent evaluation strategy for 2013-2017.

5.2 Recommendations for DFATD

Canada, in concert with other donors and AfDB Board members, should support the Bank’s efforts to strengthen its evaluation function.

Canada should support the AfDB’s efforts to strengthen its evaluation function and the design of its evaluations, as outlined above. One possibility, among others, would be an annual analysis of the AfDB’s evaluation coverage to be included as part of the AfDB’s evaluation work plan. For example, the AfDB’s rolling work plan could provide some measure of the target for, and actual implementation of, program level evaluation, perhaps as a percentage of total expenditure.

In working with other Board members and donors, Canada should consider among others supporting the strengthening of the AfDB’s evaluation function. Important elements would include: peer review; capacity development to strengthen evaluation coverage, design, and timeliness; and strengthened program-level development effectiveness reporting.

Annex 1: Evaluation Universe

No 1

No 2

No 3

No 4

No 5

No 6

No 7

No 8

No 9

No 10

No 11

No 12

No 13

No 14

No 15

No 16

No 17

No 18

No 19

No 20

No 21

No 22

No 23

No 24

No 25

No 26

No 27

No 28

No 29

No 30

No 31

No 32

No 33

No 34

No 35

No 36

No 37

No 38

No 39

No 40

No 41

No 42

No 43

Annex 2: Evaluation Coverage and Quality Screening Scoring Guide

Evaluation Title:

Reviewer:

Coverage1:

Financial2:

Table A: Evaluation Coverage and Quality Screening Scoring Guide
CountryRegionGlobal
Priority areas and strategic objectives were taken from the AfDB Medium Term Strategy 2008 - 2012.
Column 1 (Evaluation) pertains to the worth of the program and Column 2 (Total) pertains to the total expenditure.
Name of country:Name of region:Multicountry: YES / NO
EvaluationTotalEvaluationTotalEvaluationTotal
      
Priority Areas
SectorCoveragePlanned Budget
Infrastructure  
Governance  
Higher Education & Vocational Training  
Private Sector  
Regional Integration  
Fragile States  
Client-responsive engagement  
Climate Change  
Gender  
Environment  
Strategic Objectives
Operational Selectivity 
Results and Performance
Decentralization
Streamlined Internal Processes
Staffing, Knowledge, Financials and Partnerships as Foundation
General Coverage
Dates covered (please indicate dates): 
Case study that addresses a larger evaluationYES / NO
Joint studyYES / NO
Institutional impact addressedYES / NO
Beneficiary impact addressedYES / NO
Table B: Quality Screening
 Points for criteria scoredMaximum PointsScore
The combined score for criteria G+H+I is used to provide an overview of the quality of the evaluation
A

Purpose of the evaluation:

  • why the evaluation was done (1)
  • what triggered the evaluation (including timing in the project/program cycle) (1)
  • how evaluation is to be used (1)
3 
B

Evaluation objects:

  • evaluation objectives is to be used (1)
  • objectives logically flow from purpose (1)
2 
C

Organization of the evaluation:

  • logical structure to the organization of the evaluation (1)
  • evaluation is well-written (1)
  • clear distinction between evidence, findings, conclusions, and recommendations (1)
3 
D

Subject evaluated is clearly described:
Evaluation describes:

  • the activity / program being evaluated (1)
  • the program's expected achievements (1)
  • how the program addresses the development problem (1)
  • the implementation modalities used (1)
4 
E

Scope of the evaluation:
Evaluation defines the boundaries of the evaluation in terms of:

  • time period covered (1)
  • implementation phase under review (1)
  • geographic area (1)
  • dimensions of stakeholder involvement being examined (1)
4 
F

Evaluation criteria:
Evaluation criteria including:

  • the achievement of development objectives and expected results (including impacts) (1)
  • cross-cutting issues: inclusive development which is gender sensitive and environmentally sustainable (1)
  • the sustainability of benefits and positive results achieved (1)
  • the relevance of AfDB activities and supported projects and programs (1)
  • the efficiency of AfDB operations in support projects and programs (1)
5 
G

Multiple lines of evidence:

  • one point (1) for each line of evidence used (case studies, surveys, site visits, interviews, etc.), up to a maximum of four points (4)
4 
H

Evaluation design:
Elements of a good evaluation design include:

  • an explicit theory of how objectives and results were to be achieved (1)
  • specification of the level of results achieved (output, outcome, impact) (1)
  • baseline data (quantitative of qualitative) on conditions prior to program implementation (1)
  • comparison of conditions after program delivery to those before (1)
  • a qualitative or quantitative comparison of conditions among program participants and a control group (1)
5 
I

Evaluation findings and conclusions are relevant and evidence based:
Evaluation report includes:

  • evaluation findings relevant to the assessment criteria (1)
  • findings that are supported by the chosen methodology (1)
  • a clear logical link between the evidence and the finding (1)
  • conclusions which are linked to the evaluation findings as reported (1)
4 
J

Evaluation limitations:

  • statement of limitations of the methodology (1)
  • impact of limitations on evaluation (1)
  • remedies of limitations (1)
3 
K

Evaluation recommendations:

  • evaluation contains recommendations that follow from findings and conclusions (1)
  • recommendations are directed to one or more organizations (1)
  • recommendations are aimed at improving development effectiveness (1)
3 
 Total (required to have a minimum of 25 points)40 
 Total for Criteria G+H+I (required to have a minimum of 10 points)*13 

Comments

Annex 3: Corporate Documents Reviewed

African Development Bank Group. (October 2012). Independent Evaluation Strategy 2013 – 2017. Tunis: African Development Bank Group – Operations Evaluation Department.

African Development Bank Group. (n.d.). Medium-Term Strategy 2008-2012. Tunis: African Development Bank Group

African Development Fund. (September 2012). Overview Note: ADF-12 Mid-Term Review, September 2012, Praia, Cape Verde.

Canadian International Development Agency. (April 2012). Development Effectiveness Review of the United Nations Development Programme (UNDP) – 2005 – 2011: Synthesis Report. Gatineau: CIDA

Multilateral Organisation Performance Assessment Network (2012). Assessment of Organisational Effectiveness and Reporting on Development Results. African Development Bank (AfDB). Volume I and II, December 2012.

OECD-DAC Network on Development Evaluation. (June 2012). Assessing the Development Effectiveness of Multilateral Organizations: Guidance on the Methodological Approach. Paris: OECD-DAC Network on Development Evaluation.

PERRIN, Burt. (July 2012). Self-Assessment of the African Development Bank Group Independent Evaluation Function: Final Report. African Development Fund. ADF/ BD/ WP/ 2012/ 84/ Add.1

Annex 4: African Development Bank – Comments on the Review of AfDB Program Evaluation Reporting

The following comments are provided by the Independent Operations Evaluation Department of the African Development Bank. August 2013.

The Independent Operations Evaluation Department (OPEV) of the African Development Bank (AfDB or Bank) welcomes CIDA’s Preliminary Review of AfDB’s Development Effectiveness Reporting. As a development institution the AfDB is committed to continuously improving its development effectiveness and welcomes all external assessments as useful means to understand better areas where it could further improve performance.

The AfDB has recognised in its Strategy 2013-2022 the important role evaluation plays for improving its operational performance and committed to strengthen this function. The implementation of OPEV’s new Strategy has already produced visible initial results in 2013 in terms of quality and coverage of evaluations, but any ambitious change programme takes time before it yields full results. Against this background, OPEV finds in CIDA’s review elements of confirmation of its own diagnosis that led to the design and implementation of its new Strategy. We feel, however, its overall conclusion fails to capture the dynamics already in place to address existing challenges, including broader reforms at Bank level.

OPEV’s comments are organised under three main categories: (i) the current context for independent evaluation in the AfDB; (ii) the scope and methodology of the CIDA review; and (iii) the recommendations.

1. The current context for independent evaluation in the AfDB

CIDA’s review was conducted after a detailed “self-assessment” of the AfDB’s independent evaluation function. The 2012 self-assessment was followed by a new Independent Evaluation Policy and separate Strategy for Independent Evaluation (2013-17). This strategic direction has in turn has been reflected in OPEV’s three-year rolling work program. Thus, the past year marks a step change in independent evaluation at the AfDB. Implementation of the new Strategy is now underway and is designed to help OPEV to address key challenges identified in the self-assessment. Features most relevant to the CIDA review include:

As well as a step-change in independent evaluation, the Bank more broadly has redoubled its efforts to improve its development results reporting. Annual development effectiveness reviews, at continental, sector and recently country level are also produced, although these were not included in CIDA’s documentary review. The Bank’s corporate results framework is also being revised to align with the new Strategy 2013-2022.

These transitions are underway. As always the AfDB welcomes the support and engagement of development partners such as CIDA to strengthen further its ability to report on development effectiveness, either through independent evaluation or otherwise.

2. Scope and methodology

Overall, OPEV agrees that independent evaluation should make a stronger contribution to reporting on development results. Nevertheless, some issues regarding the scope and application of the methodology in the CIDA review are worth highlighting since they may affect the overall conclusions of the review. These issues were raised with CIDA when a first draft of the report was shared with OPEV, but still remain concerns in the final draft. In particular:

3. Recommendations

The Bank welcomes the support of all development partners for strengthening its independent evaluation function. In late 2012 the new evaluation Policy and Strategy were discussed by the Board’s Committee on Development Effectiveness, to which OPEV reports. The Strategy in particular sets out how the independent evaluation function of the Bank can be strengthened; it has been widely endorsed and in 2013 was formally approved by the Bank’s Board.

Recommendation to the Bank: Strengthening its independent evaluation function should be a priority for AfDB.

Response: Strengthening the evaluation function is at the heart of the Independent Evaluation Strategy 2013-2017. OPEV voluntarily launched a detailed self-assessment to diagnose weaknesses and identify strengths and opportunities. On this base of solid evidence OPEV revised its Independent Evaluation Policy and prepared a first Independent Evaluation Strategy. The Strategy highlights actions to strengthen the quality of evaluations including preparation of an evaluation manual, establishing quality standards and a more systematic quality assurance process. These are being implemented during the course of 2013 and beyond. It also proposes more explicit coverage of the themes outlined in the Bank’s Ten Year Strategy and has increased evaluations at country, sector and thematic level. Support from Canada to the implementation of OPEV’s Strategy is most welcome.

Recommendation to the Bank: The AfDB should consider engaging a peer review of its evaluation function.

Response: The peer review process is already planned to take place within the next two years. It is expected that the peer review will form the basis for the next set of reforms for OPEV.

Recommendation to CIDA: CIDA, in concert with other donors and AfDB Board members, should support the Bank’s efforts to strengthen its evaluation function.

Response: Support is most effective when aligned with the Bank’s own Strategy and Policy for independent evaluation. Discussion on entry points for CIDA is very welcome; discussions with other interested partners are on-going. Regarding the specific suggestion to assess the coverage of OPEV reports, coverage is looked at through the process of developing the rolling work program; which involves consultation within the Bank and with its stakeholders, including Board members. Canada’s involvement in this process through its Board membership is most welcome.

Annex 5: Management Response

The Preliminary Review of the African Development Bank (AfDB) Development Effectiveness Reporting prepared by CIDA’s Evaluation Division, provides an assessment of the Bank’s program-level evaluation function.

In 2011, the AfDB financed 184 operations in regional member countries at a cost of UA 5.72 billion. The AfDB was founded by signatory agreement between regional member states in 1963. It is one of three entities that together comprise the African Development Bank Group, the other two being the African Development Fund and the Nigeria Trust Fund. As a regional multilateral development finance institution, the mission of the AfDB is to contribute to the sustainable economic development and social progress of African countries. Membership of the AfDB comprises 77 countries, of which 54 are regional member countries. The AfDB provides development assistance to its regional member countries through the public and private sectors, in the form of financial instruments, policy advice, and technical assistance.

Canada’s financial support to the Bank consists of: (i) contributions to the Bank’s capital through share purchases; (ii) voluntary support to specific trust funds; and, (iii) voluntary support to the African Development Fund – the Bank’s concessionary funding window. The African Development Bank continues to be a key partner for CIDA in helping to achieve poverty reduction in the African region.

The Review conducted a meta-synthesis of the African Development Bank’s Operations Evaluation Office program evaluations between 2007 and 2012, and did not include a review of project-level or trust fund evaluations. Based on the quantity and quality of program-level evaluations, it was determined that the AfDB’s reporting on development effectiveness is not adequate and that its evaluation function does not provide an adequate information base on development effectiveness. Therefore, the report presents recommendations aimed at supporting improved evaluation and development effectiveness reporting, as suggested by the OECD-DAC evaluation methodology.

The Review’s conclusions are broadly consistent with the views expressed in the 2012 MOPAN assessment of the Bank, which concluded that the AfDB’s Evaluation Unit does not appear to have a formal process for reviewing the quality of evaluations, and that the frameworks and systems it uses to manage for, and report on, organization-wide results need further improvement.

CIDA accepts the Preliminary Review’s recommendation for improving the African Development Bank’s evaluation function, while noting that there have been some improvements since the review period. In particular, the AfDB has initiated a process to enhance its evaluation function through ensuring methodological rigor and improving the quality assurance process, as proposed in the 2011 self-assessment of the AfDB’s evaluation function, and the AfDB 2013-2017 independent evaluation strategy.

CIDA’s interventions to address the recommendations will focus on the following approach: 1) Issues related to improving the Bank’s evaluation function will figure prominently in Canada’s interventions in the upcoming meetings for the replenishment of the African Development Fund; 2) The issue will also be raised at the Bank’s Board of Directors in the context of the approval of the Three-Year Rolling Evaluation Work Programme; and 3) CIDA will assist with capacity development, based on opportunities identified by the Bank. This could be done, for example, through expertise provided via Evaluation Division standing offers.

Recommendations

1. Canada, in concert with other donors and AfDB Board members, should support the Bank’s efforts to strengthen its evaluation function.

Canada should support the AfDB’s efforts to strengthen its evaluation function and the design of its evaluations, as outlined above. One possibility, among others, would be an annual analysis of the AfDB’s evaluation coverage to be included as part of the AfDB’s evaluation work plan. For example, the AfDB’s rolling work plan could provide some measure of the target for, and actual implementation of, program level evaluation, perhaps as a percentage of total expenditure.

In working with other Board members and donors, Canada should signal its readiness to support the strengthening of the AfDB’s evaluation function. Important elements would include: peer review; capacity development to strengthen evaluation coverage, design, and timeliness; and strengthened program-level development effectiveness reporting.

Commitments/measures

Agree

1.1 CIDA, along with other donors, will urge the African Development Bank (AfDB) at the upcoming African Development Fund (AfDF XIII) replenishment meetings to strengthen its evaluation function. The AfDF XIII negotiations are scheduled to conclude in the Fall of 2013.

Responsible: CIDA Multilateral and Global Programs Branch IFI Division

Completion date: March 2014

Status: Ongoing. Was raised at the February 21-22, 2013 meeting in Tunis.

Commitments/measures

1.2 The Bank has indicated that evaluation coverage is already considered in the process of developing its Three Year Rolling Work Programme, which is adjusted annually. This is also considered within the Bank’s Independent Evaluation Strategy.

Responsible: CIDA Multilateral and Global Programs Branch IFI Division

Completion date: Completed

Status: The Three-Year Rolling Work Programme.

Commitments/measures

1.3 In working with other Board members, and in line with the Bank’s evaluation strategy, CIDA will assist with capacity development, based on opportunities identified by the Bank. This could be done, for example, through expertise provided via Evaluation Division standing offers.

Completion date: March 2014

Date modified: