Rolling Five-Year Development Evaluation Work Plan 2012-2013 - 2016-2017

June 2012

List of Abbreviations

Executive Summary

This Rolling Five-Year Evaluation Work Plan (“the Plan”) was prepared by the Evaluation Division (ED) of CIDA’s Strategic Policy and Performance Branch (SPPB). It is an updated version of the Plan from fiscal year (FY) 2011/12, which was approved by the Evaluation Committee (EC) in February 2011.

The Plan lays out the scheduled evaluations for FY 2012/2013 to 2016-2017. Execution of this Plan will enable CIDA to move towards achieving evaluation coverage of 100% of its direct program spending, as required by the 2009 Treasury Board (TB) Policy on Evaluation. The Plan provides an overview of the ED’s achievements in FY 2011/12. It outlines the evaluation work planned by the ED for fiscal years 2012/13 to 2016/17, with a focus on FY 2012-13, and a detailed list of resources required. It also includes a list of decentralized^{Footnote 1} evaluations planned for FY 2012-13. This Plan was developed using the best available information regarding the availability of the Agency’s resources for evaluation in FY 2012/2013.

Evaluation Division’s Accomplishments – FY 2011/2012

The ED completed several evaluations in FY 2011/2012 and provided input to numerous corporate initiatives. Highlights of these accomplishments include:

• Initiation of reviews of the World Health Organization (WHO) and the Asian Development Bank (ADB), as well as evaluations of the Mali, Colombia^{Footnote 2} and Caribbean^{Footnote 3} country programs.
• Completion of the following evaluation activities:
  1. Ukraine, Honduras, and Peru Country Program Evaluations; the Regional Inter- American Program Evaluation^{Footnote 4}; and CIDA’s Humanitarian Assistance Evaluation.^{Footnote 5}
  2. Development Effectiveness Review of the United Nations Development Programme (UNDP)^{Footnote 6} and the World Food Programme's Humanitarian Assistance^{Footnote 7}; and a Review of Evidence of the Effectiveness of CIDA’s Grants and Contributions.
• Support to the renewal of CIDA’s terms and conditions, based on the review of the effectiveness of CIDA’s grants and contributions;
• Two joint evaluations with international donors: Mali and the Paris Declaration;
• A scoping/evaluability assessment of CIDA’s largest aid program (Afghanistan) and development of terms of reference for the evaluation;
• Development of evaluation strategies for the Muskoka Initiative and CIDA’s three thematic priorities;
• A strategy and work plan for the dissemination of evaluation knowledge;
• Completion of a quality assurance tracking system for decentralized evaluations, along with the provision of coaching, advice, and guidance to program branches, with a 113% increase in the number of inquiries during FY2011/12;
• Improvements to the generic terms of reference for decentralized evaluations and other guidance documents;
• Provision of quality at entry services to over 70 branch-led (decentralized) evaluations;
• Input into several corporate initiatives (e.g., Report on Plans and Priorities, Departmental Performance Report);
• Contribution to 8 TB submissions for new initiatives, with a total value of $528 million;
• Effective management of the evaluation standing offer and design of the upcoming new supply arrangement;
• Termination of three branch-led evaluation mandates due to insufficient quality of deliverables; and,
• Endorsement of a new approach for assessing the development effectiveness of multilateral organizations by the DAC-EVALNET.

Planned Evaluation Work – FY 2012/2013

In 2012-2013, the ED will initiate and/or complete the following evaluations:

• Countries of concentration and regional programs: Bolivia, Colombia, Mali, Tanzania, Pakistan, Indonesia, and Caribbean Regional
• Fragile States: Afghanistan and Haiti
• Multilateral Organizations: African Development Bank, Asian Development Bank, World Health Organization and UNICEF
• Partnerships with Canadians Branch: Governance programming
• Corporate evaluations: a meta-evaluation of branch-led evaluations, and a report on the state of performance measurement at CIDA

In addition the ED will:

• Provide advice and support to program branches on decentralized evaluations;
• Complete an annual report on CIDA’s evaluations, which will be entitled CIDA Learns;
• Continue to implement the dissemination strategy for evaluation knowledge; and,
• Serve as Secretariat for CIDA’s Evaluation Committee.

Resources

Implementation of this evaluation work plan will require a budget of approximately $2.2 million for FY2012/2013, including a base salary budget of $1.4 million for 18 FTEs. In order to achieve cost savings while meeting evaluation coverage requirements, the UNICEF review will be conducted jointly with the Netherlands, and coordination between centralized and decentralized evaluations will be maximized.

The Evaluation Division has a base salary budget of about $1.4 million for 18 FTEs. She also knows that the effects of the action plan for deficit reduction could impact on the resources needed to implement the plan. This could be changed to reflect a different scenario.

1.0 Background

1.1 Introduction

This document describes the Rolling Five-Year Evaluation Work Plan (“the Plan”) for the Evaluation Division (ED) of CIDA’s Strategic Policy and Performance Branch (SPPB). The Plan takes into account a range of considerations, including:

• The evolving international, Canadian and CIDA contexts;
• The achievements, experiences and lessons learnt from previous years;
• Treasury Board of Canada requirements;
• Accountability, value for money; and,
• A strategic assessment of risk^{Footnote 8}.

The scope and content of the Plan are consistent with the requirements of the 2006 Federal Accountability Act (FedAA) and the 2009 Treasury Board (TB) Policy on Evaluation. The Plan proposes an approach that moves CIDA towards achieving 100% evaluation coverage of direct program spending as required by the FedAA and the 2009 TB Policy on Evaluation, while contributing to the Agency’s learning and decision-making needs and processes.

1.2 International Context

In 2005, the Paris Declaration on Aid Effectiveness emerged from a growing consensus on reforming donors and developing countries’ approaches to delivering and using aid in order to ensure more and better development results. The Paris Declaration committed developing countries and donors to hold each other accountable for implementing its principles at the country level through a set of clear indicators, with targets to be achieved by 2010.

In 2008, the first phase of the Paris Declaration evaluation assessed changes in behavior and identified better practices for partners and donors in implementing the Paris commitments. The results of this evaluation contributed constructively to the ongoing aid effectiveness policy debate and, in particular, to the 2008 Accra Agenda for Action. The Accra Agenda for Action further built on the Paris Declaration by promoting policies and actions that improve transparency and accountability in strengthening aid effectiveness. In 2011, the second phase of the Paris Declaration evaluation contributed towards the aid effectiveness debate, ahead of discussions at the 4th High Level Forum on Aid Effectiveness. The evaluation highlighted that most principles and commitments in the declaration have proven relevant to improving the quality of aid. However, evidence gathered on the implementation of the Paris Declaration shows that while progress has been made, it has not been to the extent and pace envisioned in 2005.

The 4th High Level Forum on Aid Effectiveness was held in Busan, Korea, at the end of 2011. The forum considered the results of implementing the Paris Declaration and charted the future course for aid effectiveness, particularly given changing global dynamics and political upheavals. At the Busan forum, decision-makers in developed and developing countries agreed on a framework for global co-operation supporting the achievement of the Millennium Development Goals. A global compact was also forged for effective development towards and beyond 2015.

1.3 Canadian Context

Similar to its peers in a globalizing world, Canada is grappling with the demands of economic recovery, demographic shifts, climate change, and fiscal restraint. The impact of social media, the special challenges in conflict and fragile states, and the influence of new types of donors active in development cooperation efforts are also having an impact on Canada and other donors’ approach to international development.

The Government of Canada has outlined its plan to return to a balanced budget by 2015–16, as indicated in the federal budget tabled on June 6, 2011. In this context, government departments underwent a strategic and operational review to inform the government’s deficit reduction decision-making process. Evaluations supported this objective by providing information on the relevance and performance of programs.

In addition, government departments and agencies have been focusing on the implementation of the 2009 TB Policy on Evaluation. The objective of this updated policy is to create a comprehensive and reliable base of evaluation evidence that is used to support policy and program improvement, expenditure management, Cabinet decision-making, and public reporting.

In line with the 2006 FedAA, the 2009 TB Policy on Evaluation requires that 100% of direct program spending be evaluated over a five-year cycle. This requirement will be in full effect by 2017-18. The TB Policy on Evaluation also introduces the use of flexible evaluation approaches guided by risk, scale, and scope. It requires departments to develop a strategically focused evaluation plan that is founded on an assessment of risk, departmental priorities, and government priorities. The expected results of this updated policy are a more robust evaluation function focused on value for money; accountability; and credible, timely and neutral information on the ongoing relevance and effectiveness of all direct program spending.

1.4 CIDA Context

With a view to improving aid effectiveness, The Budget Plan 2007: Aspire To A Stronger, Safer, Better Canada laid out a three-point program for enhancing the focus, efficiency and accountability of Canada’s international assistance efforts. To facilitate this objective, the government committed to "examining options to ensure the independent evaluation of our aid program, providing parliamentarians and Canadians with an objective assessment of the results we achieve with our international assistance" (pp. 263).

In response to this commitment, the Agency agreed to strengthen the independence of its evaluation function in a manner consistent with the framework of the FedAA and the TB Policy on Evaluation. Changes include strengthening the governance, oversight, and independence of CIDA’s Evaluation Committee (EC), which is now comprised of six members from outside government and five CIDA senior executives (including the President, who chairs the committee). The EC acts as an advisory body to the President on evaluation activities and is a central part of the Agency’s evaluation function.

Over the last three years, CIDA has re-examined its programs, policies and operations. The Agency has implemented a comprehensive aid effectiveness agenda, strengthened its geographic and thematic focus, completed a multilateral effectiveness review, and modernized its partnerships with Canadians programming.

In the context of the Government of Canada’s deficit-reduction action plan, CIDA continues to work towards achieving further efficiencies. The Agency’s business modernization initiative is an attempt to streamline the Agency’s core business processes through the implementation of a system to manage all of CIDA’s policies and procedures (e.g. a Guide to Rules and Tools), and decentralization of program management and project delivery functions to field offices.

This Plan has been developed using the best available information regarding the availability of Agency resources for evaluation in FY2012/2013.

2.0 CIDA’s Evaluation Function

Evaluation provides Canadians, parliamentarians, beneficiaries, ministers, central agencies, and CIDA’s senior management with evidence-based timely, credible, and neutral information on the relevance, effectiveness, sustainability, efficiency, coherence and cross-cutting issues of the Agencies policies and programs. The key objective of CIDA’s evaluation function is to create a comprehensive and reliable base of evaluative evidence for the purposes of accountability, expenditure management, decision-making and learning.

CIDA’s evaluation function is governed by the Agency’s Evaluation Policy, which was updated to reflect the new requirements outlined in the 2009 TB Policy on Evaluation. The policy is currently going through the Agency’s internal approval process. The policy is implemented through the Rolling Five-Year Evaluation Work Plan, which is approved by the President on advice from the EC.

The policy and the Plan are executed through the Agency’s centralized and decentralized evaluation functions. The ED leads the centralized evaluation function, focusing on corporate (Agency-level) and program-level assessments of relevance and performance. The evaluations led by the ED respond to the requirements of the 2009 TB Policy on Evaluation.

Program branches lead and manage (alone or jointly with other donors) decentralized evaluations, while the ED provides technical quality assurance services, fostering an environment of quality improvement. These evaluations cover investments below the sub-sub activity level of the Program Activity Architecture. Decentralized evaluations support decision- making, program and project improvement, mutual accountability and learning. While the results of these evaluations improve CIDA’s aid effectiveness, the primary beneficiaries are developing country partners. These decentralized evaluations also serve as building blocks for centralized evaluations.

Centralized and decentralized evaluations are integral to CIDA’s solid oversight function while simultaneously contributing to learning and the adoption of better practices. Recognizing the duality of the evaluation function within the Agency, this Plan will continue to focus on both centralized and decentralized evaluations, including some multi-donor initiatives.

The key responsibilities of both the Evaluation Division and program branches are outlined below.

2.1 Responsibilities of the Evaluation Division

The Evaluation Division is responsible for^{Footnote 9}:

• ensuring that the evaluation function is appropriately staffed with competent professionals;
• developing a rolling five-year evaluation plan, which covers 100% of CIDA’s direct program spending;
• conducting centralized evaluations in a neutral and cost-effective manner;
• disseminating evaluation knowledge to increase the visibility, accessibility and use of evaluation results and lessons, both within and outside the Agency;
• tracking the implementation of commitments in management responses and action plans to address the recommendations of evaluations;
• strengthening CIDA’s evaluation function through evaluation quality assurance services, by providing: advice, coaching, training, guidelines, tools, and templates to the Agency’s branches;
• maintaining standing offers for evaluation services;
• building and maintaining strategic alliances with partners in Canada and internationally; and,
• supporting the Evaluation Committee to ensure effective governance and oversight.

2.2 Responsibilities of Program Branches

Program branches are responsible for^{Footnote 10}:

• ensuring that credible and reliable performance data are being collected to effectively support evaluation;
• developing multi-year evaluation work plans (for decentralized evaluations) and submitting them to the ED for inclusion in CIDA’s Rolling Five-Year Evaluation Work Plan;
• managing decentralized evaluations, providing management responses to these evaluations, and ensuring follow-up and tracking of recommendations;
• disseminating the results of their decentralized evaluations, for learning and accountability purposes;
• preparing management responses and/or actions plans in response to centralized evaluations of their programs; and,
• implementing actions based on the approved management responses and/or action plans.

3.0 Overview of Achievements and Challenges - FY 2011/2012

3.1 Achievements

3.1.1 Evaluations of Geographic Programs

During FY 2011/2012, the ED completed the Peru evaluation, which was presented to the EC during its February 2012 meeting. The joint evaluation of general budget support (Mali) was also completed, as well as the Ukraine, Honduras, and Inter-American program evaluations.

In terms of evaluations in fragile states, the ED completed a scoping/evaluability assessment of CIDA’s largest aid program (Afghanistan) and developed terms of reference for this evaluation.

3.1.2 Evaluations of Multilateral and Global Programs

The approach to assess the development effectiveness of multilateral organizations was finalized under the guidance of the Development Assistance Committee Network on Development Evaluation (DAC/EVALNET) of the Organization for Economic Development and Cooperation (OECD) and was endorsed by its members as an acceptable approach for this type of review.

The approach was applied to two multilateral organizations in FY 2011/2012, namely the United Nations Development Programme (CIDA-only initiative) and the World Food Programme (jointly with the Netherlands). The reports from these reviews were presented to the Evaluation Committee during its February 2012 meeting.

3.1.3 Corporate Evaluations

The ED completed A Review of Evidence of the Effectiveness of CIDA’s Grants and Contributions. This review covers CIDA’s recent investments provided through its three programming channels – bilateral, partnership and multilateral. The results of this review also fed into the renewal process of CIDA’s Terms and Conditions.

The Corporate Evaluation of CIDA’s Humanitarian Assistance was initiated in 2011 and its final evaluation report was presented to the EC during its February 2012 meeting.

The division finalized its participation in the joint evaluation of the Paris Declaration, including assisting Senegal and Ghana to complete their country evaluations as case studies while strengthening their evaluation capacity.

The ED completed evaluation strategies for the Muskoka Initiative, the Sustainable Economic Growth Thematic Priority Strategy, the Children and Youth Thematic Priority Strategy, and the Food Security Thematic Priority Strategy.

3.1.4 Evaluations of Partnerships with Canadians Programming

The ED has worked, in conjunction with PWCB, to develop an approach suitable for evaluating PWCB programming at the corporate level. The ED has identified five evaluation units within the new PWCB organization and thematic focus, all of which are integrated within this Plan. These evaluation units correspond to the themes of PWCB's logic model.

3.1.5 Support Provided to Program Branches

The ED continued to provide support and advice to program branches on decentralized evaluations, with a 113% increase in the number of inquiries during FY2011/12. Other support included:

• reviewing 75 evaluation TORs, 30 evaluation work plans, and 20 draft evaluation reports, of evaluations which range from smaller mid-term formative evaluations ($60,000 to $90,000) to large and complex evaluations ($200,000 to $400,00);
• providing support to programs in their selection of consultants for branch-led evaluations, ensuring the best fit between mandated requirements and the experience and knowledge of consultants;
• effective management of the evaluation standing offer and design of the upcoming new supply arrangement; and,
• improving the generic terms of reference for decentralized evaluations and other guidance documents.

All of these efforts are intended to support the program branches attempts to strengthen their decentralized evaluations, which will also create stronger “building blocks” for upcoming ED evaluations.

3.1.6 Support to Other Corporate Initiatives

The ED provided support and input to corporate initiatives such as the Report on Plans and Priorities (RPP), the Departmental Performance Report (DPR), the OECD/DAC Peer Review exercise, the Busan 4th High-Level Forum on Aid Effectiveness and the Management Accountability Framework (MAF). In FY2011/12 CIDA received a “strong” rating in the evaluation area of management, as part of the TBS MAF assessment. This marked the sixth consecutive year that CIDA’s evaluation function has received this rating. The ED also provided advice and recommendations for the review of proposals for grants and contributions under the authority of the SPPB, and continued to contribute to high-level decision-making by CIDA’s executives.

3.1.7 TB Submissions

In collaboration with the Performance Management Division, the ED assisted programs with preparation of their TB submissions. In particular, the ED supported preparation of 8 TB submissions this year. Table 3 (at the end of the Plan) provides a list of evaluations that are required as a result of TB submissions.

3.1.8 Dissemination and Use of Evaluation Results

The ED developed and initiated the implementation of a strategy and work plan for the dissemination of evaluation knowledge. The strategy and work plan are aligned with the Government of Canada’s commitment (in the 2007 budget) to improve the accountability to Canadian taxpayers of international assistance programs. The strategy and work plan will contribute to the Agency’s knowledge management and lessons learned agenda.

The accompanying work plan provides further details on the specific activities under each of the areas of focus to be undertaken during the next two fiscal years, including deliverables, costs and timelines.

The ED launched the first Evaluation Learning Café in September 2011 with the support of the Agency’s evaluation knowledge champion, the Vice-President of the Strategic Policy and Performance Branch. The learning event was on the evaluation of the Paris Declaration on Aid Effectiveness with a presentation by Bernard Wood, Team Leader for the evaluation. Approximately 100 participants from headquarters and the field attended the event, as well as colleagues from other departments. The learning café was the first in a series of events held in FY2011/12, which were designed to help disseminate evaluation results and promote learning within the Agency.

3.1.9 Building and Maintaining Strategic Alliances

The ED strengthened strategic alliances through domestic joint efforts, such as the horizontal evaluation of the Canadian Police Arrangement, which was led by the Department of Foreign Affairs and International Trade (DFAIT) with participation from CIDA, the Royal Canadian Mounted Police and, Public Safety Canada. Internationally, the ED has continued to actively collaborate with other donors, and participate in the DAC-EVALNET. The ED’s participation in the DAC-EVALNET has included promotion of the common approach for assessing the development effectiveness of multilateral organizations.

3.2 Challenges

During the past year, the ED faced specific challenges that have impacted its work. These challenges constitute key considerations for developing the division’s work plan for FY 2012/2013 and beyond. These challenges include:

• Evaluation Capacity: The shortage of skilled evaluators (both consultants and staff) has been a challenge for the evaluation function in Canada; CIDA’s evaluation function is no exception. This problem is further exacerbated by the departure of experienced evaluators due to retirement. To address this challenge, the ED is currently working on the Agency’s standing offer arrangement to improve the capacity of consultants. Additionally, the ED intends to hire four junior evaluation officers, and provide them with a clear path towards career progression and development. These officers will receive training in evaluation methods, which will serve to enhance the ED’s capacity while contributing to a pool of qualified evaluators within CIDA and the public service at large.
• Evaluation Coverage: Another challenge is the fulfillment of evaluation coverage requirements with multilateral organizations. The division’s leading role in the current OECD/DAC joint donor exercise to develop and test a methodology for evaluating the development effectiveness of multilateral organizations is bringing to success in this area.
• Evaluation of Fragile States: The ED faces particular challenges in delivering evaluations on fragile and conflict-affected states (i.e., Haiti, Sudan, West Bank/Gaza, Afghanistan). Both natural disasters and ongoing security issues make field-based evaluations problematic and expensive. This challenge is being addressed in various ways, depending on the specific situation. Joint evaluations are one way to leverage the resources of multiple donors (e.g., Multi-Donor Evaluation of Conflict Prevention and Peacebuilding Activities in Southern Sudan) and mitigate security risks. In other situations, the ED will engage consultants with significant experience in fragile states (e.g., Afghanistan).

4.0 Strategy and Key Considerations in Developing the Work Plan

In developing this Plan, the Evaluation Division considered the following:

1. the requirement to achieve 100% evaluation coverage of CIDA’s direct program spending over 5 years by 2017/2018;
2. the evaluation approach used in different types of programming (geographic, fragile states, multilateral, and partnerships);
3. assessing and managing levels of risk for specific evaluations;
4. the duality of the evaluation function at CIDA (i.e., centralized and decentralized evaluations);
5. the commitments in Treasury Board submissions; and
6. the need to further promote the use of evaluation results within the Agency.

4.1 Evaluation Coverage

The first challenge in addressing the TB evaluation coverage requirement is to establish an evaluation universe that provides for full coverage of the Agency’s direct program spending during the 5-year cycle. A detailed description of the Agency’s evaluation universe is provided under Annex 1: “Establishing the Evaluation Universe”. In the current environment of fiscal restraint, the ED will give utmost priority to meeting the coverage requirements stipulated in the 2006 Federal Accountability Act and the 2009 TB Policy on Evaluation.

The second challenge is the appropriate sequencing of evaluations so that their usefulness and value are optimized. The sequencing is determined by factors such as program priorities; materiality (importance in financial terms); and the contribution of the evaluation to decision- making processes (e.g. renewal of the programming strategy or replenishment of funding to partner organizations and institutions). The sequencing of evaluations over a five-year cycle is outlined in Table 2.

Based on the Agency’s corporate risk profile, a preliminary risk assessment has been undertaken^{Footnote 11} to assess the operational, financial, reputational, and development risks of undertaking a specific evaluation. The ED’s risk assessments, which are listed in Annex 1, paid particular attention to stability, level of security, and complexity.

4.2 Evaluation Approach for CIDA’s Programs

Given the variety of CIDA’s programming channels, the ED has adopted a multi-pronged evaluation strategy:

4.2.1 Geographic Programs Branch (GPB)

The approach taken to cover geographic programs will continue to give priority to countries of concentration (priority level I). The Minister of International Cooperation has announced that CIDA will prioritize recipients by allocating 80% of its geographic program resources to 20 such priority countries. Country program evaluations in countries of concentration with greater political, social and economic stability (e.g., Vietnam, Mozambique, Senegal, Mali, Honduras) have been assessed as medium risk.^{Footnote 12}

For selected countries that are not countries of concentration, the ED proposes to conduct cluster evaluations at the sub-regional level in order to maximize the cost-effectiveness of evaluation resources. As this category also includes regional programs, an approach and methodology to undertake these evaluations will be developed, based on an examination of themes across these countries through a sample of projects, using various sources of information and evaluations that have already been undertaken by the Program Branches. For these selected countries of more modest investment, the risk is considered low to medium. The level of priority for these same countries is low to medium (priority level II and III).

4.2.2 Fragile States

Fragile states evaluations include consideration of the extent to which the program is aligned with the OECD/DAC Principles for Good International Engagement in Fragile States and Situations. These principles direct international interventions in fragile states towards governance and state building for long-term stability. The Evaluation Division is developing generic TORs for program evaluations in fragile states that fully integrate the OECD/DAC principles and provide guidance on risk mitigation strategies. The ED considers fragile states evaluations a high priority (level I), given the large investment made in these programs. The level of risk to undertake program evaluations in fragile states has been assessed as high, given security concerns and/or the complexity of programming.

4.2.3 Multilateral and Global Programs Branch (MGPB)

MGPB provides support to a large number of international institutions and funds. The ED has developed, under the guidance of the DAC/EVALNET, a joint approach aimed at assessing the development effectiveness information from these organizations themselves, through their evaluation work and their performance reporting. The approach, which was endorsed by the DAC-EVALNET following two pilot tests, includes a common methodology, which allows greater coverage of more institutions over a shorter period, at less cost to each shareholder, and with greater leverage for affecting needed changes.

This approach is complementary to the work done by the Multilateral Organization Performance Assessment Network (MOPAN), which until 2011 focused mainly on organizational effectiveness^{Footnote 13}. The levels of priority and risk for multilateral organization evaluations is low to medium (priority level II or III).

4.2.4 Partnerships with Canadians Branch (PWCB)

In September 2010, the branch underwent a transformation and refocused its approach. The new branch, renamed “Partnerships with Canadians Branch,” is comprised of two overarching programs: "Partners for Development" and "Global Citizens."

Through these programs, PWCB supports development projects and programs that are designed and implemented by Canadian partner organizations, in co-operation with their counterparts in developing countries. PWCB also aims to engage Canadians and build knowledge around international development.

The ED has identified five evaluation units within the new PWCB organization and thematic focus, all of which are integrated within this Plan. These evaluation units correspond to the themes of PWCB's logic model. The five evaluation units are:

1. Governance;
2. Economic Growth and Environmental Sustainability;
3. Human Development;
4. Engaging Canadians; and,
5. Knowledge Creation and Sharing.

Country program evaluations will also cover investments by PWCB, in whole-of-agency assessments of CIDA program spending. Regarding the above-listed evaluation units, PWCB has been consulted on the order in which the units will be assessed. Given PWCB’s previous level of implementation and achievement of results in the area of governance, the first evaluation unit will focus on this thematic area. This evaluation will be implemented in FY 2012/2013 and will use decentralized evaluations and project reports generated by PWCB, among other sources of information. Although PWCB programming has limited financial implications, the level of risk and priority is considered medium due to high levels of public interest in Canadian partnership programming.

4.2.5. Corporate Evaluations

In addition to performance information at the program-level, the Agency also requires information on the performance of cross-agency initiatives such as policies, thematic issues and delivery mechanisms. Corporate evaluations include periodic meta-evaluations to assess the quality of program-led evaluations, and thematic evaluations, which can cut across various country programs and/or priority themes. Given the complexity and variety of situations emerging from corporate and policy evaluations, the level of risk has been assessed as medium. The priority for corporate evaluations is medium (priority level II).

4.3 Alignment between Centralized and Decentralized Evaluations

As mentioned above, decentralized (branch-led) evaluations are “building blocks” to centralized evaluations (ED-led). In addition, they complement centralized evaluations by providing a more detailed picture of the Agency’s performance.

The ED is moving towards establishing better alignment by: (1) including the evaluation plans of the program branches within the ED’s Rolling Five-year Evaluation Work Plan and, (2) providing technical advice, coaching, and ensuring that decentralized evaluations are carried out in accordance with the OECD-DAC’s 2010 Quality Standards for Development Evaluation.

4.4 Commitments in Treasury Board Submissions

Over the past few years there have been several evaluation commitments made as a result of Risk and Results-Based Management and Accountability Frameworks (RRMAF) and TB Submissions for the approval of major initiatives. Many of these initiatives are program sub- components, which could have been covered through the respective program evaluations, instead of separate evaluations –as per commitments made to TB. These evaluation commitments may be fulfilled through existing planned program evaluations, rather than conducting separate evaluations. The level of risk of such evaluations depends on the country, program and institution under consideration. The numbers of these evaluations are expected to decline in the future. The ED will finalize discussions with TBS to facilitate this objective.

4.5 Optimizing the Learning / Knowledge Benefits from Evaluations

Evaluations at CIDA create a source of credible and neutral information, which feeds into strategic reviews, renewals of terms and conditions, Treasury Board submissions, the Report on Plans and Priorities, the Departmental Performance Reports, and memoranda to Cabinet.

Though CIDA has traditionally benefited from an evaluative culture, there increasingly remains a need to better optimize the learning and knowledge benefits from evaluations. The Agency’s strategy and work plan for the dissemination of evaluation knowledge will increase the visibility, accessibility and use of evaluation findings and lessons, both within and outside the Agency. An example is Agency committees, which can be a useful venue for discussing evaluation results and promoting learning across the Agency.

5.0 Evaluation Division Priorities for FY 2012/2013

Based on the evaluation context, the lessons and challenges from the previous year, and the considerations outlined in Section 4.0, the evaluation work plan (FY 2012/2013 –FY 2016/2017) will focus on the following broad areas:

1. Evaluations planned for FY 2012/13;
2. Provision of advice, support and evaluation quality assurance services to program branches;
3. Dissemination and use of evaluation results; and
4. Governance structure and strategic alliances.

5.1 Evaluations Planned for FY 2012/2013

5.1.1 Countries of Concentration and Regional Programs

In selecting countries to be the evaluated, the ED will give priority to the countries where CIDA has concentrated its efforts. These countries have received significant investments over the past five years. In FY 2012/2013, the ED will finalize the Mali, Colombia and Caribbean regional program evaluations, which were initiated in FY 2011/2012, and will launch and conduct country program evaluations in Tanzania, Bolivia, Indonesia and Pakistan.

5.1.2 Fragile States

The ED will undertake evaluations of the Afghanistan and Haiti programs, as per commitments made to the TB. These countries constitute a high priority given the importance they have for the Canadian government’s policy agenda and the magnitude of resources invested over the past five years.

Conducting evaluations in these countries is considered a very complex undertaking given the level of insecurity and instability and the number of actors involved (including CIDA, Foreign Affairs and International Trade Canada, the Department of National Defence, etc.). In preparation for the Afghanistan program evaluation, the ED completed a scoping/evaluability assessment, which determined the scope of the evaluation, and prepared TORs for the evaluation. The Afghanistan program evaluation will be implemented over two fiscal years, starting in July 2012 with a planned duration of 15 months. The evaluation of CIDA's Program in Haiti will be implemented during 2012/13, and the final report will be produced early in the following fiscal year.

5.1.3 Multilateral Programs

Multilateral partners are excellent candidates for joint evaluation with other bilateral shareholders, many of whom share Canada’s interests in finding out more about their effectiveness. In FY 2012/13, CIDA will conduct reviews of the African Development Bank and UNICEF, and finalize reviews of the Asian Development Bank and World Health Organization. The UNICEF review will be conducted jointly with the Netherlands, using the DAC/EVALNET common approach. Given the close linkage between the joint approach and the work of the MOPAN, the ED will continue to collaborate with the MGPB to maximize complementarities, avoid duplication of efforts, and consider how to institutionalize the approach.

The ED will continue to offer assistance and provide evaluation services to CIDA’s large multinational partners. For example, the ED will continue to comment on our multilateral partners’ evaluations policies and plans; evaluation mandates; TORs; and, evaluation reports and management responses, while continuing to participate in multi-donor evaluation committees.

5.1.4 Partnerships with Canadians Programming

In FY 2012/2013, the ED will conduct the first evaluation of partnerships with Canadians programming, following its modernization in 2010. This program evaluation will focus on PWCB’s governance programming, and serve to inform related thematic programming across CIDA.

5.1.5 Corporate Evaluations and Initiatives

In FY 2012/2013, the ED will conduct a meta-evaluation of branch-led evaluations, and complete a report on the state of performance measurement in the Agency. In addition, the ED will continue to provide advice in relation to the thematic evaluation strategy of the Muskoka Initiative.

5.2 Advice and Support to Program Branches

In FY 2012/2013, the ED will continue to provide evaluation quality assurance services, advice and support to program branches on ongoing evaluations, and other related needs. The ED will also develop new tools to ensure that CIDA’s evaluations properly assess gender equality performance and develop guidance documents for the implementation of project-level evaluations. In addition, the ED will help program branches streamline their evaluation plans and better align them with the ED Rolling Five-Year Evaluation Work Plan.

Following the deficit reduction action plan, a number CIDA programs will be closing. As a result, there will likely be an increase in demand for the ED to provide evaluation services and support to program branches. Additional resources may be required to support this demand.

5.3 Corporate Initiatives

The ED will also continue to provide support and input to corporate level initiatives, such as the Departmental Performance Report; the Report on Plans and Priorities; and, the Management Accountability Framework (MAF). The ED will also provide advice and recommendations for the review of proposals for grants and contributions under the authority of the SPPB. Moreover, the ED will continue to participate in various committees and working groups (e.g. Program Committee) that offer opportunities for the knowledge from ED’s evaluations to feed back into the policy and planning processes of the Agency.

5.4 Dissemination and Use of Evaluation Results

The ED will continue to implement the Agency’s strategy for the dissemination of evaluation knowledge, which seeks to increase the visibility, accessibility and use of evaluation results and lessons, both within and outside the Agency. The first phase of the implementation was initiated during the fiscal year 2011/2012 with a focus on evaluation knowledge from ED-led evaluations and on dissemination efforts within the Agency.

In 2012/2013, the ED will continue to phase in the implementation of the dissemination strategy. The first phase focused on evaluation knowledge from ED-led evaluations and on dissemination efforts within the Agency. The second phase will involve expanding the work on dissemination of evaluation knowledge to program-led evaluations and spreading dissemination efforts more widely to reach audiences outside the Agency. Priority will be given to the dissemination of evaluation knowledge within CIDA with a focus on improving CIDA’s planning, programming and performance.

Based on the accompanying work plan, one of the key deliverables for fiscal year 2012/2013 will be the preparation of the annual report on CIDA’s evaluation function for dissemination of evaluation knowledge from ED- and branch-led evaluations. The report will present key lessons learned from ED- and branch-led evaluations conducted during the previous fiscal year. Specifically, the report will contain an overview of key lessons learned from evaluations, improvements via ED-led evaluations, corporate evaluation resources, and one-to-two pagers on each ED-led and branch-led evaluation included in the analysis for the report, describing purpose, findings and lessons learned.

5.5 Governance Structure and Strategic Alliances

5.5.1 Support to the Evaluation Committee

The ED, serving as the Secretariat to the EC, will continue to dedicate the necessary human, technical and financial resources to ensure that the EC functions efficiently and effectively.

5.5.2 Building and Maintaining Strategic Alliances

Outside the Agency, the Canadian evaluation community of practice consists of the TBS Centre for Excellence in Evaluation (CEE), heads of evaluation from other government departments and the Canadian Evaluation Society. Internationally, the ED is a member of the DAC Evaluation Network (DAC-EVALNET), the Nordic Plus sub-group inside the DAC-EVALNET, and the Multilateral Organization Performance Assessment Network (MOPAN). CIDA has been particularly active within the DAC-EVALNET as a champion of the common approach for assessing the development effectiveness of multilateral organizations.

Liaising with these communities of practice is important to maintain and strengthen Canadian leadership in evaluation, and provides the ED with sources of best practices and lessons learned. Additionally, it fosters opportunities for joint evaluations with other donors on initiatives such as the evaluation on general budget support; the DAC initiative on multilateral effectiveness; the joint evaluation of the Paris Declaration; and the International Initiative for Impact Evaluation (3IE).

6.0 Program Branches’ Evaluation Plans for FY 2012/2013

As described in previous sections, CIDA maintains a dual evaluation function (centralized and decentralized). The centralized evaluation function, maintained by the ED, focuses on corporate (Agency-level) and program-level evaluations. The decentralized evaluation function, in the program branches, conducts evaluations to support decision-making at the program and project level (i.e. sub-sub-sub activity level). As the functional lead for evaluation across the Agency, the ED provides tools, advice, guidance, training and referral of evaluation expertise to program branches on their decentralized evaluations.

The following section describes the evaluation activities planned by the branches during the FY 2012/2013 period. Information for this section was provided directly by each respective program branch. The ED recognizes that not all branch-led evaluation activities can be classified as evaluations, as per the TB Policy on Evaluation. However, branch-led activities complement the ED’s evaluation work by providing performance assessments, monitoring data, and other relevant information.

6.1 Geographic Program Branch (GPB) – Decentralized Evaluations Planned for FY 2012/2013

GPB has approximately 30 branch-led evaluations currently scheduled for FY 2012/2013. These evaluations will include consideration of project relevance, efficiency, effectiveness, and sustainability. While GPB work plans may be modified prior to the start of the fiscal year, the list below provides provisionally planned evaluations.

6.2 Partnership with Canadians Branch (PWCB) – Decentralized Evaluations Planned for FY 2012/2013

PWCB started several thematic evaluations in FY 2011-12, which will be completed and disseminated in FY 2012/2013. The thematic evaluations are: cooperatives; water and sanitation; sports for development; and, the volunteer cooperation program.

In FY 2012/2013, PWCB will initiate thematic evaluations on Basic Education and the Short- Term Volunteers Model.

6.3 Multilateral and Global Programs Branch (MGPB) – Decentralized Evaluations Planned for FY 2012/2013

At the end of FY2011/12 MGPB completed a mid-term formative evaluation of the Micro- Nutrient Initiative. This evaluation provided performance information for the Micronutrient Initiative to fulfill a Treasury Board Submission commitment. In FY 2012/2013, MGPB will also implement a mid-term evaluation of the $120 million TB Reach Program, which is housed within the Stop TB Partnership of the World Health Organization. No linkages have been identified between these decentralized evaluations and the corporate evaluations that will be led by the ED in the upcoming fiscal year.

7.0 Resource Estimates and Challenges

7.1 Non-Salary O&M

As shown in Table 2 at the end of this document, the Plan anticipates a budget of approximately $2.2 million for FY 2012/2013 to implement the proposed plan. This amount covers the costs of engaging external professional expertise, as well as other non-salary administrative costs such as staff travel, translation and printing. In order to achieve cost savings while meeting evaluation coverage requirements, the Plan takes into account that the UNICEF review will be conducted jointly with the Netherlands. Additionally, the ED will strive to improve coordination between centralized and decentralized evaluations in an effort to maximize resource efficiency.

7.2 Salary Budget

The division now has a base salary budget of approximately $1.4 Million and a staff complement of 18 FTEs: the Director, 16 evaluators and 1 support staff.

7.3 Challenges

The Evaluation Division recognizes that the results of the deficit reduction action plan may have an impact on the resources necessary to deliver this Plan. Therefore, this Plan may be altered to account for a different scenario. Options may include proposing alternative implementation approaches or reducing the scope of planned evaluations.

CIDA Evaluation Initiatives (FY 2012/2013 – FY 2016/2017)

Notes:

• Only initiatives for FY 2012/2013 are costed. Amounts include all non-salary O&M (such as professional services, printing, travel etc.).
• The Total Program Level of Expenditures amount reflects investments from FY 2006/2007 to FY 2010/2011. It does not include expenditures from closed programs.
• FY 2011/2012 is retained in the table order to provide a more comprehensive picture of the evaluation universe. However, it is understood that the 5-Year Rolling Plan includes FY 2012/2013 to FY 2016/2017.

Codes:

• E = Led by Evaluation Division. (Evaluation activities for the upcoming FY are highlighted in green)
• S = Led by Program Branches and supervised (quality-controlled) by the Evaluation Division
• D = Dissemination / knowledge sharing.
• C = Support to the Corporate level (Evaluation activities for the upcoming FY are highlighted in blue.)

Evaluation Commitments to TBS^{Footnote 16}

The table below presents only specific evaluation commitments indicated in the TB Submissions that were approved between FY 2006/2007 and FY 2010/2011. It excludes evaluation commitments that have already been fulfilled.

Annex 1: Establishing the Evaluation Universe

Taking into consideration the PAA, the Performance Measurement Frameworks approved by TBS, and the commitments made in TB Submissions, the ED arrived at a tentative Evaluation Universe of 65 units for coverage during the next five (5) years:

• 5 evaluation units related to Fragile Countries and Crisis-Affected Communities;
• 14 evaluation units related to Low-Income Countries;
• 15 evaluation units related to Middle-Income Countries;
• 21 evaluation units related to Global Engagement and Strategic Policy;
• 5 evaluation units related to Canadian Engagement;
• 5 evaluation units related to Corporate Initiatives; and,
• specific commitments to the TB (see Figure 2), which will be covered as part of our Program and Corporate Evaluations.

The number of units in the evaluation universe have been reviewed and adjusted in the Plan, based on CIDA’s PAA, which was approved by TB on September 15, 2011.

Table 4 identifies the evaluation units stated above. Given the large number of evaluations to be completed over the next five years, the ED is exploring the possibility of a division of labor between CIDA and partner organizations. Some of the evaluations will be conducted jointly with other partners (Geographic and Multilateral Programs) while others (Partnerships) will necessitate the development of new methodologies to review the new organizational structure.

In the case of multilateral organizations, the ED will be using the common approach endorsed by the DAC-EVALNET.

Following the deficit reduction action plan, CIDA will need to, in consultation with TBS and legal services, determine its evaluation obligations in regard to the following:

1. Closing the Cambodia, China, Malawi, Nepal, Niger, Rwanda, Zambia and Zimbabwe programs.
2. Country programs, which are facing reduction, particularly: Bolivia, Pakistan, Mozambique, Ethiopia, Tanzania, and South Africa.
3. Regional programs that are consolidating or reducing; in particular African programming will be consolidated into the pan-African regional program, and the Southeast Asia regional program will be reduced.
4. Multilateral and global programs, which are being reduced or consolidated.
5. Partnerships with Canadians programming that will be reduced or consolidated.