Evaluation of Global Affairs Canada’s Partners for Development Program – FY 2010/11-2013/14

September 2016

Acknowledgments

We would like to acknowledge the work of the team of consultants from Universalia Management Group who conducted this evaluation: Marie-Hélène Adrien (evaluation team leader), Halcyon Louis, Eric Abitbol, Jérôme Gandin, Mia Choinière, and Lorenzo Daîeff.

The Development Evaluation Division would also like to thank all those who contributed to this evaluation. This includes Partnerships for Development Innovation Branch, in particular the Impact and Analysis group, who provided support throughout the process as well as Country Program staff at headquarters and in the field who provided documentation and participated in interviews during field visits in Bangladesh, Burkina Faso, and Ethiopia. Many other stakeholders made themselves available for meetings, interviews and project site visits, including Government officials, donor representatives, implementing partners, civil society organisations, and individual consultants who were generous with their time and provided the evaluation team with valuable insights into their programs and projects.

From the Development Evaluation Division, the evaluation was managed by Sandra Gagnon, and supervised by Tara Carney.

David Heath
Head of Development Evaluation

List of Acronyms and Abbreviations

APP

Authorized Programming Process

CCIC

Canadian Council for International Co-operation

CIDA

Former Canadian International Development Agency

CSO

Civil Society Organisation

DAC

Development Assistance Committee (of OECD)

DFAIT

Department of Foreign Affairs and International Trade

DFATD

Foreign Affairs, Trade and Development Canada

FRET

Fiduciary Risk Evaluation Tool

FMA

Financial Management Advisor

FY

Fiscal Year

GAC

Global Affairs Canada

Gs&Cs

Grants and Contributions

KFM

Partnerships for Development Innovation

n.d.

No date available

NGO

Non-governmental Organisation

MFM

Global Issues and Development

MNCH

Maternal, Newborn and Child Health

ODA

Official Development Assistance

OECD

Organisation for Economic Cooperation and Development

PMF

Performance Measurement Framework

RBM

Results Based Management

RMNCH

Reproductive, Maternal, Newborn and Child Health

SCM

Corporate Planning, Finance and Information Technology (Chief Financial Officer)

SDGs

Sustainable Development Goals

SGD

Grants and Contributions Management

Executive Summary

This report presents the findings, conclusions, lessons and recommendations of the evaluation of Global Affairs Canada’s Partners for Development Program during the 2010/11 to 2013/14 period, including disbursements made under the Calls for Proposals mechanism.^{Footnote 1} The evaluation was conducted from March 2015 to January 2016.

Rationale, Purpose and Objectives

The evaluation was undertaken for learning and accountability purposes in order to support program improvement and provide an evidence-based assessment of the Partners for Development Program’s relevance, effectiveness, efficiency and sustainability in achieving its expected results.

The objectives of the evaluation were to:

• i. Examine the results achieved by Partners for Development program between 2010/11 and 2013/14 in relation to: relevance, effectiveness, efficiency, sustainability, crosscutting themes, and performance management;^{Footnote 2} and
• ii. Assess findings, conclusions and lessons, and formulate recommendations for program improvement.^{Footnote 3}

Partners for Development Program Context

The Partners for Development Program (the Program)^{Footnote 4} was established in July 2010 under the Department’s Partnership Modernisation and Effectiveness Framework, to support the engagement of Canadian development actors. This Framework aimed to concentrate 50% of the funding available to Canadian development actors (hereafter referred to as Canadian partners) in 20 focus countries. The Program provided funding for the most meritorious proposals submitted by Canadian partners of all sizes to create enhanced opportunities for focused, results-oriented development programming. Beginning in 2010, projects were selected through a Calls for Proposals mechanism, which was used to increase transparency during project selection.

Methodology

The Program was evaluated through a utilisation-focused approach that incorporated mixed methods and participatory techniques. Evaluation activity was conducted over three phases: inception; data collection; and data analysis and reporting. Data was collected through stakeholder consultations; document review; a portfolio review of projects; electronic survey; and country visits. A purposeful, non-randomised sample of 50 out of the 322 projects funded by the Program during the evaluation period was selected. A workshop to reconstruct the Program’s Theory of Change was held on April 2015.

Main Messages

Overall, the quality of the partnership between Global Affairs Canada and Canadian partners improved over the evaluation period. There were, however, questions around the flexibility of some operational modalities used by the program (e.g. the use of Calls for Proposals), as well as the capacity of the Program to implement practices that could contribute toward the achievement of results and sustainability. Improvements are required in these areas in order to help increase the Program’s capacity to achieve its expected results, both at the level of outputs and outcomes. In general, the Program contributed towards the achievement of the Aid Effectiveness Agenda of the Canadian government, and demonstrated relevance to the needs of partner countries in its programming. Program activities enabled the Branch to steer partners towards key Departmental priorities, encouraged creativity in project design and allowed for the selection of the most meritorious projects.

The Program leveraged Canadian resources for North-South technical cooperation, supported the development effectiveness agenda of the Government of Canada, promoted partnerships between Government and Canadian civil society, and supported the maintenance and development of North-South partnerships, and contributed to the achievement of sustainable development results, and the provision of humanitarian aid.

The Program was relevant to both Canadian priorities and partner countries’ needs. The management and implementation of the Program was strategically aligned with planning at the Branch level, the Department level, and with Canada’s Aid Effectiveness Agenda. The Program was also relevant to fulfilling the needs of partner countries and recipient communities in terms of the implementation of results-oriented development programs.

Partnerships between Canadian partners and their country level counterparts included diverse partners (non-governmental organisations, universities, private sector, etc.) and reflected good practices. The quality of the partnership between the Branch and the Canadian partners declined with the initial introduction of the Calls for Proposals mechanism.

The achievement of expected Program-level results was mixed. While there was evidence of the achievement of outputs and immediate outcomes, there was limited information on the achievement of intermediate and ultimate outcomes at the Program-level. The wide variety of project types, objectives and indicators, and the finite reporting capacity of implementing partners, particularly local implementing partners which were already struggling with requirements to present detailed results data according to Program reporting guidelines were some of the practical constraints to aggregation of results at the Program level.

The evaluation identified a number of enabling factors (e.g. long-standing partnerships and strong project management capacities of local implementing partners) that facilitated the achievement of results. Conversely, it also identified factors that impeded results achievement (e.g. short timelines of project activities). Variations in levels of project effectiveness were not easily explained, and could not be attributed to any one factor.

For the period under review, the integration of crosscutting themes across the Program was mixed. Despite a successful integration of gender equality and environmental sustainability as crosscutting themes at Program and project levels, the integration of governance at project level lacked of clarity and consistency because it was understood differently by different partners.

The Program experienced delays in project approval. Reasons for delays were often multi-pronged, from technical issues in the proposal process, to the decision-making process. Human resource capacity, among others, was also insufficient to address the administrative requirements of the Calls for Proposals mechanism.

The Program added value to Canada’s development agenda though its capacity to respond to the needs of reach underserved populations in partner countries and by complementing bilateral aid programming. This agenda has multiple levels and the Program contributed to the potential achievement of development results in developing countries, and engaged Canadian civil society in development programming. Through the support provided to small projects, the Program also created an opportunity for communities to benefit from innovative ideas and best practices.

Although Canadian partners and local implementing agencies demonstrated a commitment to achieving development results, the Program was challenged in terms of sustainability at the project-level. There was a lack of a clear strategy for ensuring the human resource and financial sustainability of projects, including the capacity of local implementing agencies to access project resources from local sources.

The design of the Program was enhanced by the emphasis on partnership with civil society organisations, and a strong rationale was provided for working with and through civil society organisations to achieve international development results. The logic model and a performance measurement framework are essential tools to allow a program to manage for development results. The absence of an approved logic model for the Program created a significant challenge for the assessment of intermediate and ultimate outcomes at Program level. The reconstructed Theory of Change was ambitious but contained inaccurate assumptions that affected the capacity of the Program to achieve its development results.

Another issue related to the design was that both Global Affairs Canada staff and Canadian partners indicated that project scale-up was rare because of a lack of communication between Global Affairs Canada’s different programming channels, and the lack of familiarity of bilateral program staff and Canadian staff in the field with Partnership programming.

Although knowledge-exchange and knowledge-sharing occurred during project-level implementation, there was less evidence of systematic knowledge management under the Program. Some examples of knowledge-exchange and knowledge-sharing were found at the project implementation level, but some local implementing agencies indicated that they believed project partners did not receive sufficient opportunities for learning.

Recommendations

Recommendation 1: Improve performance management by (a) reviewing the Program’s logic model and Theory of Change, and (b) integrating the guidelines for the crosscutting themes into its programming. In particular, it will be important to ensure robust monitoring and reporting on intermediate outcomes at Program-level.

Recommendation 2: Ensure that Canadian partners integrate exit plans and sustainability strategies that are adequately resourced into all project designs to support sustainable results.

Recommendation 3: Collaborate with Canadian partners to support systematic, as opposed to ad hoc, knowledge-transfer at the project level of the Program.

Recommendation 4: Improve communication between the different programming channels within Global Affairs Canada in order to facilitate the scaling up of projects.

Recommendation 5: Undertake efforts to limit those delays in project approvals that fall within the Program’s control. This may include reviewing and streamlining internal processes or by managing internal capacity better.

Management Response

Overview

The Development Evaluation Division of Global Affairs Canada undertook a corporate evaluation of the Partners for Development Program (2010/11-2013/14) over the past year. The evaluation employed a systematic methodological approach that included the review of a large volume of corporate, program and project documents, field visits in three developing countries relevant to the program, interviews with over a hundred and sixty stakeholders, as well as a survey of dozens of Canadian organizations in order to generate reliable data that informed the findings, conclusions and recommendations of the evaluation report. The evaluation draws many positive conclusions about the Program in terms of: 1) advancing Canada’s aid effectiveness agenda; 2) filling gaps in the development process by focusing on the needs of the poorest and most vulnerable communities; 3) bringing together a broad range of Canadian and Southern partners to support international development initiatives; 4) transferring skills and resources to help build the capacity of Southern partners; and 5) investment in gender equality. Of the five recommendations formulated in the report, all are accepted, including three with modifications.

The Partnerships for Development Innovation Branch (KFM) takes note of the performance gaps identified by the evaluators regarding the Partners for Development Program. Many of those gaps are similar to those found in other project and program evaluations of Global Affairs Canada development programming, most notably in the area of sustainability, results measurement, development synergy and other operational issues. While it is apparent that these issues are not necessarily unique to partnerships with Canadian civil society, as they are prevalent features of project management challenges in the wider field of international development, KFM management welcomes the opportunity to review and improve the performance of its programming. The recommendations identified by this evaluation give management further impetus to build on past achievements and address persistent gaps in all KFM programming, as reflected in the commitment and action items proposed in the table below. Moreover, as part of its change management exercise, KFM is already engaged in reflections on how to strengthen many of the areas identified. KFM will establish an internal working group - composed of at least one representative of each of its three Bureaus^{Footnote 5} - to oversee and monitor the implementation of this management response.

1. Introduction

This report presents the findings, conclusions, lessons and recommendations from the evaluation of Global Affairs Canada’s^{Footnote 6} Partners for Development Program for Fiscal Year (FY) 2010/11 to 2013/14.^{Footnote 7} The evaluation was conducted from March 2015 to January 2016. See Appendix II for an overview of the Program during the evaluation period.

1.1 Evaluation Rationale, Purpose and Objectives

In accordance with the Financial Administration Act, and the Treasury Board Policy on Evaluation (2009), the evaluation had a learning and accountability purpose in order to support policy and program improvement and to provide Canadians with an evidence-based assessment of the Program’s relevance, effectiveness, efficiency and sustainability in achieving international development results.

The specific objectives of the evaluation were:

• i. To examine the results achieved by the Program between 2010/11 and 2013/14 based on relevance, effectiveness, efficiency, sustainability, crosscutting themes, and performance management,^{Footnote 8} and
• ii. To assess and disseminate findings, lessons and conclusions, and formulate recommendations for program improvement.^{Footnote 9}

1.2 Evaluation Object and Scope

The object of the evaluation was the Global Affairs Canada’s Partners for Development Program (hereafter the Program). The scope covered program disbursements between FY 2010/11 and 2013/14, including disbursements made under the Calls for Proposals mechanism.

2. Methodology

The evaluation used a utilisation-focused approach that incorporated mixed methods and participatory techniques. It was based on the Organisation for Economic Cooperation and Development (OECD) - Development Assistance Committee (DAC) key evaluation criteria: relevance, effectiveness, efficiency and sustainability. These are consistent with the Treasury Board of Canada’s Policy on Evaluation. The evaluation focused on 33 evaluation questions included additional questions on partnerships, program design, performance measurement, and knowledge sharing. See Appendix IV for an overview of the evaluation methodology.

Evaluation activity was conducted over three phases: inception; data collection; and data analysis and reporting. Data was collected through stakeholder consultations at the Program and project levels; document review; a portfolio review of projects; an electronic survey; and country visits. Program staff based at the headquarters also participated in a Theory of Change Workshop on April 2015, which was designed to increase the ownership of the evaluation at the Program level in order to support forward-looking program improvements.

As the evaluation is not intended to establish generalisations based on statistical significance, purposeful sampling allowed the evaluation design to focus on pre-defined criteria and categories to maximise response quality to the key evaluation questions. The use of this strategic approach to sampling contributed to the utility of the evaluation for the identified end users. Based on the identified criteria – reflection of the various project categories, a range of sizes and values, Global Affairs Canada’s thematic priorities, projects meet evaluability criteria, specifically, adequate project documentation is available for review; they have reached a sufficient degree of completion, and other criteria identified by Global Affairs Canada as being relevant for project selection – a purposeful sample of 50 projects, out of a total of 322 projects funded by the Program, was selected using the approved sampling criteria. This formed the base for the portfolio review and country field missions to Bangladesh, Burkina Faso and Ethiopia.^{Footnote 10} The final selection of projects was based on a strategic approach to ensure that there was adequate variety in the sample to reduce evaluator bias and to increase the validity and reliability of data analysis results, and by extension, the main evaluation findings.

Stakeholder engagement through consultation and workshop participation was an important part of this evaluation. This engagement did not compromise the integrity of the evaluation process.

3. Context of the Partners for Development Program

The Partners for Development Program^{Footnote 11} was established in July 2010 under the Department’s Partnership Modernisation and Effectiveness Framework. The framework was introduced to support the engagement of Canadian development actors through a comparative assessment process. It aimed to concentrate 50% of available development funding to Canadian development actors (hereafter referred to as Canadian partners) in 20 focus countries.^{Footnote 12} Initiatives in other countries that were eligible for official development assistance (ODA), benefitted from the remaining 50% of development funding. The framework also enabled the alignment of program activities with the development goals of the international community. The Program design therefore reflected Canada’s intention to strengthen its alignment with the Paris Declaration on Aid Effectiveness (2005), and other global agreements on aid effectiveness and poverty reduction, in particular, the Accra Agenda for Action (2008) and the Millennium Development Goals (2000).

Beginning in 2010, projects were selected through a Calls for Proposals mechanism. The Partnership for Development Innovation Branch (the Branch) used a comparative assessment process to select projects through which effective development results could be obtained, and create increased transparency. Under this mechanism, the Program provided Canadian partners with equal access to information on available funding opportunities. During the evaluation period, the aim of the Program was to leverage the development expertise of Canadians to support delivery of development results on the ground and to contribute to poverty reduction. The Program provided funding for the most meritorious proposals submitted by Canadian development organisations of all sizes, to create enhanced opportunities for focused and results-oriented development programming.^{Footnote 13} Canadian partners selected for funding under the Program were required to work with one or more local implementing agencies based in developing countries.

Between 2010/11 and 2013/14, the Program disbursed approximately $924 million, which accounted for 94% of Branch disbursements. Eighty percent (80%) of the disbursements were allocated in response to unsolicited proposals by Canadian partners before the Partnership Modernisation and Effectiveness Framework was implemented. The remaining 20% of program disbursements were allocated to projects selected under the new Calls for Proposals mechanism.

In 2010, the Branch launched Calls for Proposals mechanism with five categories:

• i. Envelope for small projects (under $2 million);
• ii. Envelope for large projects (over $2 million);
• iii. University Partners envelope;
• iv. Volunteer Cooperation envelope; and
• v. Knowledge Partners envelope.

A total of seven Calls for Proposals were issued during the evaluation period, three of which were from the five categories listed.^{Footnote 14} The other four Calls for Proposals were Special Calls, issued in response to emerging development needs and Canada’s development agenda (see sidebar). During the same period, a total of 124 projects, totaling $355 million, were selected and implemented through the Calls for Proposals mechanism in 58 developing countries.

4. Findings

4.1 Relevance

The evaluation assessed the relevance of the Program in relation to Canada’s Aid Effectiveness Agenda, Departmental objectives and policies, and partner country national priorities at the project-level.

Finding 1: The Program provided strategic support to realizing Canada’s Aid Effectiveness Agenda.

The management and implementation of the Program was strategically aligned with three levels of planning: Branch level, Department level, and with Canada’s Aid Effectiveness Agenda. As part of the evaluation, Canadian partners that were funded between 2010/11 and 2013/14 were asked to participate in an online survey. Over 70% of the survey respondents (29 out of 40 respondents) indicated that the Program was relevant to the mandate and needs of their respective organisations, as aligned to the strategic objectives of the Branch, and by extension, Global Affairs Canada.

The Program was aligned to Canada’s five thematic priorities for the International Assistance Envelope: i) increasing food security; ii) securing the future of children and youth; iii) stimulating sustainable economic growth; iv) advancing democracy; and v) promoting stability and security^{Footnote 15} (80% of Partners for Development programming was intended to focus on the first three thematic priorities listed). The Program also adhered to the Official Development Assistance Accountability Act (2008), which mandated all official development assistance to focus on aid effectiveness and poverty reduction principles in a manner that is consistent with Canadian values, such as global citizenship, equity, and environmental sustainability.^{Footnote 16}

Within the global context, Canada adhered to several international development agreements for aid effectiveness and poverty reduction.^{Footnote 17} In order to comply with Global Affairs Canada’s commitments, the Program funded projects that were aligned with Canada’s Aid Effectiveness Agenda (2007). As part of the evaluation process, the review of projects selected for funding confirmed that all projects adhered to the principles of the 2008 Accra Agenda for Action.

The Program continued to be relevant to Global Affairs Canada’s global development efforts in the post-FY 2013/14 period. As part of Canada’s support for the Global Partnership for Effective Development Cooperation, Global Affairs Canada launched a Civil Society Partnership Policy in February 2015 that outlined its guiding principles for partnerships with Canadian development partners to alleviate global poverty and deliver humanitarian assistance.^{Footnote 18} Global Affairs Canada’s priorities and crosscutting themes, which were reflected in the Program, were also thematic areas of focus within the global Sustainable Development Goals, as established by the international development community in September 2015.^{Footnote 19}

Finding 2: There was close alignment between projects funded through the Program and thematic priorities within Canada’s Aid Effectiveness Agenda.

As noted above, in 2010/11, a new funding modality, the Calls for Proposals mechanism, was introduced to support project selection for the Program. One of the assumptions of this new process was that a focused Call for Proposals would ensure that projects would be increasingly aligned with Global Affairs Canada’s thematic priorities and countries of focus (see section 4.7, Design, for a critical review of the Calls for Proposals mechanism). At the project level, multiple lines of evidence showed a strong alignment between projects selected under the Calls for Proposal mechanism and Canada’s Aid Effectiveness Agenda.

The new process established a target of 50% of funding to Global Affairs Canada countries of focus by 2012/13 (with the remaining 50% being allocated to ODA-eligible countries where Canada had a presence), and a target of 80% of programming to three thematic priorities of the Department (food security; children and youth; and sustainable economic growth).

The target of increased alignment between Partners for Development programming and the Canadian Aid Effectiveness Agenda was successfully achieved. The portfolio review^{Footnote 20} provided evidence of geographic and thematic alignment between project selection and Global Affairs Canada’s strategic objectives. The strategic alignment of the project selection process was also reflected in the reconstructed Theory of Change for the Program,^{Footnote 21} as developed during a Theory of Change workshop at Global Affairs Canada. Included among the factors required for the Program to achieve its ultimate outcome, the Canadian International Development and Policy Objective (see Reconstructed Theory of Change, appendix IV) was the allocation of at least 80% of Program funds to Global Affairs Canada’s development thematic priorities.^{Footnote 22}

The Program exceeded its target of allocating 80% of its funding to projects developed around its three thematic priorities of focus, as shown in Table 1.

Source: Partnerships for Development Innovation Branch, Internal Document, n.d.

Finding 3: Projects funded through the Partners for Development Program were relevant to the needs of recipient countries and communities.

The Program was designed to foster partnerships between Canadian partners and local implementing agencies in developing countries for the implementation of results-oriented development programs. A central feature of the Program was North-South collaboration, built on the exchange of knowledge and expertise, to support sustainable development results. Canadian partners and local implementing agencies included a range of civil society organisations, such as non-governmental organisations, non-profit and voluntary-driven organisations, universities, foundations and professional associations, who may also have collaborated with government agencies or the private sector during project implementation.

The evaluation found that applications to the Program were required to provide information on the relevance of project results at the country-level. The portfolio review identified examples of alignment between project objectives and country-level investment in the social sector (in particular, national poverty reduction strategies). For example, in both Ethiopia and Zimbabwe, a CARE-led project, the African Regional Nutrition Initiative, supported national objectives related to health and nutrition, specifically the roll-out and scale-up of national-level programs.

In order to meet the Program’s strategic objectives, the main criterion used to select projects for funding was evidence of a partnership between Canadian partners and local implementing agencies to support the achievement of development results in project countries. The review of project proposals indicated that applications were required to confirm whether local implementing agencies had conducted a needs assessment within the communities of the project’s intended beneficiaries. This information was used to determine community need, as well as existing capacities within communities to support the implementation of project activities. One example of a community needs assessment is seen in the Disability, Maternal and Child Health project implemented by the Centre for Rehabilitation of the Paralysed (with support from Queen’s University).

Specifically, the portfolio review assigned relevance scores based on a combined assessment of several criteria that included: relevance to country/context, relevance to Global Affairs Canada strategic priorities, relevance to Aid Effectiveness Agenda, relevance to beneficiaries, and complementarity of the Program to other donors’/government’s efforts. Of the 50 projects reviewed, 34 earned a rating of high/substantial on relevance; 16 earned a rating of modest/low. The overall average for relevance across all projects was close to “substantial” (see appendix VIII). Based on the analysis of the portfolio review, interviews, and fieldwork, the following qualitative observations regarding projects’ relevance can be added:

• Projects appeared to be highly relevant and aligned to Global Affairs Canada’s strategic objectives (a direct consequence of project selection mechanisms); most interviewees and survey respondents concurred.
• Projects were usually better-aligned with Global Affairs Canada priorities, and/or with local partner priorities, than with national government priorities (e.g. Poverty Reduction Strategy Papers) – or at any rate the link was not always made explicit. That being said, a select number of projects cooperated closely with local/national authorities.^{Footnote 23}
• The concepts of ‘needs assessments’/‘baselines’ were not always clear, as evidenced by the different standards of interpretation across projects. In many cases, needs assessments were understood as the Canadian partner consulting local partners (who were assumed to be in tune with beneficiary needs), rather than a formal (and participatory) assessment of the needs of beneficiaries. Interviewees suggested, however, that Global Affairs Canada was tightening the guidelines on needs assessments/baselines by the end of the evaluation period.
• The evaluation did not find projects that were clearly irrelevant to beneficiaries. However, field missions visited projects that were of ambiguous relevance (i.e. projects which catered to a need that were not necessarily a priority, or which did not appear to have significantly improved the situation), or projects which did not appear to have reached the poorest of the poor (e.g. urban education projects).
• There was limited evidence of coordination/alignment with other donors, or with other development work in the area.
• There was only partial evidence of the Program funding complementing other Global Affairs Canada funding channels (as many interviewees suggested). However, some interviewees also suggested that this problem had been partially recognized by the time of the evaluation, and steps were being taken to address it, including resolving the disconnect between headquarters and the field, which may have had a limited knowledge of partnership programming.

4.2 Partnership

The evaluation identified several types of partnerships that had the potential to affect Program implementation. Based on the centrality of the following two types of partnerships to results achievement at Program and project levels, evaluation findings were derived from an analysis of the partnership agreement between: 1) Global Affairs Canada and the Canadian partners; and 2) Canadian partners and the local implementing agencies.

Finding 4: The partnership between Canadian partners and their counterparts at the country level reflected good practices.

Project implementation under the Program benefitted from good partnerships at the project level. Consultations with Canadian partners and local implementing agencies provided examples of projects that adhered to several good practices for development partnerships (see sidebar).^{Footnote 24} For example, the implementation of the ADRA Securing Mothers’ and Infants’ Lives with Equity Project in Cambodia (2012-2015) benefitted from collaboration between the local implementing agency, a local non-governmental organisation and the Provincial Health Department. It drew on established practices with the creation of Mother and Child Health Groups to share information on reproductive health and child development issues, and led to the development of a pilot community based transportation system for mothers and children, for which high satisfaction levels were reported.

The evaluation found that partnerships between Canadian partners and local implementing agencies were diverse and included partners drawn from international aid and development organisations, coalitions, independent non-governmental organisations, universities and professional associations. In some instances, partners also collaborated with the private sector or government agencies in project countries, to support the implementation of project activities.

Many partnerships were also based on a long history of collaboration (on average two to four decades of collaboration) between specific Canadian partners and local implementation agencies. Out of 50 projects reviewed, the evaluation identified only two examples (4% of all projects reviewed) of challenging partnerships. These challenging partnerships were affected by delays to project implementation and an inability to achieve their development results.

Finding 5: The introduction of the Calls for Proposals mechanism increased the neutrality of the process, but weakened the quality of the partnership between the Branch and Canadian partners.

Prior to the introduction of the Calls for Proposals mechanism,^{Footnote 25} Canadian partners submitted unsolicited proposals for consideration. They received support from Program staff, in the form of open consultations, as well as training and coaching during proposal development. The introduction of the Calls for Proposal mechanism was designed to ensure that administrative processes were more streamlined, and to support increased efficiency and transparency during the application through project selection and funding processes (see table 2 for more details). As confirmed through consultations with Program staff and Canadian partners, before the Calls for Proposals mechanism was introduced, Program staff were more accessible to funding applicants and provided clarification during the proposal development process.

The accessibility of Program staff to potential applicants was reduced between FY 2010/11 and 2011/12, in order to ensure that desired levels of transparency and integrity during project selection were maintained.^{Footnote 26} As a result, the perceived quality of the partnership between the Branch and Canadian partners, including new and long-standing partners, declined during this period according to interviews and focus groups. The survey conducted in October 2015 as part of this evaluation indicated that Canadian partners were “somewhat dissatisfied” with the Calls for Proposals mechanism. Specifically, 64% of respondents somewhat or fully disagreed with the statement that their organisation was “satisfied with the pace of processes for selection and approval of projects under the Calls for Proposals mechanism”. An additional 51% of respondents stated that the workload of their organisations had increased because of the reporting requirements of the new mechanism.

Similar findings emerged from a 2014 Canadian Council for International Co-operation (CCIC) survey of Canadian civil society organisations (CSOs), which was conducted to assess CSO experience with the Calls for Proposals mechanism. According to this survey, the “once stable institutional funding relationship between CIDA/DFATD and civil society has been seriously undermined for all organisations, including those with a long historical relationship.” The CCIC survey results also indicated that the Calls for Proposals mechanism affected the capacity of Canadian partners to maintain their organisations, and sustain programming and partnerships in project countries.

Sources: Multiple sources – Document review; Stakeholder consultations

4.3 Effectiveness

The evaluation based its assessment of effectiveness on the extent to which the Program achieved the objectives of its draft logic model. As the Program logic model was not approved at the branch or Departmental-level, the evaluation also drew on the reconstructed Program Theory of Change, which was informed by a workshop with Program staff.^{Footnote 27}

Finding 6: Results achievement at the Program level was mixed. There was strong evidence of the achievement of outputs and immediate outcomes, and negligible evidence of the achievement of intermediate outcomes.^{Footnote 28}

The Program was successful in reaching its expected outputs during the evaluation period with close to 100% of sampled projects achieving high results. The project portfolio review indicated that immediate outcomes were achieved by one-third of the projects reviewed. However, the Program did not aggregate outputs and immediate outcomes at the Program level. As a result, the Program was not able to report on the intermediate and ultimate outcomes of the projects.

There were practical constraints to aggregation of results at the Program level given the wide variety of project types, objectives and indicators, and given the finite reporting capacity of implementing partners, particularly local implementing partners which were already struggling with requirements to present detailed results data according to Program reporting guidelines. In addition, reporting on intermediate and ultimate outcomes was constrained by the existence of broad Program outcomes that occur months or years after projects ended. As such, in the absence of human and financial resources for tracking results achievement, the responsibility for monitoring and reporting on project intermediate outcomes remained unclear. It was therefore very difficult for partner organisations to establish a link between outputs and intermediate outcomes achieved, and broader outcomes which were often tied to national-level indicators, over which the small projects funded by the Program would have had very little influence.

In the annual Departmental Performance Report, for example, reporting on the Program focused typically on perception-based data (such as the level of satisfaction with progress achieved), but did not present results at the Program level. Figure 1 provides some examples of results achievement under the Program.

Figure 1: Examples of Results Achievement

Finding 7: The overall effectiveness of the Program was affected by several factors that enabled or impeded project-level results achievement. Variations in levels of project effectiveness were not easily explained and could not be attributed to any one factor.

Findings from the portfolio review and field missions to Bangladesh, Burkina Faso and Ethiopia identified a number of enabling factors that facilitated the achievement of results. These included strong political support for the projects and a long history of collaboration between project partners. Factors that impeded progress towards expected results for projects and the Program included constraints related to program design, such as the short duration of project activities as well as constraints linked to external factors such as contextual challenges at project level (political, administrative, economic, security-related challenges).

Using a scale of low, modest, substantial and high, the evaluation rated the performance of the 50 sampled projects to determine which combination of factors supported greater effectiveness in relation to the achievement of results. This portfolio review found that local implementing agencies that were well-established and had developed a long history of collaboration with Canadian partners were better equipped in managing for development results. Table 3 below presents some key examples of factors that enabled or challenged project implementation.

On a four-point summary scale – from low (1) to high (4) – the average effectiveness across the fifty projects examined was “substantial (3)”, in line with the observation that most projects achieved almost all of their outputs, but that there was less evidence to show achievement of outcomes. The average effectiveness of projects by distinct characteristics is illustrated in the “effectiveness” graph: Overall, most projects performed relatively close to average, independent of their characteristics.

Effectiveness

Behind this average lies a wide dispersion in projects’ effectiveness, largely in the range from ‘‘modest’’ to ‘‘high’’ effectiveness. As previously explained, the reasons for delays or difficulties in achieving results varied across projects, but recurrent factors included political delays (bureaucracy, change in government, local conflict), economic challenges (including inflation and foreign exchange fluctuations), and weather extremes.

When aggregated by project characteristics (outlined in Table 4 below), the major differences in performance regress to “substantial” effectiveness. In some cases, the variations in effectiveness between sub-categories, though small, aligned with other evidence gathered for this evaluation. For instance, projects selected before the Calls for Proposals mechanism was introduced (and projects of longer duration) were more effective on average than projects selected through Calls for Proposals mechanism, and projects of shorter duration. This may capture the constraints on project planning introduced by the Calls for Proposals mechanism as well as the time constraints on shorter projects, especially in the face of unexpected events.

In other cases, the small variations in average effectiveness by sub-category were not easily explained. It was not clear, for example, why projects under certain thematic areas were systematically more effective than projects implemented under other thematic areas. This variation in average effectiveness by thematic priority may therefore be an artefact of the sample (for instance, “Advancing Democracy” projects may perform better because five out of the seven were selected on the basis of unsolicited proposals, which appear to be slightly more effective for other reasons).^{Footnote 29} Similarly, the uneven relationship between project budget and effectiveness – with small and large budgets both outperforming medium budgets – could not be firmly explained. The effectiveness of small-budget projects may have been influenced, however, by a set of small community projects that were highly effective as a result of their integrated holistic approach, and despite their small budget.

4.4 Crosscutting Themes

Global Affairs Canada’s three crosscutting themes for development programming are gender equality, environmental sustainability and governance.

Finding 8: There was strong evidence of the integration of gender equality across the Program.

The portfolio review assessed gender equality based on available project documentation (including annual reports, partner gender policies, the DFATD Gender Assessment Form) and comments from the DFATD gender specialist. Findings from the review provided evidence of the strong integration of gender equality as a crosscutting theme at both Program- and project-levels. Gender equality inclusion involved:

• The implementation and dissemination of Global Affairs Canada’s Policy on Gender Equality;^{Footnote 30}
• Capacity-building for Global Affairs Canada staff, to ensure they understood and applied gender equality principles (capacity-building in gender equality for staff is a central element of the Gender Equality Action Plan);
• The appointment of a dedicated gender specialist for the Branch; and
• Awareness-raising among Canadian partners and local implementing agencies on the importance of gender equality integration at project level.

At the project-level, the integration of gender equality contributed to the inclusion of women in key decision-making processes on use and access to services and resources that support development results. Findings from the portfolio review revealed, however, that women’s economic or political empowerment was usually a project sub-component rather than a central component supporting increased decision-making by women.

The evaluation also noted that the measure of the extent of which gender equality was integrated was not always applicable at the project-level. For example, it would be unrealistic for Program staff to anticipate that projects would initiate new gender equality legislation, as this goal would be beyond the influence of individual projects, as well as ambitious for a three- to four-year project cycle. One example can be the IHAA project (International HIV/AIDS Alliance). As was the case with many projects, there is mention of gender equality in the logic model but little evidence provided of actual behavioural changes (outcomes). The logic models mentions men and women and how IHAA activities seek to improve the health of women and their children, but gender equality considerations are not explicit in project documents, despite the fact that gender equality is nominally integrated and woven into the design of IHAA.

Finding 9: The integration of environmental sustainability was not applied to all projects.

The Global Affairs Canada’s Policy on Environmental Sustainability (1992)^{Footnote 31} was designed to support the integration of environmental considerations into Program-level decision-making and project-level activities, including collaboration with partners and developing countries to increase their capacity to promote environmental sustainability. Within the context of the Program during the evaluation period, applications for funding needed to outline the procedures that were to be established for integrating environmental sustainability considerations during project implementation. Out of the 50 projects reviewed, few (3) had an explicitly environmental focus (in terms of overall final outcome or of project rationale) – this being in large parts due to sustainability per se not being a strategic DFATD priority (although it appears under the ‘sustainable economic growth’ priority – and as a crosscutting theme).^{Footnote 32} Seven of the reviewed projects were classified (by DFATD) as emanating from the ‘‘environmental sustainability’’ fund centre, but few of those projects had an explicitly environmental focus (some examples included the Ceegag Somalia wellhole drilling project, and the Résilience Climatique Boucle du Mounhoun project in Burkina Faso). Two of the reviewed projects emanated from the Canada Fund for African Climate call for proposals.

Many more than three projects, however, integrated environmental sustainability as a crosscutting factor, as was evident from project reports, comments by DFATD environmental specialists, and DFATD strategic environmental assessments. Our document review indicated, however, that in many cases environmental considerations were centred on compliance concerns, such as the Canadian Environmental Assessment Act (CEAA). Since the CEAA did not apply to many projects (e.g. projects without construction work), environmental considerations were often rather limited, with at least one project suggesting ‘‘printing less paper’’ as a part of its sustainability strategy (a valiant effort, but not an active step towards building sustainable livelihoods). Environmental analysis and indicators were also sometimes absent from project RBM frameworks, although this was usually addressed following comments from a DFATD environmental specialist.

Finding 10: The integration of governance at project level lacked the clarity and consistency observed in the integration of gender equality and environmental sustainability.

The integration of governance as a crosscutting theme during the evaluation period was largely informed by guidelines developed in 2013.^{Footnote 33} The purpose of these guidelines was to help Canadian partners integrate governance as a crosscutting theme at the project-level using the governance considerations identified by the Department.^{Footnote 34} As Departmental guidelines for the integration of governance at the project-level were issued in the final year of the period under review, there was less clarity and consistency for the integration of governance across projects reviewed.

Findings from consultations with Canadian partners and local implementing agencies, as well as the portfolio review of projects, indicated that the integration of governance was understood differently by project partners. For example, while some Canadian partners and local implementing agencies understood governance to mean processes (participatory or inclusive), others interpreted governance as referring to effective advocacy for human rights. At the project level, the evaluation found examples of capacity-building to support project governance. As an example, the World Renew Food and Economic Security Project in Bangladesh developed a People’s Institution Model to empower community groups to become autonomous, self-help mechanisms for mobilising community resources and developing relationships in support of improved health outcomes.

The portfolio review assessed the governance crosscutting theme based on available project documentation, such as annual reports, project applications, and other relevant documents. The review sought to establish to what extent a project integrated or promoted governance considerations, understood as ‘‘capacity and timely program responsiveness’’; ‘‘efficiency and effectiveness’’, ‘‘transparency and accountability’’, ‘‘equity, equality, and non-discrimination’’, and ‘‘participation and inclusion’’. The review also identified evidence of the absence/presence of transparent, inclusive, consultative processes, and/or of contributions towards the strengthening of the rule of law, institutions, and human rights.

As described above, a key finding was the lack of consistency in the application or interpretation of governance as a crosscutting theme (also because, whereas nearly all projects had gender and environmental assessments with dedicated forms, no standardised governance review process was apparent across project documentation). In general, relatively few projects – with the exception of certain dedicated rule of law projects by the Canadian Bar Association – explicitly targeted governance as a final outcome. Many projects were also somewhat unclear as to how they interpreted it as a crosscutting theme, and in many cases there was little evidence of open, transparent consultations with beneficiaries (interviewees suggested that unexpected calls and short application deadlines often did not leave the time for in-depth needs assessments, leaving Canadian partners to rely on local partners for an understanding of local needs). However, certain projects demonstrated strong inclusive features, but more as a consequence of project design itself than because of the influence of DFATD’s crosscutting theme. This was particularly the case for smaller community projects, such as the Youth-Our Future project in India, which operated in a highly inclusive manner, and benefited from its small size, careful implementation, generous timeline, and crucially, the assistance of an experienced and locally highly respected and legitimate partner.

4.5 Efficiency

Finding 11: The financial management of the Program met Treasury Board of Canada standards.

Program administration complied with Treasury Board requirements for the administration of grant contribution funding. In 2012, the “Internal Audit of Grant and Contributions”^{Footnote 35} in the Branch concluded that the Program had established effective measures to remain compliant with the Policy on Transfer Payments and the Directive in Transfer Payments. The Program also aligned with Departmental policies and established a system for risk-based asset management.

At the level of project implementation, with emphasis on the framework of contribution agreements between Canadian partners and local implementing agencies, the portfolio review found no evidence of financial mismanagement and cost overruns. Regular project reports were submitted by local implementing agencies to Canadian partners, which informed reporting by the Canadian partners to Global Affairs Canada. The internal audit noted, however, that the Branch did not have a system in place to enforce the administrative requirements of the Program.

Finding 12: The Program experienced delays in project approval.

The internal audit attributed delays in project selection to weak internal systems for Program oversight within the Branch. Program officers, in particular, were noted as not having access to detailed procedures and tools to support program management responsibilities. Other challenges noted by the Internal Audit included the launch of several concurrent calls, which led to bottlenecks in the proposal review process, as conducted by sector specialists and financial analysts. Evidences pointed out that reasons for delays were often multi-pronged, from technical issues in the proposal process to the decision-making process.

The internal audit also found that Branch human resource capacity was insufficient to address the administrative requirements of the Calls for Proposals mechanism, in particular, project screening and selection, given the volume of applications received.

The portfolio review and consultations with local partner agencies also noted lengthy delays in project implementation, as a result of the length of the project approval process, which created a need for local implementing agencies to adjust project activities and resources in some instances.^{Footnote 36} The median amount of time for project approvals was 20.2 weeks (141.5 days). The range was from 7.9 weeks to 42.6 weeks (for an aimed-for services standard of 16 weeks). It is to note that different calls received a different number of proposals. Relative response times are thus more appropriately compared if measured relative to the number of proposals received for any given call. The median amount of weekdays per proposal was 3.0 days, and the range was from 1.0 days to 9.0 days.^{Footnote 37} The CCIC Survey (2014) also provided evidence of increased unpredictability of funding created by the Calls for Proposal mechanism, which resulted in the need for Canadian partners to reduce the scope of their programming activities in project countries, and at times, their core staff in Canada.

4.6 Sustainability

Finding 13: While the Program added value to Canada’s development agenda by complementing the bilateral aid programming, it had limited effect in leveraging resources to support the sustainability of development results.

A notable feature of the Program was its capacity for outreach to unreached and under-served populations. The Program supported project implementation at the community level and had the potential to contribute to the achievement of development results within vulnerable communities. Of note, many of these communities were not accessible by services provided by their respective country governments and were therefore unreached by Canadian bilateral aid program activities. The portfolio review and consultations with Global Affairs Canada program staff indicated that the Program complemented the activities of the bilateral aid program.

In addition, the Program added value to Canada’s development aid efforts on multiple levels. It contributed to the potential achievement of development results in developing countries, and engaged Canadian civil society in development programming. Through the support provided by the Program to small projects, the Program also created an opportunity for communities to benefit from innovative ideas and best practices.

It was unclear, whether the Program was able to leverage resources to support the sustainability of development results. This finding draws on the observation that the portfolio review did not provide any evidence that the Program invested in project scale-up or replication. In the context of this evaluation, the question of leveraging-resources was understood as leveraging resources beyond a Partnership for Development Innovation Branch project itself, i.e., scaling up, replication, new donors, etc.

The basic cost of a project (shared between the Program and the Canadian partner) was taken as a given. Cost-sharing requirements were not raised as a major issue by partners, partly because they have been a long-established feature of Global Affairs Canada programming. Established partners were thus likely used to cost-sharing requirements, while new partners benefitted from a discounted rate for their first project under recent calls.

Finding 14: Although Canadian partners and local implementing agencies demonstrated commitment to achieving development results, the Program was challenged by the sustainability of results at the project-level.

All stakeholders acknowledged that financial resources originating at the Program level were not provided indefinitely. Given that, local partners should have committed financial and human resources to be able to maintain benefits and results after the Program involvement ended. However, there was no indication of future scale-up or replication for 32% of the projects selected for the portfolio review or site visits during country missions that had already been completed (i.e. the disbursement rate was 100%). While Global Affairs Canada maintained a commitment to its thematic priorities, including the allocation of funding aligned to each thematic there was, nevertheless, a consistent decrease in financial allocations to the Program (see Appendix I).

Consultations with Canadian partners and local implementation agencies indicated that project partners were interested in a continuation of project activities, and by extension the sustainability of results, after formal programming ended. That being said, the evaluation found that projects lacked a clear strategy for ensuring the financial sustainability of projects, including the capacity of local implementing agencies to access project resources from local sources. It was also unclear whether core project staff within local implementing agencies would have been available to support results sustainability at community-level if systems for sustainability were implemented by community members.

The evaluation found some examples of project partnerships that were successful in leveraging additional funding from grant-making institutions. The portfolio review and consultations with Canadian partners and local implementing agencies further indicated that some project activities were embedded in existing systems within communities, an approach that was used to encourage increased project ownership by community members, and accountability for results achievement and sustainability.

Figure 2: Examples of Steps taken towards Project/Results Sustainability

Financial support for project scale-up and replication to encourage greater results sustainability was not incorporated into the funding provided by Global Affairs Canada. The lack of access to funding earmarked for sustainability limited the capacity of the Program to achieve its ultimate outcome. Project scale-up would have been particularly feasible for projects that demonstrated a strong contribution to the achievement of development results at the outcomes level, and would have contributed to results sustainability at project level.

4.7 Design

Finding 15: The design of the Program was enhanced by the new emphasis on partnership with civil society organisations.

The global development agenda provided a strong rationale for working with and through civil society organisations (CSOs) to achieve international development results. Both the 2008 Accra Agenda for Action (2008) and the Istanbul CSO Development Effectiveness Principles (2010)^{Footnote 38} recognised CSOs as valuable and effective partners in achieving development results. The OECD also noted that all members of its Development Assistance Committee (DAC) work with civil society organisations, in particular, ‘‘to reach a specific development objective (implement aid programmes) linked to service delivery’’, and ‘‘to promote awareness about development co-operation in the DAC member country’’. These objectives were strongly in line with the Branch’s approach to service delivery through the Program, and the engagement of Canadians through its Global Citizens Program.^{Footnote 39} Canada’s Partners for Development Program has been less explicitly oriented, however, towards other objectives mentioned by the OECD/DAC.

Although enhanced capacity of local implementing agencies was an immediate outcome in the Program logic model^{Footnote 40} and the reconstructed Theory of Change, it was not identified or monitored as a high-level outcome. Similarly, neither the Program logic model nor the reconstructed Theory of Change identified the general civil society space as a Program outcome. Nonetheless, there were strong reasons for Global Affairs Canada to develop viable partnerships with civil society to support the achievement of development results. Of note, many Canadian CSOs, including the Canadian partners that applied to the Program, had decades of experience in their respective sectors and countries of focus, strong community and partnership ties with local implementing agencies, and a world-recognised experience in humanitarian affairs. Through its partnerships with Canadian partners, Global Affairs Canada was supported to meet its development objectives, as articulated in Canada’s Aid Effectiveness Agenda.

Finding 16: The absence of an approved Program-level logic model created a challenge for the assessment of results at Program level.

Although the Program logic model provided an indication of Program-level outcomes, it was in draft format at the time of the evaluation. The evaluation was therefore constrained in its efforts to measure results achievement at the Program level. The evaluation noted that the period for which the logic model was designed, 2013/14 to 2017/18, was applicable to the final year of the period under review only. Departmental Performance Reports for 2012/13 and 2013/14 reported on the Program using expected results that were not articulated in the draft logic model.

Of equal implication for the evaluation was the absence of quantitative baseline measures and targets, as the draft logic model was structured around qualitative results statements at output and outcome levels. This posed a significant challenge for results measurement as the use of qualitative targets increased the potential for inaccurate conclusions generated by self-reported anecdotal data.

Finding 17: The communication between the Program and other Global Affairs Canada funding channels was not used to support capacity-building and scale-up at the project level.

Canadian partners surveyed for the evaluation identified an important synergy between Global Affairs Canada’s bilateral (regional), and partnership Program funding. Thirty-six percent (36%) of survey respondents stated that they had received funding through Global Affairs Canada’s bilateral (regional) aid program in addition to funding provided through the Program.

Survey respondents had a positive perspective on the degree of synergy between both the Program and Global Affairs Canada’s bilateral funding, and commented that:

• Funding from the two programs helped to deepen partners’ scope in project countries;
• Synergy/complementarity between the programs helped partners develop methodologies that suited both programs; and helped partners to support capacity development in local contexts.

Global Affairs Canada staff and Canadian partners indicated, however, that project scale-up was rare because of a lack of communication between programs, and the lack of familiarity of bilateral program staff and Canadian staff in the field with Partnership programming. Within this context, the lesson from the Partnerships with Canadians Governance Program Evaluation 2008-2012 remains relevant. Specifically, “there is significant potential for cross-fertilization between projects… partners could also cross fertilize with CIDA’s multilateral and bilateral projects. There is also the potential for projects to work together jointly from the same technical assistance”.^{Footnote 41}

Finding 18: The Program filled a gap in the Government of Canada’s development agenda but provided limited support for small-scale project implementation.

Consultations with Canadian partners and sector specialists revealed that some stakeholders perceived the flexibility of the Program before the renewal as a distinct strength since funding was provided to projects that were otherwise ineligible to receive funding. Included among these projects were small-scale projects, and projects that were outside Global Affairs Canada’s thematic priorities. Both types of projects were usually funded through the submission of unsolicited proposals. In particular, the difference of stakeholders’ opinion has raised questions around the worth of small projects; whether they have contributed to a valuable enrichment of a diverse portfolio or have led to an investment in small initiatives with little impact. The portfolio review indicated that larger projects tended to be more effective in terms of results achievement. A key argument noted in favour of smaller, demand-driven projects was the opportunity to leverage Canadian partners’ experience. In addition, smaller, grassroots projects often have the capacity to reach the most marginalised and under-served populations.

Finding 19: The Program’s reconstructed Theory of Change identified ambitious but inaccurate assumptions that affected the ability of the Program to achieve its development results.

Overall, the Program was built on a sound Theory of Change that demonstrated the program’s complementarity with Global Affairs Canada’s bilateral aid programs. The Program was unable to achieve some results at each level on account of ambitious, but inaccurate assumptions included in its original design. The following assumptions within the reconstructed Theory of Change were built into the program and were particularly problematic:

• i. Program design, management and activities will be improved if application processes are clearly defined, competitive, objective and predictable;
• ii. Immediate outcomes will have more chance to be achieved if results mapping is strengthened and exit strategies are clearly articulated;
• iii. Intermediate outcomes will be favoured if lessons are captured and disseminated;
• iv. The generation/identification of innovative projects requires an enabling environment for innovation (leadership, common definition, incentives, etc.) and such an environment exists at Global Affairs Canada; and
• v. The achievement of intermediate outcomes and their contribution to higher level impact suggest that agreed-to results frameworks exist, and that resources are commensurate with the expected intermediate outcomes (these assumptions are only partially verified).

Evaluation findings from portfolio review found that as conceptualised, these assumptions were not validated during program implementation. In particular, the evaluation found, from the portfolio review, that the Calls for Proposals mechanism was unpredictable, there was evidence of a lack of clearly articulated exit strategies, and there was no system for the capture and dissemination of lessons learned at the project level. Consequently, Program-level achievement of intermediate and ultimate outcomes was weak.

4.8 Performance Management and Knowledge-Sharing

The evaluation of the Partners for Development Program draws on the notion that performance management goes beyond the measurement of Program/project performance. To measure this adequately, the evaluation purpose was tailored to reflect: the inclusion of the reconstructed Theory of Change, the existence and use of a results framework at the Program and project levels, the reporting on results achievement, and the extraction of key lessons from programming activity to inform future programming.

Finding 20: Results monitoring and reporting at project level was strongly supported by the use of the project monitoring framework, but was weakened by insufficient reporting on project outcomes.

Findings from portfolio review indicated that local implementing agencies attached considerable importance to results-based management (RBM) during project implementation. The evaluation found strong evidence of results monitoring and reporting at project level, which were supported by the use of performance measurement frameworks (PMFs). Interviews conducted during country visits also indicated that local implementing agencies had benefitted from RBM coaching from their Canadian partners to support results monitoring during project implementation. The portfolio review found that 100% of the reviewed projects had developed and incorporated RBM tools in project activities, including logic models and PMFs. This process was also supported by coaching from Canadian partners, which enabled local implementing agencies to develop capacities in managing for development results, including the systematic reporting on results achievement. As indicated earlier, project reporting indicated high levels of achievement at the level of project outputs. The incorporation of RBM into project design, in combination with significant achievement of project outputs, complied fully with the requirements of the Program. The Program provided support for results-oriented development programming, including support for training to local organisations (implementing agencies) to lead project development and delivery.

Consultations with Canadian partners indicated that the Branch also supported the development of project PMFs during the period under evaluation by providing extensive feedback on draft versions to inform PMF finalisation. While the usefulness of the feedback process to project planning cannot be denied, findings from portfolio review and stakeholder consultations noted that most PMFs that were developed to guide project implementation lacked quantitative baseline measures.

Notwithstanding the evidence provided on the incorporation of RBM at the project level, questions about existing capacities within projects for results monitoring and reporting were raised. Firstly, there was a question of whether local implementing agencies amassed adequate data about project communities during the design stage to inform the establishment of project baselines, as the absence of this data created a challenge for attaining accurate assessments of project achievements. Secondly, a question arose on the capacity of local implementing agencies to collect data on project outcomes. This situation was particularly applicable to projects of shorter duration (for example, one-year projects funded through the Haiti Recovery and Reconstruction Fund), at the end of which, local implementing agencies lacked the budget and time needed for data collection.^{Footnote 42} Similar to reporting at the Program level, reports provided on the achievement of outcomes was negligible.

Finding 21: Although knowledge exchange and knowledge-sharing occurred during project-level implementation, evidence suggests this was primarily informal in nature.

The evaluation found examples of knowledge exchange and knowledge-sharing at the project implementation level in the form of informational exchange, which was initiated using diverse vehicles, including informal discussions, workshops, study exchanges, coaching, and innovative “unconferences” (or participant-driven meeting). Consultations with local implementing agencies and Canadian partners also indicated that while certain forms of knowledge exchange and knowledge-sharing were unique to individual project designs, most projects benefitted from informal discussions, coaching and the sharing of relevant documents. Canadian partners and Program staff indicated that the exchange of technical information and experiences during project implementation was based on the structural design and objectives of individual projects.

Local implementing agencies and Canadian partners were appreciative of opportunities created for informational exchange. Some local implementing agencies indicated, however, that they believed project partners did not receive sufficient opportunities for learning.

5. Conclusions, Discussion, Lessons and Recommendations

5.1 Conclusions

The Program was relevant during the evaluation period (2010/11 to 2013/14). The evaluation findings confirmed that the Program was consistent with Global Affairs Canada’s international development agenda, with particular reference to the Department’s commitment to sustainable international development results. By extension, the Program was aligned with the global agenda for development effectiveness, as supported by Canada’s status as signatory to several global agreements for aid effectiveness and poverty reduction. Through its partnerships with Canadian civil society, the Program contributed to technical cooperation efforts that leveraged Canadian resources for the achievement of development results on the ground. As the Program had evolved in parallel with the global agenda for development effectiveness, it continued to be relevant for high-level global development strategies, as well as project-specific programming in developing countries.

Overall, the Program:

• Leveraged Canadian resources for North-South technical cooperation;
• Supported the development effectiveness agenda of the Government of Canada;
• Promoted partnerships between Government and Canadian civil society, and supported the maintenance or development of North-South partnerships; and
• Contributed to the achievement of sustainable development results, and the provision of humanitarian aid.

Evaluation activities involved an independent and unbiased assessment of the evidence that was available on Program performance. The evaluation recognised that there was no single format for an effective partnership. In order for the Program to continue to contribute towards its higher-level development outcomes, there was a need for it to evolve in parallel with various factors, including resource availability, the mandate and interests of stakeholder groups, the needs of project communities and the Aid Effectiveness Agenda. At the centre of this process, was the need, for the alignment of development efforts with the principles of good partnerships. Significantly, these principles were not transferrable across all partnerships, but needed to be tailored to the unique characteristic of each joint venture or developmental partnership. This approach was necessary for the maintenance of healthy collaboration to generate the high degree of mutualism and interdependence required for the achievement of development results.

The evaluation concluded that the quality of the partnership between Global Affairs Canada and the Canadian partners significantly improved over the period evaluated. There were questions around the flexibility of some operational modalities used by the Program (such as the Calls for Proposals mechanism), as well as Program capacity to finalise or enforce practices that contribute toward results achievement and sustainability (e.g. the final approval of the Program logic model; and sustainability planning at project-level). Program advancements in these areas would be required to support increased Program capacity to achieve development results at the level of outputs and outcomes.

The evaluation concluded that the Program successfully engaged with civil society to enhance the relationship between the Branch and Canadian partners under the Program. Stakeholders commented that the Program’s reliance on the Calls for Proposals mechanism contributed to deteriorations in the quality of partnerships between Global Affairs Canada and Canadian partners. There was an acknowledgement by Program stakeholders, however, of Branch efforts to increase its engagement with Canadian partners on account of the launch of the Partnership Policy in 2015.

The evaluation concluded that the Program should continue to investment in both larger and small-scale projects to address a range of needs continues to be a viable approach to programming.

The evaluation also concluded that while the Program filled a gap in development programming, it did not leverage resources to support sustainable development, in particularly, the investment in innovative practice through project scale-up or replication.

Program activities further enabled the Branch to steer partners towards key Departmental priorities, encourage creativity in project design and allow for the selection of the most meritorious projects. Data generated from the CCIC Survey (2014), and the evaluation’s data from interviews and its survey, also showed that Canadian Partners had a renewed optimism in relation to the commitment of Global Affairs Canada to engage in an open, collaborative and meaningful partnership with Canadian civil society.

5.2 Discussion

One element emerged from the consultation with partners for this evaluation: the improvement in the quality of the relationship between the Program and Canadian partners.

There was an improvement in the quality of the relationship between the Program and Canadian partners because of recent steps taken by Global Affairs Canada to improve its civil society engagement.

Civil society and sector experts consulted during the evaluation agreed that the relationship between civil society organisations and Global Affairs Canada has improved over fiscal years 2014/15 and 2015/16. Global Affairs Canada has taken steps to improve its engagement with civil society, which has coincided with the announcement of the International Development and Humanitarian Civil Society Partnership Policy in 2015. The final version of this Policy was informed by contributions from 25 civil society organisations, during an online public consultation facilitated by Global Affairs Canada in 2014.

Civil society representatives noted that the consultations helped to re-establish a stronger sense of partnership between civil society and government, and that meetings between both parties have occurred regularly since then. The new Civil Society Partnership Policy articulated nine goals and objectives for Global Affairs Canada’s relationship with civil society. Chief among these is the objective to “augment the voice of poor and marginalized people, including women and girls”, to “facilitate an enabling environment for civil society in developing countries” and to “foster Canadian CSO leadership in international development and innovation”. Responding to critiques of the manner in which initial Calls for Proposals were carried out, this new Policy states that it will “provide merit-based, predictable funding opportunities through equitable, flexible and transparent modalities that will support the diverse roles and types of CSOs in Canada and in developing countries”^{Footnote 43} [bolded for emphasis]. CCIC noted that the government committed in this Policy to meeting with CSO representatives on an annual basis to discuss the implementation of the Policy.^{Footnote 44}

5.3 Lessons

Effective development partnerships are attainable through the integration of good principles for development partnerships, and the consideration of practical issues at the operational level. There is increased likelihood for the achievement of sustainable development results if collaborative arrangements between development partners integrate tailored good practices for development partnerships into their collaboration and operations. As an example, partnerships can be integrated into project design, as well as the ongoing risk assessment of relationships among partners.

The logic model, complemented by the Program/project Theory of Change, is an essential tool when managing for development results. The propensity for achieving development results at the Program or project level diminishes significantly if programming is not structured around a logic model and Theory of Change that articulates how results will be achieved and measured.

Clear, standardised guidelines for the integration of crosscutting themes are conducive for increased consistency and coherence during integration. In order to achieve the quality of integration required, directives for the integration of crosscuttings themes should include clear guidelines to inform the integration process.

The sustainability of development results is highly dependent on the design and implementation of an exit plan or sustainability strategy at the project level. As project funding is not infinite, it is crucial for project partners to incorporate a strategy for results sustainability into project design. This should be followed by strategic implementation and refinement, as required, throughout the project lifecycle.

5.4 Recommendations

Recommendation 1: Improve performance management by (a) reviewing the Program’s logic model and Theory of Change, and (b) integrating the guidelines for the crosscutting themes into its programming. In particular, it will be important to ensure robust monitoring and reporting on intermediate outcomes at Program-level.

• (a) Reviewing the logic model and Theory of Change will help support the establishment of realistic program-level indicators which in turn will help increase the likelihood for the Program to achieve expected results. The absence of an approved logic model for the Program limited the ability of Program staff to monitor and evaluate Branch performance in the achievement of development results. In addition, notwithstanding the Program’s application of results-based management principles at project level, the Program did not have a robust performance measurement framework and did not accurately evaluate its performance. It is therefore recommended that the Program revisit its framework for performance measurement, including the Program logic model and Theory of Change. The revision of the performance tool would also improve the ability of the Program to explain the effectiveness of its disbursements.
• (b) The Program should integrate guidelines for the crosscutting themes into its programming. Noting that the development of guidelines on crosscutting themes is the responsibility of Global Issues and Development Branch, the Program should integrate available guidelines into its programming as an important requirement of the Program and its funded projects. A close collaboration with Canadian partners is also required to ensure that project monitoring complies with integration guidelines and the wider regulations for each thematic area, as applicable.

Recommendation 2: Ensure that Canadian partners integrate exit plans and sustainability strategies that are adequately resourced into all project designs to support sustainable results.

Applications to the Program were required to outline an exit plan or sustainability strategy to support results sustainability beyond the funded period. The Program should ensure that a viable plan for results sustainability is developed by Canadian partners, in partnership with the local implementing agencies, at the project design stage, and that this is enforced during project implementation. The Program should also work closely with Canadian partners to ensure that the partnership at project level invests adequate resources in sustainability planning and ultimate implementation. This is essential for the sustainability of results, given the relatively short time frame of most projects funded through the Program, and the lack of any indication that additional support will be provided for project scale-up or replication.

Recommendation 3: Collaborate with Canadian partners to support systematic, as opposed to ad hoc knowledge-transfer at the project level of the Program.

The process of knowledge transfer at project level was ad hoc and fragmented. Local implementing agencies, Canadian partners, and Program participants indicated that they were satisfied with benefits gained from knowledge transfer within individual projects. Some local implementing agencies expressed an interest in further opportunities for acquiring new learnings, and suggested that there were inadequate opportunities for sharing knowledge and experiences, in particular during the implementation of multi-country projects. In the interest of building on the momentum created at the individual project level, it is recommended that the Program support the development and dissemination of guidelines for systematic knowledge transfer at project-level. These guidelines would not only support increased structure for knowledge-transfer during project implementation, but could be used to support capacity building for future project implementation and/or scale-up.

Recommendation 4: Improve communication between the different programming channels within Global Affairs Canada in order to facilitate the scaling up of projects.

Although Canadian partners surveyed for the evaluation identified synergy between Global Affairs Canada’s bilateral (regional), and partnership program funding, they indicated, however, that project scale-up was rare because of a lack of communication between programs, and the lack of familiarity of bilateral program staff and Canadian staff in the field with Partnership programming.

Recommendation 5: Undertake efforts to limit those delays in project approvals that fall within the Program’s control. This may include reviewing and streamlining internal processes or by managing internal capacity better.

Multiple lines of evidence cited delays in project implementation as a result of the length of the project approval process. This created a need for local implementing agencies to adjust project activities and resources.