Formative Evaluation of Canada’s Contribution to the Maternal, Newborn and Child Health (MNCH) Initiative

2010/11 – 2013/14

December 2015

Acknowledgments

The Development Evaluation Division would like to thank all those who contributed to this evaluation. Staff from Foreign Affairs, Trade and Development Canada’s Global Issues and Development Branch, Geographic Program Branches, Partners for Development Innovation Branch and Corporate Planning, Finance and IT Branch provided invaluable support throughout the process. We especially thank those who hosted the field missions and facilitated data gathering.

We would like to acknowledge the work of the team of consultants from hera: Josef Decosas and Gretchen Roedde; as well as those consultants who provided support for country case studies: Marieke Devillé, Marie Collard, Azam Ali, Abebe Alebachew, and Javed Rahmanzai.  Anneke Slob and Leo Devillé conducted hera’s internal quality control.

From the Development Evaluation Division, the evaluation was managed by Pamela Nibishaka and Deborah McWhinney, and supervised by Andres Velez-Guerra.

David Heath
Head of Development Evaluation

List of Acronyms and Abbreviations

Executive Summary

Context and Description

Canada has played a key role in establishing the foundations for a global initiative to improve the health of women and children in the world’s most vulnerable regions. In particular, the Maternal, Newborn and Child Health (MNCH) Initiative was Canada’s contribution to the G8 Muskoka Initiative and the UN Global Strategy for Women’s and Children’s Health. The Initiative was implemented by Canada’s Department of Foreign Affairs, Trade and Development (DFATD) from 2010/11 to 2014/15 and was comprised of $1.1 billion of new funds (known as Muskoka funds) and $1.75 billion of existing program funds (known as baseline funds).

To achieve the overall goal of increased survival of mothers, newborns and children, the Initiative focused on strengthening health systems; reducing the burden of disease; and improving nutrition. Newly committed Muskoka funds were primarily allocated to ten focus countries with high rates of maternal and child mortality.^{Footnote 1} The selection of the ten focus countries was made by the former Canadian International Development Agency (CIDA)^{Footnote 2} in 2010 based on epidemiological parameters, the history of Canadian cooperation in the health sector, and Canada’s commitment to program 80% of the MNCH Initiative funds in Africa.

The MNCH Initiative was not designed as a “program”. Instead, it was designed as a thematic initiative with a strategic framework that is implemented through many autonomous program implementation strategies.  In contrast to other thematic initiatives, it has a unified logic model, a defined budget envelope and a performance measurement framework. However, the MNCH Initiative does not have a management unit that is in charge of applying these instruments or that is directly accountable for program performance.

The Initiative’s logic model is specific to the Muskoka-funded component only ($1.1 billion). However, performance monitoring and reporting covers the $2.85 billion (both Baseline and Muskoka funding). The logic model has three intermediate outcomes that are defined as program paths leading to the achievement of the overall goal of increased survival of mothers, newborns and children. These paths were 1) health systems strengthening; 2) reducing the burden of disease; and 3) improving nutrition. In the implementation of the Initiative, DFATD largely followed the logic of the health systems building blocks developed by World Health Organization.

Objectives, Scope and Methodology

This formative evaluation of the MNCH Initiative was conducted for accountability and program improvement purposes. It covered disbursements by all DFATD program branches over the first four years of the Initiative (fiscal year 2010/11 to fiscal year 2013/14) with a particular emphasis on the ten focus countries.

A total of 73 projects implemented in the ten focus countries were reviewed for the evaluation. Country missions were conducted in Bangladesh, South Sudan and Tanzania. An additional sample of global and multilateral projects was reviewed covering the full range of Global Issues and Development Branch (MFM) project partners. The evaluation team conducted 76 individual and group interviews. Feedback from Canadian civil society stakeholders was collected during an evaluation workshop. Case studies were prepared for each of the ten focus countries.

Summary of Key Findings

Relevance

The MNCH Initiative is aligned with the G8 commitments and the principles of development effectiveness and contributes to the achievement of the 4th and 5th Millennium Development Goals (MDGs) on maternal and child health in low- and middle-income countries. The Initiative ultimate outcome is defined as “increased survival of mothers, newborns and children under five in Muskoka funded countries”. This is fully in line with global MNCH goals.

The predominance of health systems strengthening path in the strategic positioning of the Initiative is aligned with the principle of the G8 Muskoka Initiative to “support country-led national health policies and plans”.^{Footnote 3} This has contributed to focusing the projects funded under the Initiative on addressing critical bottlenecks in the delivery of maternal and child health services, such as human resources, infrastructure and management inefficiencies.

However, the three paths of strategy for the Muskoka-funded component of the MNCH Initiative neither provide a transparent picture of what is being funded, nor do they present a logical structure on which to build a performance measurement framework. In particular, the outcomes of activities to strengthen the health systems building blocks are distributed across all three programming paths. This distinction decreased the capacity of the logic model to provide strategic guidance to implementing programs and weakened the link between objectives and performance monitoring indicators.

Given the focus on strengthening the supply side of health services, issues of timely care-seeking and access to services for pregnant women (the demand side) receive relatively less attention, although they are known contributing factors to maternal mortality. In addition, initiatives to address root causes of high maternal and child mortality at the community level such as adolescent pregnancy, gender violence, women’s lack of power in household decision-making and unmet need for family planning are not prominent in the logic model and are relatively underrepresented among the implemented programs.

MNCH programming by the Partnerships for Development Innovation Branch (KFM) was well aligned with the strategy of the Initiative. KFM included objectives for the community and demand-side component in its own MNCH strategy under each intermediate outcome. This is reflected in the projects implemented with Muskoka Initiative Partnership Program funding (Muskoka funds).

Programming by the MFM Branch focused on nutrition and child health. Emphasis on maternal health was strengthened in Muskoka-funded projects compared to those that were baseline-funded, although it continued to be a relatively weak focus.

Effectiveness

There is evidence of progress towards achievement of results, based on evidence from the sampled projects. Specifically, most projects achieved their objectives for outputs (such as number of people trained or number of children immunized) and immediate outcomes (such as coverage of skilled attendance at delivery). However, information on higher-level outcomes of the MNCH Initiative was generally only available from projections and mathematical models. Higher-level information (i.e. the eleven UN indicators) on maternal and child mortality is collected through population surveys, which are conducted every three to six years by national governments. Updated information on high-level results was not available at the time of the evaluation in the majority of focus countries.

The emphasis of Muskoka-funds on ten focus countries contributed to strengthening Canadian participation in the health sector policy and partner coordination dialogue. It was used effectively to promote Canadian priority development themes such as gender equality, transparency and accountability.

Thematic management of the MNCH Initiative at corporate level was strong in the first year of the Initiative but had weakened considerably at the time of the evaluation. Planned or previously existing central governance and management structures for MNCH could no longer be identified at the time of the evaluation.

MNCH projects at country level focused on relevant priorities adapted to local context. This approach, and the volume of available funds, also determined the mix of project delivery channels. Country programs with sufficient resources effectively balanced their support by working with government, multilateral implementing agencies and Non-Governmental Organizations (NGOs).

Efficiency

Although DFATD was on track to meet or exceed the Canadian financial commitments of the Muskoka Summit, the dual accounting system for baseline- and Muskoka-funded disbursements generated inaccurate estimates for MNCH Initiative expenditures.

The MNCH Initiative supported cost-effective high impact interventions to improve maternal and child health through strengthened health systems. High impact interventions for the prevention of maternal and child mortality, however, received less support. These included, for instance, projects to reduce adolescent pregnancy and address the unmet need for family planning.

Cross-Cutting Issues

Gender analyses and gender equality strategies were particularly strong in NGO projects and weaker in projects with pooled contributions to the health sector. DFATD representatives in national and international policy fora and committees consistently (and often successfully) focused attention on gender inequality as a factor contributing to poor maternal and child health.

In terms of governance, the implementation of the MNCH Initiative strengthened the engagement of DFATD in initiatives to improve health sector governance in focus countries, and to some extent also at the level of multilateral organizations by increasing the cooperation among United Nations agencies under the H4+ umbrella. ^{Footnote 4}

In terms of the environment, the limited number of construction projects supported under the MNCH Initiative included environmental assessments, as required by Canada’s legislation. Medical waste disposal was addressed in the majority of health service projects. Some NGOs working in dry rural areas noted that water and sanitation facilities in health centres should receive greater attention in the effort to improve MNCH services.

Sustainability

The sustainability approach of the MNCH Initiative largely relied on support to strengthening health systems and aligning projects with national priorities and strategies. In many contexts, this has been an effective approach to the achievement of sustainability. This approach also entails contextual risks, such as funding unpredictability by recipient national governments. These risks could be mitigated by long-term involvement in health sector cooperation, close cooperation with international partners, and participation in the health sector dialogue at national and sub-national level to ensure local ownership of interventions. At the same time, several KFM’s Muskoka Initiative Partnership Program projects face major sustainability issues because of the short cooperation time frame.

Performance management

DFATD monitored the performance of the MNCH Initiative, albeit with a high level of systemic inefficiencies because the planned performance monitoring system was not implemented. Performance data for MNCH projects were to be collected annually through a Program Management Reporting Tool, which would collect performance data at the project level and aggregate the data at headquarters. However, the tool was never used systematically throughout the Department. This generated costs which were partially born by project partners and service providers. It also weakened DFATD’s delivery on its commitment to reducing the reporting burden on developing countries.

Conclusions

DFATD’s MNCH Initiative has effectively contributed to the UN Global Strategy for Women’s and Children’s Health and is on course to deliver upon the Muskoka commitments of the Government of Canada.

1. The MNCH Initiative effectively supported efforts to reach the goals of MDGs 4 and 5 in ten focus countries with high rates of maternal and childhood mortality.
2. The Department’s MNCH strategy focused on the supply side of health interventions, namely improving the availability, quality and equity of health services.
3. Programs that address the root causes of high maternal and child mortality at the community level were underrepresented in the MNCH Initiative.
4. The MNCH Initiative aggregated general interventions to strengthen health systems (such as the support of Ministries of Health through pooled health sector funds) with targeted interventions to improve maternal health service delivery (such as salary incentives for birth attendants). The aggregation of disbursements for systems strengthening and for service delivery reduced the clarity of the Canadian support for MNCH.
5. The Department is on track to meet or exceed the Canadian resource commitments of the Muskoka Summit. This was facilitated by the decentralized institutional design of the MNCH Initiative, which requires strong technical leadership and coordination at the central (headquarters) level. This leadership was present at the start of the MNCH Initiative but weakened over time.
6. The cross-cutting themes of gender, governance and the environment were addressed appropriately.
7. The sustainability strategy of MNCH projects has largely relied on the support to health systems strengthening and on the alignment with national priorities and strategies. This is an effective approach to promote sustainability, but it also entails major contextual risks.
8. DFATD monitored the overall performance of the MNCH Initiative, albeit with a high level of systemic inefficiencies because the planned performance monitoring system was not implemented.

Recommendations

The following recommendations build on Canada’s leadership role and significant contribution to the G8 Muskoka Initiative and the UN Global Strategy for Women’s and Children’s Health.

1. The Department should consider balancing its support for interventions targeting the demand and supply side of MNCH services.
2. The Department should consider widening the scope of the MNCH Initiative programming by placing greater emphasis on addressing factors contributing to high maternal, newborn and child mortality, such as reproductive health.
3. The Department should strengthen its capacity to provide horizontal leadership and coordination, as well as technical guidance, on health to all programming branches. This should be provided by a specifically mandated MNCH group at DFATD headquarters.
4. The Department should continue harmonizing financial reporting for the Initiative.
5. The Department should continue to improve the sustainability of the Initiative by maintaining a long-term strategic horizon and working closely with international partners and national governments in focus countries to develop, implement and sustain effective interventions.
6. The Department should consider aligning the MNCH Initiative Performance Management Strategy more closely with the World Health Organization framework of health systems building blocks.
7. The Department should consider establishing and implementing a robust MNCH system of performance monitoring and reporting.

Management Response

Preamble to Recommendations

The following recommendations from the formative evaluation of the MNCH Initiative covering fiscal year 2010/11 to fiscal year 2013/14 build on Canada’s leadership role and significant contribution to the G8 Muskoka Initiative and the UN Global Strategy for Women’s and Children’s Health. Canada’s key role in establishing the foundations for a global initiative to improve the health of women and children in the world’s most vulnerable regions is well documented and recognized. In particular, Canada made a significant contribution to creating frameworks that encouraged accountability, pragmatism and focus among partners and within its own international development program for MNCH.

Introduction to Management Response

DFATD welcomes this evaluation, for which an initial report was completed in October 2015, and agrees with its findings and recommendations.

This evaluation provides relevant guidance for DFATD to consider as the Government of Canada follows through on its $3.5 billion commitment and continues to lead global efforts to reduce the number of preventable maternal, newborn and child deaths made at May 2014 Saving Every Woman, Every Child: Within Arms’ Reach Summit. The findings and recommendations in this evaluation will help inform the Department’s efforts to accelerate progress on achieving Canada’s goals and maximize the impact and sustainability of our future MNCH investments.

DFATD has identified and begun to implement a number of actions and commitments in response to the evaluation’s recommendations. This includes the development a new performance management strategy, full implementation of the OECD-DAC^{Footnote 5} RMNCH marker, and the convening of an ADM Committee for greater leadership and accountability.

1. Introduction

At the 2010 G8 Muskoka Summit, the Canadian Government committed to spending $2.85 billion over five years to contribute to the achievement of the 4th and 5th Millennium Development Goals (MDGs) on maternal and child health in low- and middle-income countries. The commitment included funding of $1.75 billion for ongoing programs (referred to as ‘baseline funding’) and $1.1 billion of additional spending (referred to as ‘Muskoka funding’). Together, the projects funded from these two envelopes are Canada’s contribution to the Muskoka Initiative on Maternal, Newborn and Child Health (MNCH). In this report they are referred to as the ‘MNCH Initiative’ or the ‘Initiative’. (Figure 1)

Figure 1: MNCH Initiative financial commitments

1.1 Purpose and Objective of the Evaluation

In accordance with the Federal Accountability Act and the Treasury Board Policy on Evaluation, the formative evaluation of the MNCH Initiative was conducted for accountability and program improvement purposes, specifically:

• to review the relevance and performance of Canada’s contribution to the MNCH Initiative in order to fulfill reporting and accountability requirements and improve the implementation of the Initiative;
• to provide Foreign Affairs, Trade and Development Canada (DFATD) with relevant information for future international development programming on MNCH; and
• to assess progress to date, learn about good practices and areas for improvement and adjust programming.

1.2 Evaluation Objectives and Scope

The evaluation covered the first four years of the five-year Initiative (fiscal year 2010/11 to fiscal year 2013/14), including both baseline and Muskoka funding. In particular, the evaluation assessed:

• DFATD’s policy and strategy decisions for maternal and child health;
• Changes in Canadian development cooperation within the ten MNCH focus countries through the implementation of the Initiative, including Canadian participation in health sector policy dialogue;
• Progress towards MDGs 4 and 5 in the ten MNCH focus countries^{Footnote 8} and, to the extent possible, an assessment of the contribution of Canadian development cooperation to this progress;
• The financial support provided by DFATD’s Global Issues and Development Branch (MFM) for a sample of global programs and initiatives; and
• The activities and role of Canadian civil society organizations in the implementation of the Initiative through the Muskoka Initiative Partnership Program (MIPP) of the Partnerships for Development Innovation Branch (KFM)

The evaluation was guided by the DFATD MNCH Formative Evaluation Advisory Committee.

1.3 Evaluation Approach, Methodology and Data Collection Methods

The evaluation covered disbursements by the Geographic Program Branches for Africa, Asia and the Americas, the Global Issues and Development Branch and the Partnership for Development Innovations Branch as recorded in the DFATD MNCH database of November 3rd, 2014. Data collection and analysis focused primarily but not exclusively on disbursements for cooperation with the ten focus countries of the MNCH Initiative.

For each of the ten focus country programs, a purposive sampling strategy was used to select projects with significant disbursements during the evaluation period representing different types of activities, funding modalities and funding mechanisms. An initial sample was discussed with the DFATD country programs and modified according to suggestions.

The evaluation team reviewed 73 projects in the ten MNCH focus countries detailed in Table 1.

Source of the sampling frame: DFATD MNCH database: Version of November 3, 2014
* includes only projects with any disbursement in one or more of the 10 focus countries during the evaluation period

For each of the projects, documentation was obtained from the responsible DFATD officers and additional information collected through internet searches, interviews and project visits.

The core evaluation team conducted three country missions for project visits and local interviews (Bangladesh, South Sudan and Tanzania). Two local consultants joined the team to conduct interviews in Afghanistan and Ethiopia. For the remaining five focus countries, case studies were prepared on the basis of document reviews and telephone interviews.

A sample of global and multilateral projects, in addition to those reviewed in the focus country case studies, was selected by purposive sampling in consultation with the MFM Branch and the MNCH Formative Evaluation Advisory Committee covering five thematic program areas and a full range of implementing partners. They included UN Agencies (UNFPA and UNICEF), International NGOs (Micronutrient Initiative), Global Health Initiatives (GAVI Alliance), and an academic institution (Johns Hopkins University). Disbursements of the MFM Branch under the MNCH Initiative were mostly (81%) made with baseline funding, which is reflected in the sample.

The evaluation team conducted 76 individual or group interviews using four different interview scripts for different types of stakeholders. Interviews were transcribed into one of four matrices corresponding to the four interview guides and analysed for common themes.

Feedback from Canadian civil society stakeholders was collected during an evaluation workshop held on March 2^nd, 2015 with logistical assistance from Plan International Canada and the Canadian Network for MNCH. It brought together 19 program and technical staff of 18 Canadian civil society organizations for a participatory review and evaluation of the implemented intervention strategy.

Case studies were prepared for each of the ten focus country programs and reviewed by DFATD country desks at headquarters and/or by the decentralized DFATD teams.

1.4 Limitations of the Evaluation

The review of the sampled projects was based primarily on documents provided by DFATD. In some cases, additional documentation was obtained from the implementing partners and from internet sites. Interviews and site visits were conducted for a limited number of projects. These data sources were not sufficient for a comprehensive evaluation of sampled projects.

The project reviews focused primarily on assessing the relevance and effectiveness of the projects, in most cases from self-reported information. Evidence on economy and efficiency could only be obtained for some projects. Consistently obtaining this information would have required a much deeper evaluation of individual projects, which was beyond the scope of this evaluation. A limited number of evaluation reports for projects or country programs were available. However, some of these evaluations were still on-going or the reports had not yet been approved.

Planned country missions to Mali and to Nigeria were cancelled because of security concerns. A mission to Tanzania was added as a replacement. Security concerns during the country missions to Bangladesh and South Sudan constrained travel within the countries and prevented several scheduled project visits.

2. Context and Description of the MNCH Initiative

2.1 Background

In June 2010, the G8 leaders launched the Muskoka Initiative, committing to spend US$5 billion of new development funds over five years for maternal and child heath in low- and middle-income countries. Canada committed $1.1 billion of new international development funds, in addition to estimated baseline expenditures on maternal and child health of $1.75 billion through existing programs.

Three months later, at the UN Development Goals Summit in September 2010, the UN Secretary General launched the Global Strategy for Women’s and Children’s Health, which integrated the G8 Muskoka Initiative into a global commitment endorsed by the UN General Assembly. A UN Commission for Information and Accountability (COIA) was convened, co-chaired by the Prime Minister of Canada and the President of Tanzania, to develop recommendations for a global accountability framework.

The former CIDA was closely involved in defining the content of the G8 Muskoka Initiative and the accountability framework of the Global Strategy, working in collaboration with the Prime Minister’s Office, an informal group of Canadian civil society experts and advocates for international development and maternal and child health, and international technical agencies.

During the technical consultations leading up to the Muskoka Summit, a framework was developed that encouraged the use of existing mechanisms and structures of development cooperation aligned around common goals and development effectiveness principles in support of country plans and systems. It recommended a focus on strengthening health systems to provide integrated and comprehensive services in countries most at need, specifically: antenatal care; post-partum care; family planning including contraception; reproductive health; treatment and prevention of diseases; prevention of mother-to-child transmission of HIV; immunization; nutrition; and other services that contribute to improving the health of women and children, such as access to safe drinking water and sanitation and gender equality.

After the Prime Minister’s announcement of Canada’s contribution to the Muskoka Initiative, the former CIDA established the programing and accountability framework of the MNCH Initiative. The envelope for the newly committed funds ($1.1 billion) was approved by Treasury Board in September 2010. It was conditional on the development of a detailed performance management strategy, which, given the very tight time line, had not been included in the submission. The performance management strategy including the logic model was approved by Treasury Board in August 2011, already three months into the second year of the Initiative.^{Footnote 9}

The profile and logic of the MNCH Initiative developed by the former CIDA was closely aligned with the principles and technical guidelines of the G8 Muskoka Initiative. The delivery of the MNCH Initiative was through existing development program channels working with a common thematic strategy but without a unified operational strategy. Newly committed funds were primarily allocated to programming in ten focus countries with high rates of maternal and child mortality. The overall goal of ‘Increased survival of mothers, newborns and children under five in Muskoka-funded countries’ was to be pursued under three strategic paths that included all high impact interventions identified by the G8. A central path to achieve this goal was the strengthening of health systems. The other two paths were reducing the burden of disease and improving nutrition.

The performance management strategy of the MNCH Initiative was developed at the same time as the accountability framework of the Global Strategy for Women’s and Children’s Health under the guidance of the Commission on Information and Accountability for Women's and Children's Health (COIA), which had identified eleven programmatic performance indicators to be tracked globally, and particularly in 74 countries with high maternal and child mortality. The eleven UN indicators were adopted by the performance measurement framework of the MNCH Initiative with some expansion of indicators for policy outcomes of the logic model.

To ensure accountability for resources to support the UN Strategy, COIA recommended that the OECD implement changes to its system of tracking disbursements for development assistance by introducing a marker for reproductive, maternal, newborn and child health (RMNCH) in its Creditor Reporting System (CRS). Procedures for the RMNCH marker were issued by the OECD DAC in February 2014 but have not yet been implemented by the DAC partners.^{Footnote 10} Meanwhile, a G8 working group had developed a weighting system to be applied to 18 DAC sector codes to estimate baseline MNCH spending. Under this system, disbursements for projects reported to the DAC under some sector codes were considered to contribute 100% to MNCH, while others were imputed to 40% or less.

Under the performance management strategy of the MNCH Initiative DFATD used a dual methodology. Disbursements for Muskoka-funded projects were to 100% imputed to the MNCH Initiative, while the contribution of baseline-funded projects was estimated by applying the G8 weighting formula. According to this methodology, total disbursements by DFATD at the end of the 2013/14 fiscal year amounted to $2.456 billion or 89.57% of the commitment for baseline and Muskoka funds.

DFATD is responsible for programming all of the committed Muskoka funds of $1.1 billion and for programming $1.64 billion of the baseline commitment of $1.75 billion. The remaining baseline commitment of $110 million is programmed through the Department of Finance and other government channels. Disbursements by Branch are presented in Table 2.

Source: DFATD MNCH database: Version of November 3, 2014

2.2 Programming profile

2.2.1 DFATD cooperation with focus countries

Canadian development cooperation with ten focus countries was central to DFATD’s implementation of the MNCH Initiative and the main focus of this evaluation. It accounted for 42% of total disbursements and for 70% of disbursements of Muskoka funds by the end of the fourth year. The selection of the countries was based on epidemiological parameters as well as the history of Canadian cooperation in the health sector and Canada’s commitment to program 80% of the MNCH funds in Africa.

Canadian development cooperation with the ten focus countries was not restricted to the bilateral program channels of DFATD. About one third of all funds for MNCH disbursed by DFATD for programming in these countries during the evaluation period were spent by the Global Issues and Development Branch and the Partnership for Development Innovation Branch. The disbursements in each focus country by the end of the 2013/14 fiscal year for baseline and Muskoka-funded projects are presented in the Table 3.

Source: DFATD MNCH database: Version of November 3, 2014

The table shows that over the four years from 2010/11 to 2013/14, the commitments of new funds announced at the Muskoka Summit added an additional 156% to the disbursements under baseline programs and commitments. These increases were not uniform. They ranged from additional disbursements of 51% over baseline in Bangladesh to 250% in Mozambique.^{Footnote 11} Overall, 80.5% of the Muskoka funds were spent in the seven African focus countries. The impact of the funding on country cooperation varied from country to country.

2.2.2 Cooperation with global partners

Disbursements for the MNCH Initiative by the Global Issues and Development Branch (MFM) over the first four years amount to $1,376.6 million or 56% of the entire DFATD expenditure for MNCH. The great majority of this amount, about 81% ($1,118 million), was spent on established programs and partnerships and therefore recorded as baseline funding. (see Table 2) In order to examine the impact of the MNCH Initiative on the MFM portfolio, the G8 methodology is applied to all projects. This lowers the imputed amount for Muskoka-funded disbursements by $95 million but has little effect on illustrating trends.

Figure 2: MNCH disbursements by MFM according to G8 imputation method (million $)

• Text version

  	Total Disbursements	MNCH Disbursements
  FY 09/10	1,608	226 (14%)
  FY 10/11	1,463	275 (19%)
  FY 11/12	2,081	377 (18%)
  FY 12/13	1,795	310 (17%)
  FY 13/14	1,758	319 (18%)

Source: DFATD historical project data set at http://www.international.gc.ca/ (accessed 14/06/2015)

Total annual disbursements by the DFATD Global Issues and Development Branch varied between $1.5 and 2.0 billion over the four years of the evaluation period, with a slight upward trend since the reference fiscal year 2009/10, the year prior to the Muskoka Initiative. MNCH spending over the four years of the MNCH Initiative averaged approximately $317 million per year, about a 40% increase over the spending in the reference fiscal year 2009/10.

2.2.3 Participation of Canadian civil society

Canadian civil society organizations participated actively in the conception of the MNCH Initiative through technical input and advocacy during the preparation of the G8 Muskoka Summit. In November 2010, four months after the Summit, the Minister of International Cooperation invited leading Canadian health experts and representatives of civil society development organizations to a roundtable discussion. This meeting produced two key outcomes:^{Footnote 12}

• The announcement of the Muskoka Initiative Partnership Program (MIPP) with an initial grant envelope of $75 million to be awarded on the basis of calls for proposals to Canadian organizations for MNCH projects in 31 eligible low income countries, primarily in Africa.^{Footnote 13}
• The start of a process that led to the development of the Canadian Network for MNCH.

The 28 successful MIPP proposals were announced in September 2011. In 2012, additional funds were committed for the support of the Canadian Network for MNCH. Several of the agreements with Canadian civil society organizations funded multi-country initiatives for a total of 51 projects. The projects ranged from $0.5 million to $18 million for a total envelope of $77 million. After the projects were announced in September 2011, the Partnerships for Development Innovation Branch (KFM) developed a logic model for the program that was closely aligned to that of the MNCH Initiative.^{Footnote 14}

The Canadian Network for MNCH emerged from the advocacy initiatives of an informal group of Canadian development CSOs and health professionals during the preparations for the Muskoka G8 summit and the subsequent consultative meetings hosted by the Minister of International Cooperation. It rapidly grew beyond an initial group of implementing CSOs and established itself as an advocate for a continued focus on maternal and child health in Canadian development policy, and as an information portal for Canadians to access information on key MNCH issues. Table 4 presents KFM disbursements to Canadian organizations for MNCH programming under Muskoka and baseline funding by the end of fiscal year 2013/14.

* does not include disbursements to international NGOs not based in Canada
Source: DFATD MNCH Database, version November 3, 2014

3. Findings

The MNCH Initiative is not a ‘program’ that can be evaluated by assessing program outcomes against targets and benchmarks established in an implementation strategy and results framework. It is a thematic initiative with a strategic framework that is implemented through many autonomous program implementation strategies.

In contrast to other thematic initiatives, for instance the former CIDA’s Children and Youth Strategy, it has a unified logic model, a defined budget envelope and a performance measurement framework. It does not, however, have a management unit that is in charge of applying these instruments and that is directly accountable for program performance. It was therefore described by senior management involved in the design of the Initiative as a ‘hybrid’: An initiative to implement defined programmatic objectives by pre-existing decentralized management units that were autonomous to varying degrees.

A further complexity arises given that the Initiative has two components. The larger component is made up of baseline-funded projects which, in a post hoc analysis, contribute to the goals and objectives of the Initiative, but which were largely beyond the reach of a strategic planning process for MNCH.

The second, smaller component of Muskoka-funded projects was more accessible to strategic planning at the central level, partially because it involved new funds that were added on top of existing (baseline) implementation strategies, and partially because the allocation of these funds was concentrated on a smaller number of programs. After the fourth year of implementation, 70% of the Muskoka envelope of funds had been spent in 10 focus countries compared to 26% of the baseline envelope which was spread more widely across countries, regions, and global initiatives.

Thematic evaluation criteria such as relevance, therefore, are applied to the MNCH Initiative as a whole, including both the baseline- and Muskoka-funded component. Operational evaluation criteria, for instance efficiency, could only be applied by focusing more narrowly on the Muskoka-funded component.

3.1 Relevance

3.1.1 The logic model of the MNCH Initiative and its alignment with G8 commitments, the MDGs 4 and 5, and the principles of development effectiveness

The logic model of the Muskoka-funded component of the MNCH Initiative outlines a common strategy that is applicable to all programming branches of DFATD.^{Footnote 15} It applied only to the Muskoka-funded component because baseline contributions to the MNCH Initiative were guided by pre-existing corporate strategies for child health and health systems strengthening. The ultimate outcome is defined as “increased survival of mothers, newborns and children under five in Muskoka funded countries”. This is fully in line with global MNCH goals. The logic model’s narrative mentions that each program would implement the strategy under its own program- or country-specific strategic framework. Only the Muskoka Initiative Partnership Program and the South Sudan Program developed specific MNCH strategies.

a) The strategic framework of the Muskoka-funded component of the MNCH Initiative

The logic model has three intermediate outcomes that are defined as program paths leading to the achievement the overall goal of increased survival of mothers, newborns and children (see table 5). The dominant path was health systems strengthening. It received 70% of DFATD resources from the Muskoka funding envelope over the first four years of the Initiative.

Source: CIDA (2011) Performance Management Strategy of the Muskoka Initiative on Maternal, Newborn and Child Health

The predominance of health systems strengthening in the strategic positioning of the Initiative is fully in line with the principle of the G8 Muskoka Initiative to “support country-led national health policies and plans”.^{Footnote 16} It has contributed to focusing the projects funded under the Initiative on addressing critical bottlenecks in the delivery of maternal and child health services, such as human resources, infrastructure and management inefficiencies.

A conceptual framework for health systems was developed by WHO as the interaction of six building blocks as presented in Figure 3.

Figure 3: WHO Health systems framework

• Text version

  Figure 3 illustrates the World Health Organization’s (WHO) health systems framework.

  The six building blocks of a health system are 1) service delivery, 2) health workforce, 3) information, 4) medical products, vaccines and technologies, 5) financing and 6) leadership and governance.

  Investing in these six building blocks determine the access, coverage, safety and quality of the health system. This leads to improved health, responsiveness, efficiency, and social and financial risk protection.

Source: WHO (2007). Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action.

Investments in the six building blocks (on the left side of the figure) contribute to the strengthening of health systems in different ways. In a simplified form, the first building block of service delivery refers to the supply-side output of health systems strengthening and covers services for maternal and child health services, reducing the burden of disease and nutrition among others.

The MNCH service delivery gaps need to be identified in order to apply a health systems strengthening framework to the goal of increased survival of mothers, newborns and children. The gaps are caused by bottlenecks in the availability of system inputs, such as human resources, medicines, vaccines or infrastructure, by inefficiencies in financing, or by cross-cutting failures of system components such as a lack of information or poor governance. Interventions to strengthen each one of these system components or ‘building blocks’ have to be specifically designed and monitored.

In the implementation of the MNCH Initiative in focus countries, DFATD has largely followed the logic of the building blocks. It has supported health reform and decentralization of budgets to improve governance and increase the efficiency of health financing; it has addressed shortages of human resources through training of midwives and task shifting; it has supported greater efficiency in the supply chain for vaccines and nutrition products, etc.

However, at the level of the strategy as presented in the MNCH logic model, the outcomes of activities to strengthen the health systems building blocks are distributed across all three programming paths. This makes the definition of the ‘health systems strengthening’ path somewhat meaningless. Furthermore, outcomes of horizontal health systems strengthening interventions, for instance contributions to pooled health sector support funds, are not included in the strategic framework. They are categorized by default under the ‘health systems strengthening’ path and are monitored with maternal health outcome indicators without consideration to which they actually contribute to improved maternal health.

The three paths of strategy for the Muskoka-funded component of the MNCH Initiative neither provide a transparent picture of what is being funded, nor do they present a logical structure on which to build a performance measurement framework.

The realized MNCH strategy (i.e. what is actually being done by the programs), as opposed to the intended strategy (i.e. what is presented in the MNCH logic model),^{Footnote 17} can be better understood in an analysis of the disbursements for Muskoka-funded projects by DAC sector codes. Figure 4 presents the volume of disbursements over the first four years of the Initiative as coded under the three paths of the MNCH strategy (on the left) and as coded by DAC sector (on the right).

Figure 4: Disbursements for Muskoka-funded projects by ‘Muskoka Paths’ and by DAC Sector Codes (million $)

• Text version

  • Muskoka Paths
    • Improving nutrition, 243.2
    • Reducing the Burden of Disease, 140.7
    • Strengthening Health Systems, 510.5
    • Total: C$ 894.4 million
  • DAC Sector Codes ^{Footnote 18}
    • Health Service Infrastructure, 51.1
    • Health System Management, 86.6
    • Basic Health Services, 174.2
    • Infectious Disease Control, 156.2
    • Reproductive Health and Family Planning, 77.4
    • Human Resource Development, 106.4
    • Basic Nutrition, 199.2
    • Basic Water and Sanitation, 33.2
    • Total: C$ 884.3 million

Source: DFATD MNCH database: Version of November 3, 2014. Total disbursement amounts differ because some projects are partially coded with DAC Sector Codes that are not recognized as contributing to MNCH (such as sustainable agriculture)

The analysis of the implemented MNCH strategy according to DAC sector codes reveals quite a different picture than an analysis according to the strategic paths:

• 28% of funds address health system weaknesses such as human resources, infrastructure and management;
• 60% of funds are almost equally distributed between three service delivery areas (i) nutrition services, (ii) basic health services which are primarily maternal and neonatal health services, and (iii) infectious disease control which are primarily child health services;
• 9% of funds support reproductive health and family planning, which include the main upstream services for the prevention of maternal and neonatal mortality; and
• 4% of funds support programs for improved water and sanitation.

This analysis indicates that the MNCH interventions implemented under the strategies of DFATD country programs are not reflected in the corporate MNCH logic model. As a result, the MNCH logic model does not provide sufficient strategic guidance to implementing programs, disassociates performance measurement indicators from funded activities, and provides little transparency about Canadian investments in MNCH.

c) The demand and supply of MNCH services

The logic model of the MNCH Initiative refers to gender inequalities, harmful practices and social determinants at the outcome level. However, at the output level, these issues are addressed in terms of health system responses, such as health education and outreach by health workers. This gives the strategy a ‘supply side’ profile. This is in line with the WHO health systems building block framework (Figure 2) which emphasises the elements required for the delivery of quality health services. WHO acknowledges that its framework “does not take into account actions that influence peoples’ behaviours, both in promoting and protecting health and the use of health-care services.”^{Footnote 19} These actions are captured under the concept of ‘demand side’ programming.

The ‘three delays model’ for understanding and addressing maternal mortality was introduced in 1994 and has become a widely accepted analytic and programmatic concept.^{Footnote 20} It postulates that maternal mortality is due to (i) delays in the decision to seek care, (ii) delays in accessing care, and (iii) delays in receiving adequate care. The focus of the MNCH strategy has been, to a major extent, on reducing the third delay by improving the availability and the quality of maternity services, and also to a significant extent on the second delay by bringing maternal health services closer to women. The evaluation also found many examples of effective initiatives that address the first delay, especially in community-centred projects implemented by NGOs. Increasing care-seeking, both for women and for children, through programs that increase women’s decision making, women’s control of household resources, subsidize transport costs for emergency obstetric care and eliminate hidden user charges for health services are not included in the corporate MNCH logic model.

There are many underlying causes of maternal and child mortality that are beyond the scope of the health sector. Adolescent girls aged 15 to 19 are twice as likely to die during pregnancy and child birth as those over age 20. Girls under age 15 are even five times more likely to die.^{Footnote 21} Gender violence is associated with significant increases in maternal morbidity as documented by research conducted under the Nigeria Evidence-based Health Systems Initiative (project A031274). Increased intra-household bargaining power of women has a positive influence on child health and nutrition.^{Footnote 22} An analysis of maternal mortality in 172 countries concluded that satisfying unmet needs for family planning would result in a 29% reduction of maternal mortality.^{Footnote 23}

Many reviewed projects, especially those implemented by NGOs with MIPP or bilateral funding, addressed community-based root causes of poor maternal and child health as well as the ‘demand side’ of health service utilization. But on the scale of the entire portfolio of programs initiated with Muskoka funding, interventions of this type received relatively little support, in part related to the fact that this type of programming is not included in the output and outcome statements of the MNCH Initiative logic model. This was recognized as a strategic gap by some programs, for instance in Tanzania, where DFATD commissioned a study that included the assessment of NGO support for the demand side of health services.^{Footnote 24}

3.1.2 The alignment of MNCH Initiative projects in the 10 focus countries with national priorities and health systems

The DFATD bilateral MNCH programs in the ten focus countries are well aligned with national health and nutrition strategies, either through direct engagement with national public sector health authorities or through UN or NGO intermediaries.

In Ethiopia and Mali, where DFATD did not fund government structures, alignment was achieved through participation in the health or nutrition sector dialogue and through intermediary implementing partners. DFATD programming showed considerable flexibility in adapting to evolving national priorities and policies. Examples include mid-stream changes in project budgets and work plans in South Sudan in response to a revision of midwifery licencing regulations, and in Tanzania in response to standardization and revision of the training curriculum for community health workers.

The MFM Branch channels most funds through implementing partners who are fully engaged in the development and implementation of national strategies. Similarly, all the MIPP projects included in the evaluation sample were implemented in close coordination with public authorities, most of them at the decentralized level. Program alignment of the Canadian portfolio of MNCH projects with national strategies and priorities was especially strong in countries where the decentralized DFATD teams made an additional effort to consult and coordinate with all Canadian-funded implementing partners, irrespective of the source of funding, as for instance in Tanzania.

The projects reviewed were generally well aligned with national monitoring and reporting systems for MNCH. However, because national systems for monitoring and reporting are often weak, several MIPP projects established parallel reporting systems. This did not contribute to strengthening national structures, and in some cases generated a considerable opportunity cost of non-realized support to these structures.

3.1.3 The alignment of MFM programming with the strategy of the MNCH Initiative

The portfolio of MFM projects contributing to the MNCH Initiative was dominated by baseline funded projects that were developed outside the strategic framework of the Initiative. Prior to the launching of the MNCH Initiative, the global and multilateral programs of DFATD already had a strong portfolio of support for nutrition and child health, which continued as baseline support under the Initiative. There was, however, little support for maternal health in this portfolio, and there were no project disbursements in the sectors of reproductive health and family planning.

With Muskoka funding, MFM approved twelve projects of which eleven had started to disburse by the end of fiscal year 2013/14. The projects did not signal a major strategic shift in the profile of the MFM health and nutrition program portfolio. They did show a tendency towards greater alignment with the MNCH strategy through three projects:

• i) support to WHO to implement the recommendations of the UN Accountability Commission (COIA) (Project M013603);
• ii) funding of the H4+ initiative of UN Agencies supporting the Global Strategy for Women’s and Children’s Health (Project M013402); and
• iii) core support to the Micronutrient Initiative for the expansion of its strategy for MNCH integration (Project M013403).

Overall, approximately 80% of MNCH funding by MFM continued to be in the areas of nutrition, child health and infectious disease control.

3.1.4 The alignment of KFM programming with the strategy of the MNCH Initiative

The Partnership for Development Innovation Branch was responsible for a relatively small funding component of the MNCH Initiative. However, it had a relatively large communications footprint through its partnership with Canadian civil society development and advocacy organizations. KFM developed a program strategic framework for the Muskoka Initiative Partnership Program (MIPP) that was closely aligned with the corporate MNCH Initiative strategy. Its main distinguishing feature is that it includes an immediate objective under each of the three paths that focuses on changes and actions at the community level.

While each MIPP-funded project had its own distinct profile, most of them focused on under-served or disadvantaged populations or regions. Common themes included community mobilization to strengthen the utilization of services for maternal and child health, activities to remove social, gender and geographic barriers of access to care, and initiatives to change community structures, norms and behaviours to improve nutrition, hygiene and sanitation. Increasing the decision-making power of women for household nutrition and health was a frequent theme, and, where relevant, projects also addressed the issue of adolescent pregnancy and early marriage.

Overall, the KFM program was well aligned with the MNCH Initiative strategy. The portfolio of activities was balanced across sectors and objectives. Initiatives addressing the community and demand side factors that contribute to poor maternal and child health were included in the projects funded under MIPP.

3.2 Effectiveness

3.2.1 Achievements of MNCH Initiative outcomes

The eleven UN (COIA) indicators that are the basis of the MNCH Initiative Performance Measurement Framework are tracked at the global and country level by Countdown 2015, a collaboration of international health partners mandated by the UN.^{Footnote 25} The 2014 report documented significant achievements at the global level and in the focus countries of the MNCH Initiative, especially in child health and, to a lesser degree, in maternal health. However, in most cases, the most recent information dated from 2011, which is too early to attribute any of the achievements to the Initiative.

For some of the ten focus countries, the evaluation team was able to collect more recent information from population surveys and the national health information system. The data shows further achievements in service indicators, such as antenatal care coverage, immunization coverage or coverage of skilled attendance at delivery. However, more recent information about achievements in higher level outcomes, such as a reduction in mortality rates, was generally not available.

Progress and evaluation reports of sampled projects included a wealth of information about achievements of outputs and immediate outcomes. These included reports on output targets, such as numbers of children immunized or numbers of health workers trained. One research project tracked intermediate outcome indicators across 10 MIPP-funded NGO projects.^{Footnote 26}A preliminary report released in June 2015 documented increases in all indicators of service coverage for maternal and child health, especially those related to nutrition. At the same time, it showed very little improvement in the prevalence of chronic malnutrition in the project areas. This finding confirms the known fact that higher level outcome indicators do not change in the short term.

3.2.2 Effectiveness of the DFATD program delivery channels

Targeting the Muskoka-funded component of the MNCH Initiative to ten focus countries strengthened Canada’s participation in the health sector and in the donor coordination dialogue at country level. The leverage effect varied from country to country.

• In Haiti and Bangladesh, Canada was already well established in the policy and partner coordination dialogue. The additional funding helped increase the profile and visibility of Canada’s position.
• In Mali and Ethiopia, effectiveness in the policy dialogue was reduced by the Canadian suspension of direct Government funding. Policy influence was primarily provided through intermediate partners, such as UN Agencies and international NGOs.
• Canadian policy work in Malawi had mainly focused on the education and nutrition sectors. It came to an end when Canada withdrew from bilateral development cooperation with this country.
• In Nigeria and Afghanistan, Canada had already established a strong policy presence in the partner fora for polio eradication. This was further reinforced by leverage gained through Muskoka funds. In Afghanistan, the additional Muskoka funding also helped to establish Canada as a lead partner in the nutrition sector.
• The greatest leverage effects were observed in Mozambique, Tanzania and South Sudan, supported by a high volume of Canadian MNCH funding and strong DFATD in-country health policy expertise.

In all countries, DFATD used the policy leverage gained from the MNCH Initiative to promote and strengthen the attention devoted to gender equality in the health sector. Other common policy themes effectively promoted by Canada included increased coordination of international development assistance, greater transparency and accountability in health systems management, decentralization of budgets and decision-making, and decreased fragmentation of the health sector, for instance by integrating nutrition into health service delivery.

There are examples of Canadian achievements in the national health sector policy and partnership dialogue in several countries although they cannot always be directly attributed to the MNCH Initiative.

Muskoka funding represented a relatively minor component of MNCH program portfolio of the MFM Branch, which was primarily baseline-funded. However, it raised the profile of Canada in international development policy fora on MNCH and increased the effectiveness of the promotion of Canadian policy priorities, such as gender equality, transparency and accountability in international organizations and initiatives. Programmatically, the most significant achievement of MFM Muskoka-funding was the increased collaboration of UN Agencies for MNCH through the support of the H4+ Secretariat (Project M013402) combined with country-level support for joint UN Agency projects by the Geographic programs in several countries.

Muskoka funding channelled through the KFM Branch served the important role of linking Canada’s MNCH Initiative to a large community of Canadian civil society organizations engaged in international cooperation. The Muskoka Initiative Partnership Program (MIPP) created leverage for both DFATD and Canadian civil society partners. First, the MIPP allowed Canadian civil society partners to expand the scale and scope of their MNCH programs. Second, the MIPP provided DFATD with access to technical expertise, public support and the contribution of additional human and financial resources towards the achievement of Canadian MNCH development objectives.

3.2.3 Management effectiveness

The MNCH Initiative was not a “managed program” and did not have a central management structure. Nevertheless, DFATD staff interviewed expressed the opinion that at the time of inception of the Initiative, the former CIDA Policy Branch provided strong thematic guidance as well as instructions about monitoring parameters, reporting requirements and imputation methods. Interviewees perceived this guidance as useful, but some expressed the opinion that concerns of decentralized staff were not always given sufficient attention. For instance, the question of how to estimate attributable results in multi-partner funded initiatives was, in the view of some interviewees, not answered in a satisfactory manner.

Prior to the amalgamation of the former CIDA with DFAIT, a governance and management structure of the MNCH Initiative had been established led by the former Strategic Policy and Performance Branch (SPPB). (See Figure 3) At the time of the evaluation, it was no longer clear how effective this structure was in guiding the Initiative. DFATD staff interviewed by the evaluation team expressed the opinion that after the first year of the Initiative, central guidance and management started to weaken. A MNCH Working Group of the Geographic Branches was formed in 2013 and met regularly until amalgamation of the former CIDA with DFAIT. It has since been recreated and is currently chaired by the Global Issues and Development Branch.

Figure 5: Governance structure of the MNCH Initiative of the former CIDA

• Text version

  Figure 5 illustrates the governance structure of the MNCH Initiatives of the former CIDA.

  At the first level, representatives from the Multilateral and Global Programs Branch, Geographic Programs Branch, and Partnerships with Canadians Branch all reported to the Health Coordination Group. Likewise, the Geographic Programs Branch Working Group also reported to the Director-General Program Support Committee.

  At the next level,  the Health coordination group from the former Strategic Policy and Performance Branch , the MNCH focal point, and the Director-General Program Support Committee all reported to the MNCH Global Coordination Group .

  This MNCH global coordination group then reported to the SPPB vice president at the top of the governance structure.

Source: DFATD (2013) Evaluability Assessments of Canada’s Contribution to the MNCH Initiative

At the time of the evaluation, the thematic leadership of the MNCH Initiative had devolved to the MFM Branch of DFATD. An Advisory Committee of senior managers continued to exist, but meetings were infrequent. Most DFATD staff interviewed during the evaluation could not identify a person or unit at headquarters who had a central coordination or management role. Financial commitments and expenditures continued to be followed closely through the Investment Monitoring and Reporting Tool, and adjustments to budget envelopes were made to keep financial commitments on track, but the planned application of a central Program Monitoring and Reporting Tool to collect and analyse programmatic information across the three Branches of DFATD for purposes of thematic guidance and management was never implemented.

3.2.4 Effectiveness of project delivery channels at country level

Weak health systems are common in all countries with poor maternal and child health indicators. Health systems strengthening was therefore a priority in all ten focus countries. However, in each country, the program approach was adapted to need, capacity and cooperation context.

In South Sudan, it started with supporting the establishment of emergency obstetric and neonatal services in a country that had almost no functional maternal health services. In Bangladesh, it focused on a chronic structural crisis in human resources for health. In Ethiopia, it aimed to integrate nutrition into primary health care. In Haiti, it became part of the post-earthquake reconstruction effort. In Tanzania, it supported the redefinition of the role of community health workers in supporting the provision of MNCH services in remote areas. Some of the project choices generated better results than others in terms of their contribution to maternal and child health, but none of the choices can be categorized as inappropriate.

The assessment of the choice of delivery channels is similarly differentiated. Country programs that had large budget allocations, for instance the Tanzania program, were able to strategically invest in complementary channels: (i) pooled public sector support to pursue a health sector reform agenda; (ii) support to UN specialized agencies to deliver technical assistance to the national strategy; and (iii) support to NGOs to cover gaps in the country’s health system and to keep the program strategy grounded in field experience.

A similar balance between different program channels was achieved in Bangladesh, although this was primarily baseline-funded from the existing health sector program budget. In South Sudan, where cooperation structures were just starting to be developed, there was gradual progress towards a similar spread of delivery channels. Other country programs that had a smaller allocation of funds and no major pre-existing health sector programming, for instance Malawi and Ethiopia, programmed most additional funds in multi-donor funded projects led by UNICEF or the World Bank, with some additional programming through NGOs. In Mali, political changes during the implementation of the initiative forced a rethinking and re-profiling of the delivery channels.

In selecting project delivery channels, the bilateral country programs made strategic decisions within their national policy and development contexts. In some countries, for instance in Tanzania, the DFATD country programs achieved additional leverage through collaboration with the implementing partners of KFM-funded projects in the country.

3.3 Efficiency

3.3.1 Meeting the targets for Canadian resource commitments

Accountability for MNCH results, according to the Performance Management Strategy, covered the entire MNCH commitment of baseline- and Muskoka-funded programs. Accountability for resources, however, uses two distinct methods for the two funding envelopes.

Disbursements of baseline funds are imputed to MNCH by applying the G8 weighting key to the sector allocations of each project according to DAC sector codes. The weighting key was developed by a working group of G8 health specialists prior to the Muskoka Summit to estimate current G8 contributions to MNCH prior to the Summit. It is based on the sector disbursements reported to the DAC by OECD member states, assigning a percentage weight to each sector according to estimated MNCH relevance. Disbursements of Muskoka funds are by definition considered to contribute to 100% to MNCH objectives. Disbursements are reported according to their estimated contribution to the three Muskoka paths (see Section 3.1.1). By the end of the 2013/14 fiscal year, DFATD had disbursed $2.46 billion, nearly 90% of the total five-year Canadian commitment to MNCH (Table 2, Section 2.1)

The two parallel methods of accounting for MNCH expenditures introduced complexity into the effort to establish accountability for resources. The G8 MNCH imputation formula used for baseline disbursements likely underestimated MNCH spending. Immunization projects, for instance, are coded under the sector of ‘disease control’ and therefore only imputed to 40% for MNCH. Reported disbursements of Muskoka funds, on the other hand, may sometimes have overestimated MNCH spending. For instance, the contributions to pooled health sector funds in Afghanistan (Project A035218) or Mozambique (Project A033033) were not MNCH specific and only contributed partially to MNCH goals. They were, however, funded from the Muskoka envelope and are therefore considered to have contributed 100% to MNCH.

If the G8 methodology were applied uniformly to all disbursements over the first four years of the Initiative, total reported DFATD spending on MNCH would be reduced from $2,456 million to $2,125 million. While the lower amount is not more valid than the higher amount, it represents about 78% of the five year DFATD spending target. This level of expenditure at the four-year mark would still be considered to be on track.

The UN Commission on Information and Accountability (COIA) recognized early that the lack of a uniform global system to account for MNCH resources was a problem. It requested the OECD to develop a system to mark MNCH expenditures in its Creditor Reporting System (CRS) Database. In February 2014, the OECD issued a directive to include a marker for reproductive, maternal, newborn and child health (RMNCH Marker) in the expenditure reports to the CRS.^{Footnote 27} This method, which is already used for other issues such as gender responsiveness, would require that all reported projects include a code identifying their contribution to MNCH goals (on a scale of zero to four). This would then be used to compute the total amount of MNCH support provided through the development expenditures of each OECD partner. This system which was developed with technical support from DFATD was tested by some DFATD programs in 2014, but by the time of the evaluation it had not yet been applied on a Department-wide scale, nor is it reported by other OECD members in the CRS database.

3.3.2 Timing of MNCH disbursements

Implementation of the MNCH Initiative officially started with the 2010/11 fiscal year, but the first half of that year had already passed when programming of the newly committed funds was approved by the Treasury Board in September 2010. It is, therefore, not surprising that the Geographic Program Branches only disbursed about 13% of targeted allocation for the ten focus countries in the first year. Two country programs made no disbursements, six programs disbursed against one project, and only the Mozambique program disbursed against two and the Haiti program against three.^{Footnote 28} All disbursements were contributions to pooled health sector funds or to multi-donor funded UN projects.

The MFM Branch was less constrained because it programmed primarily with baseline funds through an existing pipeline of projects and core contributions to multilateral partners. It was therefore responsible for the largest proportion (56%) of reported MNCH expenditures during the first fiscal year. It also managed to sign agreements and start disbursements for four of its twelve Muskoka-funded projects during the first fiscal year.

The apparent rapid disbursement of multilateral funds may, however, be partially misleading because the Branch often disburses advances for multilateral projects with slow implementation start-ups. One reviewed project, for instance, (Project M013596) was pre-financed with $9 million in March 2012, and eighteen months later reported expenditures of only $0.5 million. Similar implementation delays were also observed for a baseline-funded multi-country project for diarrhoea and pneumonia treatment in Tanzania. (Project M013810).

The KFM Branch announced the selection of MIPP projects in September 2011, already well into the second year of the Initiative, and started disbursements for all selected projects in the last quarter of the 2011/12 fiscal year.

For the 10 focus countries of the initiative, it meant that more than half of all MNCH Initiative programming in the first fiscal year was financed with baseline funding. Disbursements of Muskoka funds started towards the end of the year.

Figure 6: MNCH disbursements (all Branches) in the 10 focus countries by fiscal year (million $)

• Text version

  Fiscal Year	Baseline	Muskoka	Total
  2010/11	100.4	90.5	190.9
  2011/12	102.1	142.1	244.2
  2012/13	112.0	207.5	319.5
  2013/14	85.7	183.2	268.9

Source: DFATD MNCH database: Version of November 3, 2014

Effectively, the time window for the implementation of projects funded with the Muskoka allocation of $1.1 billion was four, rather than five years, and for partnership projects only three years or less. For the bilateral and multilateral projects, this is not a major issue since the Geographic and MFM Branches have on-going cooperation programs with their country and global partners. They can thus renew and continue to commit funds. Furthermore, many of the disbursements for multi-year projects of UN and other global partners were made up-front, keeping the disbursement schedule on target. It is, however, a major issue for the MIPP-funded projects. These projects had relatively fixed endpoints in 2015, and the late start plus any further inception delays meant that most had three years or less in which to implement the agreed program.

3.4 Economy

Because of the nature of the MNCH Initiative, there was no uniform mechanism or process to select inputs. The selection was country- and context-specific, and it ranged from high level systems inputs, such as interventions to improve State-level budgeting in Nigeria, to highly specific inputs, such as the support for the introduction of misoprostol (a medication to control post-partum haemorrhage) in South Sudan.

The range of interventions supported by the Initiative was in line with the list of high impact interventions identified by the G8 Development Ministers in 2010 with some expansion due to the focus on health systems strengthening.

The main factors that assured the economic implementation of the MNCH Initiative were (i) the selection of the MNCH Initiative focus countries on parameters of need and existing Canadian capacity to respond, (ii) the alignment of interventions in these countries with national strategies, and (iii) the responsiveness of the country programs to the epidemiological and programmatic context facilitated by the decentralized management of the Initiative.

In the focus countries, several of the DFATD health systems strengthening interventions supported the decentralization of planning, budgeting and governance of health services to the district level. DFATD promoted decentralization through programmatic interventions, for instance, through the financing of local level planning grants in Bangladesh, and through policy support in the health sector and partner coordination fora in several of the focus countries.

The DFATD promotion of decentralization was based on evidence generated by the Tanzania Essential Health Interventions Project. It indicated that better health outcomes can be achieved at lower costs when budgets and decision-making is decentralized to the district level.^{Footnote 29} The project started in the 1990s and was supported, among others, by the former CIDA. It is an example of how evidence generated with Canadian support was used to inform the MNCH Initiative.

The evaluation found in the review of sampled projects that in practically all cases the types of interventions funded were highly effective and cost effective according to published evidence of the impact and cost of different maternal and child health interventions.^{Footnote 30} Very few exceptions were noted, all of them among project components rather than entire projects. For instance investments in high technology special care neonatal units in district hospitals in Bangladesh (Project A035530) were in some cases questionable because the technology was not adapted to the technical and human resource capacity of these facilities. Comparable results at lower costs were achieved by the implementation of ‘kangaroo care’^{Footnote 31} in NGO hospitals in the same region (Project S065372).

All potential gaps in the selection of inputs that were mentioned by stakeholders interviewed during the evaluation referred to programs addressing upstream factors that contribute to maternal and child mortality. These included initiatives addressing adolescent sexual health, and reducing early marriage and adolescent pregnancy, as well as initiative’s targeting increased female literacy and economic empowerment and reducing gender-based violence. Some stakeholders also mentioned water and sanitation as a relative blind spot in the MNCH Initiative, although about 4% of Muskoka-funded expenditures were made in this sector. (see Section 3.1.1)

3.5 Cross-cutting themes

3.5.1 Gender equality

During project development and the negotiation of MNCH project agreements, the issue of gender equality was clearly in focus. Project partners responded with different levels of rigour.

Some NGO partners prepared detailed and well researched analyses of gender equality issues affecting maternal and child health in their implementation area, for instance for the Improving MNCH project in Afghanistan (Project A035242) or the Maternal and Child Health Enhancement project in South Sudan (Project S065383).

Textbox: Addressing gender equality in MNCH projects

Gender equality issues affecting the risks to maternal and child health and the utilization of services differ from country to country and within countries. Some projects meticulously documented these issues and brought them to the attention of national and international MNCH programs, for instance through research on the association of gender-based violence and maternal mortality in Nigeria by the Evidence-based Health Systems Initiative (Project A031274). In some countries, the projects and the DFATD country teams continue to face resistance by the professional leadership in the health sector in terms of recognising the impact of gender inequality on maternal health.

Gender equality issues also affect the level of service provision, notably human resources for health. In Afghanistan, the program contributed to increasing the proportion of female health providers, resulting in an increase in the utilization of services for maternal health. In Bangladesh, systemic gender discrimination affects the status of nurses and midwives, an issue addressed by the human resources for health project (Project A034608). In South Sudan, the midwifery project prepared a gender analysis claiming a ‘sudden influx of males’ into the midwifery profession (Project A035518). There was a concern that raising the professional profile of midwives may lead to a crowding out of female training applicants by males, an issue that was taken up in the gender strategy of the project.

Pooled health sector support projects generally addressed gender equality issues superficially despite considerable efforts by DFATD staff to increase the gender focus. The attention of the pooled projects was often limited to the collection of sex-disaggregated data or to counting male and female staff members, as for instance in Bangladesh, Afghanistan and Mali.

The issue of gender equality was raised systematically by DFATD staff in the health sector policy and the partner coordination fora at country level, and by DFATD MFM representatives in the governance and coordination committees of UN Agencies and Global Health Initiatives, for instance at the Board of the GAVI^{Footnote 32}- The Vaccine Alliance, where it contributed to an evaluation of the GAVI’s gender policy.^{Footnote 33}

3.5.2 Governance

Stakeholders interviewed at country level confirmed that the Muskoka Initiative contributed to increasing the impact of Canadian participation in national dialogues on reproductive, maternal and child health. The additional program funds further increased the leverage in the policy dialogue, with DFATD country teams generally focusing their effort on transparency, accountability and coordination of stakeholders in the health system.

The issue of budget decentralization and adequate funding of health districts was an important theme of the DFATD contribution to the health sector dialogue, for instance in Bangladesh, Tanzania and South Sudan. At the global level, one of the major achievements of DFATD under the MNCH Initiative was the Canadian contribution to increasing the cooperation among UN agencies under the H4+ umbrella in their support to the UN Strategy for Women’s and Children’s Health. (Project M013402).

3.5.3 Environmental sustainability

Support of projects to improve water and sanitation was limited under the MNCH Initiative, but nevertheless accounted for 4% of Muskoka funding. Medical waste disposal was addressed in the majority of health service projects according to the reviewed project documents. The evaluation was not able to determine if this was implemented with sufficient rigour and quality. Some interviewed project partners, for instance several NGO partners working in rural Tanzania, stated that the issue of water and sanitation in health facilities should receive greater attention in the effort to improve facility-based MNCH services.

3.6 Sustainability

Baseline-funded and Muskoka-funded projects were developed within frameworks of country or multilateral cooperation strategies that shifted towards an MNCH focus and that received a significant increase in funds through the MNCH Initiative in some countries. Most in-country stakeholders who were interviewed stated that they saw little change in the Canadian cooperation with the start of the Initiative other than increased DFATD interest, support and involvement in MNCH issues.

The sustainability strategy pursued by DFATD relied mainly on an alignment with national health, nutrition, and reproductive health strategies, and on a strong focus on systems support. Overcoming management inefficiencies, increasing public participation/utilization of health services through decentralization and increased public accountability, improving the quality of services and overcoming human resources bottlenecks are all approaches that promote sustainability.

Where the national policy context was favourable, as for instance in Tanzania, DFATD included the support to public-private partnerships in health service delivery through cooperation with the Aga Khan Foundation (Project A035252) and with Comprehensive Community-based Rehabilitation (Project D000164), a national NGO that provides hospital care through cross-financing between for-profit and not-for profit services.

Supporting changes in health systems towards greater sustainability is a long-term strategy. It is not without risks. The success depends on a country’s stability which, for fragile states like South Sudan is not a short term prospect. It also depends on political stability and the continued prioritization of health by the political leadership. An example is Ethiopia which, through the politically driven prioritization of primary health care, has made major progress towards a more robust and sustainable health system.^{Footnote 35} Short-term political decisions like the proportion of public spending allocated to the health sector, also have major effects on the success or failure of the sustainability strategy. The mitigation strategy for these contextual sustainability risks is a long-term engagement in cooperation and policy processes in the country.

The review of MIPP-funded projects in the focus countries raised critical issues of sustainability. The projects were awarded on the basis of call for proposals. Because of the unavoidable time lag for the design of the program, calls for proposals, selection of projects and signing of contracts, many of the projects had less than three years of implementation. These three years were further shortened by the need to conduct baseline and end-line studies and to set up local cooperation structures. During this short implementation period, each of the projects attempted to implement an ambitious agenda of affecting change at the community and at the service level. Some of the projects were embedded in established long-term health and community systems development projects and served to orient existing cooperation activities towards MNCH. But others only had a short MIPP-funded program window to build a partnership, implement a program and phase-out the cooperation. In such a context, the prospect of sustainability is unrealistic.

3.7 Performance management

3.7.1 Performance monitoring

The performance management strategy of the MNCH Initiative was developed in 2011.^{Footnote 36} It includes a logic model and a performance measurement framework. . While the logic model is specific to the Muskoka-funded component, “performance monitoring and reporting outlined in the strategy covers the $2.85 billion. Tracking and reporting on the results will represent both funding together. It is impossible to distinguish results attributed to each funding amount since higher-level results are the objectives of all initiatives under Muskoka.”^{Footnote 37} The performance measurement framework does not include any targets. According to the performance management strategy, the logic model and the performance management framework were meant to be ‘navigation maps’ to orient country programs.

Performance data for MNCH projects were to be collected annually through a Program Management Reporting Tool which would collect performance data at the project level and aggregate them at headquarters. The evaluation found a small number of completed performance reports using the tool. However, the tool was never used systematically.

In the absence of a system for the routine collection of program performance data, information was collected and compiled as needed. For instance in 2013, on the Minister’s request, the Geographic Branches compiled an MNCH progress report by consolidating information from sixteen country programs and one regional program.^{Footnote 38} Progress reports were also prepared by the then Multilateral and Global Programs Branch (MGPB) and by the then Partnerships with Canadians Branch (PWCB). A plan to synthesize these three reports in a Department-level MNCH progress report was not realized.

Most sampled projects had detailed performance measurement frameworks and most submitted regular reports. Reporting by NGOs was generally more regular and complete. Most projects implemented by UN partners provided semi-annual or annual narrative reports that included some but not all of the performance indicator data. Projects of contributions to pooled health sector funds did not have DFATD-specific logic models or performance measurement frameworks, but they had regular mechanisms for joint partner performance monitoring.

Throughout the first four years of the MNCH Initiative, DFATD succeeded in compiling and reporting performance information, although the Department was not successful in establishing a streamlined system. The transaction costs of collecting and compiling this information was high. One project manager interviewed estimated that supported health staff spent up to 40% of their time preparing reports and completing statistical information sheets. A group of NGOs with MIPP-funded projects engaged a research organization to track performance data for 10 projects over a period of three years at a cost of $1.1 million.^{Footnote 39} In interviews, some decentralized DFATD staff stated that performance reporting took much of their time at the start of the Initiative but had since become manageable.

Monitoring results that are attributable to DFATD support was an issue raised in several interviews. Output-level performance data, such as the number of health workers trained or the number of children immunized, were usually compiled from project reports. For multi-partner funded and pooled health sector support projects, reported results were prorated according to the proportion of Canadian contribution to the fund. Either method discounts the national contribution and the community efforts, which are often much higher than the international partner support. For higher level outcome indicators, for instance for the number of deliveries attended by skilled personnel, this is even more of an issue because achievements are not driven by project inputs alone.

The overall assessment is of a performance monitoring system that provided accountability for results, however with a high level of systemic inefficiencies. These inefficiencies generated costs that were to a large extent born by project partners and frontline health staff. They also generated opportunity costs of foregone investments in strengthening routine national health management information systems.

DFATD is committed to reducing the reporting burden on national health systems and service providers. This is restated in the MNCH performance management strategy. However, this commitment was not fully translated into in the performance monitoring practice of the MNCH Initiative.

3.7.2 Performance management

The evaluation did not identify a specific performance management system for the MNCH Initiative. The performance of projects funded under the Initiative was managed at the project and at the program level using well established and generally robust Branch- and Program- specific processes including reporting, on-site monitoring, contracted project monitors, project management and oversight committees as well as project and country program evaluations.

4. Conclusions and Recommendations

4.1 Conclusions

DFATD’s MNCH Initiative has effectively contributed to the UN Global Strategy for Women’s and Children’s Health and is on course to deliver upon the Muskoka commitments of the Government of Canada. In the implementation of the MNCH Initiative, DFATD has lived up to the high standards of accountability for results and resources for MNCH that were established under Canadian leadership at the global and the G8 level.

1. The MNCH Initiative effectively supported efforts to reach the goals of MDGs 4 and 5 in ten focus countries with high rates of maternal and childhood mortality. Programming in these focus countries was well aligned with national priorities and systems and allowed DFATD to concentrate resources and promote Canadian priority development themes, such as gender equality, transparency and accountability. This was facilitated by the decentralized institutional design of the MNCH Initiative, which placed decision-making about priorities and areas of investment in the hands of established country- and multilateral programs. Such an institutional model, however, requires strong technical leadership and coordination at the central (headquarters) level to maintain the overall coherence of the Initiative. This leadership was present at the start of the MNCH Initiative but weakened over time.
2. The Department’s MNCH strategy focused on the supply side of health interventions, namely improving the availability, quality and equity of health services. Maternal and child health outcomes are, however, also dependent on demand-related factors, such as care-seeking behaviours and accessibility of services. This is captured in the “three delays” model of the causes for maternal deaths: The delay in seeking care (demand), the delay in reaching a health facility (demand and supply); and the delay in receiving appropriate treatment (supply). The main focus of the MNCH Initiative was on the third delay, namely the provision of adequate care.
3. Programs that address the root causes of high maternal and child mortality at the community level were underrepresented in the MNCH Initiative. Reducing maternal and childhood mortality cannot be achieved through health care services alone. They also require effective efforts in prevention, such as reducing adolescent pregnancy, gender violence, women’s lack of power in household decision-making and unmet need for family planning.
4. The Department’s investments in health systems strengthening have contributed to achievements towards MNCH goals at the country level. Strengthened health systems (i.e. improvements in health sector governance, human resources, financing, technology, and information management) are required to deliver effective services for maternal and child health, reducing the burden of diseases, and improving nutrition. By defining health systems strengthening as one of three intermediate objectives, the logic model of the Muskoka-funded component of the MNCH Initiative aggregated general interventions to strengthen health systems (such as the support of Ministries of Health through pooled health sector funds) with targeted interventions to improve maternal health service delivery (such as salary incentives for birth attendants). The aggregation of disbursements for systems strengthening and for service delivery reduced the clarity of the Canadian support for MNCH, and weakened the role of the logic model as an instrument for strategic guidance to country- and multilateral programs.
5. The Department is on track to meet or exceed the Canadian resource commitments of the Muskoka Summit. However, the dual accounting system for baseline- and Muskoka-funded disbursements for MNCH generated under- and over-estimations of expenditures. Notwithstanding the accounting challenges, the MNCH Initiative provided program and policy inputs at country level that were cost-effective and appropriate to the context.
6. The cross-cutting themes of gender, governance and the environment were addressed appropriately. Many projects implemented under the MNCH Initiative included well researched analyses of gender equality issues as well as detailed strategies on how to address them. Similarly, the limited number of construction projects supported under the Initiative included environmental assessments. DFATD engaged meaningfully in initiatives to improve health sector governance in focus countries, and to some extent at the level of multilateral organizations.
7. The sustainability strategy of MNCH projects largely relied on the support to health systems strengthening and on the alignment with national priorities and strategies. This is an effective approach to promote sustainability, but it also entails major contextual risks. These risks can be mitigated by extending the horizon of health sector cooperation, ensuring funding predictability, and working closely with international partners and national governments in target countries to develop, implement and sustain effective interventions. Several MIPP-funded projects faced major sustainability issues because of the short cooperation time frame.
8. DFATD monitored the overall performance of the MNCH Initiative, albeit with a high level of systemic inefficiencies because of non-implementation of the planned performance monitoring system. This generated costs which were partially born by project partners and service providers. It weakened DFATD’s delivery on its commitment to reduce the reporting burden on developing countries. Performance management of projects funded under the MNCH Initiative, on the other hand, used Branch- and Program-specific processes which are well established and robust.

4.2 Recommendations

The following recommendations build on Canada’s leadership role and significant contribution to the G8 Muskoka Initiative and the UN Global Strategy for Women’s and Children’s Health. Canada’s key role in establishing the foundations for a global initiative to improve the health of women and children in the world’s most vulnerable regions is well documented and recognized.  In particular, Canada made a significant contribution to creating frameworks that encouraged accountability, pragmatism and focus among partners and within its own international development program for MNCH.

1. The Department should consider balancing its support for interventions targeting the demand and supply side of MNCH services. Maternal, neonatal and child mortality is overwhelmingly due to delays in (1) deciding to seek appropriate medical help for an emergency (demand); (2) reaching an appropriate facility (demand and supply); and (3) receiving adequate care when a facility is reached (supply). The evaluation found many examples of effective programs to overcome the first two delays. However, the main focus of the MNCH Initiative was on the third delay, namely the provision of adequate care. Greater impact could be achieved by extending the focus to cover all three delays.
2. The Department should consider widening the scope of the MNCH Initiative programming by placing greater emphasis on addressing factors contributing to high maternal, newborn and child mortality, such as reproductive health. Evidence indicates that adolescent pregnancy, gender-based violence, poor access and low uptake of reproductive health services, including family planning, as well as low levels of female education are significantly correlated with maternal, neonatal and child mortality. Efforts to address these issues would contribute to reduced mortality for both mothers and children, and ease the resource requirements for the provision of adequate health care. 
3. The Department should strengthen its capacity to provide horizontal leadership and coordination, as well as, technical guidance on health to all programming branches. This should be provided by a specifically mandated MNCH group at DFATD headquarters. The institutional design of the MNCH Initiative as a thematic strategy delivered across existing DFATD program channels and focusing on a limited number of countries was effective and well aligned with national priorities and systems. Building on this experience, greater emphasis on providing clear and timely direction to programming branches on key issues that arise during implementation will further reinforce the effectiveness of the Initiative.
4. The Department should continue harmonizing financial reporting for the Initiative. DFATD, in consultation with other OECD DAC partners, is moving forward with implementing the DAC’s Reproductive, Maternal, Newborn and Child Health (RMNCH) marker system in its expenditure reports to the DAC. Maintaining momentum and ensuring full implementation of this approach will increase the transparency of accountability for resources disbursed in support of the MNCH Initiative.
5. The Department should continue to improve the sustainability of the Initiative by maintaining a long-term strategic horizon and working closely with international partners and national governments in focus countries to develop, implement and sustain effective interventions. Changes in recipient countries’ health system requires time, predictable funding, as well as local ownership and political support. Addressing these enabling issues will sustain and accelerate MNCH results.
6. The Department’s strategy for achieving MNCH results through the strengthening of health systems was effective. When pursuing this strategy in the future, the Department should consider aligning the MNCH Initiative Performance Management Strategy more closely with the World Health Organization (WHO) framework of health systems building blocks. This would create greater transparency of investments and provide better strategic guidance. Under such a framework, improved service delivery is a central output of investments in governance, financing, human resources, information, technology and commodities.
7. In order to further enable transparency and demonstrate results to Canadians, the Department should consider establishing and implementing a robust MNCH system of performance monitoring and reporting. This system should adhere to the greatest degree possible to Canada’s commitments to “increasingly use shared mechanisms for managing and accounting for funds, reporting on progress and reviewing performance”^{Footnote 40}. Based on this principle, the performance monitoring and reporting system should allow for aggregating information at the Departmental level while recognizing the limitations of directly attributing results to Canada’s intervention.