Summative Evaluation of Canada's Afghanistan Development Program 2004-2005 — 2012-2013 - Synthesis Report

March 2015

Table of Contents

Acknowledgments

We would like to acknowledge the independent evaluation team from Ecorys for their work in the conduct of this evaluation, including data gathering and analysis, and technical reporting. Anneke Slob led the team, which included Alessandra Cancedda, Josef Decosas, Khadijah Fancy, Ivo Gijsberts, Yvan Conoir and Anette Wenderoth. We thank them for their diligence and professionalism in undertaking this challenging assignment. We are also very grateful to Ted Kliest for his detailed feedback as the Peer Reviewer for this evaluation.

Particularly appreciated was the time afforded to the evaluators by the various stakeholders involved during the course of this initiative. This included government officials, representatives of international organizations, civil society organizations, the private sector, and other development partners, who graciously made themselves available for interviews in Canada and Afghanistan. In particular, we would acknowledge the Afghanistan Development Program, the International Humanitarian Assistance Bureau, the Development Policy Directorate, the Stabilisation and Reconstruction Task Force, and the Department of National Defence, all of whom were represented on the Internal Evaluation Advisory Committee.

From the Development Evaluation Division, Vivek Prakash managed much of the evaluation process, succeeded by Deborah McWhinney who made a major contribution to drafting the final synthesis report. Michelle Guertin and then Tara Carney, Evaluation Team Leaders, were responsible for overall supervision.

James Melanson
Head of Development Evaluation

List of Acronyms and Abbreviations

ADB
Asian Development Bank
AIHRC
Afghanistan Independent Human Rights Commission
AIRP
Arghandab Irrigation Rehabilitation Program
AKF
Aga Khan Foundation
ANDS
Afghanistan National Development Strategy
APEI
Afghanistan Polio Eradication Initiative
APHI
Afghan Public Health Institute (Ministry of Public Health)
ARTF
Afghanistan Reconstruction Trust Fund
ATF
Afghanistan Task Force
AusAid
Australian Agency for International Development
BEGE
Basic Education and Gender Equality
BRAC
Bangladesh Rural Advancement Committee
CARE
Cooperative Assistance and Relief Everywhere (CARE International)
CBE
Community Based Education
CDC
Community Development Council
CEAA
Canadian Environmental Assessment Act
CEDAW
Convention on the Elimination of all Forms of Discrimination against Women
CF
Canadian Forces
CIDA
Canadian International Development Agency (now DFATD)
CIMIC
Civil Military Cooperation
CPSU
Canadian Program Support Unit
CRS
Catholic Relief Services
CSO
Civil Society Organization
DAC
Development Assistance Committee
DANIDA
Danish International Development Agency
DDR
Disarmament, Demobilization and Reintegration
DFAIT
Department of Foreign Affairs and International Trade (now DFATD)
DFATD
Department of Foreign Affairs, Trade and Development Canada
DFID
Department for International Development (United Kingdom)
DND
Department of National Defence
DoWA
Department of Women Affairs
EQUIP
Education Equality Improvement Program
EU
European Union
FIDH
International Federation for Human Rights
FY
Fiscal Year
GAVI
Global Alliance for Vaccines and Immunization
GBV
Gender-based violence
GDP
Gross Domestic Product
GE
Gender Equality
GEP
Girls' Education Project (BRAC)
GESP
Girls' Education Support Programme (AKF)
GHD
Good Humanitarian Donorship
GIRoA
Government of the Islamic Republic of Afghanistan
GoC
Government of Canada
HQ
Headquarters
I-ANDS
Interim Afghan National Development Strategy
IASC
Inter-Agency Standing Committee
INGO
International Non-Governmental Organisation
ISAF
International Security Assistance Force
KLIP
Kandahar Local Investment Program
KPRT
Kandahar Provincial Reconstruction Team
LOTFA
Law and Order Trust Fund for Afghanistan
M&E
Monitoring & Evaluation
MACCA
Mine Action Coordination Centre in Afghanistan
MAPA
Mine Action Programme in Afghanistan
MDG
Millennium Development Goal
MFI
Microfinance Institution
MGPB
Multilateral and Global Partnerships Branch
MISFA
Microfinance Investment Support Facility for Afghanistan
MNCH
Maternal, Newborn and Child Health
MOPH
Ministry of Public Health
MORE
Ministry of Women's Affairs Organizational Restructuring and Empowerment Project
MoWA
Ministry of Women Affairs
NABDP
National Area-based Development Programme (UNDP)
NAPWA
National Action Plan for Women of Afghanistan
NATO
North Atlantic Treaty Organization
NESP
National Education Sector Plan
NGO
Non-Governmental Organisation
Norad
Norwegian Agency for Development Cooperation
NPP
National Priority Program
NRVA
National Risk and Vulnerability Assessment
NSP
National Solidarity Program
OCHA
United Nations Office for the Coordination of Humanitarian Affairs
OECD
Organisation for Economic Cooperation and Development
PACE-A
Partnership for Advancing Community Based Education in Afghanistan
PCO
Privy Council Office
PRT
Provincial Reconstruction Team
PSU
Program Support Unit
RFAW
Responsive Fund for the Advancement of Women
RRMAF
Results and Risk Management Accountability Framework
RoCK
Representative of Canada in Kandahar
SCR / UNSCR
(United Nations) Security Council Resolution
SIDA
Swedish International Development Cooperation Agency
SME
Small and Medium Enterprise
ToR
Terms of Reference
UN
The United Nations
UNAMA
United Nations Assistance Mission in Afghanistan
UNDP
United Nations Development Programme
UNFPA
United Nations Population Fund
UNHCR
United Nations High Commissioner for Refugees
UNICEF
United Nations Children's Fund
UNMAS
United Nations Mine Action Service
US
United States
USAID
United States Agency for International Aid
WB
The World Bank
WFP
World Food Programme
WoG
Whole of Government

Executive Summary

Objectives, scope and methodology

This evaluation was undertaken for accountability and learning purposes, to assess the performance and results of the Afghanistan Development Program in a fragile state and complex environment. The evaluation was implemented from April 2013 to June 2014 by an international team from Ecorys Netherlands. This synthesis report was prepared by the Development Evaluation Division, based on the data gathering and analysis that Ecorys undertook.

The evaluation covers the period from fiscal year 2004-05 to 2012-13. In June 2013, as this evaluation was starting, the Department of Foreign Affairs and International Trade (DFAIT) and the Canadian International Development Agency (CIDA) were amalgamated into Foreign Affairs, Trade and Development Canada (DFATD). The object of this evaluation was Canada's Afghanistan Development Program, implemented by the then CIDA, recognising that it needed to be understood in the context of both Canada's "Whole of Government Approach" and the efforts of the wider international community.

The project sample assessed for this evaluation covered 55% of the project portfolio, including all main programming sectors, cross-cutting themes and a specific focus on Kandahar.

The evaluation design was based on internationally agreed principles for evaluations in fragile states. Four main evaluation questions addressing the criteria of relevance, effectiveness, efficiency, impact and sustainability were consolidated under one overall evaluation question: 'To what extent has CIDA contributed to a more secure and democratic Afghanistan, able to deliver key services to Afghans and better provide for its longer-term stability and sustainable development?' Program and sector evaluation matrices were developed to answer the evaluation questions based on solid evidence.

The evaluation team applied a multi-method approach to gather and analyze both qualitative and quantitative information. More than 2,000 documents were analyzed. Field visits to Kabul, Kandahar, Herat and Jalalabad were carried out and over 220 interviews were conducted with those involved in the Canadian engagement in Afghanistan during the evaluation period.

Background and basic characteristics

The evaluation covers a period in which development activities were carried out in three broad phases – i) 2004 to 2007; ii) 2008 to 2011; and, iii) 2011 to 2013.
From 2004-05 to 2012-13, the total volume of international assistance disbursed to Afghanistan by CIDA amounted to $1,546 million, with 310 initiatives implemented. This was the largest Country Program implemented by CIDA up to that time. Disbursements were highest from 2007-08 to 2011-12 – between $215 and $280 million per year – when the Whole of Government approach was implemented. Before and after that, annual disbursements were approximately $100 million per year.

Canada's Afghanistan Development Program was part of the unprecedented international community involvement in Afghanistan after 2001 when the Taliban were ousted in a military effort – Operation Enduring Freedom, led by the USA. Canada was among sixty bilateral donors and forty-seven troop-contributing countries. 

There was a growing resurgence of the Taliban between 2006 and 2009, particularly in the south of the country. From 2010 to 2013, the insurgency continued throughout the country and resulted in significant loss of life. A transition to full Afghan leadership and responsibility for the country's security, development and reconstruction in all spheres became the focus of attention from 2010 onwards. By early 2014, the withdrawal of most international troops was underway and many countries, including Canada, had completed their withdrawal as of March 2014.

A focus on state-building and long-term development support was reflected in Canada's Afghanistan Development Program with economic growth and democratic governance representing the most important sectors in terms of disbursements (each 22% of the total). Emergency assistance represented 12% of total disbursements, which is comparable to contributions of other international agencies in this area.

CIDA developed its first strategy for the Afghanistan Development Program in 2003. When Canada assumed leadership of the Kandahar Provincial Reconstruction Team in August 2005 and sent 2,500 soldiers to secure this province, CIDA began programming there as well. However, insurgency increased and fighting in Kandahar was quite intense, which affected development activities. In October 2007, the government commissioned an Independent Panel to examine Canada's future role in Afghanistan. In January 2008, the Independent Panel issued its report, commonly known as 'The Manley Report'. The Government of Canada's response to the Manley Report was multi-pronged, including: the definition of six policy priorities and three signature projects to focus Canada's efforts; a shift from development programming at the national level to 50% of programming focused in Kandahar; and a Whole of Government approach with involvement of the Privy Council Office in the planning and management of Canada's engagement in Kandahar. After the withdrawal of the Canadian military from Kandahar in 2011, the Afghanistan Development Program returned to its national focus and development activities specific to Kandahar were phased out.

Main findings and conclusions

On continuity and change in programming

Canada's Afghanistan Development Program over the 2004 to 2013 period can be characterized by continuity in involvement on the one hand, and by clear changes in strategy and focus on the other. There was: i) an initial focus on state-building at the national level from 2004 to 2007; ii) a concentration on stabilization in Kandahar from 2008 to 2011; and, iii) a humanitarian, social sector and gender equality-oriented Program after 2011.

On short-term achievements and longer-term results

Canada, together with other donors, contributed to important short-term achievements in various sectors, ranging from the construction and rehabilitation of thousands of schools, increased enrolment, especially of girls, improved access to health facilities, construction of community infrastructure, delivery of food to millions of people and support to the Afghanistan Independent Human Rights Commission and women's organizations.

Given Afghanistan's low level of development in 2002, and ongoing insurgency, long-term development results were difficult to realize. Nevertheless, real gains were made, especially in the social sectors. Access to and use of health and education services have increased and a considerable amount of land has been cleared of mines. However, in economic growth, human rights and governance, little substantive change beyond the project level was observed. There is contradictory evidence on the level of awareness of human rights. In most sectors, issues of distribution and equity remain unaddressed.

Gender equality results are mainly concentrated in the social sectors through improved access to services. Limited improvements for women related to human rights and their role in decision-making were observed, though long time frames are needed for these kinds of societal changes.

Canada contributed to strengthened national capacity – particularly in the health, education and demining sectors. However, there is evidence of missed opportunities, especially in the work to strengthen sub-national governance and establish adequate linkages to national government. Generally, there were more short-term achievements than long-term development results.

On risk awareness and efficiency

Programming in Afghanistan entails certain risks that cannot be completely mitigated. There is evidence that the Afghanistan Program analyzed programming risks and developed mitigation strategies throughout the evaluation period but was, at times, quite risk averse. A project-level focus precluded the ability to address sectoral or cross-sectoral issues effectively. Despite CIDA's comprehensive toolkits for results and risk management, there was a lack of specific guidance on how to identify, document or manage risks in a fragile and conflict context beyond the project level. A very prominent and effective risk mitigation measure taken by the Program was the choice made to diversify the range of implementing partners.

Canada's Afghanistan Development Program addressed some efficiency considerations at the level of projects and implementing partners. However, major efficiency issues that affected overall Program performance, such as staff mobility in a conflict environment, rotation, and centralized decision-making, were only addressed from 2008-2011 when the Whole of Government approach was implemented.

On unintended impacts and sustainability

There was evidence of unintended impacts of the aid provided by the international community in Afghanistan – both destabilizing effects, including the shrinking of humanitarian space, and stabilizing effects, especially in urban environments where access to health and education facilities increased.

With the benefit of hindsight, it can be concluded that sustainability of development results – in particular, building the necessary capacity and local ownership – requires more time than foreseen in military stabilization theories. Furthermore, Canada's contribution as part of the broader international community's capacity building efforts in this area led to positive results but also to the creation of a parallel civil service and excessive use of costly technical assistance.

On Canada's role in the international community

Canada has been a consistent and reliable donor working within negotiated international frameworks of engagement with Afghanistan. In recent years, Canada has led the dialogue on human rights and the elimination of violence against women, which was in line with its strategic focus. Canada's role in general policy dialogue, as well as on education, health, human rights and gender equality policies, has been appreciated by other development actors and by the Government of the Islamic Republic of Afghanistan. Progress by the Afghan government in some key areas supported by Canada, such as human rights and anti-corruption, was seen by some members of the international community to have been limited, although good progress was noted in other areas, such as public finance management.

Canada improved the alignment of its support with Afghan Government priorities, but fell short of the internationally agreed 50% target for on-budget support (see more detailed explanation of this in Chapter 3). The relatively low proportion of on-budget support and the decline in the absolute amount of aid provided by Canada is likely to affect its influence in policy dialogue.

On development in Kandahar as part of the Whole of Government Approach

The start of the development Program in Kandahar in 2005-06 was slow. Despite huge efforts to speed up implementation, fundamental absorption capacity problems made it impossible to reach the target of spending 50% of all disbursements in Kandahar. From 2008 to 2011, 29% of the development program disbursements were in Kandahar.

Understanding the political economy and main drivers of conflict and fragility received relatively little attention in Canada's Afghanistan Development Program, but Canada was not exceptional in this regard. The situation improved when the Whole of Government approach was implemented in Kandahar, as considerable effort was made by various Canadian actors to better understand the context by using situational awareness mapping and other tools. A genuine attempt was made to identify and address grievances and to deal with drivers of conflict to the extent possible. However, this understanding remined incomplete. The principles for engagement in fragile and conflict-affected states call for a thorough understanding of the context, including the conflict. In practice, the international community, Canada included, was more focused on implementation.

Within a very insecure environment, impressive short-results were realized and documented in the 14 reports to Parliament on Canada's engagement in Afghanistan from 2008 to 2012. One of the reasons for these achievements was that the various Canadian actors on the ground worked closely together from 2008 to 2011 in Kandahar. Work in silos and 'stove-piping' were overcome to a significant extent. Furthermore, institutionalized mechanisms for learning were established. There was a clear drive among development professionals and their colleagues, especially the ones that were present in the field, to learn from this important and unique experience of working jointly in a conflict zone. In 2010 and 2011, very interesting lessons for development programming in a conflict environment were drawn, touching on basic questions of the possibility of long-term development in a conflict zone that went beyond counterinsurgency and stabilization theories. Unfortunately however, these considerations and lessons were not taken up at a more strategic level.
The implementation of the development Program in Kandahar showed that long-term development cannot be accomplished with an emphasis on short-term implementation strategies, which sped up project delivery considerably, but failed to ensure sustainable, long-term development results in more than a few areas.

Main lessons and recommendations

Need for a broad strategic vision and the establishment of an institutional mechanism to build on the lessons learned from the Whole of Government effort in Afghanistan and elsewhere

The history of Canada's Afghanistan Development Program from 2004-2013 indicates that an overarching strategic development vision, based on a Whole of Government approach and principles for engagement in fragile states, provided a clear basis for planning and Program implementation. The transition away from a Whole of Government approach as of 2011 created a void in this respect and there remains a need for such an overarching strategic vision.

Counter-insurgency and stabilization approaches failed to address long-term development requirements. Complementarities between the "Three Ds" – development, diplomacy and defence – should have been further explored and strengthened. Canada's continued engagement in Afghanistan should be based on lessons learned from its past involvement, including proven models and effective approaches.

Greater attention to drawing lessons on development programming and good practices in civilian-military cooperation from Afghanistan and other fragile state involvements, and an appropriate institutional mechanism to ensure that they are retained and used to inform future engagements, would be beneficial.  Whereas each mission is unique and tailor-made approaches should be developed accordingly, it is clear that important efficiency gains could be realized if knowledge were preserved, maintained and built upon.

Balancing channels of support and working with different levels of government

The criteria and indicators for the choice of aid delivery channels changed over time and were not always clearly articulated. While in the first phase of the response there was an emphasis on the use of multilateral channels, a tendency towards the funding of international NGOs developed in the third phase of programming. However, disproportionate reliance on NGOs may mean that linkages to national policies, strategies and implementation are often not sufficiently addressed.

A lesson learned from an over-emphasis on national programs during the early years of Canada's development efforts was that the sub-national level - the "missing middle" - also required attention to improve service delivery at local level, something that was more directly addressed during the Whole of Government phase of assistance.

Using political and policy dialogue to achieve results

In line with an overarching strategic vision, it is important to clearly align non-funded and funded activities to achieve one's stated goals. Crucial non-funded activities include political dialogue (primarily a responsibility for diplomats) and policy dialogue (primarily carried out by senior development officials) in key areas of performance of the Afghan Government.

Two main cross-cutting issues: governance and gender equality

The cross-cutting nature of governance could be further enhanced in the Afghanistan Program. Decisions on the type of support to be provided – on-budget versus off-budget support, for example – should be linked to political and policy dialogue.

Capacity building at both the national and sub-national level is a key factor in the realization of long-term, sustainable results in Afghanistan. Innovative approaches to capacity building have to be developed given the context. Many evaluations and studies on Afghanistan, including this one, refer to the 'missing middle' as the absence of a sub-national level of government. A lesson learned from program activities in Kandahar was that simultaneous capacity building at various levels of government is essential to the realization of long-term and sustainable results.

During the evaluation period, the importance of gender as a cross-cutting issue was noted. For the period 2014-2017, gender equality is meant to become an 'integrating factor' across the Program, which is positive. Therefore, a continued focus on gender mainstreaming is recommended while also "Afghanizing" the approach, to the extent possible.

Respect for humanitarian principles while strengthening the linkages between humanitarian and development assistance

The "blurring" of lines related to the politicization and militarization of humanitarian assistance has led to a reduction of humanitarian space in Afghanistan. Humanitarian actors have been unable to secure access to all parts of the country. The transition period following the withdrawal of international troops offers unique opportunities to regain humanitarian access. Canada's provision of humanitarian assistance throughout the evaluation period and its respect for humanitarian principles and donor coordination places it in a good position to promote the redefinition of the humanitarian space in the country.

While constructive steps were taken to link relief, reconstruction and development, including by giving the Afghanistan Program direct responsibility for humanitarian response, in practice there was limited success in doing so.  

Improved efficiency and performance management

There were times during the evaluation period when the Afghanistan Program was quite efficient at disbursing funds (notably, the second phase from 2008-2011). At other times heavy procedures and centralised decision making resulted in inefficiencies. There is a clear need to identify, track and measure long-term results beyond the output level to the extent possible. Strengthening of national statistical systems would allow for improved basic data collection through the Central Statistics Office and the monitoring systems of the ministries, while also reaching out to provinces and districts.

Recommendations

Recommendation 1 – Establish an institutional mechanism to capture lessons from the implementation of the Whole of Government approach in Afghanistan and elsewhere, to better inform future Canadian engagement in fragile states.

Recommendation 2 –Develop a vision for Canada's future engagement in Afghanistan, taking lessons from the implementation of the Whole of Government approach into account.  

Recommendation 3 - The crosscutting nature of governance should be further enhanced in the Afghanistan Program, including the strengthening of linkages between political dialogue and development policy dialogue with Afghan government partners. Programming decisions on the type of support to be provided – on-budget versus off-budget – should be based on clear targets and directly linked to on-going political and policy dialogue.

Recommendation 4 - Continue the focus on gender mainstreaming while adapting it to ensure improved responsiveness to socio-cultural values and principles, to the extent possible.

Recommendation 5 – For future investment in key sectors, ensure clear strategic direction, including a realistic risk analysis and robust risk mitigation strategy:

1. Introduction

1.1 Purpose

The specific evaluation objectives (Terms of Reference, Appendix 1) were to:

1.2 Scope

The initial scope defined in the terms of reference (ToR) for this summative evaluation was fiscal year 2004-05 to fiscal year 2011-12. It was extended to include fiscal year 2012-13 following delays to the start of the evaluation. In addition, changes in the Afghanistan context from 2001-2004 were taken into account in so far as they affected the evaluation period. Developments from early 2014 have been mentioned to the extent possible.

The object of the evaluation was Canada's Afghanistan Development Program that, in line with the ToR, had to be evaluated in the context of the Whole of Government (WoG) Approach and that of the international community towards Afghanistan.

To improve the readability of this report, the short phrase, Afghanistan Program (or Program) has been used to refer to the evaluation object.

The total volume of external assistance provided to Afghanistan by CIDA from 2004-05 to 2012-13 amounted to $1,546 million (total funds disbursed). A total of 310 initiatives were implemented by project partners during this evaluation period.Footnote 1 The evaluation focused on the analysis of the following sectors: economic growth including private sector development; education; health; humanitarian assistance (combining emergency assistance and peace and security); and human rights (sub-sector of democratic governance also including democratic participation). In addition, specific attention was given to gender equality as a cross-cutting theme and to development initiatives in Kandahar. The sampling resulted in a selection of 50 initiatives, covering a total disbursement of $852 million, and approximately 55% of overall disbursements made from 2004-05 to 2012-13. Detailed information on sampling is provided in Appendix 6.

1.3 Methodological approach

This summative evaluation developed a detailed methodological approach and work plan based on internationally agreed principles for evaluations in fragile states (see Appendix 6 for more details).Footnote 2 The evaluation team prepared a theory of change and linked it to a consolidated intervention logic framework (see Appendix 6, A6.2), which formed the basis for field work in Afghanistan.

Evaluation questions

In the ToR (see Appendix 1), preliminary evaluation questions were formulated and grouped under six evaluation criteria and specific topics. Upon discussion, the ToR evaluation questions were regrouped in line with the four main evaluation criteria, and four lead questions were defined. In addition, one overarching question was formulated that was directly linked to the overall evaluation objective of the Afghanistan ProgramFootnote 3, as shown below:

Figure 1.3.1 Evaluation questions

Figure 1.3.1 Text Alternative

To what extent has CIDA contributed to a more secure and democratic Afghanistan, able to deliver key services to Afghans and better provide for its longer term stability and sustainable development?

  • Relevance
    To what extent has the programming and implementation of the CIDA support to Afghanistan responded adequately to the specific context of Afghanistan, and was it well aligned with CIDA’s overall strategies, with federal government priorities within the whole of government approach, and harmonised with the interventions of other donors?
  • Effectiveness
    To what extent have the planned outputs and outcomes of the CIDA Afghanistan Program been realized?
  • Efficiency
    Were the financial resources and other inputs efficiently used to achieve the expected results?
  • Impact and Sustainability
    What are the impacts of the Afghanistan Program and what is the likelihood that these impacts (and outcomes) will be sustained?

A total of 39 sub-questions were formulated in relation to lead questions (see Appendix 6).

Approach to data collection and data analysis

The evaluation questions had to be answered at three different levels: Afghanistan Program level; sector/thematic level; and project/intervention level.
Both a top-down and bottom-up approachFootnote 4 were applied. Evaluation matrices were elaborated for both the Program and sectoral levels, and evidence gathered at the project level informed higher level analysis.

Methodological challenges and limitations

During the evaluation, the team identified various methodological challenges, some of which could be overcome, while others related to contextual factors that could not be changed. These challenges included missing documents and incomplete files, rapid staff turnover and lack of institutional memory, some resistance to the evaluation, the "fishbowl effect",Footnote 5 availability of data, a challenging security situation, and incorporation of evidence from the evaluation of the Arghandab Irrigation Rehabilitation project in a timely manner. A more detailed description of these challenges and mitigation measures can be found in Appendix 6.

1.4 Organization

Phasing of the evaluation

The evaluation was structured in several phases, as illustrated in the following figure (more details in Appendix 6):

Figure 1.2 Evaluation phases Text Alternative

2011 - Evaluation preparation phase
April 2013 - Evaluation design phase
Oct. 2013 - Data collection phase
Dec. 2013 - Data analysis and reporting phase
June 2014 - Technical Report to Development Evaluation Committee
Sept. 2014 - Finalization Technical Report
Jan.2015 - Synthesis Report to Development Evaluation Committee

2. Canada's International Development Program in Afghanistan

2.1 Context

A Communist Coup in 1978 ended 230 years of dynastic rule in Afghanistan. The Soviet Union invaded the country in 1979 and withdrew a decade later, after which time civil war developed from 1992 to 1996. The Taliban seized power in 1996 and ruled Afghanistan for 5 years.  International assistance was limited to humanitarian aid during this time.

A US-led military effort in October 2001 – Operation Enduring Freedom – overthrew the Taliban for sheltering Osama bin Laden. This marked the start of a period of reconstruction and state-building in Afghanistan with intensive international support. The scale of international involvement in Afghanistan was unprecedented with around sixty governmental donors and forty-seven troop-contributing countries engaged.

By the end of 2001, the Afghan government consisted of a new Cabinet that brought together warlords, returned refugees from Pakistan and technocrats from the diaspora. The first international conference on Afghanistan took place in December 2001 and led to the Bonn Agreement, which included a framework for the establishment of a new Constitution and national government. A Constitutional Jirga held in December 2003 led to the decision to form a centralised state with a powerful President. Elections were successfully held in 2004 and Hamid Karzai was elected President. Foreign donors concentrated their efforts on much-needed reconstruction and the International Security Assistance Force was formed.  

Foreign donors supported the Government of the Islamic Republic of Afghanistan's development of a 'COMPACT' identifying the main government priorities. An Afghanistan National Development Strategy (ANDS) 2008-2013 was formally approved by the President in 2008 and National Priority Programs were defined. The goal during this period was to develop a vibrant, equitable and sustainable economy with infrastructure, private sector development and the expansion of the rural economy as its foundational pillars. This focus was accompanied by attention to the role of and participation by women in the economic and social growth of the country. A National Action Plan for Women in Afghanistan 2008-2018 was prepared and followed up with a Law on the Prohibition of Violence against Women in 2009.

Challenges with security continued, particularly in the southern and eastern regions, as the Taliban insurgency regrouped and gained momentum. There were substantial amounts of international financial assistance and the development of Provincial Reconstruction Teams (PRTs) in a number of provinces.  The number of national and international troops increased steadily from 2005 to 2009 – from an estimated 92,825 troops (65,250 Afghan security troops ,18,200 US troops and 9,375 other foreign troops) in 2005 to over 300,000 troops (195,089 Afghan, 67,400 US and 38,370 other foreign troops) by December 2009. Footnote 6 At the national level, elections in 2009 did not lead to increased political legitimacy for the government and were deemed neither free nor fair.

From 2010 onwards, transition to full Afghan leadership and responsibility for the country's security, development and reconstruction in all spheres became the focus of attention. This led to the development of concrete targets and indicators both for the Afghan government and the international community. In 2010, donors agreed to align at least 80% of development funds with Afghan priorities within two years and to channel 50% of their development funds to the Government budget.

Canada withdrew its troops from Kandahar in 2011 and handed over responsibility for that province to the United States.  Insecurity continued in many parts of the province.

In 2012, the Tokyo Mutual Accountability Framework (TMAF) was agreed upon, defining five areas for action by the Afghan Government and additional commitments for the international community. The five priorities were elections, improved public finance management, anti-corruption measures, human rights and inclusive growth. Progress in all areas was closely monitored by the international community and the GIRoA. Most foreign troops withdrew and international assistance continued to decline into early 2014 despite the continuation of a protracted, complex emergency in the country.

There are no reliable figures on the total amount of international assistance to Afghanistan after 2001, particularly because the majority of the expenditures have been in security. From 2002 to 2012, approximately US$50 billion of official development assistance was provided, of which 87% was development assistance and 13% humanitarian assistance. This reflects the strong focus of the international community on state-building and long-term development support. Canada's disbursements from 2002 – 2011 were higher than those for most bilateral donors, with the exception of the United States, the United Kingdom, the European Union and Japan (see Appendix 10 for more details). 

2.2 Canadian government engagement in Afghanistan

The main trends, key international agreements, and Canada's engagement in Afghanistan from 2001 to 2013 are outlined in the following chart:

Key International Trends and Canadian Engagement in Afghanistan Chart Text Alternative
Key International Trends and Canadian Engagement in Afghanistan
 Key Afghan and International agreementsCanadian Engagement in Afghanistan
2001

December 2001 - Bonn Agreement

  • Lay out of a framework for the political transition
  • Establishment of an interim authority
 
2002

Multilateral Engagement

  • Mainly reconstruction aid
  • ISAF starts

Canada's Involvment Increases

  • Canada re-establishes diplomatic relations with Afghanistan
  • Canadian troops are deployed to Afghanistan
  • Afganistan Programtransitions from relatively small humanitarian budget to large development program
2003Afghan Constitution

Diplomatic relations and development program

  • Canada opens embassy in Kabul
  • First interim Afghanistan Program strategy 2003-2005
2004Free and fair Presidential Elections 
2005

January 2006 - Afghan Compact

  • Three areas and benchmarks for reconstruction
Canada assumes Leadership for PRT in Kandahar
Taliban Resurgence
2006 New Afghanistan Program Strategy with focus on Kandahar and role of women and girls
2008

Afghan National Development Atrategy

  • Focused on security, governance, economic growth

Manley Report and Official Response:

  • Whole of-Government approach
  • Increase of development budget to more than $200 million per year
  • Increased development focus on Kandahar
2009Presidential elecctions, problematic process 
Continued Insurgency; Troop Surge
2010

Agreement on Transition Process

  • NATO's intention to transition responsibility for security
  • Kabul Process - Introduction of the National Priority Programs
  • Agreement on 80% alignment and at least 50% on-budget support
 
2011

since 2011 - Various International Conferences

  • Addressing the transition process including withdrawal of intl. military forces by Dec. 2014 discussed a Chicago Summit in 2012

Canada Withdraws Combat Troops, new Afghanistan Program Strategy 2011-2014

  • Exit from Kandahar;
  • Focus on education, health, humanitarian assistance, human rights and gender equality;
  • Exit from Economic growth and democratic governance;
  • Reduction in development budget to appr. $100 million
2012

Tokyo Conference; Tokyo Mutual Accountability Framework

  • Discussing Afghanistan's future 2015-2024
 
2013 DFATD: Amalgamation of DFAIT and CIDA
2014 - Today Canada's military ends on March 31, 2014, Preparation of new DFATD Afghanistan Strategy

Prior to 2001, the Canadian International Development Agency's assistance to Afghanistan consisted of humanitarian aid delivered through multilateral organizations, ranging between $10 and $20 million per year to address basic human needs.Footnote 7  

In line with the agreements reached at international conferences, Canada's post-2001 mission in Afghanistan was initially characterised by military initiatives, with reconstruction support also provided. In February 2002, as part of the Operation Enduring Freedom, 850 Canadian troops were sent to Kandahar and roughly 1,700 to Kabul to join NATO's International Security Assistance Force. In 2002, diplomatic relations were re-established and in August 2003, a small contingent of Canadian diplomats was sent to Kabul to establish the Embassy of Canada in Afghanistan.

There were then 3 phases of Canadian development assistance programming in Afghanistan from 2004 to 2014, the time frame of this evaluation.

2004 - 2007

In March 2004, the Government of Canada committed $250 million in aid to Afghanistan with a focus on national programming and state-building. In August 2005, Canada assumed leadership of the Kandahar Provincial Reconstruction Team (KPRT) and command of a challenging military mission – securing a large rural province (Kandahar) the size of Nova Scotia with just 2,500 soldiers.Footnote 8 At that time, the security situation in the country, including in Kandahar, was still relatively stable. However, from the summer of 2005 onwards, insurgency in Kandahar increased after the regrouping of the Taliban. Fighting in Kandahar was quite intense in the first six months of 2006, followed by Operation Medusa in September 2007 involving 1,400 international, mainly Canadian, troops, with many casualties.

2007 - 2011

In October 2007, the government commissioned an Independent Panel to examine Canada's mission in Afghanistan to make recommendations on the future of Canada's role in Afghanistan. In January 2008, the Independent Panel issued its report, more commonly known as, 'The Manley Report'. Key findings from this report are presented below.

Key Findings of the Manley Report (2008)

The Manley Report created the basis for a clear, centrally-led, Whole of Government approach with an important civilian presence, including CIDA staff in Kandahar.

The response by the Government of Canada to this report was multi-pronged. First, it resulted in the definition of six policy priorities (grounded in ANDS and the Afghanistan Compact) and three signature projects to focus Canada's efforts.Footnote 9

Six government priorities and three signature projects 2008-2011

The first four priorities focus primarily on Kandahar:

The last two priorities have a national focus:

The three signature projects are:

Second, the government declared that Canada's combat mission in Afghanistan would end in 2011. Third, there was an increased level of involvement of the Privy Council Officein the planning and management of Canada's engagement in Afghanistan.

Canada's initial Whole of Government (WoG) Approach was launched in 2005 and sought to bring development, diplomacy and defence ("3D") together in a coherent vision and delivery framework, especially for work in fragile states with on-going crises. From 2005 onward, there were attempts to formalize and implement the 3D approach by various government actors. However, after the publication of the Manley Report, the WoG Approach became a defining characteristic of Canada's engagement in Afghanistan until the withdrawal of Canadian troops from Kandahar in 2011. The first report to Parliament in June 2008 stated that "Canadian contributions will significantly benefit the people of Kandahar with a shift from 17 to 50 per cent of programming focused in the province".Footnote 10

2011 - 2014

From 2011 to 2014, Canada's policy for engagement in Afghanistan shifted to a focus on transition, handover, and development. During this period, Canada's activities were national in scope but operations were concentrated in the capital city, Kabul, and in four sectoral areas: children and youth through education and health; security, rule of law and human rights; promotion of regional diplomacy; and humanitarian assistance. Canada continued to support the long-term objective of transferring governance and security responsibilities to GIRoA.

The overall Canadian aid commitment was $1.9 billion for 2001 to 2011 of which $1.64 billion was disbursed by CIDA.  Annual disbursements represented 10% of the total aid commitment ($190 million annually), which is in line with the reported average by other bilateral donors in Afghanistan at that time. An estimated $18 billion was to be spent on the whole government engagement in Afghanistan for the period 2001 to 2011 according to the Parliamentary Budget Officer in 2008.Footnote 11

2.3 Canada's development strategies and priorities in Afghanistan

CIDA developed various strategies and Results and Risk Management and Accountability Frameworks (RRMAFs) for the Afghanistan Program. They defined objectives, sector choices, aid modalities, aid channels and specific targets. Most of the Afghanistan Program strategies were never formally approved, which limited the possibilities for sharing these documents formally with the GIRoA and other development partners. The RRMAFs, however, were formally approved, although often with considerable delay. The linkages between the strategies and RRMAFs were not clear, particularly in the early years. The first RRMAF was meant to cover 2004-2009, while there were two strategies in that time: 2003-2005 and 2006-2008.

The overall objective for the Canadian development Program in Afghanistan has been reformulated several times:

2.3.1 Overall objectives of the Afghanistan Program

RRMAF 2004-2009

To support the efforts of the Afghan government, the Afghan people and the international community in stabilizing Afghanistan through the consolidation of the GIRoA's authority and legitimacy across the nation and through improvements in people's well-being.

New Strategy 2006-2008

Support Afghanistan's National Development Strategy through state-building and stabilization, particularly in Kandahar.

Logic Model 2007-2011

A more stable, self-reliant, and democratic Afghanistan that contributes to national, regional, and global security

Logic Model and RRMAF 2008-2011

A more secure Afghanistan, with a focus on Kandahar, able to deliver key services to Afghans, and better provide for its longer term stability and sustainable development.

Strategy and RRMAF 2011-2014

Fulfillment of basic needs and reduced vulnerability of the people of Afghanistan, with a focus on women and girls.

Given differences in the strategy documents and RRMAFs described above, the evaluation team prepared a consolidated intervention logic, describing inputs, outputs and outcomes for the main sectors of intervention over the evaluation period.

This overview shows continuity over time, as well as some shifts in priority; from a focus on state-building at the national level to stabilization in Kandahar. A people and needs-focused approach was adopted in more recent years. Afghanistan Development Program priorities changed across, as well as within, the noted phases. From 2004 to 2011, priority was given to democratic governance and economic development sectors, although different names were used; whereas from 2011 onwards, health, education and humanitarian assistance became the priority sectors (having received some support in the early years as well).

The sector analyses carried out for this evaluation, based on document review and interviews, provided insight into the overall strategic direction for the Afghanistan Program:

The strategic documents reviewed contained some information on aid channels and aid modalities, which was complemented and validated through interviews:

While strategic documents vary considerably in length and depth of analysis provided, the RRMAFs provide information on context, risk assessment and risk mitigation, monitoring and evaluation, as well as detailed logic models.

Canada's Afghanistan development program portfolio

The Government of Canada committed $1.9 billion to development and reconstruction in Afghanistan for the period 2001 to 2011. According to official information, $1.969 billion was provided during the period 2001-02 to 2010-11, of which CIDA accounted for $1.652 billion.Footnote 14   Canada was among top five bilateral donors to Afghanistan after the United States, the United Kingdom, the European Union and Japan largest bilateral donors (see Appendix 10 for more details).

Table 2.3.2 Canada's Official Development Assistance (ODA) disbursements to Afghanistan,
2001-02 to 2012-13, in million $
Aid Budget ($M)2001-20022002-20032003-20042004-20052005- 20062006- 20072007- 20082008- 20092009- 20102010- 20112011-20122012- 2013Total
Afghanistan - Canada471221021151152303402913172901631322,264
of which CIDA471181011121051842892272382311271071,886

The portfolio information shows that a total of 310 initiatives with project partners were signed and implemented during the period 2004-05 to 2012-13.Footnote 15 From 2004-05 to 2005-06, Canada's annual aid disbursements for Afghanistan were around $100 million, with a clear increase from 2006-07 onwards, reflecting the focus in the 2006-2008 New Strategy for Afghanistan to expand the activities with specific emphasis on Kandahar. Disbursements to Afghanistan reached a peak in 2007-08 at $280 million. During the three year period 2008-09 to 2010-11, aid remained well above the $200 million mark annually and from 2011-12 started to decline again to the 2004-2006 levels of $100 million annually.

The following figure presents the shift in sectors of focus over the evaluation period.

Figure 2.3.1 Annual disbursements by Principal Sector of Focus, 2004-05 to 2012-13

Figure 2.3.1 Annual disbursements by Principal Sector of Focus, 2004-05 to 2012-13
Figure 2.3.1 Text Alternative
Row Labels# projects2004/20052005/20062006/20072007/20082008/20092009/20102010/20112011/20122012/2013Total disbursements over 2004/2005 to 2012/2013
Democratic governance67$28 M$31 M$44 M$101 M$42 M$53 M$26 M$5 M$3 M$333.04 M
Emergency assistance48$13 M$5 M$19 M$30 M$39 M$18 M$29 M$17 M$14 M$184.13 M
Health33$1 M$0 M$6 M$8 M$37 M$34 M$23 M$33 M$25 M$167.13 M
Multisector and Other68$34 M$24 M$50 M$33 M$22 M$25 M$29 M$23 M$15 M$253.97 M
Peace and security18$19 M$13 M$7 M$25 M$26 M$10 M$17 M$3 M$121.09 M
Economic Growth54$8 M$28 M$51 M$51 M$34 M$68 M$63 M$25 M$6 M$333.37 M
Education22$0 M$1 M$33 M$23 M$21 M$29 M$13 M$33 M$153.52 M
Grand Total310$103.36 M$101.09 M$178.86 M$280.39 M$223.75 M$230.08 M$215.48 M$118.66 M$94.59 M$1,546.26 M

This figure illustrates the change in strategic direction as indicated in section 2.3 above. Democratic governance was the most important sector for five years (from 2004-05 until 2008-09), while economic growth took over in 2009-10 and 2010-11. Health became the largest sector, as measured by disbursements, for the last two years of the evaluation period (2011-12 and 2012-13).Footnote 16

Total disbursements over the entire evaluation period by principal sector of focus is presented in the following figure, showing that democratic governance and economic growth are the two most important sectors, each representing 22% of total disbursements. Emergency assistance represents 12% of total disbursements, which is very comparable to the percentage of humanitarian assistance in total ODA (see Table 2.2). Emergency assistance and peace and security combined represent 20% of total disbursements. Health and education comprised 11% and 10% of total disbursements, respectively.

Figure 2.3.2 Total disbursements by Principal Sector of Focus, 2004-05 to 2012-13

Figure 2.3.2 Total disbursements by Principal Sector of Focus, 2004-05 to 2012-13
Figure 2.3.2 Text Alternative

22% - Democratic governance
12% - Emergency assistance
11% - Health
16% - Multisector and Other
8% - Peace and security
22% - Economic Growth
10% - Education

Recent priority areas, such as community-based education, maternal, newborn and child health, and human rights, do not figure very prominently in the overall Afghanistan program portfolio from 2004-05 to 2012-13.

Over the entire evaluation period, it is estimated that $273 million of the $1.9 billion portfolio was spent in Kandahar. Disbursement for projects with a partial Kandahar focus are based on estimates by the Program staff as recorded in the financial statistical system, and are not based on verified disbursement figures. In absolute and relative terms, Kandahar disbursements were highest in 2010-11 with $72 million or 33% of the total Program. From 2008 to 2011, 29% of all disbursements were made in Kandahar.

3. Relevance to Afghanistan and the international context – Main Findings

Canada's Afghanistan Development Program was clearly focused on state-building from 2004 to 2007, but lacked a clear vision of its role and function in a conflict environment. This changed in 2007 with the planning and implementation of engagement in Kandahar Province – an active conflict zone – as one of many components in the Whole of Government approach. After Canada's withdrawal from Kandahar and from the democratic governance sector, the Program resumed its national level development focus and attention to peace-building and state-building goals became less prominent.

Relevance to Afghanistan

The Afghanistan Program appears to have been well aligned with GIRoA priorities as reflected in the ANDS and NPPs. Canada reached the 80% target of alignment agreed upon in the London and Kabul conferences in 2010. However, the very comprehensive and general character of ANDS and NPPs made it easy to assert that donor programs were "aligned". This was less true for humanitarian assistance, which was not explicitly addressed in the ANDS or NPPs. 

Canada's development Program addressed important humanitarian and development needs in this protracted, complex emergency. The assessment of needs, however, required constant attention. Throughout the evaluation period, the Program was aware of the necessity to both update and deepen its understanding of the evolving context. That being said, there were issues with the quality of assessments used to this end. Humanitarian needs analyses were quite weak initially, and the specific needs of women and girls were not always sufficiently taken into account. Over time, needs analysis improved. Understanding the political economy and main drivers of conflict and fragility received relatively little attention in Canada's Development Program, but Canada is not exceptional in this regard.

Aid Effectiveness

The Afghanistan Program mitigated risks appropriately by using a diversity of implementing partners and the balance of support among delivery channels varied substantially over time. From 2004-05 to 2008-09, 90% of disbursements were made through multilateral channels (in particular, multi-donor pooled funds and the United Nations). Civil society and the private sector became more important partners from 2008 onwards, as reflected in the figure below. However, the share of private sector funding declined again from 2011-12, which is related to the Canadian transition away from the economic growth sector. In 2012-13, civil society became the most important aid channel for the first time with 37% of total disbursements.Footnote 17

 Figure 3.1 Disbursements by type of executing agency, 2004-05 to 2012-13, per year

Figure 3.1 Disbursements by type of executing agency, 2004-05 to 2012-13, per year
Figure 3.1 Text Alternative
Type of Executing Agency# projects2004/
2005
2005/
2006
2006/
2007
2007/
2008
2008/
2009
2009/
2010
2010/
2011
2011/
2012
2012/
2013
Total disbursements over 2004/2005 to 2012/2013
Civil Society121$3.89 M$9.86 M$9.65 M$17.70 M$36.13 M$39.52 M$50.18 M$29.48 M$35.32 M$231.73 M
Private Sector38$0.19 M$0.08 M$0.21 M$2.67 M$13.00 M$17.98 M$37.46 M$11.78 M$1.13 M$84.50 M
United Nations90$50.34 M$36.65 M$49.70 M$73.50 M$123.74 M$116.55 M$80.52 M$42.10 M$18.63 M$591.74 M
International Financial Institution23$45.00 M$52.00 M$117.30 M$183.30 M$43.87 M$45.70 M$34.81 M$32.00 M$34.50 M$588.49 M
Other38$3.94 M$2.49 M$1.99 M$3.22 M$7.00 M$10.33 M$12.51 M$3.30 M$5.01 M$49.81 M
Grand Total310$103.36 M$101.09 M$178.86 M$280.39 M$223.75 M$230.08 M$215.48 M$118.66 M$94.59 M$1,546.26 M

In addition to finding a suitable mix of partners and delivery channels, maintaining a balance between national and sub-national efforts was also challenging. Whereas using civil society organisations enabled access to certain populations, the implication of a disproportionate reliance on NGOs was that linkages to national policies, strategies and implementation were often not sufficiently taken into account.

The proportion of on-budget support provided by Canada varied between 19% and 34% from 2010 to 2013, which was less than the 50% target agreed to in 2010 in the London and Kabul conferences (see text box below).On-budget support was provided in two ways:

Figure 3.2 On/off budget support, 2004-05 to 2012-13, by year in %

>Figure 3.2 On/off budget support, 2004-05 to 2012-13, by year in %
Figure 3.2 Text Alternative
2004/20052005/20062006/20072007/20082008/20092009/20102010/20112011/20122012/2013Total
Non-ARTF  support59,356,78959,088,42676,863,06797,093,544180,754,713184,580,941178,975,43086,657,15762,587,431985,957,497
ARTF Recurrent Cost Window32,000,00022,000,00048,000,00030,000,00013,000,00011,000,00018,000,00019,000,00010,000,000203,000,000
ARTF Investment Window12,000,00020,000,00054,000,000153,300,00030,000,00034,500,00018,500,00013,000,00022,000,000357,300,000
Total Funding103,356,789101,088,426178,863,067280,393,544223,754,713230,080,941215,475,430118,657,15794,587,4311,546,257,497

On-budget Support

Canada’s decision-making on the provision of on-budget support and the contribution to incentive programs has been linked to the assessment of progress on the Tokyo Mutual Accountability Framework (TMAF) hard deliverables by the international donor group responsible for monitoring, in which Canada very actively participated. The international community agreed that progress had been made in some areas of the TMAF, such as public finance management, but it lagged behind in others, such as human rights.

Although joint assessments of progress on the hard deliverables were made, every donor interpreted the assessment in line with its own policies and decided the amount and proportion of on-budget support accordingly. Despite limited progress in some areas, a number of bilateral donors met the 50% on-budget support target. Canada was of the opinion that insufficient progress had been made, especially in the areas of elimination of violence against women, establishment of the Afghanistan Independent Human Rights Commission (AIHRC) and anti-corruption measures. However, the way that the assessment of progress influenced actual funding decisions was not entirely clear.

Since 2013, decision-making became more transparent and linked to the TMAF hard deliverables.

Figure 3.2 shows that between 2004 and 2008, when there were no international agreements, more than 50% of total support consisted of on-budget support (up to 65% in 2007-08). However, as noted above, from 2008-09 onwards, the proportion of on-budget support (programming and recurrent costs) varied between 19% and 34%, which was quite far from the 50% target.

The Canadian Program has been a regular supporter of the Afghanistan Reconstruction Trust Fund (ARTF) and was its third largest donor after the US and the United Kingdom for the entire period of the evaluation. After 2008, however, Canada's contribution dropped sharply in absolute and relative terms. Slightly more than one third of the Afghanistan Program portfolio ($560 million) was channelled through the ARTF, supporting the Afghan governments' recurrent costs (recurrent cost window) and key national development programs (investment window) (see Figure 3.2 where on-budget support, including ARTF funding, is presented).

Canada and other donors criticized the management of the Fund for being insufficiently results-oriented and paying inadequate attention to gender equality. In response to the criticisms, the ARTF paid more attention to outcome reporting and gender, for which evidence was seen after 2010. Various evaluations indicate that the ARTF donors' consistency with the principles of the Paris Declaration (and follow-up agreements in Accra and Busan), including harmonization, alignment and donor coordination, but appears to have been less oriented to issues of sub-national governance, fragility and conflict.

According to ARTF rules, 50% of donor funding could have been preferenced for specific programs, although theWorld Bank has always publicly stated that the ARTF succeeded in meeting any preferencing requests of donors. In fact, Canada preferenced 64% of its funding. The decreased Canadian funding of ARTF after 2008, in absolute and relative terms, and the high proportion of preferencing, may affect Canada's negotiation position in the ARTF in the medium-term.

Canada also aimed to earmark part of its ARTF contribution to activities in Kandahar despite the fact that, in principle, the ARTF did not allow earmarking. In practice, Canada contacted ministries to negotiate that specific activities from national programs be implemented in Kandahar.

This points to an interesting challenge regarding aid effectiveness principles in a highly centralized, but fragile, state. On the one hand, there was a necessary focus on enhancing the role and capacity of the national government, which was reflected in the establishment of the ARTF. On the other hand, there was a necessity to pull-down national programs to provincial and district levels in order to address issues of equity, sustainability and efficiency of service delivery.

Cross-cutting themes

Gender equality has always been a specific priority in the Afghanistan Program, and over time the strategic focus on gender equality increased. The first gender equality strategy was elaborated by the Program for the 2011-2014 cycle and undertook a thorough gender analysis, including elements specific to fragile states, as its point of departure. There was a sharp increase in gender-integrated projects in that period, although the share of projects integrating gender equality results never reached half of the portfolio. This was partially because of the lack of gender equality integration in multilateral delivery channels (World Bank and UN), but also due to the difficult context and limited gender expertise available to the Program for significant periods of time. It should be noted that the gender coding of projects was based on an ex-ante assessment of expected gender results. Over the life of a project, gender aspects may have been addressed that were not foreseen at the outset. In practice, women-specific approaches have been the main mechanism used to address gender equality, while only limited attention has been paid to changing the behaviour of men and addressing underlying cultural constraints. The Afghanistan Program could have done more to encourage female participation in reconciliation and peace-building processes.

Canada Environmental Assessment Act (CEAA) requirements, applicable to off-budget projects, appear to have been formally met and strategic environmental assessments were conducted. However, in interviews, the CIDA environmental specialists involved questioned the quality of the assessments carried out by some implementing partners – in the case of the Dahla Dam, for example. In practice, environmental issues were only addressed to a very limited extent.

Within the cross cutting theme of governance,Footnote 19 democratic governance as a sector figured most prominently from 2004 to 2011. Attention was also paid to strengthening the capacity of the government in the health and education sectors. However, no explicit strategy for mainstreaming governance issues into the overall Program was found.

Policy Dialogue

Canada was considered to be a pro-active and constructive donor on coordination, aid effectiveness principles, and policy dialogue, including the discussions around the 2012 Tokyo Mutual Accountability Framework. Further, Canada was very active in improving donor coordination, especially in health and education, where it led donor coordination for a number of years. It played a key role in establishing dialogue and coordination among international health sector partners in the country, which was widely acknowledged by key stakeholders, including the Ministry of Public Health. The same was not necessarily true at the sub-sector level. While Canada was recognised for its role in global polio eradication, this has not been the case for maternal, newborn and child health under the Muskoka Initiative where, despite considerable investment, it has yet to realize its goal of becoming a lead international partner.

4. Effectiveness – Main Findings

As discussed in section 2.3, the Afghanistan Program elaborated various logic models and detailed performance frameworks (RRMAFs) over the evaluation period. The analysis of effectiveness in this summative evaluation is based on objectives set in the various strategy and performance documents. The objectives were consolidated into one intervention logic (see Appendix 6) in order to focus on the Program's key outputs and outcomes over the evaluation period.

4.1 Short-term results

Impressive short-term achievements have been realized. In many cases, Canada was not the sole funder of projects, but contributed to multi-donor programs.Footnote 20 Although Canada's contribution was analysed in a systematic manner,Footnote 21 its specific contribution was difficult to measure. Nonetheless it is clear that Canada, together with other donors, contributed significantly to the following results: 

Reports to Parliament between 2008 and 2012 describe many of these results, but new reporting has been included in this evaluation based on an updated review of project documents. Some short-term achievements make a clear distinction between men and women. While gender-disaggregated reporting improved over time, there were still weaknesses in gender-specific reporting in some sectors, such as humanitarian assistance.

4.2 Longer-term results and synergies

Given ongoing conflict and instability in Afghanistan, as well as the very low level of development at the start of the evaluation period, long-term results are challenging to realize in the timeframe available. Nevertheless, good outcomes were achieved in some sectors, such as health, education and mine action.

Health

The national target of 90% access to essential quality health services within two hours walking distance was reached in 2011-12 according to the National Risk and Vulnerability Assessment.Footnote 22 This is due to the rapid expansion of the number of primary health facilities from 2001 to 2012, which has had a considerable positive impact on national statistics. Despite this achievement, issues of equitable access to services remain. Access to health services relates not only to the physical distance and travel time to health facilities, but also to the cost of travel and services, as well as opportunity costs. The cultural responsiveness of the health sector – for example, the provision of female health care providers – also remains an important obstacle for effective access to and use of health care.Footnote 23

Thequality of essential health services provided to vulnerable populations, especially women and children, also increased.The most consistent and impressive results are reported by the National Risk and Vulnerability Assessment (NRVA) 2011-12 on the use of skilled birth attendants at delivery.

Reported rates of increased use of health services, especially by women and children, are unreliable because of uncertain population denominators; however, trends in the total number of outpatient visits and the distribution of visits by age and sex indicate continued positive progress in health service coverage and use. The point has now been reached where a more refined social equity analysis is needed to identify the specific groups and communities who may be excluded from this progress.

Progress towards polio eradicationhas been realized although a major setback occurred when the number of reported cases of infection suddenly jumped from 25 cases in 2010 to 80 in 2011. A change in strategy was introduced that appears to have been effective in containing further spread of the polio virus. Nevertheless, until full country-wide routine immunisation coverage is achieved, a serious threat remains.

Education

There is ample evidence indicating an increase in access to education services between 2002 and 2013. The last NRVA suggests that the largest improvements in primary education school attendance were recorded before 2007-08 and that since then progress has been modest. In fact, the NRVA suggests that at the current rate of improvement, Afghanistan National Development Strategy enrolment targets for 2020 will not be met. The reasons that children are not attending school vary but include place of residence, education level and gender.Footnote 24 Canadian support made a positive contribution to education, although relatively limited funding was provided during the period that major improvements were realized.

Community-Based Education (CBE): Successful interventions at the community level

Since 2006, Canada has funded community-based schools through various projects and programs implemented primarily by NGOs, such as BRAC and AKF. More than 5,000 community-based schools have been established with Canadian support where thousands of girls and boys in remote areas, in particular, have gained access to education and enjoy active support from their communities. These schools build on culturally appropriate approaches where schools and teachers are embedded in the community with community elders (mostly men) in the driving seat.

Learning outcomes appear to be positive. However, outcome tracking needs to be improved and long-term impact cannot be guaranteed. More evidence-based lessons should be drawn and disseminated, while linkages with the formal education system to facilitate transition should be strengthened.

There has been limited attention given to assessing the improved learning of children in primary education as an indicator of the quality of education services. Some small-scale and scattered studies point to very low levels of achievement in primary education, but only in 2014 was the Education Equality Improvement Program (EQUIP) planning to undertake a major learning assessment study, which should provide real insight. In community-based schools, levels of achievement appear to be higher and this is quite promising.

Improvements in teacher methods and materials are other indicators of the quality of education for which only limited evidence is available. There was a clear increase in the number of teachers, of which about one third were women. There were some issues with the training and deployment of teachers to where they were needed; in particular, the concentration of female teachers in some areas remains a problem. In rural areas and in the south, there were fewer teachers and even fewer female teachers. Teacher/pupil ratios were good in some areas, but not in others.

Economic-growth

Short-term employment and income opportunities have increased for many Afghans as a result of economic growth projects in the Afghanistan Program portfolio. In many villages, community infrastructure works financed through national community-based programs, such as National Solidarity Program (NSP) and the National Area-based Development Programme (NABDP), have resulted in millions of labour days for the community. However, this employment creation has been mainly temporary in nature. Aid money created temporary jobs that have proven hard to sustain without continuing aid flows, as indicated in various studies and evaluations. The Program also invested in projects that have tried to create more sustainable income opportunities and jobs, through microfinance and enterprise development, but so far only output-level results have been reported (for example, the number of savings groups or the number of loans provided). Exact data on numbers of new businesses as a result of these projects was not available. Finally, there is limited evidence that vocational training or advisory support has resulted in additional jobs or higher incomes.Footnote 25

Regarding improved access to financial and business services, there is some evidence that the Program contributed to this outcome, but there is no recent or solid evidence regarding the extent to which this was the case. An impact study of microfinance carried out in 2007 showed very positive results, but this was before the contraction of the sector from 2008 onwards. Nevertheless, the decline of the microfinance sector in Afghanistan did lead to an increased level of 'Afghanization' of the management of microfinance institutions and this was seen as a positive trend.

It is difficult to measure the enhanced capacity of communities to identify and implement development activities. The community development programs NSP and NABDP were supposed to lead to positive long-term democratic governance and economic growth effects. Important community decision-making apparatus have been established: the Community Development Councils (CDCs) at village level and the District Development Authority at district level. Both organizations aim to promote the integration and inclusion of women and have given voice to women. It was the first time that women's opinions have been systematically included in the design of village level projects. Moreover, it is reported that 70% of CDCs actively form linkages with government partners and civil society to support development activities in their village. Whether these linkages are successful and result in concrete projects still needs to be investigated. NSP impact evaluation resultsFootnote 26 capture the economic perceptions and optimism of villagers, especially of women, but there is no evidence that NSP has generated general production and marketing outcomes or increases in agricultural yields, productivity, or harvest sales.

Dahla Dam/Arghandab Irrigation Rehabilitation Program – Signature Project

The difficulty of realizing and measuring long-term results

Goals: The project aimed to improve irrigated agricultural production in the Arghandab Valley by rehabilitating canals and irrigation works, improving water management practices and developing sustainable community management systems. A series of related projects was also funded by Canada in and around this valley to improve the entire agriculture value chain. The Afghanistan Program spent considerable time and energy in tracking outputs and outcomes to the extent possible in this very insecure environment.

Short-term results: Precise information is available that shows that the majority of planned outputs were realized, although not all secondary canals could be rehabilitated and only 7 of the 28 planned Water User Associations were established by the end of the project. Outcome information had to be collected after the closure of the projects when there was no longer staff on the ground.

Efficiency: The project evaluation notes that a second implementing agency, which was contracted locally to complete work on the secondary canals, appeared to be have a much better cost-output ratio than the initial implementing agency.Footnote 27

Different baselines and targets: One main outcome indicator for this Signature Project was the increase in irrigated acreage as a result of project interventions. Documents consulted present different baseline data and targets. The pre-feasibility study carried out in 2007 and finalized in 2008 estimated that: 40,000 hectares would be irrigated for at least one crop per year; there would be a 50% increase in overall production; 60,000 additional farm jobs would be created; and, a sound Arghandab Valley Authority would be established. The 2009 inception report mentions that 27,600 families who had 20,000 hectares of irrigated land would benefit from 16,000 hectares of fallow land being turned into irrigated land, i.e. 0.6 hectare of additionally irrigated land per family. Finally, the Quarterly Reports to Parliament indicate a target of 30,000 hectares to be irrigated.

Measuring longer-term results: At the end of the project, no information was available on the increase in irrigated acreage. As a result of the security situation, the Afghanistan Program decided to track this outcome in an innovative yet costly manner using satellite images from March 2011 onwards. Satellite images show variations in vegetation, but cannot distinguish between rain-fed and irrigated agriculture via canals or wells. The increase in vegetation in April-May each year corresponds to the rainy season. The main findings from satellite image coverage of three seasons in the Arghandab Valley indicate an increasing trend of overall vegetation in the first two agricultural seasons – 2011 and 2012 – and a decline in 2013. They also show structural changes around the rehabilitated canal system, which likely indicates that there is a significant increase in agricultural, farming and land use activities over the AIRP study area. The positive trend of vegetation and land use is more significant in those areas that are close to some of the rehabilitated canals.

There is significant evidence of a positive contribution of the AIRP project to increases in vegetation. However, the interpretation of satellite data is extremely complicated in the absence of baseline satellite images and without additional verification on the ground. Therefore, while no reliable estimate can be made of the increase in irrigated area with one or two agricultural seasons per year, very rough estimates made towards the end of project implementation point to a total irrigated area of approximately 30,000 hectares, which is probably a maximum figure.

Distribution and sustainability of benefits: There is no information on land ownership or who benefited from the improvements in the irrigation system. Early in project implementation, crop diversification was introduced (e.g. pomegranates and saffron), but there is no information on the continued production of these crops. The Arghandab Sub-Basin Valley Authority has been set-up and is functioning; it is responsible for maintenance of the primary and secondary canals, but has a minimal and insufficient budget for this. Water User Associations should be responsible for the maintenance of tertiary canals, but they are few in number. Mirabs – the traditional system for irrigation water management – are controlling the distribution of water to farmers, as was the case before the project. There is no information on actual maintenance of the canals and organizations that were established during the project face challenges with maintenance or no longer exist. Further, the linkages between the organizations in charge of irrigation management and the agricultural departments appear weak.

In conclusion, the results of the Dahla Dam Signature project illustrate quite well the challenges related to achieving long-term outcomes in an insecure environment when no follow-up can be provided to the organizations on the ground following project completion. There is clear evidence that most outputs were achieved, but there is no evidence that the system is being effectively operated, which may lead to a rapid deterioration if no follow-up is provided through the new project funded by USAID.

Humanitarian assistance

In 2004-05, the Afghanistan Program provided support to UNHCR's Refugee Return and Reintegration program, which facilitated the return of over half of a million Afghans to all areas of the country, a little less than the 778,000 returnees that were anticipated according to project documents.

Effectiveness of the projects implemented by the UNHCR and WFP are particularly difficult to measure partly because Canada's contribution is part of a large international funding effort, but also because information beyond the output level is largely absent. Regarding the longer-term effects of food aid, there is limited or no information on lasting changes in the nutritional status of women and children. The NRVA 2011-12 even indicated that food security deteriorated slightly between 2007-08 and 2011-12, with 30% of Afghanistan's population being food insecure in 2012. Food security varies widely by residence, household characteristics, season and geographical region.

There are some positive indications of improved resilience or an increased capacity to withstand emergencies, particularly at the community level and less so at a provincial or national level. While interventions were scattered and evidence limited, there was one clear positive example: the Mine Action Program of Afghanistan (MAPA) demonstrated strong capacity to address mine action concerns in various areas. The Mine Action Coordination Centre in Afghanistan (MACCA) has clear national ownership and has established positive and essential working relations with different ministries and a range of government and non-government actors.

Clearance of mines improved access by farmers and communities to the land and allowed for the resettlement of internally displaced persons and refugees. Approximately 100 km2 was cleared of mines each year, but given the level of contamination, it is envisaged that total eradication of mines will only be realized towards 2022. Thus far, 123 districts and 2,243 communities are no longer affected by known landmines and explosive remnants of war as a result of mine action activities.

Human rights

The Afghanistan Independent Human Rights Commission (AIHRC) has gained the capacity to exert leadership, as well as to promote, monitor and protect human rights. Annual reports indicate a consolidation and expansion of the scope of the Commission's activity and capabilities. There is a clear increase in the number of complaints of domestic violence received (from 915 in 2008 to 2,468 in 2012), more active reporting on detention centres, media monitoring, and improvements in planning and reporting. Active monitoring of detention centers, including female prisons, as well as orphanages and hospitals, contributed to the improvement of standards and the elimination of apparent human rights violations. Hundreds of men and women, boys and girls, who were illegally arrested, detained or imprisoned, were released following the AIHRC's intervention.Footnote 28 The main potential weakness of the AIHRC, highlighted in an impact evaluation, was their lack of focus. As demands grew over time, so did the need to focus on those tasks for which the Commission has a unique mandate (e.g. monitoring of detention centres) and outsource or leave to civil society organisations those activities which can be also implemented by partners (e.g. awareness raising and education).

There is contradictory evidence on the level of awareness of human rights. Surveys indicated a decreasing level of acceptance of women's equal rights among the population between 2006 and 2012; however, there is some positive qualitative evidence on the human rights awareness of duty bearers. The AIHRC impact evaluation found that, "a range of key interlocutors (Ministry of Justice detention authorities, legal aid providers and other CSOs)….confirm with a degree of consistency that general awareness of detention-related rights has improved among detention personnel and detainees".

Kabul Widows: Women's Empowerment

This project, implemented by CARE, originated as an emergency food assistance project to a particular vulnerable population group. It is one of the few humanitarian projects classified as a gender-specific project that broadened its gender equality focus over time. Attention was given to empowerment of women in various areas, such as human rights, conflict resolution and economic activities. This approach led to a decrease in the amount of food assistance provided and a corresponding increase in human rights training.

The Widows' Association for Advocacy in Afghanistan was established to gather widows in Kabul and advocate for their rights. More than 2000 widows graduated from food assistance to income-generation activities, such as tending livestock. At the time of the evaluation, activities in Phase VI of this project were ongoing.

Increased participation of women and men in political and electoral processes at national, provincial and local levels was not fully realized. Voter turn-out was above expectations in 2004, but low in 2009 as those elections experienced challenges. The share of women voters remained constant. The first round of the Presidential elections in 2014, which is formally beyond the scope of the evaluation, led to higher voter turn-out. Canadian-supported activities appear to have contributed to a higher number of female candidates, but there is no evidence of their higher electoral success rate.

Gender equality

The assessment of gender equality outcomes is linked to the three gender equality objectives defined by CIDA in its Gender Equality policy: decision-making, human rights and access/control of resources. Effectiveness appears to be stronger in terms of access to resources and services (education, health services, the AIHRC) and less evident when it comes to real improvements in women's status and rights. This might be explained by the fact that changes in women's status and rights require behaviour change that takes considerably longer to occur.

In sectors where the greatest gender equality results were reported, the picture is promising though with further room for improvement. On the demand side in health, national out-patient statistics show that women use primary health facilities more frequently than men and that female attendance at clinics is growing at a faster pace than male attendance. On the supply side, the high percentage of males among health personnel is slowly decreasing due to pre-service training programs for female nurses and midwives. However, the issues of gender inequity in the assignments, task recognition and positions of female and male staff in the health sector remain.

Some longer-term gender equality results have been realized, particularly in community-based education, the Human Rights Commission, the Responsive Fund for the Advancement of Women, and Kabul Widows project, but there is limited evidence of positive results for the main ARTF programs in various sectors.

Synergies within and among sectors

There are some areas where synergies were achieved or where there is potential for synergies. For example, the ARTF support to the GIRoA via the recurrent cost window – in particular, the Incentives Working Program – is clearly linked to progress on the policy benchmarks agreed to in the Tokyo Mutual Accountability Framework. Whereas this is an international agreement led by the World Bank to ensure support to the ARTF, Canada is actively participating in the ARTF Incentives Program Working Group and in the TMAF policy dialogue, which has been quite important for the overall focus on state-building.

There are also synergies between Canada's co-Chairing of the ARTF Gender Working Group and its support to EQUIP and the Strengthening Health Activities for the Poor initiative. Furthermore, Canada has served as donor lead for Area 2 of the TMAF focused on elimination of violence against women and human rights, which is complementary to its financial support in these areas. Canada has stressed the issue of human rights and, in particular, women's rights in donor coordination meetings and in policy dialogue with the Afghan government. The combination of financial support to specific organizations, such as the Human Rights Commission, and dialogue on legislation and law enforcement is more likely to lead to long-term results than if only financial support were provided. However, these are very difficult areas where it will take time for lasting change to be realized.

From 2008 to 2011, the intensive Whole of Government approach and focus on Kandahar led to considerable attention being paid to synergies within and across sectors aiming to stabilize the province. The mine action program is one very positive example, where the program tried to reach out to various ministries and sectors to realize long-term mine education results and re-integration of mine victims into society. Also, land was cleared of mines in the Arghandab Valley in order to enhance the effects of improved irrigation systems in that area, which is a positive example of synergies in Canadian programming.

The analysis of the education sector has shown that more linkages need to be established between the formal basic education system supported through EQUIP and community-based education. Canada's role in donor coordination in this sector placed it in a unique position to establish these linkages and strengthen the CBE unit of the Ministry of Education, but this did not get much attention. Transition from community-based schools to primary and secondary education, but also from primary education into technical and vocational training, are areas where potentially important synergies could have been realized.

Finally, the linkages between development and humanitarian assistance in a complex emergency require continued attention. The Program was well aware of this challenge when humanitarian assistance was integrated as a responsibility of the Afghanistan Task Force, but it appears that, over time, less attention was given to synergies, with the exception of mine action. Very limited evidence of positive synergies between humanitarian and development assistance was found.

While the focus of the Program has become sectoral and project-driven over time, it is very important to avoid a silo-approach, and to track cross-sectoral outcomes as much as possible so as to enhance broader development results.

Explanatory factors

The main internal factors contributing to, or limiting, effectiveness, which could be influenced by the Afghanistan Program over the three phases of programming, were:

Factors contributing to positive results:

Factors contributing to limited or negative results:

5. Efficiency – Main Findings

Efficiency is a measure of how economically resources, including funds and time, are converted into results. In a conflict environment, the costs of implementing an aid program in a situation with evolving security needs tend to be quite high because of staff and project security costs, which negatively affects efficiency.  

Implementing partners

The Afghanistan Program diversified its choice of implementing partners in order to mitigate the risk of results not being achieved. Canada accepted that the risks presented by on-budget and multilateral funding were both expected and worthwhile in support of Afghan ownership and capacity building. Multilateral pooled funds, such as ARTF and the Multilateral Voluntary Trust Fund for UNMAS mine action, were meant to reduce the administrative costs for all donors involved, as well as to provide good oversight mechanisms and robust internal and external audit systems. The ARTF and the UNMAS Fund were considered to function very efficiently, although some ARTF-funded programs, such as the education program EQUIP, suffered from high overhead costs. In principle, UN agencies were also supposed to work efficiently, although their overhead costs were considered to be relatively high. Overhead costs for NGO projects are supposed to be relatively low. Nevertheless, the costs to manage these NGO projects appear to be higher than for multilateral organizations. The management of Local Funds was quite time consuming for the Program. Despite this, these Funds offered unique opportunities to work with local civil society actors, as was shown in the case of the Responsive Fund for the Advancement of Women and the Kandahar Local Initiatives Program. It is not possible to draw firm conclusions regarding the comparative efficiency of the various aid channels used by Canada.

In practice, although due attention was given to timely implementation of projects and audits, relatively little attention was given to efficiency considerations in terms of value for money, i.e. a comparison of the costs for various items and calculations of the costs per output and per outcome. Some positive measures were taken, such as the untying of food aid after 2008, which allowed for a more efficient procurement of food items.

Program management

CIDA's centralized procedures negatively affected efficiency. This was the case for delays throughout the program management and project cycle, including decisions related to the freedom of movement for staff based in Afghanistan, especially after the withdrawal of the military, as well as project approvals and contracting. Although some authority was delegated to the field when the WoG approach was in place, after 2011 the Local Funds were phased out and decision-making was centralized again.

There is evidence that the Program was quite risk-averse, and focused too much attention on the project-level, which did not leave enough time for sectoral and cross-sectoral issues. A clear risk going forward is that the relatively large proportion of off-budget support will continue to place the management burden on the staff given the intensive procedures in place for the identification, approval, monitoring and closure of these projects.

Staffing

The Afghanistan Program has been the largest program in CIDA's existence, both in terms of volume of aid as well as in number of staff involved. The Afghanistan Program occupies a unique place in CIDA's history, as a special Afghanistan Task Force was established in 2007 and continued until 2011. It had its own corporate services, including human resources, communications, and other programming departments, and was not housed in the CIDA building for most of its existence. During the period of implementation of the Whole of Government approach and heavy focus on Kandahar, the Afghanistan Task Force functioned as an organization within an organization. The majority of the Afghanistan Development Program staff was headquarters-based. From 2002 onwards, development staff were also based in Kabul, and from 2006 to 2012 CIDA staff were deployed to Kandahar. The Canadian Embassy in Kabul deployed local development staff in addition to the expatriate staff. In the early years, the number of professional staff engaged in the planning and implementation of this rapidly expanding program in a complex emergency situation was very limited – one senior development professional was based at the Embassy and approximately five people were working on the Program at CIDA headquarters. Subsequently the staffing of the Afghanistan Program showed significant variation: from less than 10 people in 2004-05, to approximately 20-25 full-time employees in 2006-07, and up to 125 people from 2008 until 2011 followed by a rapid contraction in 2011 to 70 people, down to fewer than 30 in 2013.

The management of a rapidly growing development program in a complex emergency led to specific human resources challenges, in particular when the Task Force was created. For various reasons, the enormous expansion from 2007 onwards could not be realized with CIDA staff only. As a result, there was a clear intention to recruit externally and develop a special esprit de corps. The idea of setting up a staffing incubator partially outside CIDA and recruiting individuals with relevant experience in the Balkans or the Middle East was launched. The Task Force was under considerable pressure to not only recruit sufficient, qualified staff, but also to put pre-deployment, deployment and post-deployment packages in place, which included 'duty of care' responsibilities.Footnote 29 Interviews showed that opinions differed on the extent to which sufficiently qualified staff was sent to the field. The idea of establishing a flexible staffing structure within CIDA or across government departments to recruit people with specific expertise in engagement in conflict zones and fragile states did not materialize. A Canadian Program Support Unit (CPSU), an arrangement also used in other countries where CIDA programs are delivered, served as the eyes and ears of the Afghanistan Program on the ground. It engaged locally contracted professionals and support staff in various disciplines. The CPSU was closed in March 2014, but given its importance, an interim arrangement has been put in place until a new model of DFATD Field Support Services is initiated.

Frequent rotation of staff combined with limited freedom of movement negatively affected institutional memory; however, some key staff had several rotations in different positions in the Afghanistan Program either at headquarters or in Afghanistan, which positively contributed to continuity and has reinforced Canada's role in policy dialogue. However, there is a clear danger that this will no longer be the case in future given the reduced size of the Program.

Learning

There are various examples of project-level learning and improvement through monitoring and evaluation, studies and other means. This applied to all aid channels, although less so to some UN agencies. There is also evidence of some learning at sector level, especially in education, health, gender equality and in the work done in Kandahar. Reviews of Program-level operations were carried out, such as the Operational Program Review from 2005 to 2009 and a desk review carried out by the Evaluation Division at the end of 2007, and show that there was clear follow-up on operational recommendations but less so on strategic recommendations.

6. Impact and sustainability – Main Findings

The evaluation design indicated that this summative evaluation would seek evidence for impact-level results to which Canada, as member of the international community, contributed.Footnote 30 The extent to which intended impacts have been achieved and how sustainable these results will be is even more difficult to assess than the achievement of immediate and intermediate outcomes, given the context of insecurity and fragility. Future developments in Afghanistan will determine whether the Canadian investments will lead to the development of robust and equitable service delivery systems and an accountable state. Some seeds of hope have been sown despite the security challenges faced throughout much of the evaluation period, but results lag behind in some areas and the challenges of governance, sustainability, inclusive growth and recurrent cost financing remain.

6.1 Progress towards intended impacts

Whether Canadian investments contributed to economic growth, democratic governance, robust and equitable service delivery systems, and an accountable state remains to be determined. There were many risks and insecurities, as well as numerous system gaps to close. The risks related to these investments have to be accepted as part of the effort to build a peaceful and stable state; they could be mitigated but not avoided.

Significant improvements in Afghanistan's health indicators have been reported over the period of the evaluation, although cultural barriers to greater progress remain. The rapid development of the demand for and supply of health services has been remarkable. Service-related statistics, such as immunization rates, outpatient consultations and supervised obstetric deliveries, are still below comparable rates in other countries in the region, but they continue to improve. The basis of this rapid development was a very early consensus by the transitional government and three large international partners, the European Union, the US Government and the World Bank on a contracting-out model for health services that had only been tested once before on such a large scale in Cambodia.Footnote 31 The results have been impressive, but challenges of governance, sustainability and affordability remain. Canada was not among the largest donors to the health sector, but has definitely contributed to these improvements.

For polio eradication, the situation is different. Canada was the main international supporter of the initiative to eradicate polio in Afghanistan since 2006 and became the lead financing agency when it launched its signature project for polio eradication in 2008. With Canadian support, Afghanistan has acquired a very functional system for the containment of polio virus infection. As long as eradication in neighbouring Pakistan remains an issue and as long as routine immunization coverage in Afghanistan remains fragile, this system of communication, surveillance and response will be required to prevent outbreaks of poliomyelitis and to contribute to the global goal of polio eradication.

For education, the NRVA 2011-12 reported improvements for all education indicators, including gender equality, although the pace of improvement has slowed. It is still quite early to measure the intended impacts of the transition of girls from primary to secondary education or to vocational training and then to the labour market.  There are various challenges facing girls in the transition from one school system to another.Footnote 32

Uneven impact of the biggest flagship project, the National Solidarity Program

According to a 2013 impact evaluation, the NSP has been very successful in sponsoring Community Development Councils across the entire country, financing local subprojects, promoting local governance, empowering women and, in turn, enhancing the legitimacy of the Afghan state.

The evaluation reports positively on the program's role in improving villagers' access to basic utilities and education and health services, as well as increased empowerment of women. But, there is no noted impact of the Program on economic activity. The Program's biggest success midway through the cycle – the promotion of local governance – had declined by the end of the evaluation period with a negative impact on the quality of local governance. NSP has produced a durable increase in the acceptance of female participation in local governance and broader political participation. However, there is no sound evidence that the NSP has changed attitudes towards broader economic or social participation of women.

The evaluation found that the NSP had impact neither on general production and marketing outcomes nor on agricultural yields, productivity, or harvest sales. Overall, the study concludes that the impact of the NSP on economic welfare appears to have been driven more by the infusion of block grants than by completed economic projects, such as irrigation canals, access roads or bridges. This is corroborated with the finding that NSP-funded village-level irrigation and transportation projects had limited success.

Evidence collected through international surveysFootnote 33 suggests that support from Canada, as well as other donors, has yet to achieve visible impacts on the actual protection of human rights, increased democratic participation or broader access by women to decision-making. It is clear that the country still faces important human rights challenges. There is recognition that with Afghanistan becoming a signatory to key international human rights instruments and legislation on elimination of violence against women, a legal framework has been provided but further effort will be required in its application. The Afghanistan Independent Human Rights Commission has been strengthened with important Canadian support and that of other donors, but there are some concerns regarding its continued independence and focus that require attention.

Canada contributed to the increased availability of microfinance services and community infrastructure. At the project level, there is some evidence that improvements in agriculture through irrigation, the introduction of new crops, vocational training or advisory support have resulted in more jobs and higher incomes. However, for important projects, such as the Dahla Dam signature project and NSP (see text box), this evidence is inconclusive. For the majority of the economic growth projects in the sample, there was no clear evidence of improved income or employment opportunities. Economic growth has been quite uneven, largely absent in the agricultural sector and rural areas where Canada's support was concentrated and has yet to lead to poverty reduction at the macro level.
Canada's efforts to link relief, reconstruction and development defined expected results at the outcome level. However, there remain few reported results of humanitarian assistance beyond the output level,Footnote 34 with some notable exceptions. One exception is the Kabul Widows project, where widows successfully earned incomes, became economically empowered and their rights and participation at the community level were strengthened. Nevertheless, it was generally difficult to estimate or measure the longer-term effects of interventions carried out as humanitarian assistance. In Afghanistan, saving lives and alleviating suffering has been a short-term activity that was threatened by on-going political and military violence.

The collective efforts of mine action organizations operating in Afghanistan, including Canada's support to MAPA, have led to a steady decline in the number of mine victims. Since 2001, the average number of mine victims each month has decreased by more than 50 percent. Teaching both rural and urban communities how to avoid the dangers of landmines has contributed to the decline. Additionally, mine-free land can now be used to grow crops, which can be sold at the market or used to feed families, or to raise livestock.

Canada's contribution to capacity building was part of the international community's efforts.  In the early years of Canada's engagement, the focus of the international community was on strengthening the capacity of national ministries through ARTF and UN agencies. Various reports mention the excessive number of technical assistants located in different ministries at that time.Footnote 35 There was also a relative absence of capacity-building at the provincial and district level in these early years, which can be understood given the focus on national state-building.

There is sufficient evidence that the capacity and ownership of the GIRoA has increased, although some ministries clearly lag behind. The evaluation of the implementation of the Paris Declaration principles in AfghanistanFootnote 36 provides clear and positive examples of strengthened capacity. The Ministry of Finance has been playing a key role in managing donor relations. In other sectors, particularly health and education, capacity has improved although the government remains quite dependent on a so-called 'second civil service' consisting of well-paid, Afghan returnees whose salaries continue to be funded by donors.

Canada initially focused on capacity-building at the national level via the ARTF and the UN. From 2008-2011, the focus shifted towards capacity building at the province and district level in Kandahar. From 2011 to 2014, capacity building efforts were directly related to those sectors where Canada concentrated its support.

In Kandahar, there is little evidence of positive longer-term outcomes of capacity building activities, despite all efforts. The support provided focused on the rehabilitation of office buildings, provision of office equipment and training of staff. However, capacity-building requires long-term involvement. In most cases, the provincial authorities that were interviewed remembered the support provided, but did not consider this support as structural capacity building as no systems or working methods were changed or improved as a result of that support. Therefore, no evidence could be found of strengthened capacity in the provincial government departments that can be attributed to Canadian assistance. This also applies to the Arghandab Irrigation Rehabilitation Program to a large extent. The Arghandab Sub-Basin Agency was set up with support from the project and people were trained. However, this Agency has a very minimal budget for operation and maintenance of the irrigation system, thus hindering its performance. Despite project intentions to establish institutional linkages between this Agency and the provincial Department for Agriculture, no clear linkages remain. Furthermore, water users' associations that the project established no longer exist. These challenges can be explained both by the centralized government structure that did not allow for the channelling of additional budgets to the provinces, but also by the fact that Canadian-funded projects were time-limited.

Finally, a last important component is the capacity of local NGOs. The Afghanistan Program admits that this is an area where it had relatively little context-specific experience and where it invested less. Nevertheless, the Program did support local NGOs as a result of funding international NGOs and by using Local Funds, which led to promising short-term results. The Responsive Fund for the Advancement of Women that supported women's organizations involved local organizations to foster local ownership, although opportunities were missed to strengthen broader institutional partnerships, which affects sustainability. In general, the support to local NGOs was too short and too limited to achieve sustainable long-term results. The local Canadian development funds no longer exist, but support of local NGOs via international NGOs and UN organizations is continuing.Footnote 37

In general, few institutional assessments have been carried out to objectively assess capacities of ministries and other Afghan organizations and this was beyond the mandate of this evaluation. Furthermore, there are clear sustainability challenges related to the political, economic and security situation of the country.

6.2 Unintended impacts

Unintended impacts are not easy to measure and direct linkages to specific projects and programs are difficult to establish. Therefore, unintended impacts cannot be specifically attributed to Canada's development efforts.

There is very little sound evidence on unintended impacts, either positive or negative, and perceptions prevail. On the one hand, optimists see signs of positive impacts in many parts of society and believe that these will only become stronger. On the other hand, critics point to negative trends, such as ongoing insecurity, growing poppy production and an increasing aspirations gap among Afghan youth, and see these as unintended consequences of international support. The World Bank, for example, has stated that aid inflows have become a source of rent, patronage, and political power, sometimes inadvertently exacerbating conflicts and grievances among different groups. More impact evaluations would be needed to shed further light on intended and unintended impacts.

In the area of community-based education, an evaluation of the Girls Education Program implemented by BRAC points to positive 'hidden outcomes' that are not being tracked: "This CBE experience should have significant development results, even for those girls who do not transfer to formal schools or drop out soon afterwards. The ability to read, write and count, and spending a few years in a structured learning environment away from home, will certainly affect the position of these young women in their households and their village. What those effects are is the hidden outcome."
Health sector development engaged communities and contributed to building trust in the public system. These are collateral impacts that are not completely unintended but part of the rationale for investing in the sector. For instance, when the group of female immunization volunteers in Jalalabad told the evaluation team that they participated in the campaign because it provided them with an opportunity to contribute and interact with the community, they did not refer to achieving polio eradication but rather to overcoming discrimination.

Although the foundations for a healthy microfinance sector have been established through the Microfinance Investment Support Facility for Afghanistan (MISFA), a severe contraction took place from 2008 onwards with a substantial reduction in microfinance institutions and borrowers, before the sector stabilized again at a lower level. The rapid client base growth generated between 2006 and 2008 came at the expense of proper due diligence in lending, compliance with control processes or internal monitoring of performance. Collectively, donors appear to have failed to invest in adequate governance structures and internal control systems in this sector. By pushing more and more money into the system, they were actually contributing to the sector's downsizing to more realistic proportions from mid-2008 onwards. In the process, millions of dollars evaporated as microfinance institutions folded.Footnote 38 Canada spent most of its money in microfinance before 2008, but still disbursed some funds after that time. As the focus turned towards new activities, MISFA meetings were neither actively followed nor documented and, thus, an opportunity to draw lessons was lost.

Development and humanitarian assistance were not considered to be neutral by non-government actors; therefore, increased hostility led to decreased access to many parts of the country by development and humanitarian organizations. There have also been an increasing numbers of attacks on development and humanitarian workers since 2006. These can be considered unintended impacts. The humanitarian space appears to have been affected by the politicization and militarization of aid; some of the possible reasons for this are discussed in section 6.3.

The poor quality of aid-funded infrastructure, especially at the community level, has been reported on in NSP evaluations and the insufficient quality of EQUIP schools, including in Kandahar, has been reported upon various times. The low quality of some of the new infrastructure was the result of a variety of factors, including limited capacities of construction companies, limited oversight and/or corruption. Poor quality of infrastructure may negatively affect the perceptions of the population on the government's ability to deliver services to the people.

In-depth research by the Feinstein Center in five provinces in Afghanistan (Balkh, Faryab, Helmand, Paktia and Uruzgan; where Canada has not carried out direct programming),Footnote 39 found that development projects were frequently described negatively by Afghans. Perceptions of the misuse and abuse of aid fuelled the growing distrust of the government and aid agencies among the population. However, the authors indicate that a perception-based study is likely to paint a more negative picture of development assistance than may be warranted.Footnote 40 That being said, the primary complaints were that projects were insufficient, both in terms of quantity (not enough) and of quality (wrong kind or poorly implemented), unevenly distributed geographically, politically and, above all, associated with extensive corruption, especially those projects with multiple levels of subcontracting. There is enormous variation across regions, sectors and projects. Given the challenges of doing a survey of direct beneficiaries of Canadian support, it is virtually impossible to assess the extent to which these complaints are also related to Canadian support. In the case of the Dahla Dam signature project, one of the lessons was that high expectations may do harm, as a large part of the population expected that the height of the dam would be raised and 100,000 jobs created. Frustration ensued when that did not occur, even if communication on the specific project goals was adjusted over time.

Widespread corruption is another unintended impact of the large flows of donor money into the country. While donors pay due attention to managing and mitigating fiduciary risks within projects, broader challenges related to corruption remain. It has been very difficult to track expenditures from central to community levels. Despite many risk mitigation measures, including audits, there is still a huge gap between accusations of corruption at the community level on the one hand, which is confirmed by corruption indices, and the absence of evidence on corruption in specific projects and programs on the other. The Afghanistan Program pointed out that mitigation of the potential risk of corruption is one of the reasons that they provided only a limited amount of on-budget support. They also selected partners carefully, according to their ability to deliver efficiently.

Contributing towards a more stable Afghanistan is part of the overall objective of Canada's development assistance, yet it poses fundamental questions regarding the approach to be followed. On the one hand, the Feinstein Center indicated that stabilization theory places a high importance on the socio-economic drivers of conflict, such as poverty, literacy and lack of social services and therefore, emphasizes socio-economic solutions. However, research shows that causes of insecurity are diverse and intertwined. On the other hand, most development efforts did not address the major drivers of conflict, including grievances related to political or identity issues. The research studies found more evidence of destabilizing than of stabilizing effects of aid, especially in insecure areas where the pressure to spend large amounts of money quickly were greatest. These reports conclude that aid projects often did not address the root sources of conflict and, in some cases, fuelled conflict by distributing resources that rival groups then fought over.Footnote 41

6.3 Development work in a conflict zone

The involvement of CIDA in Kandahar was directly linked to the decision that Canada would lead the Provincial Reconstruction Team in that province from August 2005 onwards, a period when the security situation deteriorated rapidly. Despite a new development strategy for 2006-2008 in which stabilization in Kandahar played a key role, the start of the Afghanistan Program in Kandahar was quite slow. Various factors explain this slow progress, both operational – a lack of knowledge of the environment, few staff on the ground, limited freedom of movement – as well as strategic, such as fundamental differences of opinion regarding the role of a development program working together with the military in a conflict environment. In 2007, CIDA was under pressure to increase its engagement in Kandahar and disburse the increased funds allocated to the Afghanistan Program. Fierce external criticism of CIDA's engagement in Kandahar also emerged at this time. CIDA responded to these criticisms by setting-up the Afghanistan Task Force in 2007 and by sending additional staff to Kandahar. Further, the Kandahar Local Investment Program (KLIP) was set up in April 2007 to fund local initiatives.

The fundamental underlying question being asked at the time was: is development in a conflict zone possible? When the Task Force had to expand rapidly, there was little or no time to reflect on appropriate development strategies as the overall policy framework was already defined based on a stabilization approach. Joint efforts were made in Kandahar to develop an advanced model for stabilization in order to achieve results for the population. Situation awareness, including an understanding of the local context, was the first step in this approach. The aim was to link short-term, quick impact projects financed by the military to CIDA's longer-term, sustainable development activities at all levels. Interviews and lessons learned exercises indicate that, over time, a better understanding of the political economy in the districts and the province was gained, but fundamental differences of opinion regarding the correct approach to achieving long-term development goals remained. The underlying assumption was that if development benefits and services to the people were delivered, the population would be less inclined to support the insurgency. In this approach, there was limited attention to address the grievances of the people and to deal with drivers of conflict, which are identified in the research literature as important factors to achieve lasting positive change. 

There is no doubt that a lot of mutual understanding and good cooperation developed between civilian and military staff on the ground in Kandahar. While civilians based in Kandahar felt they were operating in a bubble with direct contact to headquarters in Ottawa, staff in the Embassy in Kabul worked to establish the necessary linkages between development activities in Kandahar and the national level. Kandahar was an important test site for the work to decentralise programs to the province and to the districts.

The development program in Kandahar was confronted with various fundamental challenges. The first was the balance needed between strengthening the capacity of the provincial and district authorities and civil society, on the one hand, and delivering immediate services to the population, on the other. This tension was reinforced by the fact that the governing structure in Afghanistan is highly centralized. The effort to decentralise national programs offered a unique opportunity to address the 'missing middle' in this governing structure. Canada, including CIDA, was aware of the 'missing middle' as illustrated in an interview with the Representative of Canada in Kandahar (RoCK) from 2008: "Governance and figuring out "how do you advance governance?" has also been a challenge. How do you make systems of government work? How do you help Kabul link to its provinces? And then, how do you make the province link down to the districts"? Considerable attention was paid to building the capacity of the government at various levels and to strengthening the linkages between these levels. However, the 2011 lessons learned exercise pointed to the insufficient emphasis on allowing the government to lead and on really understanding the needs at the local level.

This leads to another important challenge between the fast-tracking of project implementation and enhancing local ownership. Instruments for understanding the local context, including the drivers of conflict, were developed during this period and served their purpose. This was reflected in the different levels of sectoral understanding. Economic growth became the backbone of the stabilization strategy and an innovative agricultural value chain approach was developed based on good insights into the main bottlenecks impeding economic growth, including poppy production. The development of alternative livelihood strategies and the diversification of agriculture were important elements. However, for various reasons, including security and lack of qualified personnel, insights into other sectoral issues were few, leading to a more scattered and less successful project implementation.

Another ongoing challenge was the provision of humanitarian assistance in a complex emergency with an active military conflict, where parties to the conflict included major donor countries. There are various components of this challenge. First, military forces were increasingly eager to work with development and humanitarian actors as part of their counter-insurgency (COIN) and stabilization strategies. Second, the politicization of assistance, including humanitarian aid, in the context of 3D (Defence, Diplomacy and Development) and Whole of Government approaches was perceived by some to affect the neutrality of humanitarian aid; and a third, more traditional blurring of the lines, was the overlap between humanitarian and development assistance. While CIDA and other donors introduced the international concepts of 'linking relief, rehabilitation and development' to move humanitarian assistance closer to development, in practice, there were important gaps to overcome. Humanitarian and non-humanitarian actors involved in the Afghanistan Program were aware of these tensions. The Kandahar Action Plan indicated that, "projects will not compromise humanitarian efforts and prevent future engagement for stabilization and development by non-military actors". The Afghanistan Program invited strategic humanitarian partners to concentrate part of their activities in Kandahar province and give more priority to Kandahar through additional funding. The response from humanitarian actors was mixed. Internal documents and interviews confirmed that humanitarian assistance in Kandahar required some form of collaboration with the Canadian or other military forces while maintaining the independence of humanitarian partners, which was considered a delicate balance and led to a "blurring" of lines. The politicization and militarization of humanitarian assistance led to a reduction of humanitarian space in Afghanistan as humanitarian actors have been unable to secure access to all parts of the country.

6.4 Sustainability

Conducive policy and institutional environment

The likelihood of continued benefits from Canada's development investments depends to a significant extent on how the transition process unfolds. Key transition points in 2014 and 2015 will be the elections, the continued withdrawal of international troops and a likely further overall drop in development funding. There are clear risks related to these transitions and different scenarios can be foreseen. For the various sectors that Canada supported, specific factors can be identified that will promote or hinder sustainability.

In education, Canada has contributed to ensuring that the Ministry of Education has strong, appropriate policies in place, which offer clear – if highly ambitious – guidance to the sector. Nonetheless, the Community Based Education policy and shura policies need to be stronger and clearer, while also developing a vision for monitoring and measuring learning. It is recognized that excessive technical assistance and other external support to the Ministry of Education are unsustainable. This is a serious concern that was raised in many interviews – the fear that a parallel education sector was being created and that no real Afghan civil service exists.

The collaboration of the Afghan Government with the international donor community in the health sector has been exemplary, which is one explanation for the very impressive level of development in the health sector over a relatively short, ten-year period. Further strengthening of Ministry of Public Health (MoPH) capacity, which is planned for 2014 under the ARTF System Enhancement for Health in Transition, should ensure continued progress in this regard. However, it must be acknowledged that the government does not control the entire territory of the country. With the complete withdrawal of the foreign military presence, there are risks that government control may shrink rather than expand. Any program aiming at system strengthening faces risks inherent in working in a politically fragile country. The World Bank, in the evaluation of its Afghanistan Program over the period 2002-2011, concluded that, "without viable district or provincial institutions, the investment in community organizations at the village level may not be sustainable, substantial project benefits notwithstanding".

Another factor that needs to be contemplated when reviewing sustainability is operation and maintenance of community infrastructure projects. In a sample of 100 NSP projects covering several provinces, 56% of projects were found not to be in good condition, and 14% were no longer functional.Footnote 42 The research found an inverse relationship between the condition and use of a sub-project and the attention to operation and maintenance. Another finding was that some projects, such as water supply networks and tertiary roads, are simply too expensive for most communities to maintain.

The assessment of the probability of long-term benefits of CIDA-supported interventions in the field of human rights and democratization suggests clear challenges. Institutional capacity has been strengthened, as illustrated in the outcomes section – AIHRC and women's organizations have been reinforced. Yet, the policy environment does not appear to be very conducive to sustaining these improvements without external support. The controversial process of appointment of new Human Rights Commissioners points to clear setbacks. The policy environment in general has become difficult for human rights, as became clear from the persistent attempts by some political parties to introduce regressive modifications to legislation (e.g. on the Elimination of Violence Against Women law, on the criminal code regarding the use of stoning). Moreover, the latest electoral law has reduced the quota of guaranteed seats for women in provincial assemblies from one quarter to one fifth.

Canada did promote local ownership to some extent, but very few institutional links were created between the local NGOs supported via the local funds and governmental counterparts. This was the case for NGOs supported by the Responsive Fund for the Advancement of Women and the Ministry of Women's Affairs at national and provincial levels. Financing through the Fund was deemed to have hampered sustainability, synergy and coordination. Opportunities to enhance sustainability by linking these projects to the responsible government institution and embedding them in a broader institutional setting were missed. Focus groups and interviews showed that beneficiary organizations have expectations of continued donor funding.

There are other factors to be considered when assessing the sustainability of results of CIDA's human rights programming, including security. The AIHRC's women's rights sections, for instance, have had problems conducting activities in provinces like Kandahar and getting suitably qualified officials to work there because of the difficult security conditions. Further, the sustainability of results achieved through civic education and women's electoral participation projects is linked, at least in part, to the existence of minimum security conditions for women to exercise a political role. Support for women's rights and leadership by men and religious leaders is another key factor for ensuring sustainability of the related programming. Surveys show that men are less inclined than women to recognize the legitimacy of women's civic engagement and economic activity.

Financial sustainability

Financial sustainability was not a primary concern of the Afghan government or the international community for quite some time. However, since the start of the discussion on the transition to increased Afghan ownership and given the decline in international assistance, this issue is now higher on the agenda, as recognised in World Bank and International Monetary Fund publications. Area 4 of the Tokyo Mutual Accountability Framework focuses on government revenues, budget execution and anti-corruption, for example.

In education, the work done by the Ministry of Education to get selected for Global Partnership for Education (GPE) funding, with the support of Canada and other donors, can be considered a clear achievement in planning for secured future funding. This shows strong planning and leadership on the part of the Ministry. Other than this, however, there is little evidence of planning for the reduction of aid flows in the next few years.

The MoPH national health priority program estimates a funding gap of US$ 255 million over the next three years.Footnote 43 The polio eradication initiative projects a gap of US$ 112 million over the same period.Footnote 44 Funding gaps of similar magnitudes are reported by other programs, including the tuberculosis control program which, according to interviews, will experience a major gap in funding when the CIDA support ends in March 2014. Health sector development in Afghanistan is highly dependent on international financing and it has very large requirements. Any additional sources of funds that may be identified will be rapidly absorbed by one of the many holes in program budgets.

Sources of funding to sustain development results brought about by the Afghanistan Program after the end of Canadian support (and donor support more broadly) have been identified to a very limited extent. In 2013-14, more than a decade after its inception, the Program is reported to be undertaking a set of studies to develop a more sustainable model, but there is very little time in which to test out and institutionalize an alternate model.

6.5 Sustainability of development results in Kandahar

From 2008 until 2012, the Canadian government reported on its engagement in Afghanistan, with a specific focus on Kandahar. A very detailed monitoring and reporting system was set up to inform the Canadian public and these reports contained a significant amount of information on output-level results. There were also various internal attempts to collect information beyond the output level. 

Signature Project: 50 schools in Kandahar

Under EQUIP, 38 schools were indicated as completed and another 13 were under construction as of July 2011. Of these 51 EQUIP schools, 26 were situated in Kandahar city. According to UNICEF in 2013, 17 schools were built in Kandahar with CIDA support. Some other schools were built by NGOs. This would bring the total number of schools, even if double counting is taken into account, to 64, which is well above the target of 50 schools.

However, monitoring reports, field visits by the evaluation team and visits by journalists to a number of schools in Kandahar indicate that there have been problems with the construction in many cases. EQUIP schools faced construction problems in other provinces as well, which EQUIP and the Ministry of Education tried to address. Therefore, the quality of construction is not a problem specific to Kandahar. Also, not all schools in rural areas appear to be functioning as a result of insecurity.

There is no doubt that the construction of these schools has increased the enrolment of boys and girls, although no reliable enrolment figures are available. There are monitoring reports for the UNICEF and NGO project and also for one or a few EQUIP schools. However, there are no specific monitoring reports available for the EQUIP Signature schools in Kandahar as the ARTF did not allow geographic earmarking.

Good outputs were achieved in Kandahar, such as hundreds of pieces of community infrastructure built, polio campaigns executed, more than 50 schools constructed, and irrigation systems rehabilitated, among other things. However, some outputs, such as the construction of certain schools, were of poor quality. The many positive outputs could be considered a solid foundation for achieving long-term development results. In Kandahar city, in particular, positive outcomes related to improved access to education and health services for girls, boys, women and men were observed. However, a few years after the Canadian exit from Kandahar, there is limited evidence of positive outcomes in terms of more jobs, enhanced income opportunities or better quality of services outside of the health and education sectors. In fact, there are some signs of potential negative impacts as a rapidly growing group of unemployed, educated youth, especially in Kandahar city, may be turning to drugs (the number of drug addicts in Kandahar city is reported to be growing rapidly), or to the insurgency.

Regarding gender equality impacts, there was some evidence of improved access to education and health services by women. However, Kandahar and other provinces in the south still lag considerably behind many other provinces, especially in gender terms; there is neither evidence of improved access by women to decision-making nor improved access to resources or better protection of women's rights, which can in any case be expected to take years to achieve.

If the security situation improves, more positive impacts may emerge, although further support might be needed to realize this. There are still positive developments at the community level as a result of improved physical infrastructure and strengthened community organizations, but there are also clear signs of frustration and anger, despite the fact that some development activities are continuing. Endemic corruption has been shown to be a factor limiting sustainable long-term results.

There is no evidence showing that the capacity of the provincial and district-level governments has been strengthened sufficiently to enable the creation of a policy and institutional environment conducive to sustaining the accomplished results. The need to "Afghanize" project activities was recognized from the start but, as mentioned, there was a tension between the time needed to strengthen local ownership of the projects and the need to show quick results. The Kandahar Local Investment Program proved to be an excellent and innovative opportunity to enhance local ownership, which is reflected in the number of local NGOs that got the chance to implement projects. However, only two out of the seven NGOs that implemented KLIP projects that were tracked by the evaluators remain active. The short period of time in which projects were implemented meant that local NGO ownership could only be strengthened to a limited extent. This applies not only to KLIP but also to other NGO projects.

The COIN/stabilization theories adapted on the ground focused on the 'build', 'enable' and, possibly, 'transition' stages in order to achieve sustainable development results. However, the challenges related to sustainability in this complex environment were not sufficiently thought through. Stabilization was supposed to enable an environment where sustainable development could occur, but the lessons learned exercise points to missing elements. It was implicitly assumed that with the withdrawal of the military, development results would be realized and automatically lead to a development exit by Canada. It was only in 2011 that attention was paid to exit strategies. Interviewees indicated that no other options to continue Canada's development engagement in Kandahar were explored. A lot of attention was paid from 2010 onwards to close collaboration with and handover to the US, which had been the political agreement reached. A pragmatic exit strategy was eventually developed and consisted of elements including: i) no premature closing of projects; and, ii) handover to USAID and the possibility of developing a bi-national economic development strategy. In practice, after the Canadian withdrawal from Kandahar, USAID started pursuing its own priorities, which were not those that had been previously held by Canada. The frequent change of American staff on the ground meant that there was little institutional memory remaining to keep the strategic Canadian legacy alive.

Many interviewees indicated that this exit strategy may have been short-sighted and that, given the enormous Canadian investments made in Kandahar, other alternatives should have been explored as was done by other bilateral donors.  While the Netherlands took a similar approach to Canada's by completely withdrawing its development assistance from Uruzgan, the United Kingdom, Germany and Norway continued to provide their development assistance to the provinces where they were active in order to realize sustainable results through 2014 and beyond.

7. Conclusions, lessons and recommendations

Canada`s Afghanistan Development program was part of the unprecedented international community involvement in Afghanistan after 2001. Throughout the three broad phases of development activities covered by this evaluation (2004 to 2007; 2008 to 2011; and 2011 to 2013) Afghanistan remained a challenging context to work within given an evolving security environment.  Despite the challenges, Canada, together with other donors, contributed to achievements in various sectors and there are lessons to be learned from work in a fragile and conflict-affected environment.

7.1 Conclusions

  1. Canada's Afghanistan Development Program was characterized by continuity in its involvement on the one hand, and by clear changes in strategy and focus from 2004 to 2013, on the other. There was evolution from an initial focus on state-building at the national level from 2004 to 2007 to a main concentration on stabilization in Kandahar from 2008 to 2011, towards a humanitarian, social sector and gender equality oriented Program after 2011.
  2. Canada was recognized as a main development partner in Afghanistan and effectively participated as a member of the international community in policy dialogue with the Afghan government to reinforce the international agreements and principles for aid effectiveness and engagement in fragile states. However, Canada has, for various reasons, provided a relatively low proportion of its aid on-budget (around 30% in the last few years) and did not meet the target set in 2010 to provide 50% on-budget support. There was a lack of clear and transparent decision-making in this regard.
  3. The Afghanistan Program was clearly aligned with the priorities of the Government of Canada, but not all strategic priorities were addressed with high levels of funding. This was the case for the focus on Kandahar, where the absorption capacity at the provincial level posed serious problems, but also for gender equality that was an important priority in policy dialogue, but was addressed in less than 50% of the projects.
  4. The Afghanistan Program developed relevant approaches to assess the enormous developmental and humanitarian needs in all sectors of focus. However, the needs assessments that were undertaken had limitations and conflict analyses used did not enable a complete understanding of the drivers of conflict and grievance, thus limiting overall development performance.
  5. Canada, together with other donors, contributed to impressive short-term achievements in various sectors, ranging from the construction and rehabilitation of thousands of schools and increased enrolment, especially of girls, to improved access to health facilities, construction of community infrastructure, delivery of food to millions of people and support to the independent Human Rights Commission and women's organizations. In education and health, as well as in mine action, long-term results have been realized through increased access to and use of health and education services and land cleared of mines. However, in the economic growth, human rights and governance sectors, few substantial positive changes beyond the project level were observed. Issues of distribution and equity remain unaddressed. Gender equality results are mainly concentrated in the social sectors through improved access to services. Limited real improvements for women related to human rights and their role in decision-making were observed, recognising that long timeframes can be required for such societal changes. 
  6. Canada contributed to strengthened capacity at the national level. However, there is evidence of missed opportunities, especially in the work to strengthen sub-national governance and establishing adequate linkages to national government.
  7. The implementation of the development Program in Kandahar showed that long-term development cannot be accomplished with an emphasis on short-term implementation strategies, which sped up implementation considerably, but which failed to ensure sustainable, long-term development results.
  8. The Afghanistan Program did address some efficiency considerations at the level of projects and implementing partners, but major efficiency issues affecting overall Program performance such as staff mobility in a conflict environment, rotation, and centralized decision-making were only addressed between 2008 and 2011 when the Whole of Government approach was implemented.
  9. Canada is recognized as a consistent and reliable donor with a clear results orientation, but there is insufficient evidence to provide a definitive answer to the overall evaluation question related to Canada's contribution to long-term stability and sustainable development in Afghanistan.

7.2 Lessons

  1. The history of the Afghanistan Program from 2004-2013 indicates that an overarching strategic development vision, based on a Whole of Government approach and principles for engagement in fragile states, provided a clear basis for planning and Program implementation.  While this was the case during the "Whole of Government" phase (2008-2011), it was less evident at other times.
  2. It is essential to align and harmonize political and policy dialogue – both overall and sector-specific – with the funding of activities, while also promoting synergies across and within sectors, based on leadership and support to the government to achieve concrete goals.
  3. A good understanding of the main demand and supply-side challenges and of the main governance and funding issues in each sector, particularly during the Whole of Government phase, positively contributed to the performance of Canada's development activities in Afghanistan while also paying due attention to the evolving context.
  4. Capacity building at both the national and the sub-national levels is a key factor to realize sustainable long-term results in a centralised state like Afghanistan. Innovative approaches have to be developed given the context of insecurity and fragility.
  5. There is a need to reiterate the commitment to respect humanitarian principles in Afghanistan in order to regain access to all areas of the country and to promote the respect for humanitarian norms with all parties to the conflict.
  6. It was a challenge for the Afghanistan Program to remain sufficiently focused on achieving its development objectives, while understanding the fluid contextual factors on the ground, learning and communicating while under pressure to implement, addressing administrative requirements, undertaking risk assessments and reporting on progress, among other things.  A key lesson is the importance of maintaining a strategic view at the program and sector levels, so that project level interventions are well informed and situated.
  7. The implementation of the Whole of Government approach in Kandahar showed that good collaboration between Canadian actors on the ground can speed up the implementation of development projects, but more attention should have been paid to the elaboration of a development approach in conflict zones as an intrinsic part of the Whole of Government approach.

7.3 Recommendations

Recommendation 1 – Establish an institutional mechanism to capture lessons from the implementation of the Whole of Government approach in Afghanistan and elsewhere, to better inform future Canadian engagement in fragile states.

Recommendation 2 - Develop a vision for Canada's future engagement in Afghanistan, taking lessons from the implementation of the Whole of Government approach into account.  

Recommendation 3 - The crosscutting nature of governance should be further enhanced in the Afghanistan Program, including the strengthening of linkages between political dialogue and development policy dialogue with Afghan government partners. Programming decisions on the type of support to be provided – on-budget versus off-budget – should be based on clear targets and directly linked to on-going political and policy dialogue.

Recommendation 4 - Continue the focus on gender mainstreaming while adapting it to ensure improved responsiveness to socio-cultural values and principles, to the extent possible.

Recommendation 5 – For future investment in key sectors, ensure clear strategic direction, including a realistic risk analysis and robust risk mitigation strategy:

Date Modified: