Evaluation of the Anti-crime Capacity Building Program and Counter-terrorism Capacity Building Program – FINAL REPORT

Global Affairs Canada Inspector General Office Evaluation Division

May 2016

Abbreviations, Acronyms and Symbols

Acknowledgements

The Evaluation Division (ZIE), Office of the Inspector General (ZID) of the Department of Global Affairs Canada (GAC), would like to thank the staff and management of the Anti-Crime Capacity Building Program (ACCBP) and the Counter-Terrorism Capacity Building Program (CTCBP) for their cooperation, and the members of the Evaluation Advisory Committee (EAC) for their guidance and advice. Special thanks to all of the representatives of ACCBP and CTCBP partner organizations who agreed to be interviewed for the evaluation.

Executive Summary

This summative evaluation of the Anti-Crime Capacity Building Program (ACCBP) and the Counter-Terrorism Capacity Building Program (CTCBP) was conducted by the Evaluation Division (ZIE), Office of the Inspector General (ZID), of the Department of Global Affairs Canada (GAC) as part of the departmental Five-Year Evaluation Plan. The evaluation was conducted according to the Treasury Board Policy on Evaluation. The target audiences for this evaluation report are GAC’s senior management, program managers, other government departments (OGDs) who partner in the delivery of the programs, as well as the Canadian public.

Background

CTCBP was created in 2005 with a mandate to provide assistance to foreign states in the form of training, provision of equipment, technical and legal assistance to enhance their capacity to prevent and respond to terrorist activities. Assistance provided by CTCBP is global in coverage and targeted to countries and regions with identified needs. A separate Sahel Program envelope was created in late 2010 for that specific geographic region. This special envelope is managed by CTCBP.

ACCBP was established in December 2009 to enhance the capacity of beneficiary states, government entities and international organizations to prevent and respond to threats posed by transnational criminal activity in the Americas. As of 2015 the ACCBP has a global scope with a focus on the Americas.  In 2011, a separate Human Smuggling Envelope (HSE) was introduced to address those operations destined for Canada. HSE had a global scope, with a particular focus on Southeast Asia and more recently West Africa. This special envelope is managed by ACCBP.

Evaluation Scope and Objectives

The evaluation focused on programming over the last five years, i.e. from FY 2010-2011 to FY 2014-2015. The evaluation also followed up on the recommendations from the 2009 formative evaluation of CTCBP, and the 2012 formative evaluation of ACCBP.

The evaluation’s key objectives were to evaluate the relevance of ACCBP and CTCBP; evaluate their performance in achieving expected outcomes; examine synergies between the two programs and other GAC programming in security and development; gather lessons and make recommendations for improved management and delivery of ACCBP and CTCBP.

Evaluation Approach and Methodology

The evaluation utilized a combination of qualitative and quantitative methods and multiple lines of inquiry. Qualitative methods including interviews with 175 stakeholders, observation through site visits to 11 countries, document review and project file review were used to respond to all of the evaluation issues. These methods were complemented by quantitative methods to assess administrative data and other program information.

Key Findings

The evaluation found that for the 2011-15 period, there was a continuing need for the Capacity Building Programs: CTCBP to combat significant ongoing threats from international terrorism networks, particularly in the Middle East, North Africa and Asia; ACCBP to address organized crime and drug trafficking in Latin America and the Caribbean; and HSE to prevent human smuggling operations in Southeast Asia and West Africa. The Capacity Building Programs aligned with GoC policies and priorities. There was good consultation among the Capacity Building Programs, however coordination could be improved with other GAC security and development programs.

There was evidence that ACCBP, HSE and CTCBP made progress towards expected results at the immediate level, but there was little data available to assess achievement of intermediate outcomes. Beneficiaries interviewed during site visits expressed a high level of satisfaction with the capacity building programs and implemented projects were deemed successful in meeting their needs. The evaluation found that longer-term capacity building projects tended to be more sustainable than one-time training. CTCBP, ACCBP and HSE increased Canadian expertise, leadership and visibility in international security programming.

In assessing efficiency and economy, the evaluation found that programming focused on fewer countries and projects was more effective than spreading funding too thinly. ACCBP and CTCBP spent the Vote 1 portion of their allocated budgets, but were unable to fully disburse the Vote 10 allocation during the last few years of the evaluation period. The Project Initiation Authorization process (PIA) introduced at GAC in 2011 adversely impacted the delivery of the Gs&Cs programs in the department, including the Capacity Building programs. The PIA process not only reduced the overall efficiency and effectiveness of the programs, but also rendered some planning functions and the role of the governance committees less meaningful. A number of other factors, specifically related to the delivery of the CPBs have additionally limited their effectiveness. For example, delays in the delivery of equipment limited the effectiveness and impact of some training in the field.

The overall project management by IGC staff was good; however the performance measurement systems and practices put in place were deemed not adequate. The CPBs did not have a project database, which impacted the overall information management. ACCBP, CTCBP, Sahel and HSE did not integrate a gender perspective in their programming. The Capacity Building Programs used an appropriate mix of delivery channels by working through OGDs, multilateral agencies, trilateral mechanisms and partnerships with Canada’s allies. ACCBP, CTCBP and HSE generally coordinated their security programming well with other donor countries through bilateral consultation and regional organizations.

Recommendations

It is recommended that ACCBP and CTCBP:

1. Focus on long-term capacity building by continuing to support multi-year projects (or a series of shorter-term projects with mutually reinforcing outcomes) within target regions/themes to ensure sustainability of results, while earmarking some funding for quick response to emerging needs.
2. Increase coordination and synergy with other security programs, and with development programming at GAC.
3. Assess efficiency at the program and project levels more systematically.
4. Continue improving performance measurement systems and practices at the program and project levels.
5. Integrate a gender perspective into program planning, monitoring and reporting.

1.0 Introduction

The Evaluation Division (ZIE) at the Department of Global Affairs Canada (GAC) is housed within the Office of the Inspector General (ZID) and is mandated by Treasury Board of Canada Secretariat (TBS) through its 2009 Policy on Evaluation to conduct evaluations of all departmental direct program spending (including grants and contributions) every five years. The Evaluation Division reports to the Departmental Evaluation Committee (DEC) on a quarterly basis, which is chaired by three GAC deputy ministers.

The evaluation of the Anti-Crime Capacity Building Program (ACCBP) and the Counter-Terrorism Capacity Building Program (CTCBP) was conducted according to the requirements of the Transfer Payment Policy (2009) and as part of GAC’s Five Year Evaluation Plan. The target audiences for this evaluation report are GAC’s senior management, program managers, other government departments (OGDs) who partner in the delivery of the programs, as well as the Canadian public.

1.1 Background and Context

The Government of Canada (GoC) plays a fundamental role in protecting the safety and security of Canadians. While strengthening Canada’s domestic response to crime and terrorism remains a priority, Canada’s security is inextricably linked to that of other states, which may lack the resources or expertise to prevent and respond to criminal and terrorist activities. When source and transit states for various criminal and terrorist activities are vulnerable, the security of Canadians and Canadian interests, at home and abroad, is threatened.

The 2008 Speech from the Throne reiterated the fact that Canada’s national security depends on global security, which ultimately depends upon the respect for freedom, democracy, human rights and the rule of law. The achievement of these values across countries is often imperilled by international organized crime and terrorist activities. Such activities place the security, safety and prosperity of Canadians and all global citizens at risk.

The Counter-Terrorism and the Anti-Crime Capacity Building Programs were established to enhance the capacity of states to fight terrorism and transnational organized crime. The beneficiaries of these programs are mainly countries and regions where criminal and terrorist activities originate and/or pass through but do not possess the resources to prevent such activities on their territory; such countries are also referred to as “source” and “transit” countries for criminal or terrorist activities.

CTCBP was created in 2005 with a mandate to provide assistance to foreign states in the form of training, provision of equipment, technical and legal assistance to enhance their capacity to prevent and respond to terrorist activities. Assistance provided by CTCBP is global in coverage and targeted to countries and regions with identified needs. A separate Sahel Program envelope was created in late 2010 for that specific geographic region. This special envelope is managed by CTCBP.

ACCBP was established in December 2009 to enhance the capacity of beneficiary states, government entities and international organizations to prevent and respond to threats posed by transnational criminal activity in the Americas. As of 2015 the ACCBP has a global scope with a focus on the Americas. In 2012, a separate Human Smuggling Envelope (HSE) was announced to address those operations destined for Canada. HSE has a global scope, with a particular focus on Southeast Asia and more recently West Africa. This special envelope is managed by ACCBP.

Both programs represent a whole-of-government approach, drawing on the expertise of other Government of Canada departments (OGDs) and agencies to effectively deliver security capacity building to beneficiary states and ensure that these states are better able to manage and respond to security threats. These programs also aim to contribute to Canada’s national security and other Canadian interests, such as the safety of Canadians abroad, the provision of a more stable environment for Canadian commerce and trade to operate internationally, as well as to Canada’s international reputation and influence.

In 2008-2009, GAC reviewed its approach to programming aimed at preventing criminal and terrorist activities as a result of a number of factors, such as the expiration of the CTCBP’s Terms and Conditions in March 2010; the approval of an allotment from the International Assistance Envelope (IAE) for a program to build anti-crime capacity; and recommendations of a formative evaluation of CTCBP, etc. GAC opted for an approach that harmonized the management and delivery mechanisms for the two capacity-building programs, including the Standard Operating Procedures (SOPs), the Priority Review, the mandate of the Interdepartmental Steering Committee, as well as the existing Annual Voluntary Contributions to the United Nations Office on Drugs and Crime (UNODC) and the Organization of the American States (OAS) Inter-American Drug Abuse Control Commission (CICAD).

ACCBP and CTCBP are managed by GAC’s Capacity Building Programs Division (IGC) in the Non-Proliferation and Security Threat Reduction Bureau (IGD) of the International Security and Crisis Response Branch (IFM). Although ACCBP and CTCBP have different geographic scopes and activities, they share governance and administration structures and have similar objectives. They will therefore be reviewed and evaluated in parallel in order to identify each program’s respective strengths and weaknesses.

1.2 CTCBP and ACCBP Overview

1.2.1 Program Objectives, Expected Results and Key Activities

The overarching mandate of ACCBP and CTCBP is to enhance the capacity of key beneficiary states, government entities and international organizations to prevent and respond to threats posed by international criminal and terrorist activity. The programs achieve this by providing transfer payment assistance through projects and initiatives, such as: needs assessments, training, legislative drafting and advice, placements of technical experts, equipment and associated material, outreach and advocacy, and operational activities. All programming activities are implemented in line with international obligations, standards and norms. The programs are also expected to contribute to an increased Canadian expertise, influence and leadership in supporting and enabling security capacity building assistance on an international scale.

ACCBP and CTCBP aim to achieve the following immediate outcomes:

• Personnel in beneficiary states and organizations are more knowledgeable and skilled in anti-crime and counter-terrorism policies, procedures and enforcement;
• New or improved anti-crime and counter-terrorism legal instruments, controls and frameworks; and
• Additional or improved anti-crime and counter-terrorism tools, equipment, networks and physical infrastructure are available and in use.

ACCBP’s and CTCBP’s expected intermediate outcomes are:

• Strengthened incident prevention, mitigation and operational readiness and responsiveness;
• Increased compliance with international anti-crime and counter-terrorism commitments; and
• Strengthened infrastructure to support anti-crime and counter-terrorism coordination and response systems.

The expected long-term (ultimate) outcomes are:

• Improved global prevention and response to threats of terrorism and transnational organized crime impacting Canada and Canadian interests.

ACCBP and CTCBP are grants and contributions (Gs&Cs) programs. The G&Cs disbursed by ACCBP and CTCBP are used for a number of activities, such as:

• Needs assessments, data collection and analysis
• Training workshops, seminars and other forms of technical instruction;
• Placement or deployment of experts (in recipient/beneficiary states);
• Legal support;
• Provision of tools, technology and equipment; and
• Physical infrastructure support
• Outreach, advocacy, prevention and awareness raising.

ACCBP and CTCBP achieve their objectives by implementing projects through four delivery mechanisms: bilateral, multilateral, trilateral and partnership programming (see section 4.5.12).

1.2.2 CTCBP

Since CTCBP was created in 2005, the threats posed by international terrorism networks have become increasingly complex, partly in response to international efforts toward their elimination. As a result, the scope and demand for resources and international assistance to build capacity to counter terrorism have increased, along with the pressures to explore the linkages between criminal activities that fund or advance acts of terrorism.

The Counter-Terrorism Capacity Building Program was established in 2005 as an international security assistance mechanism and a key part of Canada’s international terrorism prevention efforts. CTCBP’s mandate is to provide assistance to foreign states that lack the resources and expertise to prevent and respond to terrorist activity in a manner consistent with international norms, standards and obligations. The Program fulfills this mandate through the provision of training, equipment, technical and legal assistance. CTCBP is focused on six key thematic areas, as summarized below.

Since CTCBP was established, it has funded more than 229 projects with a total value of over $90 M. During the evaluation period, the majority of CTCBP funding went to LEMI (76%), followed by border and transportation security (10%), as summarized below.

CTCBP Sahel Envelope

In response to the deteriorating security situation in the Sahel region of Africa and increasing threats from Al Qaeda in the Islamic Maghreb (AQIM), the international security donor community (including Canada, the US, France and the EU) set out to strengthen and coordinate international efforts to build regional civilian capacity to combat terrorism. As part of this international effort, and to mitigate the threats to the security of Canadians and Canadian businesses in the region, the Government of Canada provided the CTCBP with an additional $42.5 million for a period of five years (2010-2015) to address terrorist threats in the Sahel. ^{Footnote 1}

The relevance and performance of the CTCBP Sahel Envelope were assessed in a separate evaluation and are not covered in this report. 

Figure 1: CTCBP Theme Funding by Geographic Area, 2011-2015

1.2.3 ACCBP

Drug trafficking, organized crime, and related violence and corruption present challenges for governments around the world and threaten Canada’s security and interests. Underlying social conditions and the lack of governance capacity in some countries inhibit efforts to improve security. Persistent poverty, inequality, and unemployment leave large proportions of the population susceptible to crime. At the same time, underfunded security/police forces and failure to fully implement post-conflict institutional reforms render the security sector weak and susceptible to corruption.

Preventing and countering crime, drug trafficking, corruption and money laundering protects Canadian communities and businesses, and reinforces the rule of law on a global scale.

In December 2009, the Government of Canada established ACCBP with the purpose of enhancing the capacity of states, government entities and international organizations to prevent and respond to threats posed by transnational criminal activity. The program was created specifically to address national, regional and international security threats posed by criminal activities such as drug and firearms trafficking, money laundering, corruption, human smuggling and urban gang violence in the Americas.

Program activities and initiatives are implemented under six thematic areas:

Human Smuggling Envelope

In November 2012, the Prime Minister of Canada announced an additional funding envelope of $12M over 2 years (FY2011/12-FY 2012/13) aimed at preventing human smuggling operations abroad. The mandate of the ACCBP-HSE is to provide capacity-building assistance, mainly in the form of training and equipment, to assist beneficiary States in Southeast Asia with the detection and prevention of human smuggling operations destined for Canada. Some HSE funding was also allocated to West Africa, where some specific marine human smuggling threats to Canada were identified.

In response to the continued threat posed by human smuggling ventures to Canada, the program authorities were renewed in 2013 for another two years (FY 2013/14 - FY 2014-2015).

Funding for ACCBP is allocated to individual projects. As of 2009, 191 projects with a total value of $76M have been implemented by the Program in the Americas and the Caribbean.^{Footnote 2} Under the ACCBP-HSE, 55 projects were implemented during the evaluation reference period with a total value of $22M, the majority of which have been in Southeast Asia. The table below summarizes ACCBP funding by theme, including the HSE projects.^{Footnote 3}

Figure : ACCBP Funding by theme and Geographic Area

1.3 Governance

ACCBP and CTCBP share a common governance structure, which consists of the following entities:

Interdepartmental Steering Committee

The Interdepartmental Steering Committee (ISC) ensures that both programs are aligned with government-wide priorities for effective global security (anti-crime and counter-terrorism) capacity building. Other federal departments and agencies with a direct mandate to address international crime and terrorism can designate one Director General-level (or equivalent) representative as a member of the Steering Committee.

ISC members contribute to discussions and exchanges on GoC priorities based on their departmental mandates and emerging international trends. Committee members also contribute to setting strategic policy directions for the two programs (both geographic and thematic). They also review and make recommendations to the Annual Strategic Plans and Annual Reports.

The ISC contributes to the strategic direction for CTCBP, and ACCBP funding envelopes. It does not, however, have a mandate to address the ACCBP Human Smuggling Envelope, nor does it provide strategic direction for the Annual Voluntary Contributions (AVCs) to the UNODC and OAS, which are managed by the ACCBP and the CTCBP Secretariats.

ACCBP and CTCBP Project Review Committees

The Steering Committee is supported by the Project Review Committees (PRC) for each program, which in turn are responsible for reviewing^{Footnote 4}project proposals and providing input on projects within their respective subject matter areas of expertise. Other government departments with a direct or indirect mandate to address international crime can designate one representative at the deputy-director level (or equivalent) to the ACCBP Review Committee. Departments with direct or indirect mandate to address international terrorism can respectively designate one representative at the deputy-director level (or equivalent) to the CTCBP Review Committee, which also reviews the proposals for the Sahel envelope. The review committees do not, however, include the ACCBP Human Smuggling Envelope or the AVCs, both being managed by the ACCBP and the CTCBP Secretariats and having distinct governance structures.

The ACCBP and CTCBP review committees are chaired by the Director for the Capacity Building Programs Division (IGC), and is supported by the Chief of each Program.

Capacity Building Programs Division

The Capacity Building Programs Division (IGC) is responsible for the day-to-day program management and reporting. The Capacity Building Programs Division is led by a director and includes a staff complement of 19 FTEs. The Chiefs of Program (equivalent to a Deputy Director) report to the Director of the Capacity Building Programs Division and are responsible for the day-to-day financial and human resources management of the programs. The Division also supports the Project Review Committee for each program and the Chair of the Steering Committee to deliver the Steering Committee’s strategic mandate.

1.4 Program Resources

CTCBP Resources

CTCBP has an approved budget of $13 million per year (ongoing). Of that total, $2.5M is allocated for Vote 1 (Operations and OGD) and $10.4M for Vote 10 (Grants and Contributions).^{Footnote 5}In addition, CTCBP includes $42.5 million for the Sahel Envelope (2010-2015).

ACCBP Resources

ACCBP has an approved budget of $15 million per year (ongoing). Of that total, $2.3M is allocated for Vote 1 and $12.6M for Vote 10. ACCBP also includes the Human Smuggling Envelope (HSE) that had a four-year (2011-2015) budget of $24 million, of which $3.8M is allocated to Vote 1 and $20.2 M to Vote 10.

2.0 Evaluation Objectives and Scope

2.1 Evaluation Objectives

In accordance with the TB Policy on Evaluation, the Summative Evaluation of the Anti-Crime and Counter-Terrorism Capacity Building Programs aimed to assess results achieved to date by both ACCBP and CTCBP and provide GAC’s Senior management with a neutral evidence-based assessment of the relevance and performance of these programs.

The two programs were evaluated in parallel, with a special focus on the results achieved by each program and the extent to which the specific design and delivery approaches of each program supported their efficient and effective implementation. The parallel evaluation assessed synergies between the two programs with respect to their strengths and areas for improvement. Through a systematic data collection process, the evaluation reviewed the achievements of each program, as well as any outstanding gaps and/or areas for improvement. Based on the analysis of the collected data and information and the main evaluation findings, specific conclusions and recommendations were derived pertaining to the relevance and performance of ACCBP and CTCBP. Lessons and opportunities for improvement were also identified to guide future projects and activities under each program.

The specific objectives of the evaluation were:

• To evaluate the relevance of ACCBP and CTCBP by assessing the extent to which they address the needs of the department and its clients, and are aligned with federal government priorities and GAC’s strategic outcomes;
• To evaluate the performance of ACCBP and CTCBP in achieving their expected outcomes efficiently and economically;
• To examine synergies between ACCBP and CTCBP, as well as with other security-related programs, such as GPP and GPSF, and identify complementarities and potential duplications;
• To reflect on the lessons learned from the management and implementation of each program and make recommendations for improved management and delivery of the capacity building projects and initiatives; and
• To review the link between policy and programming in the IFM Branch and the extent to which policy priorities have been reflected in the design of the capacity building programs, as well as how programming results inform policy considerations.

2.2 Evaluation Scope

The evaluation focused on programming developed over the last five years, i.e. from FY 2010-2011 to FY 2014-2015. The evaluation also followed up on the recommendations from the 2009 formative evaluation of CTCBP, and the 2012 formative evaluation of ACCBP.

3.0 Evaluation Approach & Methodology

3.1 Evaluation Design

The evaluation was designed to assess five key evaluation questions:

• Continued need for the program
• Alignment with government priorities
• Consistency with Federal roles and responsibilities
• Achievement of expected outcomes
• Demonstration of efficiency and economy.

The evaluation was guided by an Evaluation Advisory Committee comprised of ACCBP ^{Footnote 6}and CTCBP program staff, staff from other related GAC divisions and representatives of other government departments. The Evaluation Advisory Committee reviewed the Evaluation Work Plan, participated in a presentation to validate the evaluation preliminary findings and reviewed the draft evaluation report.

3.2 Approach and Methodology

To ensure a systematic evidence-based process for data collection and analysis, the evaluation employed a combination of qualitative and quantitative methods and multiple lines of inquiry. Qualitative methods including stakeholder interviews, document and project file review, and observation through site visits were used to respond to all of the evaluation issues. These methods were complemented by quantitative methods to assess administrative data and other program information. The available data from various sources was triangulated to identify trends, similarities and points of convergence in order to develop the evaluation findings.

3.3 Evaluation Matrix

An evaluation matrix was developed as part of the Evaluation Work Plan to act as a framework for the evaluation process. It was used to guide data collection and facilitate analysis and the development of findings and recommendations. The matrix outlined the key questions to be answered by the evaluation as well as the related performance indicators, data sources and data collection techniques. (See Appendix [7])

3.4 Data Collection Methods

3.4.1 Primary Data Collection

Key Informant Interviews

A total of 175 key informants were interviewed for the evaluation in Ottawa and during field visits. In Ottawa, evaluators interviewed 38 key informants including current or past staff of ACCBP or CTCBP (n=13), other GAC staff at headquarters (n=12), representatives of other government departments (n=13). The interviews assisted evaluators in analysing performance of ACCBP and CTCBP and identifying key issues and areas for improvement.

Site Visits

The evaluators conducted site visits to observe project implementation and meet with implementing organizations, project partners and beneficiaries in eleven countries (Colombia, France, Guatemala, Indonesia, Jamaica, Jordan, Malaysia, Mauritania, Singapore, Thailand, and Trinidad). During those field visits, 137 key informants were interviewed including Mission staff working for GAC (n=18), other Canadian government departments (n=9), and representatives of recipients, implementing partners, allied countries and other local experts (n=110). Evaluators also attended a Steering Committee meeting for the regional UNODC Container Control Program in Thailand.  The field visits focused mainly on relevance and program effectiveness.

3.4.2 Secondary Data Collection

Document Review

Evaluators reviewed pertinent documents and literature to assess the relevance and performance of ACCBP and CTCBP. These included GAC corporate documents, descriptive and analytical reports, previous program evaluations, progress reports, and relevant planning, policy and priority review documents. Evaluators also reviewed international studies pertaining to global and regional crime and terrorism to assess the effectiveness of capacity building programming, and the relevance and continued need for ACCBP and CTCBP.

Project File Review

Evaluators conducted a file review of 40 projects ^{Footnote 7}to assess the extent to which ACCBP and CTCBP were achieving their expected outcomes in a cost-effective manner. The project sample selection was based on the following criteria:

• distribution among program areas and across geographic regions
• materiality (dollar value)
• implementing methods and partners
• status (completed or in progress)
• duration (one or multi-year).

The project sample was prepared in consultation with the ACCBP and CTCBP Secretariats. (See the project sample list in Appendix [3].)

3.5 Limitations to Methodology

The evaluation team used mitigating strategies to address methodological challenges or limitations, as summarized below.

4.0 Evaluation Findings

The following findings are based on the triangulation of evidence from relevant literature, project documents, project file reviews, key informant interviews and observation during site visits.

4.1 Relevance Issue 1: Continued Need for the Programs

Finding #1: The evolving nature of the threats posed by international terrorist networks and their increasing frequency and severity require governments to develop respective programs, initiatives and strategies to counter these threats. CTCBP, ACCBP and HSE continue to be a relevant GoC response to these threats and to effectively adapt to the changing needs related to the fight against international crime and terrorism.

Based on evidence from interviews, site visits and document review, CTCBP and ACCBP (including HSE) continue to be relevant and adapt well to changing needs or threats related to international crime and terrorism. The relevance and evolution of each Capacity Building Program^Footnote8(CBP) is assessed separately below.

4.1.1 Counter-Terrorism Capacity Building Program

During the evaluation period, half of CTCBP’s programming (excluding the Sahel Program) was in the Middle East and North Africa (MENA), and Asia.^{Footnote 9}Given that geographic focus, the section below highlights the main security needs or threats that CTCBP aimed to address in those regions.

Middle East and North Africa

The threats posed by international terrorism networks increased in frequency and severity over the last few years. Recent attacks in North America and Europe revealed the danger that violent extremists pose to any country. The rise of ISIS in the Middle East demonstrated the continued ability of terrorist groups to expand rapidly and recruit globally. The conflict in Iraq and Syria attracted Foreign Terrorist Fighters (FTFs), particularly from the MENA region, into the ranks of ISIS and other armed terrorist groups. Many countries lacked the laws, immigration system or border security necessary to control the flow of FTFs. Thus controlling FTFs was among the highest global security concerns for Western governments and destination countries. A related global priority was Countering Violent Extremism (CVE) through policies to discourage recruitment and improve outreach in communities at risk.

CTCBP programming focused on blocking the transnational transit routes of FTFs, and countering financing of terrorism (CFT). The Program started CVE projects in 2014-2015, in response to the increased need to prevent recruitment, and rehabilitate returned fighters.

Asia

Some countries in South Asia, e.g. Afghanistan and Pakistan, have faced an ongoing Taliban insurgency that has threatened the region’s stability. Central Asia was a source and transit region for violent extremists who went to join ISIS and other groups. CTCBP had significant programming in Afghanistan that focused on bilateral security training, although that decreased following the military and political drawdown in 2012.

Southeast Asia has been threatened by violent extremism from separatist insurgencies, particularly in the Philippines, Malaysia and Indonesia. CTCBP programming there contributed to building the skills of law enforcement officers, Countering Improvised Explosive Devices (C-IED), supporting INTERPOL’s counter-terrorism activities, and combating the financing of terrorism

Assessment of Continued Relevance of CTCBP

All 15 CTCBP projects reviewed during the evaluation showed evidence that they responded to the needs of beneficiary states. Twelve projects provided data on how they identified and responded to potential security threats to Canada, while the other three did not explain that explicitly.

Evidence from interviews and project documents showed that CTCBP analyzed and responded to emerging security needs well. For example, CTCBP partners identified Middle Eastern financial institutions as a potential source of terrorist funding in 2010, a few years before that issue became prominent within the anti-ISIS coalition. Responding to the increased threat posed by ISIS at the regional and global levels, CTCBP introduced FTF and CVE programming in 2014, and increased programming in Iraq and Jordan.

4.1.2  Anti-Crime Capacity Building Program

During the evaluation period, more than 70% of ACCBP’s programming was in Central America and the Caribbean, with the remainder being spent in South America and Mexico. Given that geographic focus, the section below highlights the main security needs or threats that ACCBP aimed to address in Central America and the Caribbean.

Central America

The impact of illicit drug trafficking increased in Central America over the last several years as transnational criminal organizations transferred production operations there from South America and Mexico. The security situation was worst in the Northern Triangle (Guatemala, Honduras, and El Salvador), which had the highest violence and homicide rates in the world, along with the lowest levels of criminal prosecution and conviction. The illicit drug trade drove related issues such as increasing levels of corruption and money laundering. Illegal trafficking in persons became an area of expanding operations and profit for transnational criminal organizations. Unaccompanied child migration was another emerging concern.

ACCBP programming in Central America focused on security sector reform (including reduced impunity and corruption), and combating illicit drug trafficking. The Program also started to fund projects related to unaccompanied child migration.

The Caribbean

The Caribbean faced several security-related problems during the evaluation period. As a transit region for illicit drug trafficking from the Andean region to Europe, the Caribbean had high rates of criminality and violence. Its financial institutions faced the risk of being used for money laundering and terrorist financing. National and regional security institutions had relatively low capacity and little coordination.

ACCBP programming in the Caribbean covered security and justice reform, combating illicit drugs, anti-corruption and anti-money laundering. ACCBP focused on building the capacity of counterparts in Jamaica and Trinidad & Tobago, which could serve as regional hubs for expertise.

Assessment of Continued Relevance of ACCBP

All of the 12 ACCBP projects reviewed during the evaluation showed evidence that they responded to the needs of beneficiary states. Eight of the projects documented evidence that they identified and responded to potential security threats to Canada.

Evidence from interviews and project documents showed that ACCBP adapted well to changing security needs. Two examples of this were:

• ACCBP shifted its resources from Mexico and South America in 2011-2012. The Program focused on Central America as it went from being primarily a transit region to a production source, with increased rates of violence and corruption. Within Central America, ACCBP concentrated on the Northern Triangle where transnational criminal activity was worst.
• Since ACCBP was established in 2009, transnational organized crime (e.g. money laundering, trafficking of persons) has become even more globalized. In response to globalized threats, the ACCBP’s Terms and Conditions were changed in April 2015 to allow programming outside the Americas. The first new projects approved were for border security in Ukraine and police reform in the Philippines.

A few interviewees questioned whether ACCBP projects were always based on real needs identified by recipient partners. In a few cases where an OGD wanted to deliver a standard course, training seemed to be more driven by supply (OGD interest in delivering the course) rather than demand.

4.1.3 Human Smuggling Envelope

Every year, hundreds of thousands of migrants are moved illegally by highly organized international smuggling and trafficking groups. This growing global threat is often linked to other criminal activities such as money laundering, corruption, drug and arms trafficking. Canada was targeted as a destination country, with hundreds of illegal migrants arriving by boat in 2009-2010.

HSE programming contributed to curbing the flow of illegal migrants by providing different levels of training and specialized equipment to over 6,000 immigration and law enforcement officers. HSE projects focused on countries that were, or were likely to become, transit points for boat smuggling operations to Canada, notably Southeast Asia and West Africa.

Assessment of Continued Relevance of HSE

Of the 13 HSE projects reviewed during the evaluation, 12 had evidence on file of the need in beneficiary states. Only one project did not have adequate documentation to demonstrate beneficiary need. However all 13 projects provided evidence that they identified and responded to threats to Canada.

Evidence from interviews and project documents showed that HSE adapted well to changing needs and priorities. For example, HSE provided support to partners in West Africa when human smuggling operations spread there from Southeast Asia.

4.2  Relevance Issue 2:  Alignment with Government Priorities and Other GAC Programs

Finding #2: CTCBP, ACCBP and HSE align with GoC foreign policy priorities, Canada’s Counter-Terrorism Strategy, as well as other whole-of-government and GAC policies and strategies.

4.2.1 Counter-Terrorism Capacity Building Program

Canada’s Counter-Terrorism Strategy^{Footnote 10}outlined the roles and responsibilities of each Department. One of GAC’s responsibilities was to “use its broad international network to enable counter-terrorism co-operation with other states and its role within multilateral organizations to enhance the security of Canadians and Canadian interests”. CTCBP contributed to implementing that Strategy by building national and regional capacity to counter financing of terrorism, foreign terrorist fighters, and violent extremism in the most vulnerable regions in the world (Middle East and Asia).

4.2.2 Anti-Crime Capacity Building Program

One of the objectives of Canada’s Strategy for Engagement in the Americas^Footnote11is “supporting efforts to improve security, especially in Central America, Mexico and the Caribbean”. ACCBP was a key element of that security pillar. The Program aimed to promote justice, security, peace and human rights; prevent violence, crime and corruption; and combat transnational organized crime, including drug trafficking.

4.2.3 ACCBP Human Smuggling Envelope

Canada’s Migrant Smuggling Prevention Strategy was put in place in 2010 as a response to the arrival of two human smuggling vessels on the West Coast. HSE contributed to implementing the Strategy by helping to prevent irregular migration through training and the delivery of specialized equipment for immigration and law enforcement officers in countries that were likely to be transit points for human smuggling operations destined for Canada.

Finding #3: While there is evidence of increased consultations and information sharing among GAC’s security and development programs, the coordination and collaboration with regard to planning and program implementation can be further increased.

Consultation between ACCBP and CTCBP

The evaluation found good coordination between ACCBP and CTCBP because they had:

• joint governance bodies (ISC, PRC).
• regular exchanges between program staff that helped reduce duplication, which were easier because ACCBP and CTCBP were housed in the same Division (IGC)
• some flow of project officers between the two Programs.
• the same  Standard Operating  Procedures  and project management tools.

ACCBP and CTCBP programming is under one Director General (DG) (IGD), while crime and terrorism policy work is handled by a division (IDT) reporting to a different DG (IDD). That split required staff to make concerted efforts to align programming and policy. According to GAC staff, IDT and IGC staff consulted regularly at the working level and prepared joint briefing notes.  Regular IDT-IGC management meetings are held but there was less coordination at the DG-level.. In some cases, coordination in the field was reported to be better than at HQ.

GAC staff noted a few cases of thematic or geographic overlap between ACCBP, HSE and CTCBP projects. For example, the financing of terrorism (CTCBP) and money laundering (ACCBP) were overlapping issues. There was some fluidity of projects between the two programs. For example, an OAS critical infrastructure project was first funded under CTCBP, and then transferred to ACCBP.  ACCBP had a sub-theme of human smuggling, although its projects were limited to the Americas, while HSE focused on Southeast Asia. Some implementing partners reported uncertainty about whether to submit proposals to CTCBP or ACCBP because the criteria or boundaries for each were not clear.

Coherence with other Security Programs and Other Divisions

The evaluation found that there was less coherence between ACCBP-CTCBP and other GAC security programs. GAC’s International Security Branch (IFM) funds security-related initiatives through seven programming channels (programs and envelopes), as outlined below.

The security programs (CTCBP, ACCBP, GPSF and GPP) were designed at different times to support GoC policy initiatives that emerged independently. The programs were not planned as a cohesive or integrated package, which created challenges for communication and coordination. Stakeholders interviewed during the evaluation noted that GAC took steps to improve coherence among its security programs, however there were still limitations, as summarized in the table below.

The steps taken helped increase communication among security programs, however some GAC staff told evaluators that there was still some confusion of roles between different corporate units. According to staff, policy and programming shops were in separate “silos”; and each security program was in its own silo. Coherence among security programs needs to be solidified and systematized.

Based on evidence from interviews and document review, the greatest potential overlap (and opportunity for coordination) of security programming was between ACCBP/CTCBP and GPSF because of their similar thematic coverage and focus on capacity building. For example, all three programs provided support for capacity building in Colombia. Those programs overlap with some areas of GAC’s development cooperation in Colombia as well (see Table 6 below).

According to interviews at HQ and in the field, ACCBP and CTCBP staff regularly consult the geographic branches; however the capacity building programs, geographic branches and OGD HQs in Ottawa had a mixed record in engaging Missions. GAC staff at some Missions reported that they were actively involved in the selection of ACCBP/CTCBP projects, while OGD representatives at some missions indicated that they were not always consulted on security programming by their HQs, even when they might have had relevant expertise and better knowledge of the situation and partners in the field.

Coordination with Development Programming

With the amalgamation of DFAIT and CIDA in 2013, coordination between ACCBP/CTCBP and development programming became a higher priority and created a challenge for the security programs. There were several possible areas of overlap (and/or complementarity) between GAC’s security and development programming. For example, ACCBP, GPSF and development programs all supported capacity building projects to combat corruption. Anti-crime activities in Ukraine were funded through GPSF^Footnote12, ACCBP (starting in 2015) and development cooperation. Projects dealing with human smuggling or trafficking were funded by HSE (criminal justice and law enforcement in Asia); ACCBP (criminal justice, law enforcement, border control in the Americas); GPSF (justice, security, reintegration of victims); and development programs (law enforcement, trafficking prevention).

That potential overlap (and/or synergy) was seen in countries or regions visited by the evaluation team, as illustrated in Colombia example below.

In the case of Colombia, stakeholders noted that coordination was quite good between ACCBP/HSE/CTCBP and development cooperation. GAC developed an integrated Country Strategy (2015) that included development and security programming and policy engagement in Colombia. Security and development programs worked well together in several African countries as well, according to CTCBP staff.

The evaluation mission to Guatemala found evidence of the complexities of coordination between ACCBP and other security programming there:

• development funding instead of ACCBP: ACCBP (or GPSF) planned to support the International Commission against Impunity (CICIG), however it could not provide the level of funding ($5M) over the longer-term period (three years) that was required. CICIG was passed to the development program for funding, even though it did not fit in the sectors outlined in GAC’s Development Country Strategy for Guatemala.
• complementary funding from both ACCBP and development: The development side had supported the Justice Education Society (JES) since 1999 to build the anti-crime capacity of police and public prosecutors in Guatemala. ACCBP provided complementary funding to JES starting in 2011, and then became the main GAC partner
• overlapping, uncoordinated funding from ACCBP and development: Through CICIG, GAC-DEV supported capacity building of the Public Prosecutor’s Office, which also received funding from an ACCBP project. Having two separate projects with different GAC managers created confusion for Guatemalan partners, who wondered why Canada had “two cooperation sections” at the Embassy.

GAC staff interviewed during the evaluation suggested that security and development programming each have comparative advantages that could be combined in a coherent and complementary package, as illustrated in Table 7 below.

During the evaluation period, GAC took some steps to improve coordination between ACCBP/CTCBP and development programming; however challenges remained as summarized below.

Several GAC staff at HQ and Missions stressed the importance of improving coordination between ACCBP/CTCBP (security programs) and development because: 

• Capacity development is a long-term process. Many countries lack the necessary expertise in counter-terrorism and anti-crime to be able to readily assimilate and implement training provided by CTCBP or ACCBP.
• Amalgamation created the opportunity, and necessity, for coherent and coordinated departmental programming.
• Several Canadian and international implementing partners such as INTERPOL or UNODC work with different GAC divisions, and find it hard to manage different sets of departmental management requirements.
• Canada’s development programming is increasingly expected to apply a “security lens” when possible (similar to the USAID model), thus needs to coordinate better with ACCBP/CTCBP and other security programs.
• GAC-DEV programs have significantly larger budgets than ACCBP/CTCBP in some “countries of focus”^Footnote13. For example, GAC has 25 people on the ground in Mali, so that Mission (or PSU) could provide valuable support to CTCBP projects. ACCBP/CTCBP should seek ways to leverage and complement those larger development investments.

Within the amalgamated department, there was growing interest in reinforcing “security value chains” in specific regions and countries through coordinated programming in areas such as conflict prevention, rule of law, corruption and human trafficking. GAC staff suggested some steps that could be taken towards the goal of having a “seamlessly coherent Canadian cooperation program,”^Footnote14 which are presented in section 6.

4.3 Relevance Issue 3: Consistency with Federal Roles & Responsibilities

Finding #4: GAC is the appropriate department to lead ACCBP and CTCBP, although it could promote better communication among OGDs.

ACCBP and CTCBP are closely aligned with GAC’s international mandate that includes supporting international security and the safety of Canadians abroad. GAC is the appropriate department to manage ACCBP and CTCBP because the programs require negotiation and coordination with governments of recipient and donor states, and multilateral organizations. GAC’s network of international Missions with contacts on the ground provides support to ACCBP and CTCBP projects abroad.

ACCBP and CTCBP projects funded through Vote 1 are delivered by other government departments (OGDs) including the RCMP, DND, CBSA, FINTRAC and CCC. The OGDs are involved in project design and selection through the annual Priority Review, the Inter-departmental Steering Committee ISC), and the Project Review Committee (PRC). Before presenting its project proposals, an OGD often undertakes a Needs Assessment Mission (NAM) to meet partners in the target country or region.

According to staff interviewed at several OGDs (RCMP, FINTRAC, Justice Canada, CBSA), their departmental priorities in international security aligned well with those of ACCBP and CTCBP.

OGD staff in Ottawa and liaison officers in the field who were interviewed indicated that they were not familiar with projects implemented by other departments in the same country or region. Each OGD tended to work with different national partners so they could not cooperate at the operational level. For example, the RCMP worked with national police forces or INTERPOL, while DND trained counterpart armed forces. Some OGDs did not inform GAC field staff of security-related missions or projects in their country.

According to some interviewees, not only was there little connection across departments, some OGDs did not even not coordinate capacity building activities within their own department. There was no whole-of-government or whole-of-department approach to security cooperation.

It would have been useful for OGDs to share information, lessons and good practices from their projects with Canadian government departments working in the same country. As a step towards establishing better communication, CTCBP organized a joint NAM on the issue of Foreign Terrorist Fighters (FTFs) in the Middle East. However CTCBP and ACCBP could do more to build connections among OGDs on security-related programming. Some OGD interviewees suggested that GAC and other OGDs should collaborate to develop a security strategy for each region (e.g. Caribbean) to “synchronize and synergize” activities, work more effectively, and have greater impact.

4.4 Performance Issue 4: Achievement of Expected Outcomes

Finding #5: There is some evidence that ACCBP, HSE and CTCBP are making progress towards expected outcomes. However the programs lacked systems or data to measure intermediate outcomes.

While the evaluation did find some evidence of progress towards expected results, ACCBP and CTCBP lacked a systematic focus on outcomes in both their programming and performance measurement (monitoring, reporting) (see 4.5.8). Evidence from interviews, site visits and project files showed that ACCBP, CTCBP and HSE concentrated on the short-term delivery of outputs such as equipment and training, with more than half (56%) of projects completed in a year or less.

The sections below provide illustrative examples of some key results achieved by projects in each Capacity Building Program. The examples were prepared based on information gathered through the evaluation site visits and the project sample document review. The examples highlight intermediate outcomes where possible, and go beyond the results of an individual project to present aggregate results over time related to a particular issue, implementing partner and/or country.

In order to demonstrate various ways to present project and program outcomes, each example has a different focus. The CTCBP example shows the combined impact that various capacity building projects had in Malaysia. The ACCBP example covers the longer-term results achieved through various projects implemented by the same Canadian and Guatemalan partners. The HSE example illustrates the outcomes of two regional projects to combat human smuggling.

4.4.1 Counter-Terrorism Capacity Building Program

CTCBP contributed to building national and regional capacities to counter terrorism. For example, the results of CTCBP projects in Malaysia are presented in the textbox below.^Footnote15

4.4.2 Anti-Crime Capacity Building Program

ACCBP achieved notable results in combating criminal activity such as drug trafficking, gang-related violence and corruption in Latin America and the Caribbean. Illustrative aggregate results of a series of projects implemented by the same Canadian and Guatemalan partners are highlighted in the textbox below.

4.4.3 Human Smuggling Envelope

Over the past four years, programming under the HSE has evolved concurrently with the evolving nature of the threats, and has led to the establishment of reliable partnerships with key beneficiaries, which in turn has helped the Program to better target and address the specific threats to Canada. Close collaboration and coordination with trusted allies such as the U.S., Australia and New Zealand, as well as with international organizations such as the International Organization for Migration (IOM) and the United Nations Office on Drugs and Crime (UNODC) have increased Canada’s image in the region as a trusted and reliable partner in preventing human smuggling, in-person trafficking, and illegal migration. Interviews with representatives from like-minded donor countries reiterated the importance of Canada’s efforts in enhancing the capacity of the maritime police and the border and immigration services in the countries of SEA.

Funding through the HSE has, for example, enabled IOM to support the governments in the region to strengthen their document examination capacity to more effectively prevent human smuggling and human trafficking by combatting the use of fraudulent documents as a means through which human smuggling flourishes.

4.4.4 Sustainability of Results

Finding #6: Longer-term capacity building projects tended to have greater potential for sustainability than one-time training.

The evaluators assessed whether the design of capacity building projects contained elements that would support or hinder the sustainability of results after GAC funding ended. Several projects assessed through the site visits or evaluation sample showed signs that results would likely be sustainable. Projects that built capacity and relations with partners over a longer-term were more likely to have sustainable results. That relationship could have been developed through one longer-term project, or a series of short projects designed to develop capacity in an incremental way over time. However it was harder to maintain continuity and momentum with shorter projects, mostly because of the potential time-gap in getting subsequent projects approved (see 4.5.3).

Some ACCBP and CTCBP projects were able to leverage funding, or to use their funding to complement larger initiatives, thus multiplying the impact and increasing the likelihood of results sustainability of the GAC investment.  For example, ACCBP anti-corruption projects in Jamaica were coordinated with larger programs supported by the US and the UK. Through ACCBP, the RCMP provided polygraph training and equipment to the Jamaica Constabulary Force (JCF). The projects helped reduce police corruption by firing (but not prosecuting) 500 officers who failed the polygraph test, improve forensic investigation techniques, and build public confidence in the JCF. The projects funded by ACCBP and other donors enabled Jamaica to become a Polygraph Center for Excellence in the Caribbean.

The main factor limiting sustainability was the complex, evolving security situation in some states where ACCBP and CTCBP operated. Changes and challenges on the ground made it hard or impossible to implement (or sustain) a few projects.

Results achieved by some projects would be difficult to sustain without continued funding from GAC. For example, ACCBP was a major funder of the UNODC’s Container Control Program (CCP) in Latin America and the Caribbean. The program aimed to build the capacity of port control units to identify and inspect high-risk freight containers that could contain illicit drugs or other black market goods. The CCP contributed to a large (twenty-fold) increase in the volume of drugs, weapons and other contraband seized by port control in several countries.^Footnote19According to a 2013 evaluation,^Footnote20the CCP made a significant contribution to “greatly improved security in the supply chain of containerised goods” in the countries where the program operated. However based on interviews done during evaluation site visits, it was difficult to sustain that level of successful results after ACBBP funding stopped because beneficiaries did not have money or inputs to continue applying and improving the training they received.

Several GAC staff and implementing partners also questioned the sustainability of one-time training courses that were not tied into longer-term project activities. According to interviewees, several training workshops and support from senior decision-makers would be required to institutionalize and sustain capacities related to anti-crime and counter-terrorism. That view was supported by evidence from international literature that indicates that short-term activities should contribute to long-term learning and change strategies.^Footnote21

Part of the reason that many ACCBP and CTCBP projects were short-term (one year or less) was due to restrictions in their funding terms, or an unpredictable approval process that limited the Programs’ ability to do longer-term planning and programming (see §4.5.3). After multi-year funding was introduced, the number of two or three-year projects increased significantly. By 2015, 36% of Capacity Building projects had funding for two to three years, and 8% was over three years. More than half (56%) of ACCBP and CTCBP projects had one-year funding.^Footnote22

Interviewees identified a few factors that promoted or limited the potential sustainability of results, as summarized below.

4.4.5 Results for Canada

Finding #7:  CTCBP, ACCBP and HSE have increased Canadian expertise, leadership and visibility in international security programming.

Benefits for OGDs

Capacity building projects had reciprocal benefits for recipient countries and for Canada. Several OGDs interviewed in Canada and the field provided examples of how the Capacity Building Programs increased their technical capacities and knowledge.

Implementing ACCBP and CTCBP projects increased OGDs’ connections with national, regional and global networks. For example, FINTRAC reported that projects with CTCBP and ACCBP helped strengthen its work with the international Financial Action Task Force (FATF), while CSBA built stronger relations with the World Customs Organization (WCO).

The international relations that OGDs developed through the Capacity Building Programs could assist them with investigating and thwarting potential criminal or terrorist activities in Canada. For example, FINTRAC projects through ACCBP targeted Caribbean countries with strong ties to Canada. By helping those countries become better partners in tracking criminal financing, ACCBP projects ensured that evidence was collected according to international standards so that FINTRAC could use it in the Canadian context.

Benefits for Canada

At the national level, CTCBP and ACCBP projects helped “open the door” and facilitate access of Canadian officials (including Heads of Mission) to senior decision-makers in several countries. For example, CTCBP projects in Kenya and Mali gave Canada access to high-level government officials there, as well as a seat at the donor coordination table. ACCBP-funded training in Jamaica gave Canada higher-level access within the government, military and police, as well as increased trust and information-sharing. That influence and access extended beyond the immediate scope of the CTCBP projects.

At the regional and international level, ACCBP and CTCBP increased Canada’s profile and influence in policy networks and coordination bodies. That was particularly evident when the Capacity Building Programs had multiple and/or longer-term projects in same region with same partners. Evidence from interviews and site visits showed that ACCBP, HSE and CTCBP helped Canada gain credibility and trust with partners and allies. That credibility increased most through focused, longer-term capacity building with partners, rather than one-off training.

Canada’s contribution to capacity building was generally well recognized and visible, even in some projects implemented by international organizations such as the INTERPOL, UNODC or IOM. A few regional projects implemented by IOs such as UNODC did not always provide visibility for Canada.

Finding #8:  Canada is a respected partner in security capacity building, however unmet expectations may have damaged its international image.

Most beneficiaries of ACCBP, HSE and CTCBP projects who were interviewed during site visits expressed a high degree of satisfaction. They found Canadian (and international) implementing partners to be competent and committed. Canada was generally considered to be a well-respected and trusted partner. However project partners and GAC staff mentioned several cases where multiple NAMs visited countries such as Malaysia, and created expectations that projects there would be supported. Then projects were not approved in Canada, or approval was delayed so long that those initiatives lost their strategic importance or relevance. (See §4.5.3 for a discussion of the project approval process.) Those unmet expectations and delays tarnished Canada’s image, created frustration for partners, and led to missed opportunities for ACCBP and CTCBP to play an even more strategic role in some countries or regions.

4.5     Performance Issue 5:  Demonstration of Efficiency & Economy

4.5.1 Program Focus

Finding #9:  ACCBP and CTCBP are most effective if/when program resources are more focused.

During the five-year evaluation period, ACCBP funded 191 capacity building projects of various types and with multiple implementing partners and recipient organizations in over 40 countries (with a regional focus on the Americas). HSE supported 55 projects in over 30 countries in Asia, and West Africa. Projects ranged in size from $11,000 to $1.9M.

According to GAC staff, ACCBP and CTCBP used to have a “shotgun approach” whereby projects were scattered widely and spread thinly. CTCBP started moving away from that shotgun approach two or three years ago. For example, CTCBP used to have projects dispersed across Southeast Asia, then shifted to a “countries of focus approach” targeting ten ASEAN member states. CTCBP staff reported that concentrating resources on bigger programs in particular countries like Jordan gave partners better access and influence in those areas.

Efforts by ACCBP and CTCBP to focus their programming were limited by the ineffective project approval process that was introduced in 2011 (see the next section). Having unpredictable and/or delayed approvals made it difficult for ACCBP or CTCBP to develop a coherent set of projects designed to build capacity over time in a particular country or region. Evidence from interviews and document review showed that ACCBP or CTCBP projects approved during the 2011-2015 period seemed ad hoc, and did not necessarily fit within top Program priorities.

ACCBP and CTCBP received requests to expand their geographic scope as anti-crime and counter-terrorism problems became increasingly globalized. For example, ACCBP was extended beyond the Americas in 2015 to cover a global mandate. However the Programs could not expect to address all anti-crime and counter-terrorism problems around the world with their relatively small annual budgets of $13M for CTCBP and $15M for ACCBP. According to many interviewees, the demand for capacity building far exceeded the funding available from the international community. Given that Canada was a relatively small player on the international stage, ACCBP and CTCBP need to find their “strategic niche” where they could bring the most value-added in terms of recipient needs and Canadian interests.

4.5.2 Resource Utilization

ACCBP and CTCBP disbursed about 20% of their funding through Vote 1 programming with OGDs, and 80% through Vote 10 grants and contributions to international organizations. Each funding channel presented different issues related to resource utilization; therefore they are discussed separately below.

Finding #10: Many OGDs did not spend their allocated Vote 1 funding, despite measures taken by the program to reduce the risk of OGDs making commitments beyond their capacity to implement projects within the fiscal year.

An expenditure review over the five-year evaluation period showed that OGD have returned on average about 10 percent of their allocated CTCBP programming budget and up to 45% of the ACCBP Vote 1 programming funds.

A number of factors were cited as reasons for the inability of OGDs to spend the resources they had initially planned and obtained through CTCBP, ACCBP and the HSE:: i) the OGDs’ lack of experience or systems to manage international projects and financial transactions; ii) their tendency to over-estimate budgets; iii) lengthy GAC process to prepare some MOUs (up to six months), which leave OGDs with less than six months to deliver programming scheduled for one year; and iv) unexpected political instability or security risks in recipient countries that prevent the implementation of projects as planned. If and when OGDs return money in March, ACCBP and CTCBP have little time to reallocate and avoid lapsing funds at fiscal year-end.

In response to the OGDs’ consistent inability to fully execute their budgets, ACCBP and CTCBP took a few mitigating steps. Program staff worked with OGD managers to improve their financial planning and develop longer-term strategies (two to three years) to avoid inflating project budgets. IGC began making activity-based payments, i.e. releasing a quarterly instalment only after disbursed funds were spent and reported on. OGDs were required to return unspent funds by February 1 so that they could be reallocated, and no OGD programming was planned for the fourth quarter. However those measures did not lead to a notable decrease in the rate of returned funds from OGDs to the Capacity Building Programs, and IGC staff began over-committing program funds by 40% to 50% to be able to absorb the returned funds

Several of those interviewed from GAC and OGDs suggested that ACCBP and CTCBP should increase the relative allocation of total program funding through Vote 1 by 20% to 30% or higher. The justification for that suggestion was partly because there was a high unmet demand for training by some OGDs, but also partly administrative, i.e. because Vote 1 funding for OGDs was easily accessible compared to Vote 10 funding that was subject to the lengthy departmental approval process.

Before increasing the Vote 1 allocation, however, Capacity Building Programs would need to assess: the absorptive capacity of OGDs in terms of technical and management staff; the absorptive capacity (institutional, financial) and real demand of potential recipients; and the comparative advantage of delivering programming through OGDs vs. other mechanisms like IOs to meet a particular need in a given context.

A few factors limited the absorptive capacity of OGDs. For example, there was high demand for counter-terrorism training provided by DND, but DND did not have the operational capacity and resources to engage in more countries. DND and other departments could not hire additional staff without financial predictability, i.e. guaranteed, medium-term (e.g. three-year) funding from the Capacity Building Programs. However ACCBP and CTCBP could not commit longer-term funding to an OGD because Vote 1 allocations had to be reconciled on an annual basis.

According to OGD interviewees, the fact that ACCBP and CTCBP did not cover the salaries of their staff engaged in international capacity building limited their ability to expand programming and led to missed opportunities for engagement. For example, Justice Canada could not accept two potential projects in Guatemala and Trinidad because it could not cover the associated salary costs.  

Finding #11: ACCBP and CTCBP were unable to spend a significant portion of their allocated Vote 10 funds over the past few years.

An expenditure review for the last five years showed an uneven rate of annual spending rate for Vote 10 budget, mainly due to the challenging project approval process. The average annual spending rate also varied among programs, as shown in the tables below.

CTCBP had an average spending rate of 76.5% (from the allocated funds) between FY2010-2011 and 2013- 2014; with disbursements reaching 136% in FY 2014-2015.

A closer look at the spending patterns shows that while all Vote Grant Funds have been allocated, there have been some considerable lapses and returned Vote 10 Contribution Funds.

ACCBP also showed an uneven rate of spending. In 2011-2012, ACCBP exceeded twice its allocated spending (204%) reflecting the high rate of project approvals in support of the Americas Strategy. In the following year, ACCBP effectively had only one project approved.

The following tables indicate the fluctuations in ACCBP Vote 10 spending over the past 5 years, mostly due to departmental priorities (e.g. Americas Strategy in 2001-2012) or challenges with the project approval process.

The Human Smuggling Envelope, albeit being a high priority for the GoC, also showed fluctuating rates of annual spending varying from 134% in FY 2011-2012, down to 29% in FY 2012-2013.  The evaluation found that while HSE projects had a much higher rate of approval, the annual fluctuations were often reflective of the challenges with designing and implementing projects in a complex environment.

According to GAC interviewees, the main reason ACCBP and CTCBP under-spending Vote 10 allocations over the last three years has been the project approval process, as discussed in the next section.

4.5.3 Project Approval Process

Finding #12:  Difficulties in the project approval process reduced the effectiveness and efficiency of Capacity Building Programs.

For the first two years of the evaluation period, the approval process for ACCBP and CTCBP was fast, and the rate of approval for projects funded through Vote 10 was almost 100% (as shown in the tables above). In 2011-2012, ACCBP was able to over-spend its allocated budget by absorbing unspent money from other programs.

From August 2011 to April 2015, a new Project Initiation Authorization (PIA) process was implemented. through which a high percentage of capacity building projects were either not approved or significantly delayed. For example, ACCBP and CTCBP would each have an average of 100 project proposals awaiting approval at any given time. In one year, ACCBP had only 1 out of 100 proposals approved, i.e. 1% (compared to 100% in 2011-2012). ^Footnote23CTCBP reportedly had a “better batting average”, although it did not track the percentage of projects that got approved. ACCBP and CTCBP sent up proposals worth about four times the available funding, with the expectation that only a small number would be approved.

HSE had the highest project approval rate, with a budget of $12M over two years. Overall, HSE and CTCBP had higher rates of approval than ACCBP since human smuggling and counter-terrorism were increasing priorities for the GoC during that period.

IGC staff reported that they had to develop hundreds of proposals to ensure that a sufficient number would be approved as per the allocated budget. That inefficient process put a heavy burden on staff, reduced the time available for monitoring ongoing projects, and had a demoralizing effect. It limited the ability of ACCBP and CTCBP to plan and implement any projects, or achieve results. In particular, the approval process hampered the Programs’ capacity to do longer-term strategic planning and programming.

The PIA process also meant that priority projects were often not approved in a timely way. Through an annual Priority Review, the CBPs and their partners jointly identified key priorities in anti-crime and counter-terrorism in each region of the world. Based on that Review, OGDs conducted Needs Assessment Missions (NAMs) in the field and developed project proposals that were reviewed by the PRC before being submitted for approval.^Footnote24

However the highest priority projects were not necessarily the ones that were approved and IGC received no justification as to why a particular project was or was not approved. Projects were approved on an “episodic basis” that was not predictable or consistent, so ACCBP or CTCBP staff could not better target subsequent proposals. The CBPs and their partners managed the situation by submitting more proposals than could be supported, in order to provide options within the PIA process, rather than strictly following the sound prioritization process they had in place.

IGC managers used a “hit and miss” approach, presenting numerous project options for approval, rather than only a select few.  In some cases, projects that were approved for implementation did not reflect the highest priorities among those set out in the Priority Review. This has led to some disconnect between the projects identified through the Priority Review and the CBPs’ final “programming footprint” in a particular country. The PIA process could take two or three years. During that time, security priorities or needs could change significantly, particularly in volatile countries. As a result, some ACCBP or CTCBP were no longer as timely or relevant as they had been when designed. In several cases, other donors had moved in to fill needs identified through the Priority Review, so Canadian support was no longer required. In a few cases, projects were approved too late in the year for international partners or OGDs to complete execution before the end of the Canadian fiscal year.

According to implementing partners interviewed during site visits, Canada’s approval process for CTCBP and ACCBP projects was much slower and more complicated than that of other donors such as the UK, EU or US. Canada’s slow, or lack of, project approval undermined its credibility, trust and profile among international partners and recipient countries. Because GAC staff could not predict whether a particular project would be approved, they could not make funding commitments to international partners or recipients. Capacity Building Program staff and OGDs were reluctant to engage recipient partners in the planning process before knowing whether a particular project would be approved.

In April 2015, the project approval process was changed for ACCBP, CTCBP and other GAC programs. To date, the new process has been quicker and provided an approval rate of about 80%.

In November 2015 the approval process was lifted, allowing the CBPs to program according to their Terms and Conditions.

4.5.4 Governance Structure

Finding #13: Governance committees were well designed, but became less effective over the evaluation period.

According to GAC and OGD staff, the Interdepartmental Steering Committee (ISC) and Project Review Committee (PRC) had well defined roles and worked effectively for the first two years of the evaluation period. However after the new approval process was introduced in 2011, the ISC and PRC lost relative importance and influence in decision-making about project selection and approval. For example, OGDs helped set the annual priorities for ACCBP and CTCBP through the ISC. However after 2011, the annual Priority Review had less impact in determining the projects presented for approval, and those approved (see §4.5.3 above). As a result, OGDs started sending lower-level representation to ISC meetings.

Through the PRC, OGDs previously played an active role in the project selection process, which sometimes took several months to reach consensus among the 18 participating OGDs. Departmental representatives initially tended to vote based on self-interest and compete for limited project funding. After the introduction of the PIA process for Vote 10 projects, the PRC became less meaningful with regard to project selection, since OGDs now had reduced influence in determining which projects were ultimately funded, and consequently their engagement in the PRC decreased. 

4.5.5 Program Efficiency

Finding #14: There was insufficient data to assess the overall efficiency of ACCBP and CTCBP at the program level.

The evaluation did not find sufficient financial or narrative data to fully assess the overall efficiency of ACCBP and CTCBP at the program or project levels. However interviewees identified some positive indicators and concerns related to the Programs’ efficiency.

There were some signs that ACCBP and CTCBP provided a good return on Canada’s investment. According to recipient organizations and international partners interviewed during site visits, Canadian programs compared well to similar mid-sized programs funded by other like-minded donors. Canada could not be measured against the US in quantitative terms because the latter allocated much more funding to anti-crime and counter-terrorism, however Canada compared well in terms of the quality of its programming.

Some ACCBP and CTCBP projects were able to leverage funding, or used their funding to complement larger initiatives, thus multiplying the impact of the GAC investment. For example, by providing equipment to support a $14M bilateral (development) project with the Public Prosecutor’s Office in Guatemala, an ACCBP project contributed to a bigger overall outcome than it would have achieved alone. JES upscaled a different ACCBP pilot project in Guatemala and replicated it in Honduras and El Salvador.

ACCBP and CTCBP projects received significant counterpart contributions from implementing bodies. OGDs covered the salaries of their staff who delivered training and other in-kind costs; international organizations provided trainers and other technical assistance; and recipient countries contributed salary costs of participating officials, office space, training facilities, logistical support etc. ACCBP and CTCBP should track the overall value of those counterpart contributions in order to have a better estimate of the Programs’ value-for-money.

Interviewees also raised some concerns about program efficiency. ACCBP and CTCBP did not assess the comparative or absolute efficiency of particular projects, implementing bodies or delivery mechanisms. The Programs’ largest Canadian and international partners were trusted and capable organizations, such as the RCMP, IOM, UNODC and OAS. However ACCBP and CTCBP did not have evidence to demonstrate that those organizations were necessarily implementing projects in the most efficient or cost-effective way (see §4.5.6 below). According to GAC staff, some IOs had high overhead and management costs to implement projects. A particular partner or program could be effective or efficient in one region but not another.  For example, the UNODC Container Control Program achieved good results in Latin America, but some UNODC projects in Southeast Asia got a lower return on investment.

4.5.6 Cost-effectiveness of Project Delivery

Finding #15:  Some projects could have been delivered in a more efficient, cost-effective way.

Only half (20) of the 40 projects in the evaluation sample provided evidence that activities were implemented on time and within budget. The remaining projects either did not provide information to assess delivery (12), or experienced delays or budget variances that included under-spending (see §4.5.2).

More than half (56%) of all ACCBP and CTCBP projects received funding for one year, which could be shortened by several months if approvals were delayed. Many interviewees at GAC and site visits noted that bigger, multi-year projects tended to be more efficient in terms of higher-level and more sustainable capacity building results. One-time projects that were dispersed by region and thematic focus were often less cost-effective than sustained, longer-term projects with the same partners.

However GAC staff cautioned that not all small projects were inefficient, nor were all large projects delivered in a cost-effective way. Several small ACCBP and CTCBP initiatives had a relatively big impact and good value-for-money, such as Transport Canada projects to strengthen aviation security in Mexico and Central America. On the other hand, some large OGD projects could have delivered training in a more efficient way. For example, some projects could have achieved a lower cost per trainee by using a “train-the-trainer” approach to develop local experts to multiply learning, rather than continually bringing in Canadian experts. GAC staff also cited cases where senior OGD officials incurred high travel costs to plan and observe training courses that they were not delivering, instead of utilizing much cheaper local resources for administrative/ logistical support or monitoring.

Based on financial data provided by the Capacity Building Programs, projects implemented by OGDs (Vote 1 funding) generally had higher transaction costs (IGC staff time, overhead ratios), but also higher dividends in terms of visibility for Canada, operational advantages for OGDs and quality of training delivered to recipients. Depending on the OGDs involved, an IGC manager could spend up to 80% of his/her time on projects that accounted for only 20% of the total program budget.

More than half (55%) of the Vote 10 annual budget for CTCBP and ACCBP was allocated for grants, rather than contributions. The percentage of the HSE Vote 10 budget allocated for grants ranged from 33% to 81% over the last three years.

4.5.7 Equipment Supply

Finding #16:  Delays in equipment delivery limited the effectiveness and impact of some training.

Most of the equipment supplied for ACCBP and CTCBP projects, was sourced by the Canadian Commercial Corporation (CCC). CCC implemented 25 ACCBP and CTCBP projects to purchase and deliver equipment, primarily for OGDs. According to GAC and OGD staff, CCC had a mixed record in equipment supply. Some projects reported a smooth sourcing process, while others experienced delays. For example, CCC effectively delivered equipment that DND needed for training in the Middle East. In another case, the equipment for DND training in Jamaica arrived after the course had been delivered, thus recipients did not receive practical training on how to use it.

According to OGD interviewees, equipment delivery through CCC typically took about 18 months. In one case, it took three years to deliver boats to combat drug trafficking in Malaysia. Other like-minded countries like the US and UK responded much faster than Canada to beneficiary needs. As a result, those countries sometimes “beat Canada to the mark” and delivered training that OGDs had intended to provide through ACCBP or CTCBP.

CCC faced internal (GoC) and external constraints affecting equipment supply. One of the internal constraints was that all equipment for OGD projects was funded through Vote 10, which required going through the approval process of GAC. Getting that approval was often slow and uncertain. Thus an OGD project could receive Vote 1 funding for training, without getting the corresponding Vote 10 money in a timely way to purchase equipment needed to deliver the training. That reduced the project’s effectiveness, impact and sustainability.

Another internal limitation occurred when ACCBP or CTCBP provided vague Statements of Work that lacked the requisite technical details. If the Capacity Building Programs or CCC did not have the in-house technical expertise to complete the proposal, they would have to hire an external adviser and/or go back to the recipient organizations for further information.

CCC and the Capacity Building Programs did not have staff in the field to facilitate or expedite the import and delivery of equipment. CCC did not have a mandate to monitor the use and maintenance of equipment after its delivery, thus it did not have data on whether the equipment was appropriate, well-utilized or maintained.

Among the external constraints, CCC faced problems with both Canadian and foreign suppliers. Some Canadian suppliers lacked experience with exporting and the related shipping, tax and Customs requirements that were often very complicated and lengthy in recipient countries. When equipment was purchased outside Canada, CCC had problems identifying, communicating with and preparing contracts with local suppliers. Other external risks included corruption among officials in recipient countries, and/or natural disasters such as hurricanes that delayed or threatened the safe arrival of shipped goods.

A few interviewees suggested that ACCBP/CTCBP and other security programs should compare lessons about mechanisms for sourcing and delivering equipment. CCC might not always be the best or most efficient mechanism. Other mechanisms might be better, depending on the type of equipment needed and location of recipients. For example, CBPs achieved good results by having equipment provided to beneficiaries through UNOPS.

4.5.8 Performance Measurement

Finding #17: Performance measurement remains a major challenge for ACCBP, HSE and CTCBP.

Previous evaluations of ACCBP and CTCBP^Footnote25identified the need for the Programs to improve their performance measurement systems and practices. That remained a challenge for both Programs during the evaluation period. Performance measurement was weak at the project and program levels.

Performance Measurement Challenges for Security Capacity Building Programs

Based on evidence from interviews, site visits and file reviews, there are many challenges to measuring the impact of any security-related capacity building program. ACCBP and CTCBP operate in complex and potentially volatile foreign beneficiary states where needs can change suddenly or over time. The Programs are designed to be responsive to emerging priorities, so it is often difficult to do detailed planning, determine baselines or set targets. Some projects relate to sensitive security issues, preventing partners from publicizing their activities or results. Measuring terrorist or criminal activities that did not occur as a result of/or despite GAC programming is hard, as is attribution to Canada when there are multiple factors and actors at play in any context. Monitoring project activities may be difficult for security reasons. Also beneficiary states often lack the human resources, expertise or tools needed for project management and reporting, particularly where there are high levels of corruption or violence.^Footnote26

Problems assessing the outcomes of security programming are compounded by the challenges of measuring the impact of capacity building. Most projects do not track the outcomes of short-term training delivered due to capacity (financial and human resources) constraints. Conversely, intermediate outcomes cannot be solely attributed to a particular capacity development initiative when several other factors intervene. Changes resulting from capacity building in sectors like judicial reform are often not quantifiable and can take five to ten years to occur.

Despite these challenges, some international organizations and other donors, such as the UK and US, have started developing systems and tools to measure the results of security-related capacity development.  

ACCBP and CTCBP Performance Measurement

CTCBP and ACCBP did not have a functional project database where key information could be stored systematically for performance measurement purposes (see §4.5.9). The Programs used a standardized Results-based Performance Framework (RBPF) template for project reporting. According to program staff and implementing partners, the RBPF was cumbersome, time-consuming to complete, and did not yield optimal information for performance measurement.

Program staff did not use the RBPF consistently. Of the 40 projects reviewed for this evaluation, only 15 (40%) had correctly completed Results-based Performance Framework (RBPF) templates. 21 projects had interim or final reports, but the other half did not have reports on file. Only one-third of CTCBP projects had reports available on file. Within the total sample, 28 projects contained some information on short-term “outcomes”, but many were actually outputs such as number of people trained or equipment delivered.

Overall, more than half of all ACCBP and CTCBP projects were completed in a year or less, and tended to report on short-term delivery of outputs. The tracking of immediate and intermediate results was “anecdotal” according to GAC staff. For example, ACCBP and CTCBP did not systematically monitor the changes (institutional, legislative etc.) that resulted from capacity building, or whether equipment remained operational and helped improve anti-crime or counter-terrorism operations in recipient countries. In cases where ACCBP and CTCBP contributed to larger IO programs, the programs did not use a contribution analysis to assess Canada’s part in achieving longer-term results. The ACCBP/CTCBP project reporting template did not include a section for results related to gender equality or instruct users to gather sex-disaggregated data (see Section 4.5.10).

As reported in evaluation interviews, the Capacity Building Programs were “chronically under-staffed” so managers did not have time to do Project Closing Reports. Many employees stayed for only one year, and there was turnover in implementing partner agencies as well, so it was hard for incoming staff to find the information needed to prepare closing reports. The Programs did not use the close-out process as a way to assess project results, impact and lessons. In many cases, IGC hired casual workers to clear the backlog of unfilled Closing Reports, thus missing an opportunity for staff learning and building corporate knowledge.

Given their heavy workload to prepare many new proposals and handle up to 30 active files each, project managers had little time (or travel budget) to do field monitoring during most of the evaluation period. Monitoring was done on an ad hoc basis. Project managers started doing more field missions over the past year, as a result of increased Program commitment to monitoring.

Evidence from interviews with GAC staff and implementing partners, as well as the project file review, indicated that the quality of reporting by implementing bodies (OGDs, NGOs, IOs) was inconsistent. OGDs often did not report on results in the same way as GAC, e.g. some ODG reports focused on activities implemented. Most IOs had different institutional reporting formats or requirements than GAC. In many cases, final project reports did not contain the data needed to assess project performance or complete the IGC Closing Reports.

Evaluators found a significant gap between the level and type of results found in IGC reports compared to the actual results they verified during multiple site-visit interviews. In general, ACCBP and CTCBP projects were achieving better results than those captured in written reports. The few cases where projects under-performed were not well documented either, in terms of reasons for poor delivery or lessons for future programming (what worked or did not, and why). In some cases, successes were communicated to GAC managers via emails, which were not archived as part of official project documentation.

A few factors made it difficult to roll up or aggregate project-level results to assess the impact of ACCBP and CTCBP at the program level:

• the wide range of projects (size, geographic location, thematic focus, implementing partner) funded, which shared few common expected results.
• limited and/or poor quality written reports on project results. Those results that were reported tended to be anecdotal, qualitative outcomes.
• lack of a project database caused a labour-intensive exercise for managers to comb through dozens of reports to find results that could be inputted into an annual Program report. The results selected and reported could have been the most accessible, rather than the most indicative or highest level ones.
• lack of good tools or systems to systematize and report program-level results. Reports prepared by GAC Corporate Services such as the Departmental Performance Report (DPR) or Report on Plans and Priorities (RPP) were too general (high-level) to capture the overall impact of ACCBP and CTCBP.

Over the past two years, ACCBP and CTCBP took steps to improve performance measurement systems and practices, however challenges remained, as summarized in the table below.

4.5.9 Information and Knowledge Management Systems

Finding #18:  While ACCBP and CTCBP made progress in implementing a project database, more work is needed to improve knowledge management systems.

The 2012 evaluation report highlighted ACCBP’s need for a comprehensive information management system. During most of the evaluation period, CTCBP and ACCBP did not have a database for project management. Key project documents were stored on the IGC system. Projects were named, filed and classified inconsistently, as decided by each manager. As noted above in Section 4.5.8, the lack of a database made it harder to maintain systematic and accessible records of project reporting, monitoring, lessons etc. Without easy access to key data, ACCBP and CTCBP staff had to spend a lot of time pulling together project information to respond to corporate requests and requirements, which took time away from effective project management.

In 2012, CTCBP and ACCBP started transitioning to the Grants and Contributions System (GCS) database of former CIDA, however, according to Program managers, the transition process was stopped due to the fact that the GCS system was not designed to capture the specificity of the smaller scale programs (e.g. Vote 1 programming, etc.). By the end of the evaluation period, CTCBP and ACCBP had entered data into the financial modules of GCS, however it was not yet functional as a project management database. The GCS had several limitations that did not fully meet the needs of ACCBP and CTCBP. It did not integrate with the existing Infobank system. GCS could only record funding through Vote 10, thus it excluded the 20% of funding administered through Vote 1. GCS did not have a separate system to file classified information generated by the Capacity Building Programs. And GCS included some modules like gender and environment that ACCBP and CTCBP did not need at that time. GAC was in the process of adapting GCS so that it could be used effectively as the project database for ACCBP and CTCBP.

While limited progress was made on information management, knowledge management systems and approaches were not optimal. ACCBP and CTCBP did not have mechanisms or a focus on systematizing and sharing lessons or knowledge among security programs, with other GAC divisions, or with/among OGDs. After OGDs participated in the annual Priority Review and PRC, they did not always receive follow-up information about the results or lessons generated by those projects.

ACCBP and CTCBP exchanged knowledge and information with other donors and international partners at regional policy meetings and conferences. The evaluation found that ACCBP and CTCBP had good operational coordination with other donors (see Section 4.5.13 below).

4.5.10  Gender-Related Issues

Finding #19:  ACCBP, HSE and CTCBP did not always consider gender-related issues in all aspects of project planning, implementation, monitoring and reporting.

According to many international organizations and donors with whom ACCBP and CTCBP work, anti-crime and counter-terrorism programming encompasses many important issues related to gender equality and women’s rights. For example, women are often targeted in human smuggling and trafficking operations, and are victims of organized crime. Emerging terrorist threats often have a particular impact on women. For example, international reports have documented violence by ISIS against women and girls in ethnic and religious communities in Iraq and Syria. Security and police forces have barriers limiting the recruitment and advancement of women. Women can play an instrumental role in CVE programs to stop radicalization of vulnerable youth in many cultures.

Promoting gender equality and gender-related results should be both an outcome in itself and a means to achieving more effective security capacity building. However gender analysis and issues were notably absent from ACCBP, HSE and CTCBP programming. For example, the annual Priority Review that guided project design and selection made no reference to gender-related issues or possible programming. Project approval documents did not require any gender analysis or related indicators/outcomes.

The ACCBP/CTCBP project reporting template did not include a section for results related to gender equality or provide instructions to gather sex-disaggregated data. None of the 40 projects reviewed in the evaluation sample collected even basic sex-disaggregated data, such as the number of women and men who were trained through a particular project. Of the projects assessed through file reviews or site visits, very few made any reference to gender-related issues or outcomes.

ACCBP and CTCBP projects did not do a gender analysis or seek gender-specific results. During site visits, evaluators found that some projects had achieved gender-related results but not measured or reported them well. For example, interviewees in Guatemala said that JES projects contributed to capacity building of the Public Ministry’s Sexual Crimes and Femicide units; provided specific training on crimes against women; supported preparation of the 2011 Femicide legislation; and donated videoconferencing equipment to Guatemalan courts serving female victims of crime.

Another example of unintended (and unreported) gender-related results was a CTCBP project with INTERPOL to train counter-terrorism officers in 10 countries in Asia. Normally all the trainees would be men for that type of capacity building. However the project had a strategy to actively recruit female participants, so women comprised one-third of all trainees. That relatively high rate was particularly notable given that several of the recipient countries had religious practices that restricted women’s participation.

ACCBP, HSE and CTCBP need to address the “gender blindness” in their programming. Fixing that gap is increasingly important given i) the GoC’s international commitment to incorporating a gender perspective into all its peace and security programming ^Footnote27; ii) emerging security threats include a strong element of gender-based violence; and iii) the amalgamation of security and development programming in GAC creates new opportunities to address gender-related issues. Many international organizations have resources available on the issue of gender analysis and engaging women in counter-terrorism and/or anti-crime programming.^Footnote28

4.5.11 Project Management

Finding #20:  ACCBP and CTCBP have good project management, and some OGDs have improved their management capacity over the last five years.

Evidence from interviews and file reviews showed that ACCBP and CTCBP generally managed projects well. According to OGDs and other implementing bodies, Program staff were competent and effective. The Programs used standardized operating procedures and management tools. The main negative factors during the evaluation period were that ACCBP and CTCBP were often under-staffed (due to personnel reductions, vacancies or rotation).  

Project management and implementation capacity varied among OGDs and fluctuated within each department according to managers assigned. During the initial years of the evaluation period, several OGDs had trouble disbursing programming money on time (as discussed in Section 4.5.2). Many OGD project reports focused on activities and outputs rather than outcomes. ACCBP and CTCBP worked with OGD staff to improve their financial planning and project management capacity. According to Program staff, OGD capacity to deliver projects improved significantly over the last few years.

For example, the RCMP was the largest OGD partner, implementing almost half of all Capacity Building projects (as presented in Section 4.5.12). At the start of the evaluation period, the RCMP had “nothing in place” to deliver international programming and could not provide the necessary level of reporting. Three years ago the RCMP created its International Capacity Building Division as a centralized and specialized unit to handle all projects implemented by different divisions. That division helped improve the RCMP’s project prioritization process, planning, management capacity and reporting.

Other OGDs were limited by staff turnover and cutbacks. For example, Justice Canada had to close its international projects department following staff reductions in 2010-2011. According to interviewees in several departments, the fact that ACCBP and CTCBP projects did not cover OGD salaries was a constraint on their project implementation capacity.

OGD and GAC staff reported that project management and results delivery improved when they had a field-based officer to assist in planning, monitoring, equipment delivery, logistics etc. The officer could be Canadian or local staff. For example, in West Africa ACCBP and CTCBP received support from a GAC employee who provided logistical help, as well as temporary liaison officers from the RCMP and CBSA. The RCMP also engaged with temporary placements of Canadian officers within recipient organizations for a few projects. Some interviewees suggested that IGC consider having regional program managers on the ground.

4.5.12 Program Delivery Mechanisms

Finding #21: ACCBP and CTCBP used an appropriate mix of delivery mechanisms.

ACCBP, HSE and CTCBP implemented projects through four main delivery mechanisms, as summarized in the table below.

According to interviewees in Canada and the field, Capacity Building Programs should work with a range of national and international partners who have different operational, policy and strategic capacities. Using a combination of delivery mechanisms contributes to the achievement of different types or levels of results. No single mechanism is inherently more effective or efficient than another. However the evaluators found no evidence that ACCBP and CTCBP had a system or criteria to determine which mechanism would be used to tackle a particular crime or terrorism-related issue in a given country. In some cases, that decision seemed to be based mainly upon the project approval process and funding allocation between Vote 1 and Vote 10.

Based upon approved allocations, about 80% of ACCBP and CTCBP funding was implemented by IOs and other partners (Vote 10 grants and contributions); and the remaining 20% by OGDs (Vote 1 funding). During the evaluation period, the three main IO implementing partners for the Capacity Building Programs were the IOM (35 projects, mainly through HSE); UNODC (24 projects) and the OAS (23 projects, most of which were funded by ACCBP). Given the relative importance and value of projects implemented by the UNODC for the three Capacity Building Programs in all regions, the evaluation team selected it for a summary review that is presented in Appendix [6].

For the Vote 1 funding, ten OGDs were involved in implementing Capacity Building projects with RCMP having the highest number of projects – 34 out of 89 total CTCBP projects, followed by DND with 12 projects.

RCMP was the largest OGD partner for ACCBP, implementing almost half (44%) of all projects funded by ACCBP and HSE, for a total value of more than $20 million.

4.5.13  Coordination with Other Donors

Finding #22:  Canada’s Capacity Building Programs generally coordinate well with security programs funded by other countries.

Based on interviews in Canada and the field, evaluators found evidence that ACCBP, HSE and CTCBP coordinated well with other donor-funded security programs. Coordination was particularly strong with like-minded donors such as the US, UK and France in several regions. Informal donor consultations and formal coordination groups were used to help identify Canada’s niche, avoid duplication and deconflict security programming.

Different modalities were used for donor coordination. In some cases, donors would select a lead country for each beneficiary state, or focus on a particular sub-region. For example, ACCBP decided to focus on the Caribbean partly because the US was more engaged in Mexico and Central America. In other cases, each donor contributed specific expertise to an integrated program..

ACCBP promoted donor coordination in the Americas by funding joint projects with allies such as the US and UK, and participating in donor coordination fora (e.g. Group of Friends of Central America and the Caribbean Basin Security Initiative Donors’ Group, currently chaired by Canada). Canada took the lead in coordinating security donors (US, EU, UK, France, Netherlands) in the Caribbean. Since 2008, Canada chaired the mini-Dublin Group for donor coordination on combating drug trafficking in the Caribbean.

Another example of donor coordination was HSE’s support to INTERPOL in Southeast Asia. Canada focused on capacity building on law enforcement issues, while Australia funded CBRN training and Japan covered cyber crime. The Canadian Missions in Southeast Asia helped ensure that HSE training projects fit with programming by the US and Australia, which were the biggest players in the region.

Donor coordination for CTCBP was evident in Malaysia, where the US found out that Canada provided the best CBRN-E training, so it redirected support to another area of capacity building. According to US officials interviewed, they had excellent coordination with Canada. Coordinating US training with Canada increased its effectiveness and helped them get buy-in from local authorities, because Canada had established better relations with Malaysian government institutions through its Capacity Building Programs.

Despite that coordination, there were a few cases of duplication of training provided by Canada and other donors. One example was when the RCMP repeated a training course delivered by the UK in Jamaica.

4.5.14  Lessons and Good Practices

Finding #23: CTCBP and ACCBP projects generated some useful lessons, but these have not been collected or applied systematically.

The Capacity Building Programs did not have a standard practice or system to collect lessons and apply them in future programming. That gap was partly due to the lack of a project database where lessons could be recorded, as discussed in Section 4.5.9. ACCBP and CTCBP did not have formal mechanisms to share lessons with other GAC security or development programs, or with/among OGDs. Information experience were shared with other donors or partners on an ad hoc basis through international conferences, regional policy group meetings or meetings at GAC

The evaluation found some evidence of lessons collected through project reports. In the evaluation sample of 40 projects, ACCBP and CTCBP did not report much information on lessons in their project templates. The exception was HSE, where all 13 of the projects in the sample provided a discussion of lessons. Based on that data, and other information gathered from interviewees at GAC and recipient organizations, evaluators compiled some key lessons and good practices related to security capacity building:

• Continuity, trust and ongoing communication among partners are key ingredients of security capacity building. It is important to maintain strong relations with partners, GAC programs and Missions, OGDs, recipient organizations and international partners.
• Effective and ongoing cooperation among all parties, including government officials and police departments (implementing agencies) of recipient states, is needed for any project related to human smuggling.
• A recipient institution should have a comprehensive capacity building strategy to ensure that training is the appropriate level/type, properly sequenced, and well-coordinated among various donors.
• Recipient partners may have limited capacity to absorb and apply training, so capacity building projects need to have realistic expectations and be flexible enough to adjust activities to suit local conditions.
• Training should be adapted and targeted to meet the specific needs of each recipient. OGDs and IOs should avoid using a “cookie cutter” approach or delivering the same “off the shelf” course to all recipients regardless of national context or priorities.
• The most effective capacity building provides practical training, builds knowledge incrementally, and has a longer duration. One-time, short-term training is not as sustainable and has less impact than longer-term, continual training, technical assistance and coaching.
• An effective approach to capacity building usually combines practical training, coaching and equipment. Many beneficiaries reported that practical, hands-on training in the field is more effective than theoretical courses in the classroom.
• Now that many recipient countries have received basic (“101”) security training, they need more advanced capacity development and long-term mentorship.
• A “train the trainer” approach usually yields more sustainable capacity building results and beneficiary engagement. This approach empowers trainers who can train others in the local language and incorporate the local context. That expertise remains in the recipient country after the Canadian OGD expert leaves.
• Local language skills and cultural sensitivity are important to be able to communicate effectively with training participants. Training materials should be available in local languages. For example, partners in Jordan noted that native Arabic speakers and experts should help deliver the training, and materials should be translated into Arabic.
• Trilateral or “south-south” cooperation is an effective way for national experts to replicate knowledge across the region. For example, through ACCBP Colombian police officers were trained to combat transnational organized crime. ACCBP then funded the Colombians to train almost 1,400 officers in Guatemala and Honduras. That project helped strengthen relations between the three countries on transnational crime and other security-related issues.
• Having enough advance preparation and planning time helps ensure training delivery. It often takes more time than expected to arrange logistics, get equipment into the recipient country etc.
• It is important to have clear criteria and processes in place to ensure the selection of appropriate participants for training.
• Projects based on participatory needs assessments with local partners tend to have the best outcomes and highest level of beneficiary satisfaction. Projects that are driven by GAC or an implementing partner are much less effective and sustainable.
• Having a field-based officer generally improves project management and results achievement. The field officer can provide ACCBP and CTCBP support in planning, monitoring, equipment supply, and logistics. The officer can be either a Canadian from GAC or an OGD partner, or locally engaged staff.

5.0 Conclusions of the Evaluation

The evaluation found that for the 2011-2015 period, there was a continuing need for the Capacity Building Programs: CTCBP to combat significant ongoing threats from international terrorist networks, particularly in the Middle East, North Africa and Asia; ACCBP to address organized crime and drug trafficking with a focus on the Americas; and HSE to prevent human smuggling operations in Southeast Asia and West Africa. The Capacity Building Programs aligned with GoC policies and priorities. There was good consultation among the Capacity Building Programs; however coordination could be improved with other security and development programs at GAC.

There was evidence that ACCBP, HSE and CTCBP made progress toward expected results at the immediate level, but there was little data available to assess achievement of intermediate outcomes. Recipients interviewed during site visits expressed a high level of satisfaction with GAC capacity building projects. Longer-term capacity building projects tended to be more sustainable than one-time training. CTCBP, ACCBP and HSE increased Canadian expertise, leadership and visibility in international security programming.

In assessing efficiency and economy, the evaluation found that programming focused on fewer countries and projects was more effective than spreading funding too thinly. ACCBP and CTCBP spent the Vote 1 portion of their allocated budgets, but were unable to disburse the full Vote 10 allocation during the last few years of the evaluation period. Difficulties in the Departmental project approval process reduced the effectiveness and efficiency of the Capacity Building Programs and their governance committees. The Capacity Building Programs did not track their overall efficiency; however some projects could have been delivered in a more cost-effective way. Delays in the delivery of equipment limited the effectiveness and impact of some training in the field.

Overall project management by GAC staff was good, however performance measurement systems and practices were not adequate. The Programs did not have a project database, and overall information management could be improved. ACCBP, CTCBP and HSE did not integrate a gender perspective in programming. The Capacity Building Programs used an appropriate mix of delivery channels by working through OGDs, multilateral agencies, trilateral mechanisms and partnerships with Canada’s allies. ACCBP, CTCBP and HSE generally coordinated their security programming well with other donor countries through bilateral consultation and regional organizations.

6.0 Recommendations

The evaluation generated five key recommendations, as presented below. Suggested steps to implement each recommendation are outlined in Appendix 2.  

It is recommended that ACCBP and CTCBP: 

Recommendation #1:  Focus on long-term capacity building by continuing to support multi-year projects (or a series of shorter-term projects with mutually reinforcing outcomes) within target regions/themes to ensure sustainability of results, while earmarking some funding for quick response to emerging needs.

ACCBP and CTCBP should continue to increase funding for longer-term capacity building projects, while earmarking a small portion to respond quickly to emerging needs. The Programs should use their limited budgets strategically by concentrating on fewer thematic priorities and regions within each fiscal year depending on the type and sources of threat (e.g. Countering Violent Extremism has become a priority last year).

Recommendation #2: Assess efficiency at the program and project levels more systematically.

ACCBP and CTCBP should assess the relative efficiency and cost-effectiveness of different delivery channels, implementing bodies, funding mechanisms and types of capacity building. The Programs should measure the efficiency and economy of CCC vs. other international mechanisms to source equipment. Tracking counterpart contributions from OGDs and international partners would help estimate the Programs’ value-for-money.

Recommendation #3: Increase coordination and synergy with other security and development programming at GAC, and with OGDs.

GAC should establish mechanisms to improve coordination between ACCBP, CTCBP, other security programs, and development cooperation at HQ and in the field. Having a “whole of department” approach would be a first step towards developing a “whole of government” strategy for Canadian support to security and development in each country and/or region.  

Recommendation #4: Continue improving performance measurement systems and practices at the program and project levels.

ACCBP and CTCBP should continue to refine their tools for monitoring and reporting longer-term (intermediate) results at the project level, and aggregating those results to assess program-level impact. The Programs need to develop a comprehensive information system that includes an integrated project database, and better knowledge sharing among GAC programs and OGDs.

Recommendation #5:  Integrate a gender perspective into program planning, monitoring and reporting.

CTCBP and ACCBP should start incorporating a gender perspective throughout the project cycle. Projects should define expected gender-related results and indicators, and collect sex-disaggregated data. The Programs could draw upon other GAC Divisions and OGDs that have relevant gender expertise and tools.

7.0 Management Response and Action Plan

Recommendation 1

Focus on long-term capacity building by continuing to support multi-year projects (or a series of shorter-term projects with mutually reinforcing outcomes) within target regions/themes to ensure sustainability of results, while earmarking some funding for quick response to emerging needs.

ACCBP and CTCBP should continue to increase funding to longer-term capacity building projects, while earmarking a small portion to respond quickly to emerging needs. The Programs should use their limited budgets strategically by concentrating on fewer thematic priorities and regions within each fiscal year depending on the type and sources of threat (e.g. Countering Violent Extremism has become a priority last year).

Recommendation 2

Assess efficiency at the program and project levels more systematically.

ACCBP and CTCBP should assess the relative efficiency and cost-effectiveness of different delivery channels, implementing bodies, funding mechanisms and types of capacity building. The Programs should measure the efficiency and economy of CCC vs. other international mechanisms to source equipment. Tracking counterpart contributions from OGDs and international partners would help estimate the Programs’ value-for-money.

Recommendation 3

Increase coordination and synergy with other security and development programming at GAC, and with OGDs.

GAC should establish mechanisms to improve coordination between ACCBP, CTCBP, other security programs, and development cooperation at HQ and in the field. Having a “whole of department” approach would be a first step towards developing a “whole of government” strategy for Canadian support to security and development in each country and/or region.

Recommendation 4

Continue improving performance measurement systems and practices at the program and project levels.

ACCBP and CTCBP should continue to refine their tools for monitoring and reporting longer-term (intermediate) results at the project level, and aggregating those results to assess program-level impact. The Programs need to develop a comprehensive information system that includes an integrated project database, and better knowledge sharing among GAC programs and OGDs.

Recommendation 5

Integrate a gender perspective into program planning, monitoring and reporting.

CTCBP and ACCBP should start incorporating a gender perspective throughout the project cycle. Projects should define expected gender-related results and indicators, and collect sex-disaggregated data. The Programs could draw upon other GAC Divisions and OGDs that have relevant gender expertise and tools.

Appendix 1: List of Findings

Appendix 2: Suggested Steps to Implement Recommendations

As presented in §6.0, the evaluation generated five key recommendations. Suggested steps to implement each recommendation are outlined below.

It is recommended that ACCBP and CTCBP:

Recommendation #1:  Focus on long-term capacity building by continuing to support multi-year projects (or a series of shorter-term projects with mutually reinforcing outcomes) within target regions/themes to ensure sustainability of results, while earmarking some funding for quick response to emerging needs.

ACCBP and CTCBP should continue to increase funding for longer-term capacity building projects, while earmarking a small portion to respond quickly to emerging needs. The Programs should continue to use their limited budgets strategically by concentrating on thematic priorities and regions within each fiscal year depending on the type and sources of threat (e.g. Foreign Terrorist Fighters become a priority last year).

• Continue to increase funding to longer-term capacity building, such as projects implemented over two years or more. Longer-term engagement is particularly important in the area of capacity development.
• Maintain some funding for responsive and rapid-response programming. For example, 25% of CTCBP and ACCBP funding could be notionally allocated for responsive programming, 5% for rapid response to emerging needs, and 70% for longer-term capacity building (as mentioned above).
• Further prioritize the Programs’ thematic areas and target countries or regions benefiting from “long-term” investments. Avoid spreading program resources too thinly to achieve significant or lasting results.
• Identify the strategic niche of ACCBP and CTCBP in relation to complementary GAC security and development programming, and to other international capacity building programs.
• Ensure a programmatic (program-based) approach to project design and selection, rather than ad hoc funding of separate projects. The overall impact of each Capacity Building Program should be greater than the sum of its individual projects.

Recommendation #2: Assess efficiency at the program and project levels more systematically.

• Assess the actual/comparative efficiency (value-for-money) and effectiveness (ability to achieve results) of different delivery channels, implementing bodies, and types of interventions.
• Review the efficiency of the Programs’ largest implementing partners (OGDs and IOs). Include other GAC funding (e.g. other security or development funding) to each partner to see Canada’s overall contribution and return on investment. That assessment could be used to help improve the efficiency and performance of individual implementing bodies, projects, and the overall Capacity Building Programs.
• Consider establishing MOUs with key implementing partners such as the RCMP in order to expedite project selection and implementation, and reduce transaction costs.
• Develop benchmark data to measure the cost-effectiveness of training delivery methods used by OGDs and IOs in GAC’s target countries.
• Track the overall value of counterpart contributions from OGDs and international partners to better estimate of the Programs’ value-for-money.
• Assess the most cost-effective and efficient way to source equipment needed for training. In some cases, international mechanisms such as UNOPS could be more economical than CCC.

Recommendation #3: Increase coordination and synergy with other security and development programming at GAC, and with missions abroad.

To improve coordination and synergy, ACCBP and CTCBP could take steps such as to:

• Map out all GAC security programs to clarify the geographic and thematic scope of each, as well as possible overlap or gaps. Because ACCBP, CTCBP and other programs were designed and developed separately, they do not necessarily form the most coherent package possible.
• Clearly define the concepts of “security” and “development” as used across GAC. Having a common understanding of departmental objectives in security and development is an essential step towards improved coordination.
• Define the responsibilities, niche and comparative advantages of development and security programming within GAC. That analysis could be enriched by looking at experiences of other agencies (e.g. UK) that do not divide development and security programming as much as Canada. Security and development programming are considered to be part of a continuum, and highly inter-related (i.e. poverty and marginalization cause security problems and vice versa).
• Establish mechanisms for coordination within GAC to enable ACCBP and CTCBP to work with other security and development programs throughout the project cycle (planning, implementation, monitoring, reporting). Such mechanisms would help ensure that security and development initiatives are complementary.
• Have joint monitoring missions between ACCBP/CTCBP and GAC-DEV so that CBP project managers learn about development programming and vice versa.
• Engage OGDs in developing a “whole-of-government” strategy for each region. For example, GAC and OGDs could jointly develop Canada’s regional security strategy for the Caribbean. Having a joint strategy would create synergy, optimize resource use, have more impact in the field, and bring greater benefits for Canada. Creating a database to capture all security information could support coordination among government departments.
• Make better use of GAC-DEV resources in the field, particularly in countries of focus that have significant resources on the ground (e.g. PSUs). Given the limited staff time and travel budget available for ACCBP and CTCBP, the Programs could arrange to use local development resources (on a cost-recovery basis) to do field monitoring or logistics to prepare training delivered through its projects.
• Provide training for both security and development staff (project managers, program directors) at GAC to acquaint them with the basics of the “other” side of programming.
• Gather and systematize lessons about security capacity building that are generated through ACCBP and CTCBP programming. Compare GAC lessons with those of other donors and international organizations.
• Promote knowledge exchange and learning through regular meetings or roundtables to share lessons and experiences about development and security projects in a particular country or region. That knowledge sharing would be facilitated through integrated information management systems that GAC is developing.
• Standardize operating procedures between ACCBP, CTCBP and other security programs; and between those programs and GAC-DEV.

Recommendation #4: Continue improving performance measurement systems and practices at the program and project levels.

To improve performance measurement, ACCBP and CTCBP could take steps such as to:

• Continue developing a comprehensive information management system, including an integrated project database.
• Incorporate a program-level analysis of results and lessons learned as part of the regular reporting process.
• Develop efficient program-level performance measurement tools thatare easy for staff and partners to use. The program-level logic model and PMF should include intermediate results and indicators (quantitative and qualitative), which could start with those contained in the current PMS. ACCBP and CTCBP could draw upon technical advisers from GAC-DEV who have experience in developing and applying RBM frameworks to diverse types of programming.
• Refine and adapt project-level monitoring and reporting tools. The resources (staff time, budget) for project-level monitoring and reporting should be commensurate with the timeframe, budget and security context of the capacity building initiative. A one-time, low-budget training course or training in a highly volatile context could have a “light” reporting template limited to immediate outcomes, while a large multiyear program involving multiple actors could have more complex monitoring/reporting requirements to capture intermediate outcomes (or impact where possible).
• Use a contribution analysis for complex interventions involving multiple actors and factors. That contribution analysis could include information from annual reports of organizations such as UNODC where Canada contributes a significant percentage of overall funding, and peer-review evaluations such as that of FATF.
• Work with OGD and IO partners to develop project-level logic models and performance measurement frameworks that include (an) outcome(s) that can be rolled up to the program-level framework. In many cases, GAC could draw upon expertise that OGD or IO partners have in developing performance frameworks (measurable indicators, realistic results) for their particular sectors of intervention.
• Set higher-level performance goals for projects, and communicate those goals to implementing partners and recipients. A few OGD interviewees noted that it would be useful to have clear goals and indicators to measure outcome-level results.
• Conduct periodic targeted reviews as one way to aggregate project results at a program level. A targeted review provides a snapshot of longer-term outcomes achieved related to a specific issue, region or implementing partner involved in Capacity Building Programs.^Footnote29For example, a review of all ACCBP and other GAC-funded initiatives to combat drug trafficking in the Caribbean. Or a review of the overall impact of projects implemented by an OGD or IO over a five-year period.
• Draw upon international and Canadian expertise and resources to strengthen performance measurement of security capacity building programs. Many international organizations and like-minded donors have developed PM tools that GAC could adapt.^Footnote30
• Provide RBM training to GAC staff and OGD partners to promote better quality data collection and reporting.
• Collect and report intermediate-level results achieved by ACCBP, HSE and CTCBP within IGC, at the corporate level, and for the Canadian public.
• Engage GAC staff from the field and other security programs inproject monitoring.Canadian and local development staff in the field could provide monitoring support to ACCBP and CTCBP. IGC could also explore the option of having staff from GPSF or GPP include ACCBP or CTCBP projects in their scheduled monitoring trips to rationalize resource use and promote knowledge sharing.

Recommendation #5:  Integrate a gender perspective into program planning, monitoring and reporting.

To integrate a gender perspective, ACCBP and CTCBP could take steps such as to:

• Review lessons, good practices and tools developed by international organizations and like-minded donors to integrate gender issues into security capacity building programs.
• Review lessons, good practices and tools developed by other GAC programs, particularly GPSF and GAC-DEV. GPSF has a thematic focus and specific programming on sexual and gender-based violence. GAC-DEV has resources for integrating a gender perspective in sectoral and thematic programs in humanitarian assistance and development. The Peace Operations and Fragile States Division (IRP) can provide technical assistance and guidance on issues related to women and security.
• Consult the Women, Peace and Security (WPS) working group, which includes experiences from the former DFAIT and CIDA, as well as other Canadian government departments. The working group could help advise ACCBP and CTCBP about integrating a gender perspective into security capacity building.
• Review ongoing programming by GAC and OGDs to identify a possible niche or gaps that ACCBP and CTCBP could fill. That programming is summarized in the GoC progress reports on Canada’s National Action Plan on women, peace and security.^Footnote31
• Consider funding gender-specific projects, and integrating gender considerations in ACCBP and CTCBP projects where possible. Seek gender-related results both as outcomes in themselves and a means to more effective security capacity building programming by ACCBP and CTCBP.
• Include gender-related results statements and outcome indicators in project design, implementation, monitoring and reporting. Start collecting and reporting on sex-disaggregated data.
• Consult GAC gender specialists in HQ and the field when designing anti-crime and counter-terrorist projects (when appropriate and feasible).
• Provide training for ACCBP and CTCBP staff, as well as for implementing partners, to build capacity and awareness of possible opportunities to integrate gender-based analysis in their security programming. The training could start with the gender courses offered by GAC-DEV, and specialized training in SGBV and the rights of women and girls in conflict areas provided by OGDs such as DND’s CANSOFCOM or the RCMP.

Appendix 3: CTCBP and ACCBP Budget Estimates

CTCBP Budget Estimates

ACCBP Budget Estimates

Appendix 4: Evaluation Matrix