Common menu bar

Breadcrumb

Formative Evaluation of the Counter-Terrorism Capacity Building Program

May 2009

(PDF Version, 828 KB) ^*

Executive Summary
Acronyms
1.0 Introduction
2.0 Description of the Counter-Terrorism Capacity Building Program
3.0 CTCBP Relevance
- 3.1 Relevance to the Government of Canada
- 3.2 Relevance to International Partners
4.0 Governance, Design and Delivery
5.0 Program Success and Cost-Effectiveness
6.0 Conclusions and Recommendations
7.0 Management Response and Action Plan

Executive Summary

Introduction

This report presents the findings of the Formative Evaluation of the Counter-Terrorism Capacity Building Program (CTCBP). The purpose of the evaluation, conducted from July 2008 to March 2009 by Goss Gilroy Inc., was to provide the Evaluation Committee of the Department of Foreign Affairs and International Trade (DFAIT) with a neutral and an evidence-based assessment of the relevance, effectiveness and cost-effectiveness of the CTCBP to findings and recommendations for improving the Program's value for money.

The CTCBP's objectives are to:

Enable other countries and governments to prevent and respond to terrorist activities in a manner consistent with international counter-terrorism and human rights obligations, norms and standards (primary objective);
Increase Canada's profile in CTCB assistance and demonstrate Canadian leadership in specialized areas of expertise (secondary objective); and
Improve domestic and international coordination and cooperation for CTCB assistance (tertiary objective).

The evaluation used multiple lines of evidence:

A document, file and administrative data review;
Key informant interviews with 38 individuals, including staff at DFAIT, Other Government Departments (OGDs) and international external stakeholders;
Case studies of five funded organizations that received funding for fourteen CTCBP projects and
A country comparative analysis to provide information from two countries (United States and Switzerland)¹ on the country's programming in the area of counter-terrorism capacity building, the relevance of this programming and the perceptions of Canada's contributions to counter-terrorism capacity building.

There were three challenges with this evaluation methodology:

Administrative data available from the CTCBP Secretariat was incomplete and inconsistent. The poor quality of the data is attributed to the challenges the Secretariat faced in setting up the Program in the first year and, at the same time, meeting expectations for project funding.
The country comparative analysis did not provide as much information as expected because respondents were generally unaware of Canada's contribution to counter-terrorism capacity building and, specifically, the CTCBP.
There were challenges, common to all capacity building programs, with the measurement of results. It was difficult to measure results beyond the level of the outputs or perhaps short-term outcomes and, even if measurement is possible, difficult to attribute any changes measured to the program's activities. The case studies were also unable to identify results, beyond anecdotal information from case study respondents.

In spite of these limitations, the evaluation team was able to collect and analyze good qualitative information on the evaluation issues and draw conclusions primarily about the Program's design and implementation and progress towards achieving results.

Program Overview

The Program is governed by an Interdepartmental Steering Committee (ISC), which includes representatives of the member departments and agencies. Day-to-day administrative and financial matters, policies and procedures are the responsibility of the CTCBP Secretariat, located in DFAIT's International Crime and Terrorism (ICT) Division, and reporting to the Senior Coordinator (ICX). There is an Online Screening Committee (OSC), made up of working level representatives of the member departments and agencies and representatives of missions abroad, which reviews applications for Program funding electronically.

The CTCBP initial funding was $15 million a year (except the first year, which was only $12 million), including funds for the management of the Program Secretariat. Funds were lapsed in 2006/07 and 2007/08 and this contributed to a decision, taken as part of DFAIT's Strategic Review, to reduce permanently the CTCBP funds to $12 million/year. The Secretariat initially included seven indeterminate non-rotational Full-time Equivalents (FTEs), but the Secretariat was not fully staffed until 20008/09 and then, one Senior Project Manager was cut in the same fiscal year.

One-third (34%) of the projects presented to the Secretariat are withdrawn, either before or at the ISC. Of the remaining projects, almost all are approved by the ISC. Of the approved projects:

Federal departments and agencies are the primary project implementing partners;
The department sponsoring the most projects is DFAIT, followed by the Royal Canadian Mounted Police (RCMP) and Transport Canada (TC);
Sponsoring departments with the highest dollar value of projects are DFAIT, RCMP and TC;
The range of project values is quite large - from $5,000 to $1,744,000;
The investment area with both the highest number of projects and the highest dollar value of projects is law enforcement, security, military and intelligence training (LEMI).

Key Findings

Relevance

Considering the importance given by the Government of Canada to responding to threats of terrorism, the Program and its specific objectives continue to be relevant. The need to contribute to the security of Canadians and inhabitants of beneficiary states and governments is still present. Canada and other countries continue to be affected by terrorist activities and key stakeholders reflected that capacity building with recipient states and governments continues to be an important mechanism to address those threats. Canada's active involvement in multilateral and bilateral relations with other partners to meet international counter-terrorism commitments and the objective of increasing Canada's profile in counter-terrorism capacity building remain relevant. This is particularly the case as Canada assumes greater leadership in some international fora. Given the involvement of many federal department/agencies in counter-terrorism capacity building activities and the engagement of the international community is this area, activities to improve both domestic and international coordination remain valid.

Governance design and delivery

Both the strengths and challenges of the CTCBP governance lie in its interdepartmental nature. As an interdepartmental initiative, it draws on the strengths and experiences of sixteen federal government departments and agencies, which serve as sponsoring organizations or implementing partners. The governance structure is based on the assumption that all departments/agencies will participate equally in the Program governance. However, the diversity in the mandates, experiences and resources of the participating departments and agencies means that it is challenging to have collective, informed decision-making for CTCBP projects. It is difficult to engage the departments/agencies in the identification of strategic priorities and to mesh DFAIT's priorities with those of other departments/agencies that are actively engaged in counter-terrorism activities. As a result, it appears that the Secretariat and, in a wider context, DFAIT play a very prominent role in the identification of strategic priorities and the approval of projects.

Beyond the governance, the overall Program design is still considered to be relevant. Stakeholders indicate that it is still relevant for the CTCBP to be an interdepartmental program, with resources held in a central pot and the Program administered by a central body. Stakeholders also support the approach of having the CTCBP as a responsive program, with parameters defining what can be funded.

There was a range of views on the effectiveness of the ISC, but concerns were expressed about the level of informed debate at the meetings and the decision-making processes. In spite of ongoing challenges with the capacity of the Secretariat, it was recognized by stakeholders for the key role that it has played in addressing these issues.

There is not a lot of competition for CTCBP funding, as the vast majority of projects submitted to the ISC are approved. The number of eligible projects is primarily limited by the capacity of the partner organizations to develop and implement projects. Most felt that the current level of funding for CTCBP was sufficient, given the capacity of the partner organizations.

The challenges with Program implementation stem primarily from two causes: the delays in setting up the Secretariat and the lack of project management capacity in some implementing organizations (notably OGDs). Set-up of the Secretariat began in 2005/06, but the full staffing has only recently been completed. As a result, a number of the components of the Program infrastructure were slow to be set up or are not yet in place - notably the development of standard operating procedures to define what can be included in projects and a database for tracking CTCBP projects. In addition, there are gaps in the implementation of the monitoring and evaluating framework. While there have been gaps in the infrastructure for Program implementation, the Secretariat is recognized by stakeholders for the key role that it has played in supporting implementation, by supporting the development of projects and the ISC.

The lack of project management capacity in OGDs, combined with the challenges of working with recipient governments and country that lack capacity themselves, means that program funds have lapsed and this contributed to a $2 million annual cut to the CTCBP in 2008/09.

Program success and cost-effectiveness

In a formative evaluation it is too early to assess the achievement of the Program's objectives. However, the key informant interviews, and particularly the case studies, provide adequate examples of projects that are contributing - at least through their outputs and, to some extent, their immediate outcomes - to the CTCBP objectives.

All but two projects reviewed appear to have contributed to the primary objective of supporting other countries and governments to prevent and respond to terrorist activities. They have done this primarily through supporting training in targeted countries - notably in the Americas. However, some have also contributed to activities to support other components of capacity building, including the development of policies, plans, procedures etc; the purchase of appropriate counter-terrorism equipment, technologies, and networks; and increasing awareness in beneficiary state or government entity of counter-terrorism issues. Only one project included specific activities to support these other components of capacity building. A primary focus on training as the mechanism for capacity building put the sustainability of the capacity building results at some jeopardy.

Almost all projects reviewed contributed, at least in the short-term, to increasing Canada's profile and demonstrating Canadian leadership. To some extent, this occurs as a direct result of funding and implementing projects with partner organizations and can occur at both the national and international level. However, key informants also noted that the CTCBP funding supports Canada's place in international fora dealing with counter-terrorism issues - for example, the CTAG and the CTC.

The Program contributes also to improving coordination and cooperation through funding multilateral organizations that have a coordination mandate, implementing regional activities that promote coordination at the regional level, building on networks with recipient countries and regions, and promoting domestic cooperation through the Program's governance mechanisms.

The Program principles emphasize the importance of sustainability. While it is too early to assess the sustainability of the CTCBP project results, there are factors that will likely contribute to, and detract from sustainability. The case studies suggest that many projects tend to focus primarily on the training of personnel, with limited activities to support directly other short-term outcomes. On the other hand, the case studies also reflected other factors that would contribute to sustainability, including synergies among projects, support of regional or global activities and support for ongoing capacity building activities being implemented by other organizations.

Recommendations

Recommendation 1:

The ISC should strengthen the CTCBP governance to make it more effective providing broader oversight of decision-making and ensure that it maximizes the value added of the OGDs with respect to their role in CTCB decision-making.

Recommendation 2:

The ISC should build on other governance mechanisms (e.g. OSC and the Secretariat) in order to maximize added value that OGDs and DFAIT missions abroad in their role as technical experts, and in project delivery.

Recommendation 3:

The ISC should better integrate Government and OGD priorities and intelligence on terrorism threats.

Recommendation 4:

DFAIT develop standard project management tools and training/coaching to support OGDs in project implementation, taking into consideration the availability of resources in the Secretariat.

Recommendation 5:

The ISC explore the possibility of providing multi-year project funding where unspent funds could be rolled over to the next fiscal year.

Recommendation 6:

DFAIT identify the necessary steps and resource requirements to finalize and populate the database for CTCBP projects that will allow the Secretariat to develop adequate profiles and track the results of CTCBP project.

Recommendation 7:

DFAIT strengthen Secretariat's capacity to monitor both the implementation (activities and outputs) and outcomes (results) of the projects and the program as a whole and develop a framework for sharing project outcome information.

Recommendation 8:

The DFAIT engage in a senior level discussion with OGD management and the ISC on the strategic focus of the program, the need (if any) to redress the balance of projects across different CTCBP short-term outcomes and investment areas and the support needed for OGD project implementation.

Haut de la page

Acronyms

ATA: Anti-Terrorism Assistance Program
CBRNE: Chemical/biological/radiological/nuclear and explosives
CFIA: Canadian Food Inspection Agency
CICTE: Inter-American Committee against Terrorism
CSIS: Canadian Security and Intelligence Service
CTCBP: Counter-terrorism Capacity Building Program
CTAG: Counter-Terrorism Action Group
CT: Counter-terrorism
CTC: Counter-Terrorism Committee
CTED: Counter-Terrorism Executive Directorate
CWC: Cricket World Cup
DFAIT: Department of Foreign Affairs and International Trade
DG: Director General
GEASSA: Group of Experts on Aviation Security and Assistance
LEMI: Law enforcement, security and military intelligence
ICAO: International Civil Aviation Organization
ICT: International Crime and Terrorism Division
ICX: Senior Coordinator, International Crime and Terrorism
IMO: International Maritime Organization
ISC: Interdepartmental Steering Committee
MOU: Memorandum of Understanding
OAS: Organization of American States
OGD: Other government departments
OSC: Online Screening Committee
PAD: Project Approval Document
RCMP: Royal Canadian Mounted Police
ToRs: Terms of Reference
UN: United Nations
UNODC: United Nations Office on Drugs and Crime
US: United States (of America)
USAP: Universal Security Audit Program

Haut de la page

1.0 Introduction

This report presents the findings of the Formative Evaluation of the Counter-Terrorism Capacity Building Program (CTCBP). The purpose of the evaluation, conducted from July 2008 to March 2009, was to provide the Evaluation Committee of the Department of Foreign Affairs and International Trade (DFAIT) with a neutral and an evidence-based assessment of the relevance and performance of the CTCBP, early in its program life. The evaluation was also designed to provide findings and recommendations for improving the Program's value for money.

As a formative evaluation, the focus was on assessing the Program's relevance, governance, design and delivery, as well as to provide a preliminary assessment of results and cost-effectiveness. The specific objectives of the evaluation were to:

Evaluate the relevance the CTCB Program by assessing the extent to which it continues to be aligned with the priorities and needs of Government of Canada, international community, implementing bodies, beneficiary states/government entities and regions;
Assess the extent to which the CTCB is achieving its immediate and intermediates outcomes; and
Assess the cost-effectiveness of the Program.

1.1 Evaluation Issues

The evaluation issues and questions were identified in the evaluation Terms of Reference (ToRs) (see Box 1.1). Given the timing of the evaluation in the third year of the program, the measurement of impacts and success focused on the short-term outcomes.

1.2 Evaluation Methodology

The evaluation was designed to collect information on each evaluation issue using multiple lines of evidence: a document, file and administrative data review; key informant interviews; case studies; and a country comparative analysis.

1.1: CTCBP Formative Evaluation Issues

Relevance

1. Do the objectives of CTCB continue to be relevant to the needs and priorities of Government of Canada, international community, implementing bodies, beneficiary states/government entities? Is further public financial support of CTCB appropriate?

Governance Structure, Program Design and Delivery

2. Is the governance structure of the CTCB Program effective in achieving CTCB Program's objectives and does in comply with the Program principles and values in the National Security Policy?

3. What are the strengths and weaknesses of CTCB Program's design, and what changes are needed, if any?

4. Is CTCB Program being implemented as intended? If not, why?

5. Does the program have an effective communications strategy? Is the CTCB Program effectively promoted?

6. Does the Program have a performance monitoring system to track progress and achievements?

7. Does the Program have a Strategic Policy and Priority Setting direction?

8. Does the project screening adhere to the Program principles and values articulated in the National Security Policy, in particular the respect for the rule of law and for human rights?

9. Are resources being targeted at priorities (geographic and investment priority areas)? Has the Program clearly articulated priorities and, if so, how are they identified?

Success

10. To what extent has the Program achieved, or is on track to achieve its short-term and medium-term outcomes?

11. Have domestic and international coordination and cooperation improved?

12. Does Canada have an increased profile domestically and internationally in counter-terrorism capacity building assistance?

Cost-effectiveness

13. Do the Program and projects result in value for money? How does the CTCB Program compare to similar national security programs in terms of level of resources, activities and overall programming efficiencies?

14.Can the required results be achieved at a lower cost?

1.2.1 Evaluation Lines of Evidence

Document, file and administrative data review

The document review consisted of a review of selected government and CTCBP documents, including reports and presentations on the Program. In addition to the general program documents, individual project files were reviewed for the case studies. The team also analyzed existing administrative data to prepare a profile of CTCBP projects. However, as will be noted in Section 1.3.2, there were limitations on the availability and quality of this data.

Key Informant Interviews

Thirty-three key informant interviews were conducted with 38 individuals (including some group interviews). The interviews included:

DFAIT staff (7);
Other Government Departments (OGDs) (24 individuals in fifteen departments/ agencies); and
International external stakeholders (3).

Case Studies

In order to provide more in-depth information on CTCBP-funded projects, their activities and outcomes, case studies were undertaken of five funded organizations that received funding for fourteen CTCBP projects.² The selection of the case studies was based on suggestions from the CTCB Program staff on studies that would provide the maximum learning for this evaluation. They were selected according to the following criteria:

Types of implementing organizations - government, non-governmental organizations (NGOs) and private sector organizations;
Dollar values, including some with single year and some with multi-year funding;
Time frames, including both single and multi-year initiatives; and
CTCBP's five investment areas.

The case studies involved a review of project documents and interviews with stakeholders, including a mix of representatives of the sponsoring and/or implementing bodies, the beneficiaries and the Secretariat.

Country Comparative Analysis

Three countries were identified for the comparative analysis, based on advice from the CTCBP Secretariat: Switzerland, the United States and Australia. Unfortunately, Australia was unavailable for interviews with the evaluation team. The expectation of the case studies was that they would provide information on the selected country's programming in the area of counter-terrorism capacity building; views of respondents on the relevance of this type of programming and the priorities for capacity building; and the perceptions of respondents on Canada's contributions to counter-terrorism capacity building . However, not all this information was available from the country comparative analysis - see a discussion of the limitations in Section 1.3.2.

1.2.2 Challenges and Limitations

In this evaluation, there were three challenges: the lack of administrative data, the lack of information from the country comparative analysis and the challenge of the measurement of results. Each is described in greater detail below. In spite of these limitations, the evaluation team was able to collect and analyze good qualitative information on the evaluation issues and draw conclusions primarily about the Program's design and implementation and progress towards achieving results.

Incomplete Administrative Data

Administrative data available from the CTCBP Secretariat was incomplete and inconsistent. Although hard copy project files exist in the Secretariat, there was no summary information that would allow the evaluation team to put together an overall profile of funded CTCBP projects or those that had not been approved. For example, in the project lists available to the evaluation team, the year of approval of a project could be derived from the project number (e.g. 2005-36) but these dates do not always correspond to the initial year of the disbursement of the project funds, and not sufficiently cross-referenced with project numbers on subsequent phases of multi-year project activities. Similarly, the information on the project values, by year, drawn from the same list of projects, was not consistent with data from the Program's financial tables.

As will be seen in the evaluation findings, the poor quality of the data is attributed to the challenges the Secretariat faced in setting up the Program in the first year and, at the same time, meeting expectations for project funding. However, it does seriously limit the extent to which this evaluation can comment on the portfolio of projects funded by CTCBP and the extent to which they contribute to meeting the Program's objectives. The Secretariat has been working with the Corporate Services Division (IXS) to develop and populate CTCBP data within the approved Branch project database. Unless historical information is entered, the summative evaluation will face the same challenges with the lack of administrative data from the early years of the Program.

Lack of Information from Country Comparative Analysis

It was expected that the country comparative analysis would identify three types of information:

The country's programming in the area of counter-terrorism capacity building;
Views on the CTCBP, or least generally on Canada's contributions to counter-terrorism capacity building; and
Views on the relevance of counter-terrorism capacity building and the priority needs.

The first challenge was that one country - Australia - was unable to participate in the case study. Respondents in the remaining two countries - Switzerland and the United States - were able to provide information on the first two subjects but were generally unaware of Canada's contribution and, specifically, the CTCBP. Since the nature of their programming is substantially different from that of Canada, the contributions of the case studies to this evaluation are limited to information on the relevance of, and priorities for, counter-terrorism capacity building programming.

Measurement of Long-term Outcomes

This challenge is common to all capacity building programs. It is difficult to measure results beyond the level of the outputs or perhaps short-term outcomes. Even if measurement is possible, it is difficult to attribute any changes measured to the program's activities. The measurement of results can be expensive and requires a commitment to follow-up long after the activities have been completed. The case studies carried out for this evaluation suggest that results are not being measured by CTCBP project implementers. As a result, the case studies were also unable to identify results, beyond anecdotal information from case study respondents.

1.2.3 Interpretation of Data

The evaluation team conducted 33 interviews with key informants. However, the responses were not quantified due to the small numbers in interviewee sub-categories and because not all questions were asked of all respondents. The structured interview guide used for these interviews included more questions than could be addressed in the time available. As a result, not all questions were posed to each respondent. A selection of questions was made, based on the profile of the respondent. With respect to a couple of issues, the evaluation team was able to quantify the responses by department/agency. For the remaining issues, the report uses the unquantifiable terms, such as "some" and "most", to reflect the relative frequency with which an issue was raised.

1.3 Report Outline

After this introduction (Chapter 1.0), the report is divided into five chapters. Chapter 2.0 presents a profile of the CTCBP. The next three chapters reflect the evaluation findings according to the three main evaluation issues:

Relevance (Chapter 3.0);
Governance, design and delivery (Chapter 4.0); and
Success and cost-effectiveness (Chapter 5.0).

The final chapter (Chapter 6.0) reflects the conclusions and recommendations of the evaluation.

Haut de la page

2.0 Description of The Counter-Terrorism Capacity Building Program

This chapter presents a description of the CTCBP objectives, the Program's governance and resources and a profile of the Programs' funded projects. This sets the context for the assessment of the evaluation issues in the subsequent chapters. It is descriptive - the analysis of the Program design is provided in Chapter 3.0.

2.1 Objectives

Much of the impetus for the establishment of the CTCBP came from the UN Security Council Counter-Terrorism Committee (CTC), which has responsibility for ensuring that states implement the UN Security Council Resolution on measures to combat terrorism.³ The CTCBP was set up in 2005 as an instrument of Canada's National Security Policy and a key part of Canada's international terrorism prevention efforts. The Program has three objectives:

To enable other countries and governments to prevent and respond to terrorist activities in a manner consistent with international counter-terrorism and human rights obligations, norms and standards (primary objective);
To increase Canada's profile in CTCB assistance and demonstrate Canadian leadership in specialized areas of expertise (secondary objective); and
To improve domestic and international coordination and cooperation for CTCB assistance (tertiary objective).

The Program provides training, funding, equipment, technical and legal assistance to other states to enable them to prevent and respond to terrorist activity, within international counter-terrorism and human rights norms, standards and obligations.⁴ It is primarily a responsive program that funds capacity building activities in seven investment areas:

Border security;
Transportation security;
Legislative, regulatory and legal policy development, legislative drafting, and human rights and counter-terrorism training;
Law enforcement, security, military and intelligence training;
Chemical/biological/radiological/nuclear and explosives (CBRNE) terrorism prevention, mitigation, preparedness, response and recovery;
Combating terrorism financing; and
Cyber security and protecting critical infrastructure.⁵

2.2 Governance

The CTCBP is an interdepartmental program administered by DFAIT on behalf of sixteen federal government departments and agencies. There are three main components to the governance structure: an Interdepartmental Steering Committee, a Program Secretariat and an Online Screening Committee.

Interdepartmental Steering Committee

The Program is managed by an Interdepartmental Steering Committee (ISC), which includes representatives of the member departments and agencies. The ISC ensures that the Program takes a 'whole-of-government' approach and is "responsible for establishing strategic policy, priority-setting and direction; review and approval of project proposals for funding through the Program; information exchange on Program-related activities; approving the terms of reference for the Program evaluations; and overseeing financial reporting and project implementation for their departments."⁶ The ISC is chaired by the Senior Coordinator, International Crime and Terrorism (ICX).

The ISC approves all projects over $35,000 and the ICX approves those projects under this limit.

CTCBP Secretariat

Day-to-day administrative and financial matters, policies and procedures are the responsibility of the CTCBP Secretariat, located in DFAIT's International Crime and Terrorism (ICT) Division, and reporting to the Senior Coordinator (ICX). Specific responsibilities include "financial administration; reporting on results; project monitoring; liaising with [DFAIT's] audit and evaluation divisions, posts and geographic divisions at the [DFAIT]; coordination with other donor states; communications and public/private sector outreach; record-keeping; coordination of responses to Access to Information Act and Privacy Act (ATIP) requests pertaining to the Program; distributing project documents; reporting on international trends and commitments stemming from international agreements; and developing and managing projects involving agreements with international organizations or for which [DFAIT] is the implementing body."

Online Screening Committee

The Online Screening Committee (OSC) is made up of working level representatives of the member departments and agencies. It also may include other DFAIT representatives, including representatives of missions abroad. The OSC does not meet in-person, but is tasked with reviewing applications for Program funding electronically. Participation in this review process may vary, depending on the subject matter of the project being reviewed. OSC members provide feedback to the Secretariat, which then provides the feedback to the project proponents to address before submission to the ISC.

2.3 Resources

The financial and human resources available to the CTCBP are described in this section.

Financial Resources

The CTCBP initial funding was $15 million a year (except the first year, which was only $12 million) (see Figure 2.1). These amounts include funds for the management of the Program Secretariat (just over $1 million/year) and the costs of monitoring, internal audits and program evaluations. The funding comes from three different Votes:

Figure 2.1: CTCBP Financial Profile

Table 1: CTCBP Financial Profile
Financial Overviews from April 2005 to January 2009 (in $ thousands)
	2005/06			2007/06
	Funding	Actuals	Variance	Funding	Actuals	Variance
Total CTCBP	$12,000.0	$208.7	-98%	$15,000.0	$9,690.4	-35%
Secretariat Programming	$1,048.5	$175.7	-83%	$1,048.5	$507.2	-52%
Votes 1 and 5 (OGD MOUs and other)	$5,651.5	$33.0		$8,451.5	$2,270.5
Vote 10 (Grants and Contributions)	$5,300.0	$0.0		$5,500.0	$6,912.7
Total Programming	$10,951.5	$33.0	-100%	$13,951.5	$9,183.2	-34%

Table 2: CTCBP Financial Profile
Financial Overviews from April 2005 to January 2009 (in $ thousands)
	2007/08			2008/09	2009/10
	Funding	Actuals	Variance	Funding	Funding
Total CTCBP	$15,000.0	$10,214.7	-32%	$12,912.9	$12,912.9
Secretariat Programming	$1,125.5	$806.7	-28%	$1,037.9	$1,037.9
Votes 1 and 5 (OGD MOUs and other)	$5,874.5	$4,168.7		$3,375.0	$3,375.0
Vote 10 (Grants and Contributions)	$8,000.0	$5,239.3		$8,500.0	$8,500.0
Total Programming	$13,874.5	$9,408.0	-32%	$11,875.0	$11,875.0

Source: CTCBP Secretariat, November 2008

Note: Expenditures for Jan-April 2006 are missing. However, expenditures for 2005/06 were very limited due to delays in the start-up of the program. These expenditures shown were primarily for one staff person.

Vote 1: Operating costs, transfers to federal departments/agencies, monitoring, audit and evaluation;
Vote 5: Capital costs, transfers to federal departments/agencies; and
Vote 10: Grants and contributions.

As can be seen in Figure 2.1, the Program under-spent its resources (primarily in the programming area, but also in the Secretariat) in each of its first three years. In the first year (2005/06), it only spent 2% of its allocated funds and then in each of the following years, the Program under-spent by 35% (2006/07) and 32% (2007/08). See Section 4.3.2 for a discussion of the issues that contributed to the under-expenditure in funds for the Secretariat in the first year and, to some extent, second year of the Program. Implementing partners (reportedly primarily the OGDs) lapsed project funds in both years and this resulted in the under-expenditure of programming funds. This recurring under-expenditure of funds contributed to a decision, taken as part of DFAIT's Strategic Review, to reduce permanently the CTCBP funds to $12 million/year.

Human Resources

The CTCBP Secretariat was initially intended to include seven indeterminate Full-time Equivalents (FTEs) - all non-rotational positions - including:

Chief of Program, who reports on all policy and human resources matters to the Director, ICT, but reports on financial matters to the Senior Coordinator, International Crime and Terrorism;
Manager, Finance;
Administrative Coordinator; and
Four Project Managers - each of whom was expected to be responsible for one or two investment priorities areas.

However, as will be discussed further in Section 4.1.2, the Secretariat was not fully staffed until 20008/09 and then, as noted above, one FTE (Senior Project Manager) was cut in the same fiscal year.

2.4 CTCBP Projects

This section presents a profile of funded CTCBP projects. However, as noted in the methodological limitations, there were challenges in getting complete and consistent information on CTCBP projects. These inconsistencies will be reflected in some information presented in this section.

2.4.1 Disposition of Project Proposals

According to one project list provided by the Secretariat, a total of 220 projects have been presented for CTCBP funding. The disposition of these 220 projects is shown in Figure 2.2.

Figure 2.2: Status of CTCBP Projects, 2005 to mid-2008

Status of CTCBP Projects

Of all the projects presented and recorded by the Secretariat, the disposition of 79% is known - the remaining projects were still in development. Of these, one-third (34%) were withdrawn either prior to or at the Interdepartmental Steering Committee (ISC) meeting.

Of the 62% of projects that were approved, 60% had been completed at the time the project list was prepared. Almost all the others were in progress.

2.4.2 Profile of Projects

Project Implementing Bodies

The total number of projects approved annually has increased from fourteen in 2005 to 43 in 2007.⁷ As shown in Figure 2.3, federal departments and agencies are the primary project implementing partners, accounting for between one-fifth (21%) and over half (56%) of all funded projects across three years.

Figure 2.3: Percentage of funded projects, by type of implementing body, 2005 - 2007

Percentage of Funded Project

Source: CTCBP Secretariat, List of Funded Projects, undated. Analysis by evaluation team

Although projects are implemented by other organizations - for example, other Canadian governments, multilateral organizations (including regional organizations) and national and international non-governmental organizations (NGOs) - they represent a smaller proportion of the projects in 2007 than they did in 2005.

Sponsoring Bodies

Most sponsoring bodies are also federal department/agencies (see Figure 2.4). From the data it would appear that DFAIT is the department that sponsors the most projects, with Royal Canadian Mounted Police (RCMP) and Transport Canada (TC) in second and third position, respectively. However, given the limitations of the data, these numbers may be misleading, since projects sponsored by a departmental Liaison Officer posted to a mission abroad would be shown as being DFAIT-sponsored, even though other departments would be primarily responsible for the project.

Figure 2.4: Number of projects, by year and by sponsoring organization, 2005 - 2007

Number of Projects

Source: CTCBP Secretariat, List of Funded Projects, undated. Analysis by evaluation team

When considering the total dollar value of the projects sponsored by these departments, the same pattern can be seen (see Figure 2.5). The sponsoring departments with the highest dollar value of projects are DFAIT, RCMP and TC. However, the same data limitations apply.

Figure 2.5: Dollar value of projects, by sponsoring organization, 2005 - 2007 (in thousands of dollars)

Dollar Value of Projects

* For all classified projects the dollar value of the projects was not shown in the data. This included all CSIS projects and one DFAIT project.

Source: CTCBP Secretariat, List of Funded Projects, undated. Analysis by evaluation team

As shown in Figure 2.6, the range of project values for each sponsoring body, with multiple projects, is quite large.

Figure 2.6: Project Values, by Department/Agency, 2005 - 2007 (in thousands of dollars)

Table 3: Project Values, by Department/Agency, 2005 - 2007 (in thousands of dollars)
Department/agency	Total value of projects	No. of projects	Range of project values
Total	$29,961	105
CBSA	$1,724	7	$28 - $821
CFIA	$660	1	$660
CNSC	$290	2	$140 - $150
CSIS*	*	7	*
DFAIT	$8,183	31	$5 - $1,002
DND	$3,550	5	$290 - $1,638
DRDC	$33	1	$33
Finance	$2,353	4	$286 - $800
FINTRAC	$545	5	$18 - $341
INTERPOL	$250	1	$250
OAS	$768	3	$92 - $487
PHAC	$907	1	$907
RCMP	$4,820	18	$24 - $625
TC	$5,813	16	$22 - $1,744
Other	$724	3	$18 - $626

* For all classified projects the dollar value of the projects was not shown in the data. This included all CSIS projects and one DFAIT project.

Investment Areas

The investment area with both the highest number of projects and the highest dollar value of projects is law enforcement, security, military and intelligence training (LEMI). These projects account for just over one-quarter (27%) of all projects (see Figure 2.7) and just under one-quarter (23%) of the total value of all projects (see Figure 2.8). the next largest investment areas are:

Transportation security - 21% of all projects and 21% of the total dollar value of all projects;
Financing of terrorism - 21% of all projects and 18% of the total dollar value of all projects; and
CBRNE - 15% of all projects and 15% of the total dollar value of all projects.

Figure 2.7:Number of projects, by year and investment area, 2005 - 2007

Number of Project - Investment Area

Source: CTCBP Secretariat, List of Funded Projects, undated. Analysis by evaluation team

Figure 2.8: Value of projects, by investment area, 2005 - 2007 ( in thousands of dollars)

Value of Projects - Investment Area

* For all classified projects the dollar value of the projects was not shown in the data. This included all CSIS projects and one DFAIT project.

Source: CTCBP Secretariat, List of Funded Projects, undated. Analysis by evaluation team

Haut de la page

3.0 CTCBP Relevance

The CTCBP continues to be relevant to the Government of Canada's priorities both with respect to its domestic agenda and its international commitments

3.1 Relevance to the Government of Canada

Finding:

The CTCBP remains relevant to the Government of Canada's priorities with respect to counter-terrorism.

The Program was designed as an instrument of Canada's National Security Policy. Although the policy is currently being revised, it is expected that the Program will still be relevant to the Government's updated national counter terrorism plan, since the importance of counter-terrorism activities continues to be highlighted in the Speeches from the Throne. The priority to be given to counter-terrorism was mentioned in the October 2004 Speech from the Throne, before the establishment of the Program, but also again in 2006 and 2007.⁸

The Program is values-based and intended to provide assistance to states to prevent and respond to terrorism "in a manner consistent with international counter terrorism and human rights norms, standards and obligations." In this respect, it is consonant with the Government of Canada's foreign policy priority of "Greater international support for freedom and security, democracy, rule of law, human rights and environmental stewardship."

The continued relevance of the Program to the activities of other federal departments and agencies was also reflected in the key informant interviews. Some OGDs have mandates in the area of counter-terrorism and the Program supports them and their existing partnerships, particularly with multilateral organizations (e.g. Transport Canada and the International Civil Aviation Organization, CBSA and the World Customs Union, RCMP and Interpol).

3.2 Relevance to International Partners

Finding:

Counter-terrorism capacity building continues to be relevant to Canada's international partners and consistent with Canada's participation in international fora.

The importance of counter-terrorism capacity building is also a priority for Canada's partners in the G8 and Canada's commitment to these priorities is particularly important as it prepares to host the G8 in 2010. In a statement after their meeting in 2008, the G8 leaders reiterated "the critical role of capacity building for countries requiring assistance to meet their international counter-terrorism commitments. In this regard, [G8 leaders] will further strengthen cooperation among the G8 and the UN, especially by enhancing efficient coordination with the Counter-Terrorism Committee/Counter-Terrorism Executive Directorate (CTC/CTED) through the Counter-Terrorism Action Group (CTAG)."⁹

The CTCBP continues to be relevant to Canada's support for the CTAG, which it will chair next year, as it assists other members states to implement UN counter-terrorism-related resolutions (UN Security Council Resolutions 1373 and 1456). It also continues to contribute to counter-terrorism activities of other multilateral institutions in which Canada participates, such as the International Civil Aviation Organization (ICAO) and International Maritime Organization (IMO).

Box 3.1: US Anti-Terrorism Assistance (ATA) Program

The ATA is a program to support the training of law enforcement personnel to protect national borders, critical infrastructure, respond to and resolve terrorist incidents, etc. with resources in the $150 million/year range. It is coordinated by the Office of the Coordinator for Counterterrorism (Department of State) but activities are delivered by a range of organizations, including the Departments of Defence and Justice, FBI, CIA, Treasury, Homeland Security and other state, and local law enforcement agencies, police associations, and private security firms and consultants. According to the ATA website, since its inception in 1983, the program has trained and assisted over 48,000 foreign security and law enforcement officials from 141 countries.

Source: Key informant interviews and
Antiterrorism Assistance Program;
Combating Terrorism: Guidance for State Department's Antiterrorism Assistance Program Is Limited and State Does Not Systematically Assess Outcomes

The CTCB also encourages and supports regional cooperation via other organizations of which Canada is a member, including the Organization of American States (OAS) and the Organization for Security and Cooperation in Europe and the Asia Pacific Economic Cooperation.

The relevance of counter-terrorism capacity building activities was also reflected in the results of the country case studies. Both Switzerland and the US fund counter-terrorism capacity building activities and target similar areas to those of the CTCBP. For example, like CTCBP, the US program - Anti-Terrorism Assistance Program (see Box 3.1) - is delivered by a range of organizations but is coordinated by a federal agency. Unlike CTCBP, the ATA appears to focus primarily on training - taking a narrower definition of capacity building than the CTCBP - and has considerably more resources annually for programming.

Box 3.2: Swiss Counter-terrorism Capacity Building Support

An interdepartmental committee, located in the Directorate of International Law in the Department of Foreign Affairs, coordinates counter-terrorism capacity building initiatives. Activities are implemented by the Departments of Foreign Affairs, Finance and Justice, but they do not have specific budgets for counter-terrorism activities and the coordinating committee does not do strategic planning - it can only encourage departments in specific areas. Switzerland appears to include a broad definition of counter-terrorism, including money laundering, criminal activities, support for information sharing/coordination but focuses its support on implementation of UN conventions, implementation of legislative frameworks and the financing terrorism activities.

Source: Key informant interviews

Switzerland, on the other hand, funds counter-terrorism capacity building activities through a number of federal departments and these activities are coordinated by a central coordinating committee (see Box 3.2). However, there is no specific budget for these activities. Nor is strategic direction provided by the coordinating committee. As an international financial centre, Switzerland's counter-terrorism capacity building funding, unlike CTCBP, appears to be more narrowly focused in areas related to legislative frameworks and the financing of terrorism activities. However, like CTCBP, the activities are funded through existing organizations.

Haut de la page

4.0 Governance, Design and Delivery

As this is a formative evaluation, considerable emphasis was given to the issues of the governance, design and delivery of the Program. The evaluation findings on these issues are reflected in this chapter.

4.1 Governance

Findings:

All OGDs have an equal stake in the ISC but this presents a challenge for the CTCBP governance. It has been difficult to engage the sixteen participating federal departments/agencies in the identification of strategic priorities and to mesh DFAIT's priorities with those of other departments/agencies that are actively engaged in counter-terrorism activities. As a result, the Secretariat and, in a wider context, DFAIT plays a very prominent role in the identification of strategic priorities and the approval of projects.

There are four components to the governance structure of the CTCBP: the ICX, the ISC, the Secretariat and the OSC. The following sections assess the strengths and challenges of these four components. The evaluation team believes however, that there are key assumptions lying behind the governance structures that need to be discussed prior to the assessment of the strengths and weaknesses.

The CTCBP is a horizontal initiative structured somewhat differently from some other federal government horizontal initiatives. In some federal government horizontal initiatives (e.g. Youth Employment Strategy) the initiative's funds are allocated to each participating department or agency and each department sets its priorities and makes the funding decisions for its allocation. As an international program, the CTCBP manages a central pot of funds, administered by DFAIT, consistent with its mandate to coordinate foreign policy matters within the interdepartmental community. The Interdepartmental Steering Committee includes all OGD partners and, unlike the Youth Employment Strategy, is responsible for setting priorities and making funding decisions within this centralized funding mechanism. The funds are then disbursed by third-party organizations. As such, the OGDs are playing a collective decision-making role for Canada's counter terrorism capacity building via the Program. However, the same OGDs are often recipients of the funding, and are directly implementing projects, either through their own department or through third-party organizations. Finally, because of the wide range of investment areas, the departments are also called up to provide technical expert advice for project funding decisions. The governance model appears to be based on the assumption that all participating departments and agencies contribute equally in the management and implementation of the Program.

However, the context for each OGD's participation varies. Notably, not all OGDs have an international mandate for counter-terrorism capacity building. For example, Transport Canada has an international mandate and has existing partnerships with international organizations for counter-terrorism activities. On the other hand, departments or agencies such as the Canadian Boarder Services Agency (CBSA) or the Canadian Food Inspection Agency (CFIA), while they have international mandates, these do not include capacity building or the provision of technical assistance.

For some departments/agencies, the contribution to counter terrorism capacity building is not a part of their primary mandate and as such, their investment in the CTCBP is not given the same priority as those that have a clear mandate and expertise in this area. Some OGDs actually implement CTCBP projects but others do not. In addition, some OGDs have strong international project management capacities (perhaps building on project work that is part of their main mandate) and others do not. Yet, the OGDs are deemed to have an "equal" stake and voice in the ISC.

4.1.1 Senior Coordinator, International Crime and Terrorism

The ICX reports to DFAIT's Assistant Deputy Minister, International Security Branch. He/she is the Chair of the ISC and has responsibility for the financial management of the CTCBP. While the Chief of CTCBP reports to the Director, International Crime and Terrorism Division (ICT) on all policy and human resources matters, he/she reports to the ICX on financial matters. This ensures there is a clear distinction between the CTCB Program and other DFAIT budgets. The ICX is also responsible for approving CTCBP projects of less than $35,000.¹⁰

The 2007 Capacity Check for the Program identified reservations about the split in reporting relationships. Although this was not specifically addressed in this evaluation, there was no evidence to indicate that this was a concern for stakeholders.¹¹

4.1.2 Interdepartmental Steering Committee

The ISC has three ongoing responsibilities with respect to the Program as a whole:

Establishing strategic policy, priority-setting and direction;
Reviewing and approving projects over $35,000; and
Sharing information on Program-related activities.¹²

The ISC has faced a number of challenges in fulfilling these functions. Although the Committee was intended to develop overall CTCBP strategy, the Program is, to a large extent, responsive - responding to the project proposals that are presented. Yet, some strategic direction is required to set the parameters within which projects will be approved and to guide outreach activities of the Secretariat to encourage project proposals that can be consistent with Canada's strategic foreign policy priorities. Developing these strategic directions has been difficult. Initially it was intended that representation on the Committee would be at the Director General (DG) level to ensure senior coordination and vision in horizontal priority-setting. However, in practice, departmental/agency representatives are rarely DGs and, in fact, tend to be working level staff - at times, reportedly relatively junior staff. There is also a lack of continuity in representation at the meetings, with new OGD representatives appearing at meetings.

A second challenge is that, since the initial impetus for the Program came from Canada's commitments to the UN Security Council Counter-Terrorism Committee (CTC), Canada's international commitments, as defined by DFAIT on behalf of the Government of Canada, have played a big role in defining CTCBP strategic priorities. It has been a challenge to mesh DFAIT's role in coordinating broad Government of Canada priorities with the focused operational activities that other departments/agencies may wish to see funded by the CTCBP. For example, the Program is very much driven by Canada's international priorities - one of which is focusing on the Americas (US, Mexico, Latin America and the Caribbean). In a presentation given by the ICX in October 2008, nearly half (47%) of funded projects in 2007/08 were implemented in the Americas.¹³ Yet some OGDs have their own, different, priority countries and regions, which are less well represented in the Program funding. For example, Transport Canada had identified the Asia Pacific region as a priority. Yet, according to the ICX presentation, over the same period of time, only 19% of CTCBP projects were funded in the Asia region.¹⁴

The OGD representatives interviewed for this evaluation expressed a range of views on the effectiveness of the ISC. Representatives of one-third of OGDs reported that the ISC was effective; one-third had reservations about its effectiveness; and the remaining one-third could not comment, often because the representative interviewed had not attended an ISC meeting.¹⁵ Of those who had reservations about the Committee, these related to:

The level of informed debate at the meetings - Given the wide range of technical areas covered by CTCBP projects, many representatives do not feel that they (or their colleagues) are well-positioned to comment on projects being proposed by other departments/agencies. As a result, there is limited debate on project proposals. The amount of debate may also be affected by the extensive turnover in, and different levels of, representation at the meeting; and
The decision-making process - In the Program design it was expected that projects would be approved by a two-thirds majority of ISC members.¹⁶ However, the current chair and his predecessor have chosen to use a consensus model for decision-making, on the grounds that it is not as divisive as a voting model would be. They reportedly also assume that when a representative makes no comment on a proposal, it is assumed that they are in agreement. This has, reportedly meant that decisions are made based on limited discussion and, because there is no vote, no position being taken (in favour or against) by many, if not most, OGD representatives. This reflects, for some key informants, a lack of accountability for decisions.

The Secretariat and the ICX have attempted to address these challenges:

They conduct two meetings annually that are intended to focus on strategic planning - encouraging, with limited success, participating OGDs to send more senior representatives to these meetings;
They encourage broad consultation on proposed projects, using a variety of different approaches. The use of the OSC as a less public mechanism to solicit preliminary comments on proposals, than the open discussion at the ISC, was appreciated by the OGD representatives. They were positive about the way in which their input had been taken into account. On the other hand, Secretariat staff noted that they often do not receive a lot of comments - perhaps again reflecting the challenge for many OGDs to provide comments on projects that cover a wide range of technical areas. Secretariat staff actively solicit feedback from those - either among the OGDs or within DFAIT particularly the missions - who have specific knowledge relevant to the proposal;
They are extensively involved in pre-screening projects. As noted in Section 2.3.1, one-third of all projects are withdrawn before a decision is made by the ISC. It is not known what proportion of these are withdrawn prior to the ISC, based on comments from the Secretariat, or due to review and comments by the OSC. However, the pre-screening process is reported to have eliminated a substantial number of proposals prior to reaching the ISC. Although there is no consistent tracking of withdrawn proposals, the Secretariat has reported some of these proposals initially withdrawn have been substantively revised and re-submitted for review and ultimately approved by the ISC; and
They recently began the practice of inviting OGDs to host the ISC meetings and make presentations on their counter-terrorism capacity building work. This provides a learning opportunity for ISC members and, based on the first meetings, was well received by some OGD representatives.

The Secretariat and the ICX have tried to overcome challenges associated with engaging the ISC in setting strategic and decision-making for the Program. However, they have not been able to overcome them all. As a result, the Secretariat and, in a wider context, DFAIT plays a very prominent role in the identification of strategic priorities and the approval of projects.

4.1.3 Secretariat

The Secretariat faced significant challenges in the first year of the Program. An allocation of $12 million was available for project funding and there was reportedly interest on the part of the participating OGDs to access these funds. However, only one staff person was assigned to the Secretariat and tasked with developing the jobs descriptions for, and staffing, the remaining positions, while at the same time trying to disburse Program funds. As a result, as noted in Section 2.2, all the programming funds lapsed in the first year, as did 83% of the funds for the Secretariat. Although the staffing process continued into the second and third year, there was considerable turnover in staff (including no less than four Program Chiefs in the period 2006-2008). The financial accountability capacity of the Secretariat was particularly strengthened with the staffing of a Finance Manager position in 2007/08. Although all positions are now staffed, the Secretariat was cut by 1 FTE (a Senior Project Manager) in early 2008/09. This has resulted in a greater portfolio load among the remaining three project managers.

In spite of the significant improvements since the launch of the Program, the Secretariat's capacity remains weak in a couple of areas:

Limited capacity of the Project Managers to review and use financial reporting information to monitor the implementation of up to 40 active projects at a time. The Finance Manager does not have the time to do this for all funded projects. This lack of capacity is a particular challenge given the recurring problem of OGD partners recurrent lapsing of funds; and
Lack of clarity on the role of the Secretariat in results monitoring and evaluation (see Section 4.3.3).

4.1.4 Online Screening Committee

The OSC provides feedback on proposals prior to their submission to the ISC or approval by the ICX (in the case of projects under $35,000). As noted in Section 4.1.1, the use of the OSC to comment on proposals prior to being submitted to the ISC is appreciated by the OGDs, even though this consultation does not often result in a lot of comments from the OGDs. Over time the Secretariat developed the approach that if no comments were received, the OGDs were assumed to be supportive of the proposal.

There have been fewer projects less than $35,000, so it is difficult to assess the effectiveness of the OSC for providing inputs to the ICX's decision on these projects.

4.2 Program Design

Finding:

CTCBP design was appropriate. Stakeholders were supportive of both the horizontal nature of the Program and the broad definition of counter-terrorism and capacity building.

Generally speaking, key informant interviews indicated that the program design was appropriate. Issues related to the design of governance have been discussed in Section 4.1.

Key informant respondents commented particularly on the horizontal nature of the Program and the definitions of counter-terrorism and capacity building. Both of these two components of the CTCBP design were reported to be appropriate. While there are challenges with the governance structure (as discussed in Section 4.1), almost all department/agencies indicated that it was still relevant to structure the CTCBP as a horizontal initiative, with centralized funding source and coordination through a central secretariat. There was very little support for the upfront allocation of funding to individual OGD partners, as opposed to the current allocation on a project basis.

Most key informants felt that broad definitions of both counter-terrorism and capacity building continued to be relevant. A broad definition of counter-terrorism allows the Program to fund activities that may overlap with other areas - for example, anti-crime activities - but it recognizes that counter-terrorism is a complex issue. A broad definition of capacity building includes activities that go beyond training and support the development of infrastructures (e.g. legislative frameworks, networks) to strengthen capacity.

4.3 Program Delivery

Findings:

There have been challenges with Program implementation stemming from delays in setting up the Secretariat and the lack of project management capacity in some implementing organizations. The lack of capacity in OGDs and in recipient governments means that program funds have lapsed and this contributed to a decision, undertaken as part of DFAIT's Strategic Review, to reduce permanently the CTCBP funds to $12 million/year.

The Secretariat was recognized by stakeholders for the key role that it has played in supporting the development of projects and the ISC. However, there remain challenges in program monitoring and evaluation and Program external reporting.

The evaluation addressed issues related to the three phases of program delivery: project identification and selection; program implementation and program results monitoring and evaluation.

4.3.1 Project Identification and Selection

The CTCBP is a responsive program. While it relies on the submission of proposals from sponsoring bodies or directly from implementing organizations, the staff of the Secretariat and DFAIT, more broadly, were engaged and, to some extent still engage, in outreach activities to promote the program with potential partners. In the first year, Secretariat and ICT staff reportedly played a major role in promoting the program to OGDs and international partners. Now the Program is apparently well-known and less promotion is required, particularly considering that program funding is limited to $12 million. The Program has reportedly been able to generate sufficient interest to be able to allocate all its funding.

The Secretariat coordinates the selection and approval of projects. The steps in the process include:

Receipt of the proposal;
Identification by project managers of the key substantive focus of the proposal and identification of the relevant people to assess the proposal;
Conduct, by the Secretariat, of a preliminary review and revision of the proposal in consultation with the project sponsor;
Circulation of the project proposal electronically for formal comment and the provision of comments to the proposal's sponsoring organization(s); and
Forwarding of the revised proposal to the ISC for review and approval.¹⁷

Key informants interviewed for this evaluation noted that the Secretariat, working with the OSC, plays a key role in screening projects. Respondents were positive about this role and the opportunity that the OSC provides for soliciting comments prior to the ISC meetings and the active role that the Program Managers play in ensuring that comments are addressed prior to the meetings. They indicated that the role of the Secretariat is to assess the extent to which the proposed project meets the Government priorities and the CTCBP T&Cs and the technical merits of the project.

In the absence of a complete project database, it is not possible to analyze in depth the types of projects that have been selected. The information that was available is presented in Section 2.4. It includes only a profile by type of sponsoring and implementing organization and investment area, and for unclassified projects only. It was not immediately apparent, based on the list of project titles provided to the evaluation team to present a complete profile of the countries and/or regions supported by CTCBP projects or to provide any other information about the nature of the projects. Instead of the review of nearly three hundred project files to reconstruct such a profile, information from key informants and the case studies was used to provide an analysis in Section 5.0 of selected projects and their results.

In any event, there does not appear to be a lot of competition for funding. The vast majority of projects presented to the ISC for approval (i.e., not withdrawn) are approved. Key informants noted that the number of eligible projects is primarily limited by the capacity of the partner organizations (notable the OGDs) to develop and implement projects. Most felt that the current level of funding for CTCBP was sufficient, given the capacity of the partner organizations.

4.3.2 Program Implementation

As the CTCBP is a project-based program, there are two components to program implementation - the implementation of projects by implementing partners and the support of the Secretariat for the Program infrastructure. This section provides feedback on issues with respect to both components.

Project Implementation

As noted in Section 2.3, a major challenge for Program implementation has been the delays in project implementation, which has resulted in the lapsing of CTCBP funds. In the three years of disbursement of Program funds (2005/06, 2006/07 and 2007/08), in spite of efforts of the Secretariat, the Program has significantly under-spent its program funds. In the first year of funding, apparently late decisions on the funding allocations left partners with very limited time to implement all their activities before the end of the fiscal year. Beyond the first year, delays can be attributed, in part, to weak project management capacity in some partner organization, particularly OGDs. Some OGDs reportedly face challenges in developing accurate project budgets for project approval, forecasting expenditures, managing human resources and monitoring projects to determine the likelihood of success. Those OGDs that have lapsed funds reported that the problems stem from the incompatibility of their corporate organizational structure for these types of projects, internal communication issues, lack of adequate priority and staff for the projects and the challenges of working in recipient countries which may be in crisis or lack capacity themselves. The OGD challenges with project monitoring are particularly critical when delays are caused by recipient countries/organizations.

In an attempt to address these weaknesses and introduce more rigour in the financial disbursement process, starting in fiscal year 2008/09 the Secretariat developed new procedures and an updated Memorandum of Understanding (MOUs) with OGDs that call for more frequent instalment payments contingent on receipt of financial reports showing that previous disbursements had been used. Also, in order to minimize the impact of the lapsing, the ISC continues its review and adjustment of notional allocation of funds throughout the fiscal year.

Support for Program Implementation

As noted in Section 4.3.1, the Secretariat plays a key role in the identification and selection of projects - preparing the groundwork for the ISC. The Secretariat also provides support during project implementation. However, the challenges of implementing a program before the staffing is complete have had a number of implications for the support that the Secretariat is able to provide. There have been delays in the development of the necessary infrastructure for effective program management. This includes such things as:

Delays in the development of program and project templates and guidelines - Some standard tools were developed and have continued to be adapted, including the project templates/Project Approval Document (PAD) and are seen favourably by some OGD representatives. The gradual development over time of more standardized and onerous program documents was seen negatively by some OGDs as an increasing rigidity in the proposal process;
Inconsistency in the interpretation of the Programs T&Cs - Some key informants noted discrepancies and/or confusion about what could be funded through the Program. The most commonly cited area of confusion was the use of CTCBP fund for the purchase of equipment. Issues were raised about whether the Program could fund equipment and, if not, the impact this would have on the sustainability of training. The Program does not yet have a standard operating procedures manual to ensure greater consistency in program administration and ensure better integration of new staff;
The lack of standard operating procedures to provide guidance on what may be funded through the CTCB Program and how to deal with specific issues such as travel expenses and currency exchange; and
Delays in the establishment of a project database for tracking projects. As a result of a recommendation in the 2007 Capacity Check for the Program,¹⁸ ICTC collaborated with the Corporate Services Division to develop a CTCBP database. However, the database design is not yet finalized and has not been fully populated.

The Secretariat continues to have challenges in monitoring project implementation. As previously noted, the Secretariat has only recently been fully staffed, with the further reduction of one Project Manager FTE, lack of resources have kept Project Managers occupied primarily with the development and approval of projects, leaving them with limited time to monitor the implementation of activities in partner organizations. The only mechanism that Project Managers have for monitoring is the review of information provided in partner financial reports. While the arrival in 2007 of the Finance Manager has strengthened the Secretariat's capacity in this area, the review of financial reports is still the responsibility of the individual Project Managers, who report that they do not necessarily have sufficient time or adequate capacity in this area. In addition, they are seen by OGDs as playing a more important role in supporting the development of projects than the monitoring of their implementation.

However, in spite of these challenges, the Secretariat was generally recognized for its support to program administration. Of the eight OGDs, whose representatives provided comments in the key informant interviews on the Secretariat's role, five were positive about the staff's contribution to program administration. They reported favourably on the support provided by Secretariat staff for the development of project proposals, including providing feedback from the preliminary screening of projects. They generally agreed that the Secretariat provides good support for the ISC, soliciting comments from key stakeholders and ensuring documents are distributed in a timely fashion. Representatives of three OGDs provided either mixed or more critical comments, noting three different issues: the need for the Secretariat to play a bigger role in filtering projects; the need for the ISC to provide more guidance to the Secretariat to ensure consistency in the determination of what can be funded; and the fact that the level of support depends on the project manager.

Finally, the Secretariat is expected to provide support for communications and public/private sector outreach. It is important to make the distinction between communications and outreach. The role of the Secretariat in outreach activities was discussed in Section 4.3.1. There has been less attention paid to communication activities to publicize the Program and its results to the general public. Although some support has been provided by ICX, no resources have been available in the Secretariat to dedicate to these activities. As a result, the activities have been limited to the development of program brochures and a CTCBP website (although the website has not been updated since April 2007).¹⁹

Yet all those who commented on the Program's communication strategy agreed that it was not appropriate to have a major communication focus for a national security program that is neither commercial, nor developmental. The Program's objective of increasing Canada's profile is explicitly articulated in its terms and conditions to mean promoting Canada within the international counter-terrorism and global security community, rather than the general public. As a result, it was felt that this objective can be achieved effectively through the funded projects and the CTCBP and ICT networks, than with a public communication strategy. Each project proposal includes a component on the project communication strategy and some partner organizations do promote Canada's involvement at a project-level or more generally in publications.²⁰ As such the current level of effort allocated to communication activities appears to be adequate, given the scope and nature of the Program.

4.3.3 Program Results Monitoring and Evaluation

In the previous section on program implementation, the team assessed the challenges of monitoring the implementation of CTCBP projects. This section deals with the monitoring not of the implementation, but rather of the achievement of the project and program results.

CTCBP results monitoring and measurement is defined in a very comprehensive RMAF/RBAF document.²¹ Some challenges in monitoring and reporting were anticipated in this document:

Given the challenges of measuring the impact of capacity building initiatives and attributing it to the CTCBP (see the discussion in the evaluation limitations covered in Section 1.3.2), the RMAF/RBAF indicates that "the Program will seek to measure and illustrate progress in beneficiary states and government entities receiving training, technical or other assistance through the Program, by way of anecdotal evidence, reports, best practices and case studies;"²² and
The RMAF/RBAF recognized DFAIT's challenges in implementing results-based management. "As with other departments and agencies, [DFAIT] has not yet fully adopted a results-oriented management culture. Not all [DFAIT] managers have the required knowledge of the importance of RMAFs and RBAFs as a management tool to increase efficiency and effective performance measurement practices."²³

For these reasons, the RMAF/RBAF proposed three different approaches to CTCBP monitoring and evaluating: ongoing monitoring and reporting, a quality-at-entry review and periodic evaluations. The document also defines the external reporting requirements for the Program.

Monitoring and Internal Reporting

The RMAF/RBAF identified approaches to monitoring at both the project and program levels:

Project level: The use of standard financial and narrative reporting from implementing bodies to identify progress towards the achievement of the expected results. It was expected that these reports would "bring to the attention of [DFAIT] and the Interdepartmental Steering Committee any impending problems and/or need for modifications."²⁴ However, it was suggested that the Secretariat also has a role in ongoing project-level monitoring and performance measurement activities, including the coordination of such activities if they are carried by other stakeholders, such as OGD representatives or officials at Canadian missions.²⁵
Program level: "Program-level monitoring encompasses viewing and analyzing the results of comprehensive performance measurement, evaluation and audit (both recipient audit and internal audit) and taking remedial action where warranted. The Program Secretariat (ICTC) is responsible for overall program monitoring and performance measurement, and is ultimately accountable for the results of such measurements."²⁶

Not all the RMAF/RBAF provisions have been implemented as planned. The project proposals identify expected results and, based on the files reviewed for the case studies, the narrative reports provided by the implementing partners do identify results achieved. However, they are primarily at the level of output results. Implementing partners, in the case studies, identified the challenges of expecting implementing partners to measure results beyond the level of outputs (see Box 4.1).

Box 4.1: Examples of Challenges with Results Measurement from the Case Studies

CAO: Anecdotal information suggests that the training/workshop initiatives funded contributed to the development of capacity in States but no follow-up evaluation is carried out to measure longer-term impacts (including impact this may have had on states' national security programs).
FINTRAC: FINTRAC does not have the capacity to monitor long-term results. If there is an on-going relationship with a region, it is possible to obtain some anecdotal information on results, but generally not much intelligence is provided by beneficiary countries.
CANADEM: The long-term impacts of the deployment of specialists are not being measured.
CICTE: The CICTE Secretariat reportedly does not carry out evaluations to assess the short- or long-term results of its projects, including the CTCBP projects. A Canadian results-based management expert deployed to the CICTE Secretariat (and funded by CTCBP) has not been able to bring about significant changes in the way OAC/CICTE evaluates its projects. The evaluations continue to be based primarily on observations of the events and anecdotal information from participants.

The database set up by Corporate Services Division (IXS)to support the Program and record project results has not been completed nor fully populated. Results information provided by implementing partners is not being aggregated at the program level or provided to DFAIT senior management or the ISC (beyond what is provided in a proposal for follow-on funding).

The lack of resources available at the Secretariat has limited the role which it can play in project monitoring.

Quality-at-Entry Review

The RMAF/RBAF recommended that DFAIT's Evaluation and Audit Divisions undertake a quality-at-entry review within the first year of the Program's operation.²⁷ This review was undertaken and resulted in a number of recommendations that have been subsequently addressed by the Secretariat including, among others, the need to complete the staffing of the Secretariat and develop a project database.²⁸

The Secretariat did not find the results as useful as they might have been because staff felt that the review had been undertaken from the perspective of a strategy-driven development program, not a responsive program. However, some issues raised in the Quality-at-Entry review are similar to those being raised in this formative evaluation, including the need to define the role of Secretariat in project monitoring and to undertaken a more systemic assessment and compilation of project results. While these recommendations may have been supported by DFAIT at the time of the Quality-at-Entry review, delays in staffing and resource shortages at the Secretariat have delayed their implementation.

Evaluation

In addition to indentifying the requirements for this formative evaluation, a summative evaluation and subsequent periodic external program-wide evaluations, the RMAF/RBAF indicates that, with respect to project evaluations:

CTCB Program projects will be reviewed, and high-risk and/or high-value projects may be subject to evaluation. … Project proposals will indicate what, if any, evaluation function the implementing body has and if it is willing or able to manage the evaluation. If the implementing body has an independent evaluation function, [DFAIT Evaluation Division (ZIE)] may arrange with them to complete the evaluation and will provide the corresponding funding. This would apply to any participating federal entities as well as other levels of government, international organizations, international non-governmental organizations and other eligible recipients of Program funding.

If no evaluation function exists and it is deemed that a project-level evaluation is desirable, ZIE will manage the evaluation and cover the costs from funds reserved for that purpose.²⁹

Little progress has been made on the implementation of project evaluations. The evaluation team is not aware of any independent project evaluations that have been carried out by implementing organizations or by DFAIT.

At the end of fiscal year 2008/09, one Project Manager from the Secretariat undertook a mission to Southeast Asia to assess the results of three CTCBP projects implemented in the region. Given the lack of resources in the Secretariat, this is not part of a systematic approach to evaluation. On the other hand, the RMAF/RBAF did not foresee a role for the Secretariat in such evaluations.

External Reporting

The RMAF/RBAF called for, beginning in 2007, the preparation of an annual report on the Program, with particular attention to identifying Program achievements and impacts. It was expected that these reports would "satisfy external reporting requirements, and will form the basis for demonstrating public accountability for results achievement."³⁰ As with other monitoring and evaluation requirements, limited resources in the Secretariat have meant that annual reports have not been finalized, although a draft report was started in 2007, and a second drafted in 2008.

Haut de la page

5.0 Program Success and Cost-Effectiveness

The CTCBP has three objectives (see Section 2.1), which can be summarized as follows:

To support other countries and governments to prevent and respond to terrorist activities;
To increase Canada's profile and demonstrate Canadian leadership; and
To improve domestic and international coordination and cooperation.

In order to assist with the analysis of the results of the CTCBP and its projects, the evaluation team organized the short-term and intermediate outcomes of the Program (as identified in the RMAF/RBAF) under each objective (see Box 5.1).³¹

It should be noted that the objectives are not mutually exclusive - that is, a project can support more than one objective. A project to conduct training in a recipient country may also contribute to increasing Canada's profile in that country or to international coordination of training activities. In fact, it would be expected that all CTCBP projects would, by their results, contribute to the first objective; whereas projects may also, both by the way they are selected and implemented and the results they achieve, contribute to increasing Canada's profile and improving cooperation and coordination.

Based on information from the key informant interviews and the case studies, preliminary results of CTCBP projects in these areas were identified for this evaluation. Given the lack of systematic results information on CTCBP-funded projects, it is not possible to determine the extent to which the case study projects are representative of all CTCBP projects. However, they do provide a "snapshot" of Program funding and examples that illustrate how the projects contribute to the expected Program outcomes.

Box 5.1: Summary of CTCBP Expected Outcomes

1. Support to Other Countries/Government to Prevent Terrorist Activities

Short-term outcomes

Increased number of trained, knowledgeable counter-terrorism (CT) personnel who may in turn train other personnel
Progress toward new/improved CT plans, policies, legislation, regulations, controls in beneficiary state or government entity
Improved risk and threat assessment, contingency and emergency planning, information management capacity in beneficiary state or government entity
Improved and appropriate CT equipment, technologies, networks being used by beneficiary state or government entity
Increased awareness in beneficiary state or government entity of need to address counter-terrorism issues

Intermediate Outcomes

New or enhanced CT infrastructure, systems, legislation, institutions in beneficiary state or government entity that integrate human rights considerations
Increased compliance of beneficiary state or government entity with international counter-terrorism standards
Beneficiary state or government entity CT plans, procedures tested through participation in exercises, simulations, assessments

2. Increase Canada's Profile and Demonstrate Leadership

Intermediate Outcomes

Increased profile for Canada in CTCB assistance; Canadian leadership in specialized areas of expertise

3. Improve Domestic and International Coordination/ Cooperation

Short-term outcomes

Increase in contact, level of cooperation between Canadian officials and counterparts in beneficiary states or government entity

Intermediate Outcomes

Improved domestic, international coordination and cooperation in CTCB assistance
Improved international and/or regional counter-terrorism cooperation by beneficiary state or government entity

A summary table was prepared that shows how the case study project results contributed to the various evaluation objectives (see Figure 5.1). The limitations of the evaluation methodology, outlined in Section 1.3.2, illustrate the difficulty of indentifying the long-term impacts of these projects. The case study results are preliminary and are based on documents reflecting actual results achieved and/or anecdotal information from key informants on actual results.

Figure 5.1: Overview of Results of Case Study Projects

Table 4: Overview of Results of Case Study Projects
	Objectives
Project	1. Prevention of terrorist activities ¹					2. Increase Canada's profile and demonstrate leadership	3. Improve domestic and international coordination and cooperation
	Trained CT personnel	Improved CT plans, policies, legislation, regulations, controls	Improved risk and threat assessment, contingency and emergency planning, information management capacity	Improved and appropriate CT equipment, technologies, networks	Increased awareness in beneficiary state or government entity of counter-terrorism issues	2. Increase Canada's profile and demonstrate leadership
* No results discernable -- Indicates that no information, beyond what was included in the project proposal, was available to confirm that these results had been achieved. ( ) Indicates that, although the specific project did not result in the expected outcome, it contributed, through subsequent projects, to the achievement of the outcome. ¹ In order to provide a more detailed overview of the types of CTCBP projects that contribute to the objective of "prevention of terrorist activities", this objective is broken down into its contributing short-term outcomes.
1. CANADEM
Establishment of Counter-terrorism Capacity Building Expert Roster (2005-007)	*	*	*	*	*	*	*
CARICOM Security Coordinator Placement for the World Cricket Cup (2005-009)	x	x	x		x	x	x
Placement of Canadian CTCB Program Manager at OAS/CICTE (2006-083)						x	x
2. FINTRAC
Risk-based Compliance Program Tool for Foreign Financial Intelligence Units (2006-067)	(x )					x
Information Workshop (compliance, IT, legislation, analysis, and security training (2007-175)	x	x				x	x
Analysis Training Course and Open Sources Link Disk (2007-187)	x					x
International Compliance Program (2008-238)	--	--	--	--	--	--	--
3. ICAO
Funding from Passport Canada to the International Civil Aviation Organization (ICAO) Public Key Directory (PKD) (2005-020)	--	--	--	--	--	--	--
ICAO USAP Action Plan (2006-032)		x			x
ICAO/Canada Security Awareness Training Program (Phase II) (2006-039)	x	x			x	x	x
Document Security and Fraud Prevention Workshop - El Salvador, Dominican Republic, Mexico (2007-225)	x				x		x
4. JCLEC
Placement of an RCMP Member at the Jakarta Centre for Law Enforcement Cooperation (Semarang, Indonesia) (2005-028)	x	x				x	x
5. AMITA
CID Incident Database Conversion to Spanish and Provision of CT tracking and Analysis Capabilities of CBRNE incident in Colombia (2006-043)	--	--	--	--	--		--
Group of Experts on Aviation Security and Assistance (GEASSA) Information Sharing Network (2007-213)				x		x	x

5.1 Prevention of Terrorist Activities

Findings:

There is evidence that, at least through their outputs and, to some extent, their immediate outcomes, projects are contributing to the CTCBP objectives. Reviewed projects contributed to supporting other countries and governments in preventing and responding to terrorist activities, primarily through supporting training in targeted countries - notable in the Americas. Some also contributed to activities to support the development of policies, plans, procedures etc; the purchase of appropriate counter-terrorism equipment, technologies, and networks; and increasing awareness in beneficiary state or government entity of counter-terrorism issues.

There are a wide range of short-term outcomes that contribute to the objective of preventing terrorist activities:

Increasing the number of trained counter-terrorism personnel;
Improved counter-terrorism plans, policies, legislation, regulations and controls;
Improved risk and threat assessment, contingency and emergency planning, information management capacity;
Improved and appropriate counter-terrorism equipment, technologies, networks; and
Increased awareness in beneficiary state or government entity of counter-terrorism issues.

Eight of the eleven projects covered by the case studies, for which results information were known, contributed to the first short-term outcome - increasing the number of trained counter-terrorism personnel. For example, the placement of an RCMP Member at the Jakarta Centre for Law Enforcement Cooperation (JCLEC) resulted in the acquisition of skills by law enforcement officials.

Although projects covered in the case studies tended to focus primarily on the short-term outcome of training of key personnel some were also expected to contribute to other short-term outcomes - such as improving counter-terrorism policies, plans etc. and increasing awareness of counter-terrorism issues. For example:

FINTRAC provided financial intelligence training in recipient countries through the development of a risk-based compliance program for financial intelligence units and regulators, which was delivered in multiple locations. In one region, it contributed to strengthening their counter-terrorism capacity as they seek membership in the Egmont Group - a membership required for Canada to establish an MOU for sharing information;³² and
ICAO Security Awareness Training Program - This project included workshops, courses and a seminar in Latin America and Caribbean to help states meet international aviation security standards. They increased participants' knowledge/understanding of ICAO's policies and the states' international obligations. They aimed to contribute to the development of national security programs in the member states.

However, it is noticeable in these two cases that, although the training was expected to contribute to other short-term outcomes (e.g. meeting international security standards or strengthening national security systems), the CTCBP does not appear to have provided any support to contribute specifically to these outcomes. The assumption appears to have been made the training would, in and of itself, contribute also to the other Program outcomes.

On the other hand, CTCBP funding for ICAO targeted directly other short-term outcomes besides increasing the number of trained personnel. CTCBP provided Canada's contribution to the implementation of ICAO's Universal Security Audit Program (USAP). Although no specific results are available for CTCBP's contribution, since the funding is pooled with that of other countries, the results of the USAP overall indicated that there has been a "markedly increased level of implementation of ICAO security Standards."³³

Some training projects supported "training of trainers", or the replication of the training activities for others - for example:

The ICAO Security Awareness training focused on training instructors to carry out aviation security training in their home country; and
Some students in the JLEC training, over the project's two-year period, have gone on to train other law enforcement officers in the region.

Perhaps the most comprehensive project in terms of addressing multiple short-term outcomes was the project to fund the one-year deployment of a Canadian expert to work with multiple Caribbean states to identify counter-terrorism needs prior to the Cricket World Cup (CWC). In addition to supporting specific training activities, the person deployed also contributed to the improvement of the overall infrastructure for counter-terrorism.³⁴ He was involved in advising on a range of activities:

Development of national and regional security plans;
Establishment of the legal and administrative frameworks for counter-terrorism strategies;
Development of policies and MOUs for sharing information among beneficiary states and joint action;
Establishment of agreed-upon levels of counter-terrorism alerts;
Development of a database of critical infrastructure with protection plans, response plans, recovery plans, command structures etc.; and
Establishment of an International Support and Advisory Group.

While there has been no measurement of the long-term impacts of this project, there is apparently anecdotal evidence of the sustainability of some results, including the ongoing operations of the Caribbean Regional Operations Center, the establishment of a "one visa region" for all Caribbean states, legislative changes that contributed to increased sharing of information and resources, and the establishment of the Advanced Passenger Information System in the region.

In addition to the capacity building benefits in recipient countries, a couple of key informants indicated that there are also capacity building benefits for the Canadians who participate. For example, as a result of the development and implementation of border enforcement training for CWC in the Caribbean, the RCMP now has regional trainers who have been used again in other training activities.

In summary, all but two of the projects that could be assessed contributed directly or indirectly to one or more of the short-term outcomes associated with preventing terrorist activities. However, as was noted in Section 4.3.3, implementing partners have limited mechanisms or plans to follow-up on these achievements and assess the longer-term results. Given this lack of follow-up, there is, and will continue to be, limited information on the achievement of intermediate and longer-term outcomes.

5.2 Increase Canada's Profile and Demonstrate Leadership

Finding:

Almost all projects reviewed contributed, at least in the short-term, to increasing Canada's profile and demonstrating Canadian leadership, either as a direct result of funding and implementing projects with partner organizations or through supporting Canada's place in international fora dealing with counter-terrorism issues.

A large majority of the projects covered in the cases studies, for which results are known, appear to have contributed to increasing Canada's profile and demonstrating Canadian leadership. In some cases, this appears to be the case because the provision of the funding alone contributes to Canada's profile - for example:

The deployment of a security expert to Caribbean prior to the CWC - The expert participated in international meetings and made presentations to senior Caribbean officials, which reportedly contributed to increasing Canada's profile in the security community in the Caribbean;
Training activities carried out by FINTRAC in the Middle East and Africa have contributed to increasing the profile of Canada's capacity building activities, as reflected in FINTRAC receiving further requests for training; and
The use of Canadian instructors, provided by Transport Canada, for the ICAO Security Awareness Training Program has raised the profile of Canada in the region with respect to counter-terrorism related issues.

One initiative that at the start appeared to have the potential to contribute to Canada's profile in the counter-terrorism community was the project funded with CANADEM to establish, populate and promote a roster of security experts (CAN-Secur). However, after funding two phases of the project, DFAIT decided to not continue its support of the database in the form of provision of an operating contribution. If, through the CTCBP funding for the database, CANADEM was able to identify security experts, whose names are not available elsewhere, then this database would have been a good source of Canadian expertise for counter-terrorism activities. However, by stopping the funding after two years, these initial potential benefits have not been realized. As a result, it would appear that this project funding did not represent a good initial investment of CTCBP funds or make the expected contribution to Canada's profile in this community.

According to key informant interviews, CTCBP funding gives Canada a profile that belies the size of the Program. It supports Canada's place at the table in international fora, such as the Counter-Terrorism Action Group (CTAG) set up by the G8 to support the UN Security Council Counter-Terrorism Committee (CTC), by coordinating counter-terrorism capacity building assistance and engaging in outreach to other states.

Key informants also indicated that the implementation of a CTCBP-funded project in a given country contributes to enhancing the profile of Canada's mission in that country. One key informant noted that being associated with CTCBP funded training open doors, by increasing the networks and buying goodwill for posted Liaison Officers. Similarly, training provided with the participation of Canadian federal departments increases the profile and credibility of that department in the counter-terrorism community. For example, CTCBP funding has reportedly increased awareness of counter-terrorism issues among Transport Canada's marine security partners and increased its credibility with its partners.

A particular aspect of Canada's profile with respect to counter-terrorism capacity building is that some countries reportedly prefer to work with Canada, rather than the United States, because Canada is perceived to be more open to working with recipient countries to define their needs, rather than setting its own expectations. On the other hand, the size of Canada's program is such that it cannot provide the breadth of support that other, larger, allies (such as the US or the United Kingdom) can.

5.3 Improve Domestic and International Coordination and Cooperation

Finding:

The CTCBP contributes to improving coordination and cooperation through funding multilateral organizations that have a coordination mandate, implementing regional activities that promote coordination at the regional level, building on networks with recipient countries and regions, and promoting domestic cooperation through the Program's governance mechanisms.

The Program appears to contribute in a number of ways to improving domestic and international coordination and cooperation:

International cooperation through support for multilateral organizations that have a coordination mandate;
Regional coordination and cooperation through countries in which projects are implemented;
Strengthening Canada's contracts and networks with other countries and/or regions; and
Domestic coordination/cooperation through the Program's governance mechanisms.

International Cooperation

According to the data shown in Section 2.4.2, CTCBP funds between 21% and 36% of its projects with multilateral organizations that have mandates for coordination of counter-terrorism activities. The two largest beneficiary organizations were:

The United Nations Office on Drugs and Crime (UNODC) was established in 1997 to assist UN member states in the fight against illicit drugs, crime and terrorism; and OAS/CICTE, which is a small Secretariat that works with countries in the hemisphere to implement technical assistance projects in the region.

The support provided to these organizations contributes to implementation of their existing mandates for counter-terrorism capacity building and, as a result, supports a more coordinated international contribution.

Regional Coordination and Cooperation

Given that often CTCBP-funded training is provided at the regional level, the training also contributed to improving regional coordination of counter-terrorism activities - for example:

The OAS/ICAO Security Awareness Training has reportedly contributed to increased cross-cultural awareness and strengthened communication and networking on security issues and the development of a more regional approach; and
JCLEC, which hosted an RCMP member posted under a CTCBP project, trains 1000 students every year, some of whom are from other Southeast Asian countries. The training has reportedly resulted in increased cooperation among law enforcement agencies within the region.

Canada's Contacts and Networks

A number of the projects covered by the case studies reflected specifically a contribution to increasing Canada's contacts and networks in the counter-terrorism capacity building community - for example:

As a result of the training provided through the JCLEC, the RCMP has created a strong network with officials in Indonesia and other Southeast Asian countries and with officers from other donor countries;
As result of the funding for the GEASSA project, the level of cooperation between Canadian officials and their counterparts in Columbia has reportedly increased; and
Through the delivery of the CWC project in the Caribbean and participation in regional meetings, the security expert increased Canada's network of contacts with security personnel in the region.

Key informant interviews also confirmed that participation in CTCBP-funded projects has contributed to strengthening Canada's contacts with groups such as the Group of Experts on Aviation Safety, Security and Assistance (GEASSA).

Domestic Coordination

Some OGD representatives indicated that their participation in the CTCBP, and particularly the ISC, contributed to expanding their contacts within the federal government. However, this was not the case for all departments, as many already had such networks in place - often with different participants than those associated with the Program. Some ISC participants also noted that the ISC, particularly since the launch of the practice of having OGDs host the meetings in 2008/09, provides an opportunity to learn more about the counter-terrorism capacity building activities of other departments and agencies.

More concrete examples of improved domestic coordination were provided by OGD representatives who indicated that, as a result of the CTCBP, they are now engaging in more joint programming - for example, Transport Canada and the CBSA, and the RCMP and CBSA.

5.4 Sustainability of CTCBP Outcomes

Finding:

The CTCBP principles emphasize the importance of sustainability, but it is too early to assess the sustainability of the project results. Many projects tend to focus primarily on personnel training, with limited activities to support directly other short-term outcomes. This does not encourage sustainability. On the other hand, there are synergies among projects, projects that support regional or global activities and projects that support ongoing capacity building activities and these do support sustainability.

The lack of information on longer-term outcomes of the CTCBP-funded projects means that it is not possible to assess the sustainability of the funded activities. However, the Program's principles include an emphasis on sustainability:

To the extent possible, ISC members will develop long-term, multi-year approaches to the provision of technical assistance and training to identified priority states. A "train-the-trainer" approach will be adopted where feasible and appropriate to ensure that learning and knowledge are adapted, owned and disseminated by the beneficiary state officials.³⁵

In addition, the Program emphasizes the importance of adapting CTCBP assistance to the needs of beneficiary states and government entities and taking a "whole-of-government" approach to assistance. This latter principle encourages departments to work together and ensure that efforts are complementary, in order to increase the likelihood that results are sustainable and Canada's profile is raised.

From the review of the case study projects, it is possible to identify a number of factors that either contribute to, or potentially detract from, the sustainability of outcomes. The major factor that would detract from sustainability is the fact that some projects (at least based on the case studies) focus primarily on training. This is a common challenge for capacity building programs. It is difficult for the impacts of training to be sustainable, given the usually frequent turnover of staff and the inability of programs to continue to provide training as new staff arrive. Similarly, unless capacity building programs support the development of counter-terrorism infrastructure, people trained will not be supported to implement what they learned in the training.

There were, however, a number of factors present in the case study projects that may, in the long-term, contribute to the sustainability of CTCBP project results:

Shifting the emphasis on programming from needs assessments to the delivery of capacity building activities. This was a shift which reportedly occurred after the first couple of years of the Program;
Providing support that build on existing organizations and partnership increases the chances that there will be long-term impacts. For example, a number of the projects funded with ICAO build on the partnership that already exists between Transport Canada and ICAO and allows the international organization to continue to implement activities with its Canadian partners. Similarly, funding projects that are implemented in collaboration with international partners, such as Australia, potentially strengthens the long-term impacts;
Providing support for "training of trainers", as opposed to "one-off" training increases the chances that the benefits of the training will be expanded and sustained. It is not possible to know to what extent the CTCBP-funded training programs were "training of trainers", but examples of this approach were reflected in the case studies and it is reportedly a key factor in the assessment of projects;
Providing support for more than one project in support of a common objective, including activities that go beyond training, increases the chances that a critical mass of activities will be implemented and lead to successful, sustainable results. For example, the funding of a package of projects to support security for the CWC increases the chances that the impacts will be sustainable (see Box 5.2);
Providing support for regional or global activities (as opposed to activities at the country-level) strengthens the chances that the benefits will be spread and sustained; and
Providing support for ongoing capacity building activities being undertaken by existing organizations provides a better opportunity to ensure the training will continue to be provided. For example, the provision of support for training as part of the ICAO's Universal Security Awareness Program, which is also supported by other countries, increases the chances that the support will be available in the longer-term.

Box 5.2: CTCBP-funded Projects to Support the Cricket World Cup (2007)

In addition to funding the placement of a security specialist at the Secretariat of the Caribbean Community (CARICOM) to advise on a wide range of security aspects prior to the CWC, the CTCBP funded border integrity training and training to detect fraudulent travel documents. Equipment and training was provided for trace detection technology for narcotics and explosives; truth verification (polygraph) units to test members of intelligence units, drug enforcement units, special branches and similar specialized entities; and regional epidemic surveillance to detect possible bioterrorist incidents.

Canadian Aid for the Regional Security Unit of the Cricket World Cup (CANADEM; $288,000)
Delivery of Border Security Training to the CARICOM Region (Public Safety and Emergency Preparedness Canada - RCMP and CBSA; $606,540)
INTERPOL Travel Document Database Extension in the Caribbean (INTERPOL; $250,000)
Permanent Allocation of Trace Detection Technology for Narcotics and Explosives for Customs Administrations (Canada Border Services Agency; $805,500 over two years)
Enhanced disease surveillance for the Cricket World Cup 2007 (Caribbean Epidemiology Centre under auspices of Pan-American Health Organization; $907.136 over two years)
CARICOM Truth Verification Program - Regional Polygraph Service (Royal Canadian Mounted Police; $93,062)

Source: CTCBP Secretariat

Note: The information comes from project proposals and is not confirmed by an assessment or project results.

Based on this analysis, it appears that a number of the CTCBP project funding decisions are supportive of the sustainability of project results. However, only a long-term evaluation, with a more in-depth assessment of project results, will be able to assess the extent to which the results have, in fact, been sustainable.

Haut de la page

6.0 Conclusions and Recommendations

This section presents the conclusions of the evaluation and the recommendations that flow from these conclusions, with respect to the three key evaluation issues.

6.1 Relevance

The need to contribute to the security of Canadians and inhabitants of beneficiary states and governments is still present. Canada and other countries continue to be affected by terrorist activities and key stakeholders reflected that capacity building with recipient states and governments continues to be an important mechanism to address those threats. Canada's active involvement in multilateral and bilateral relations with other partners to meet international counter-terrorism commitments and the objective of increasing Canada's profile in counter-terrorism capacity building remain relevant. This is particularly the case as Canada assumes greater leadership in some international fora. Given the involvement of many federal department/agencies in counter-terrorism capacity building activities and the engagement of the international community is this area, activities to improve both domestic and international coordination remain valid.

No recommendations are made with respect to the relevance of the CTCBP.

6.2 Governance, Design and Delivery

Both the strengths of the CTCBP and the challenges in its governance lie in its interdepartmental nature. As an interdepartmental initiative, it draws on the wide-ranging strengths and experiences of sixteen federal government departments and agencies, which serve as sponsoring organizations or implementing partners. The governance structure - including the ISC, the OSC and the Secretariat - is based on the assumption that all departments/agencies will participate equally in the governance of the Program. However, the diversity in the mandates, experiences and resources of the participating departments and agencies means that it is challenging to have collective, informed decision-making for CTCBP projects. It is difficult to engage the departments/agencies in the identification of strategic priorities and to mesh DFAIT's priorities with those of other departments/agencies that are actively engaged in counter-terrorism activities. As a result, it appears that the Secretariat and, in a wider context, DFAIT plays a very prominent role in the identification of strategic priorities and the approval of projects.

The evaluation team recommends that:

Recommendation 1:

Recommendation 2:

Recommendation 3:

The ISC should better integrate Government and OGD priorities and intelligence on terrorism threats.

One way in which the Program could strengthen the governance would be to reduce the size of the ISC, making it a more strategic body, composed of experts who would participate as experts in the counter-terrorism field, rather than as representatives of their individual departments. The ISC would call upon those individuals who have information and analytic capabilities in the area of counter-terrorism and consideration might be given to including external stakeholders for the counter-terrorism field.

The overall Program design - beyond the governance issues already discussed - is considered to still be relevant. Stakeholders indicate that it is still relevant for the CTCBP to be an interdepartmental program, with resources held in a central pot and the Program administered by a central body. Stakeholders also support the approach of having the CTCBP as a responsive program, with parameters defining what can be funded.

The CTCBP Secretariat housed at DFAIT is recognized by stakeholders for the key role that it has played in supporting the implementation of the Program, primarily by supporting the development of projects and the ISC. However, there have been gaps in the infrastructure for the implementation of the Program.

The ongoing challenges with Program implementation stem primarily from two causes: the delays in setting up the Secretariat and the lack of project management capacity in some implementing organizations (notably OGDs, but also present to some extent in partner organizations, such as the OAS CICTE). Set-up of the Secretariat began in 2005/06, but the full staffing has only recently been completed. As a result, a number of the components of the Program infrastructure were slow to be set up or are still not yet in place - notably the development of defined standard operating procedures, and a database for tracking CTCBP projects. The lack of project management capacity in OGDs, combined with the challenges of working with recipient governments and country that lack capacity themselves, has contributed to recurring program funding lapses resulting in a $2 million annual cut to the CTCBP in 2008/09.

The evaluation team recommends that:

Recommendation 4:

DFAIT develop standard project management tools and training/coaching to support OGDs in project implementation, taking into consideration the availability of resources in the Secretariat.

Recommendation 5:

The ISC explore the possibility of providing multi-year project funding where unspent funds could be rolled over to the next fiscal year.

In addition to the challenges with program implementation, there are significant gaps in the implementation of the framework for monitoring and evaluating Program results, which meant that there was limited administrative data on CTCBP projects for analysis in this evaluation and, if the situation continues, will limit the data available for the summative evaluation.

The evaluation team recommends that:

Recommendation 6:

Recommendation 7:

6.3 Program Success and Cost-effectiveness

All but two projects reviewed appear to have contributed to the primary objective of supporting other countries and governments to prevent and respond to terrorist activities. They have done this primarily through supporting training in targeted countries - notable in the Americas. Some have also contributed to activities to support other components of capacity building, including the development of policies, plans, procedures etc; the purchase of appropriate counter-terrorism equipment, technologies, and networks; and increasing awareness in beneficiary state or government entity of counter-terrorism issues. However, only one project reviewed in the case studies included specific activities to support these other components of capacity building. A primary focus on training as the mechanism for capacity building put the sustainability of the capacity building results at some jeopardy.

While it is too early to assess the sustainability of the CTCBP project results, there are factors that will likely contribute to, and detract from, the sustainability. The case studies suggest that many projects tend to focus primarily on the training of personnel, with limited activities to support directly other short-term outcomes. On the other hand, the case studies also reflected other factors that would contribute to sustainability, including synergies among projects, support of regional or global activities and supports for ongoing capacity building activities being implemented by other organizations.

Recommendation 8:

Haut de la page

7.0 Management Response and Action Plan

Table 5: Management Response and Action Plan
Recommendations	Commitments/Action	Responsibility Centre	Target Completion Date
Recommendation #1 The Interdepartmental Steering Committee (ISC) should strengthen the CTCBP governance to make it more effective providing broader oversight of decision-making and ensure that it maximizes the value added of the Other Government Departments (OGDs) with respect to their role in CTCB decision-making. Recommendation #2 The ISC should build on other governance mechanisms (e.g. Online Steering Committee (OSC) and the Secretariat) in order to maximize added value that OGDs and DFAIT missions abroad in their role as technical experts, and in project delivery.	Recommendations #1 and #2 are seen to be linked and therefore our response addresses the two together for a more efficient action plan. Response: The CTCBP concurs with the evaluation recommendations. Commitment: The CTCBP Secretariat is currently preparing a Treasury Board (TB) Submission requesting renewal of the CTCBP and its Terms and Conditions, which expire at the end of FY 2009-10. As part of this process, core Program documents, including the Terms and Conditions, Audit, Risk and Accountability Framework (ARAF) and Terms of Reference for the Governance Structure are being revised based on lessons learned, best practices, and the recommendations contained in the formative evaluation report. In response to recommendations #1 and #2, the revised core Program documents outline a stronger and more streamlined Program governance structure. The proposed structure would maintain Program oversight by the Senior Executive responsible for Counter-Terrorism Programming (currently ICX) and overall program/project management by a Secretariat housed at DFAIT. Rather than maintaining a single Interdepartmental Steering Committee, however, the proposed structure would clearly separate the responsibilities of a Director-level Interdepartmental Steering Committee from a Deputy Director-level Interdepartmental (Project) Review Committee. The Steering Committee would focus on strategic matters, while the Review Committee would focus on operational aspects of program implementation and delivery. Invitations to meetings and circulation of information and documentation regarding specific projects would be extended to experts and observers with knowledge and expertise pertinent to the subject matter (e.g. DFAIT missions abroad). Draft Terms of Reference for the Committees have been prepared and, if approved, would be reviewed periodically and updated if necessary to improve on the effectiveness and efficiency of the Program's governance structure. In addition, the CTCBP Secretariat will design and implement a targeted outreach strategy (intra-DFAIT, OGDs, international institutions), a key expected outcome of which is to increase participation of OGDs and DFAIT missions in the CTCBP as technical experts and/or project delivery partners. These actions are expected to strengthen the CTCBP governance structure and maximize the added value of OGDs and DFAIT missions abroad.
	Action Plan:
	FY 2009-10
	1. Complete intradepartmental consultations on TB Submission and related documents, including Terms and Conditions, Terms of Reference for Governance Structure and ARAF	1-6: ICTC Chief, ICX	1. Quarter 2
	2. Complete interdepartmental consultations on TB Submission and related documents		2. Quarter 2
	3. Develop targeted outreach strategy		3. Quarter 2
	4. Finalise TB Submission and related documents		4. Quarter 3
	5. Begin implementation of targeted outreach strategy		5. Quarter 3
	6. Strike Steering and Review Committees		6. Quarter 3
	FY 2010-11
	7. Undertake year-end review of governance and management structures and integrate lessons learned	7. ICTC Chief, ICX, Steering Committee, Review Committe	7. Quarter 3 (or one year after implementation)
Recommendation #3 The ISC should better integrate Government and OGD priorities and intelligence on terrorism threats.	Response: The CTCBP concurs with the evaluation recommendation, and will continue to closely align projects with existing and emerging Government of Canada and OGD geographic (i.e. Americas, Afghanistan and Pakistan, Asia and Africa) as well as investment (i.e. Transportation Security, Financing of Terrorism) priorities. Commitment: Under the proposed governance structure, the Steering Committee would become a truly strategic body. Comprised of departments with a direct mandate for addressing international terrorism, the Steering Committee would be responsible for planning and conducting a Strategic Planning Exercise focused on leveraging scare resources, avoiding duplication of efforts domestically and internationally, aligning CTCB programming with Government and OGD priorities, integrating intelligence on terrorism threats into the planning process, and ensuring an appropriate balance of projects across CTCBP short-term outcomes and investment areas. Complementing the work of the Steering Committee, the CTCBP Secretariat will design and implement a targeted outreach strategy (intra-DFAIT, OGDs, international institutions), a key expected outcome of which is increased integration of Government and OGD priorities and intelligence on terrorism threats. These actions are expected to strengthen the CTCBP's responsiveness to intelligence on terrorism threats and OGD and government priorities and ensure an appropriate balance across CTCBP short-term outcomes and investment priority areas.
	Action Plan:
	FY 2009-10
	1. Develop targeted outreach strategy	1-2: ICTC Chief, ICX	1. Quarter 2
	2. Begin implementation of targeted outreach strategy	1-2: ICTC Chief, ICX	2. Quarter 3
	3. Design Strategic Planning Exercise	3-4: ICTC Chief, ICX, Steering Committee	3. Quarter 4
	FY 2010-11
	4. Implement Strategic Planning Exercise		4. Quarter 1
Recommendation #4 DFAIT develop standard project management tools and training/coaching to support OGDs in project implementation, taking into consideration the availability of resources in the Secretariat.	Response: The CTCBP concurs with the evaluation recommendation. Commitment: The CTCBP is in the process of developing standard project management tools and training/coaching materials and workshops to support OGDs in project implementation. For example, the Program is re-designing templates and instructions for OGDs in an effort to make them more user-friendly, organising a project management workshop for OGDs in the Fall 2009 and planning a series of "initial visits" aimed at providing coaching/training to support OGDs and other implementing bodies at the start of project implementation.
	Action Plan:
	FY 2009-10
	1. Finalize initial visit plan for 2009-10	1-3: ICTC Chief	1. Quarter 2
	2. Develop methodology for initial visits with a narrative and financial reporting component		2. Quarter 2
	3. Deliver training workshop(s) and/or guidance material to ODG implementing bodies on aspects of project management		3. Quarter 3
	4. Develop Standard Operating Procedures for CTCBP	4. ICTC Chief	4. Quarter 4
	5. Revise templates and instructions for implementing bodies	5. ICTC Chief	5. Quarter 4
Recommendation #5 The ISC explore the possibility of providing multi-year project funding where unspent funds could be rolled over to the next fiscal year.	Response: The CTCBP accepts the evaluation recommendation. Commitment: The CTCBP will explore the possibility of providing multi-year project funding where unspent funds could be rolled over to the next fiscal year. For transfers to OGDs, options could include identifying funds to be re-profiled via the Annual Reference Level Update (ARLU) process. For Grants and Contributions, the CTCBP will be governed by the new Transfer Payment Policy.
	Action Plan:
	FY 2009-10
	1. Consult with IAM, IXS, JUS, SMFH as part of internal TB Submission consultations	1-2: ICTC Chief	1. Quarter 2
	2. Develop options and provide recommendation to Senior Executive responsible for CTCBP	1-2: ICTC Chief	2. Quarter 2-3
Recommendation #6 DFAIT indentify the necessary steps and resources requirements to finalize and populate the database for CTCBP projects that will allow the Secretariat to develop adequate profiles and track the results of CTCBP project.	Response and Commitment: The CTCBP concurs with the evaluation recommendation.
	Action Plan:
	FY 2009-10
	1. Summer co-op student to populate database with project-level narrative information	1. ICTC Chief	1. Quarter 2
	2. Identify the necessary steps and resource requirements to finalise and populate the database	2. ICTC Chief	2. Quarter 3
	3. Populate database with project-level financial information	3. ICTC Chief, ICTC Finance Manager	3. Quarter 4 with additional resources/ Quarter 1 FY 2010-11 without
	4. Request authority from TB, as part of Treasury Board Submission requesting renewal of the CTCBP and its related Terms and Conditions, to create a new position to provide ongoing database support to a) input into the database b) field requests for information c) track results of CTCBP projects and d) generate reports	4. ICTC Chief, ICX	4. Quarter 2
	FY 2010-11
	5. Implement necessary steps to align database with revised ARAF and reporting templates (contingent on available resources)	5. ICTC	5. Quarter 2
Recommendation #7 DFAIT strengthen Secretariat's capacity to monitor both the implementation (activities and outputs) and outcomes (results) of the projects and the program as a whole and develop a framework for sharing outcome information.	Response: The CTCBP concurs with the evaluation recommendation. Commitment: At a program level, a revised ARAF is being developed for the CTCBP, which includes performance measurements strategies designed to assess the degree to which the desired results are being achieved and provide the ongoing information needed to support evaluation of the program as a whole (e.g. evaluations by CTCBP theme, region, or country). The ARAF will document the Program's activities, outputs and outcomes. The CTCBP Secretariat will produce an annual report for the Program's Committees summarising overall Program performance. At a project level CTCBP management is committed to undertaking initial visits, project monitoring missions and recipient audits as the start of a systematic approach to monitoring and evaluation. Initial visits/conference calls will be conducted when possible to provide coaching/training to support OGDs and other implementing bodies at the start of project implementation. Project monitoring will be conducted on an ongoing basis to confirm that approved projects are conducted and that the planned outputs are produced. The CTCBP Secretariat is responsible for ongoing project-level monitoring and the coordination of such when undertaken by other stakeholders or independent assessors. In doing so, they work in close collaboration with DFAIT divisions responsible for audit and evaluation. In addition to the review of Project Reports submitted by the implementing bodies/recipients, the CTCBP will also conduct visits (monitoring missions) with some recipients to look at various project performance elements such as project design, strengths and weaknesses of implementation, lessons learned, beneficiary feedback and risk mitigation. The CTCBP will also work with the Program Services Division on ways in which to use corporate systems for planning and reporting on outcomes of these monitoring activities to optimize program performance and risk management. These actions are expected to strengthen the CTCBP Secretariat's capacity to monitor both the implementation (activities and outputs) and outcomes (results) of the projects and the program as a whole.
	Action Plan:
	i) Program Level:
	FY 2009-10
	1. Finalise ARAF	1-3: ICTC Chief, ICX	1. Quarter 2
	2. Develop annual report framework		2. Quarter 3
	FY 2010-11		3. Quarter 2
	3. Complete 2009-10 annual report
	ii) Project-level:
	FY 2009-10
	4. Finalise audit and monitoring plan for 2009-10	4. ICTC Chief, ZIV	4. Quarter 2
	5. Develop methodology for monitoring visits	5. ICTC	5. Quarter 2
Recommendation #8 That DFAIT engage in a senior level discussion with OGD management and the ISC on the strategic focus of the program, the need (if any) to redress the balance of projects across different CTCBP short-term outcomes and investment areas and the support needed for OGD project implementation.	Response: The CTCBP agrees with the recommendation. Action Plan: The actions identified in response to Recommendations #1 and #2 (value-added of OGDs and their role in decision-making), #3 (integrate priorities) and #4 (OGD project implementation) will address this recommendation.	See response to recommendations #1, #2, #3 and #4	See response to #1, #2, #3 and #4

Haut de la page

Footnotes:

¹ Australia was also selected for a case study but representatives were not able to participate.

² Classified projects were not included in this evaluation.

³ Counter-Terrorism Capacity Building Assistance as of March 2009

⁴ Ibid

⁵ Counter-Terrorism Capacity Building Assistance as of March 2009

⁶ "Integrated Results-based Management and Accountability Framework and Risk-based Audit Framework (RMAF/RBAF), Counter-terrorism Capacity Building Program", Department of Foreign Affairs, International Crime and Terrorism Division, September 26, 2005, p. 10

⁷ As noted in the limitations of the evaluation methodology, there is some inaccuracy in the data. For example, for lack of other information, the number of projects funded in a given year is based on the project number assigned to the project, which includes the year (e.g. 2005-036). As a result, these numbers indicate that fourteen projects were funded in the calendar year 2005, yet the financial reports do not shown any program expenditures in 2005/06. Data was analyzed for 2005 to 2007 because, at the time of the analysis, 2008 data covered only part of the year.

⁸ For 2 February 2004, see Welcome to the Privy Council Office. For 4 April 2006, see Welcome to the Privy Council Office. For 2007, see CTV news

⁹ See "G8 Leaders Statement on Counter-terrorism", 8 July 2008 G8 Research Group

¹⁰ "Interdepartmental Steering Committee of the Counter-Terrorism Capacity Building Program: Terms of Reference", p. 10

¹¹ "Follow-Up Capacity Check Report, Counter-Terrorism Capacity Building Program", Audit Division (ZIV) Office of the Inspector General, Department Of Foreign Affairs and International Trade, Draft 1.1, May 23, 2007, p. 10

¹² "Integrated Results-based Management and Accountability Framework and Risk-based Audit Framework (RMAF/RBAF), Counter-terrorism Capacity Building Program", p. 10. In addition, as noted in the RMAF/RBAF, the ISC has periodic responsibility for approving the terms of reference for the Program evaluations.

¹³ "Counter-Terrorism Capacity Building Program, Presentation to the Policy and Program Board, October 21, 2008", Mark Gwozdecky, Senior Coordinator, International Crime and Terrorism Bureau (ICX), Slide 10.

¹⁴ "Counter-Terrorism Capacity Building Program, Presentation to the Policy and Program Board, October 21, 2008", Mark Gwozdecky, Senior Coordinator, International Crime and Terrorism Bureau (ICX), Slide 10.

¹⁵ This illustrates the challenge for the ISC in the lack of continuity of attendance.

¹⁶ "Integrated Results-based Management and Accountability Framework and Risk-based Audit Framework (RMAF/RBAF), Counter-terrorism Capacity Building Program", p. 10

¹⁷ Based on key informant interviews and ISC minutes, 13 December 2007

¹⁸ "Follow-Up Capacity Check Report, Counter-Terrorism Capacity Building Program", Audit Division (ZIV) Office of the Inspector General, Department Of Foreign Affairs and International Trade, Draft 1.1, May 23, 2007, p. 16

¹⁹ Counter-Terrorism Capacity Building Assistance The site promotes the CANADEM security expert roster. While the link to the CANDADEM roster site is still valid, the roster is no longer being actively promoted by CANADEM (see Section 5.2).

²⁰ For example, the IMO publically acknowledged the CTCBP contribution to regional seminars and workshops on maritime security in the Caribbean. See Canada supports IMO maritime security activities Similarly the Centre on Global Terrorism Cooperation recognized Canada's CTCBP contributions in a paper on "Implementing the Global Counter-terrorism Strategy in the Americas", Discussion Paper, New York, NY, 31 March 2008. p. 19 See Implementing the UN Global Counter-Terrorism Strategy in the Americas (PDF version, 318 kb)

²¹ "Integrated Results-based Management and Accountability Framework and Risk-based Audit Framework (RMAF/RBAF), Counter-terrorism Capacity Building Program"

²² Ibid, p. 13

²³ Ibid, p. 35

²⁴ Ibid, p. 44

²⁵ Ibid, p. 11 and p. 31

²⁶ Ibid, p. 31

²⁷ Ibid, p. 35

²⁸ See "Follow-Up Capacity Check Report, Counter-Terrorism Capacity Building Program"

²⁹ Ibid, p. 41

³⁰ Ibid, p. 44

³¹ Ibid, p. 15

³² The Egmont Group is a global network of 108 financial intelligence units (FIUs) responsible for following the money trail in order to counter money laundering and terrorism financing. About the Egmont Group

³³ "Report on the ICAO Universal Security Audit Programme", Agenda Item 16: Universal Security Audit Programme (USAP), International Civil Aviation Organization, Working Paper, Assembly - 36th Session

³⁴ The term "infrastructure" is used here to describe, not physical infrastructures, but rather other components of counter-terrorism capacity building including, for example, the development of counter-terrorism plans and policies, an appropriate legislative and regulatory framework for counter-terrorism activities, establishment of networks, the acquisition of necessary equipment and technologies to implement counter-terrorism activities etc.

³⁵ "Interdepartmental Steering Committee Counter-terrorism Capacity building Program: Terms of Reference", p. 3

Office of the Inspector General

^* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Primary navigation

About the Department

Footer

Date Modified:: 2013-01-02

Top of Page

Important Notices

Foreign Affairs and International Trade Canada

Common menu bar

Breadcrumb

Formative Evaluation of the Counter-Terrorism Capacity Building Program

Table of Contents

Executive Summary

Introduction

Program Overview

Key Findings

Relevance

Governance design and delivery

Program success and cost-effectiveness

Recommendations

Recommendation 1:

Recommendation 2:

Recommendation 3:

Recommendation 4:

Recommendation 5:

Recommendation 6:

Recommendation 7:

Recommendation 8:

Acronyms

1.0 Introduction

1.1 Evaluation Issues

1.2 Evaluation Methodology

1.1: CTCBP Formative Evaluation Issues

Relevance

Governance Structure, Program Design and Delivery

Success

Cost-effectiveness

1.2.1 Evaluation Lines of Evidence

Document, file and administrative data review

Key Informant Interviews

Case Studies

Country Comparative Analysis

1.2.2 Challenges and Limitations

Incomplete Administrative Data

Lack of Information from Country Comparative Analysis

Measurement of Long-term Outcomes

1.2.3 Interpretation of Data

1.3 Report Outline

2.0 Description of The Counter-Terrorism Capacity Building Program

2.1 Objectives

2.2 Governance

Interdepartmental Steering Committee

CTCBP Secretariat

Online Screening Committee

2.3 Resources

Financial Resources

Figure 2.1: CTCBP Financial Profile

Human Resources

2.4 CTCBP Projects

2.4.1 Disposition of Project Proposals

Figure 2.2: Status of CTCBP Projects, 2005 to mid-2008

2.4.2 Profile of Projects

Project Implementing Bodies

Figure 2.3: Percentage of funded projects, by type of implementing body, 2005 - 2007

Sponsoring Bodies

Figure 2.4: Number of projects, by year and by sponsoring organization, 2005 - 2007

Figure 2.5: Dollar value of projects, by sponsoring organization, 2005 - 2007 (in thousands of dollars)

Figure 2.6: Project Values, by Department/Agency, 2005 - 2007 (in thousands of dollars)

Investment Areas

Figure 2.7:Number of projects, by year and investment area, 2005 - 2007

Figure 2.8: Value of projects, by investment area, 2005 - 2007 ( in thousands of dollars)

3.0 CTCBP Relevance

3.1 Relevance to the Government of Canada

Finding:

3.2 Relevance to International Partners

Finding:

Box 3.1: US Anti-Terrorism Assistance (ATA) Program

Box 3.2: Swiss Counter-terrorism Capacity Building Support

4.0 Governance, Design and Delivery

4.1 Governance

Findings:

4.1.1 Senior Coordinator, International Crime and Terrorism

4.1.2 Interdepartmental Steering Committee

4.1.3 Secretariat

4.1.4 Online Screening Committee

4.2 Program Design

Finding: