Foreign Affairs, Trade and Development Canada
Symbol of the Government of Canada

Foreign Affairs, Trade and Development Canada

international.gc.ca

Archived Document

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated since it was archived. Web pages that are archived on the Web are not subject to Government of Canada Web Standards; as per the Communications Policy of the Government of Canada, you can request alternate formats by contacting us.

Canadian Consortium on Human Security

(July 2007)

(PDF Version, 1.01 MB)  *

Table of Contents


Top of page

Executive Summary

Background:

Established in December 2001, the Canadian Consortium on Human Security (CCHS) is an academic-based network supporting policy-relevant research on human security. CCHS's primary mission is to facilitate the exchange of information, analysis and dialogue on human security issues between non-governmental organizations (NGOs), academics, research institutes, "think tanks" (domestic and international), government and trans-national organizations involved in human security studies and policy development. The CCHS is funded by the Human Security Program of Canada's Department of Foreign Affairs and International Trade (DFAIT).

On the request of the Human Security Policy Division (GHS) the Evaluation Division (ZIE) of the Department of Foreign Affairs and International Trade (DFAIT) undertook in the fall of 2006 through to March 2007 a summative evaluation of the Canadian Consortium on Human Security (CCHS). The objective of the research was to provide information with respect to three key evaluation issues:

  • Relevance
  • Success
  • Efficiency and Cost Effectiveness

The evaluation findings are based on multiple lines of evidence including extensive literature and document review, analysis of the CCHS budget information, key informant interviews and an online/telephone survey with CCHS constituents (individuals identified as using/having some interaction with CCHS program/services). Where possible, reference has been made to a comparison group (i.e., non-funded fellowship applicants) to provide perspective on the net or incremental impact of the program in terms of supporting the growth/expansion of Highly Qualified People (HQP) in the area of human security.

Key Findings:

Program Relevance

The CCHS is consistent with DFAIT's goal of supporting and promoting research and dialogue in the area of human security.

From the perspective of the majority of informants, the activities of the CCHS are consistent with and supportive of the Canadian government's human security agenda. That is, 90% of stakeholders interviewed felt strongly that the activities and services administered by the CCHS are consistent with, and supportive of, the GOC's human security agenda, and 63.3% of constituents felt that the CCHS has made a positive contribution to shaping the human security research agenda in Canada.

It should be noted that the CCHS is consistent with DFAIT's practice of supporting institutions that can undertake and coordinate research related to Department issues and priorities. For example, the Canadian Peacebuilding Coordination Committee (CPCC) represents a structure through which DFAIT can access research and policy advice from non-government organizations (NGOs) regarding a range of issues (including human security).

The majority of stakeholders interviewed felt strongly that the CCHS serves as an effective vehicle to facilitate dialogue, research, and to generate new ideas and approaches to address the challenges of human security relevant to Departmental policy development and programming.

Although the broader constituent groups were very supportive of the work done by the Consortium in facilitating dialogue and policy advice regarding human security, DFAIT stakeholders interviewed expressed a lower level of satisfaction with the extent to which the CCHS has been responsive to the relevant policy needs of the Department and the extent to which it has contributed to program development. For example:

  • In comparison to the CPCC, DFAIT stakeholders felt that the CCHS was less responsive to Departmental needs regarding policy consultation and/or policy development than was the case for the CPCC;
  • Some stakeholders felt that the CCHS did not reach all academic institutions, and was initially weak in engaging the Francophone academic community; and
  • DFAIT staff felt that the CCHS could not easily identify and/or mobilize a wide range of academic experts who could comment on pressing and/or newly identified policy issues that were a priority for the Department.

It should be emphasized, however, that the limited ability for the CCHS to respond to emergent DFAIT policy requirements also reflects the very limited formal communication channels/structures established between DFAIT and CCHS. In many instances, the CCHS had only a limited insight as to Departmental needs and priorities.

In terms of the relevance of the CCHS in meeting Departmental needs, it is clear that DFAIT must also examine what role(s) it feels is appropriate for the Consortium. In retrospect, DFAIT needs to also identify whether the Department is supportive of the CCHS filling the role as a networking/information sharing institute, working to provide long-term goals of capacity building, or whether the CCHS should be more proactive in terms of addressing DFAIT policy requirements in the capacity as a "policy think tank."

Program Success

The products and services developed and maintained by the CCHS are viewed by stakeholders and constituents as valuable resources that have contributed to research and dialogue in the area of human security.

Information obtained from the key informant interviews, constituent survey and a review of administrative data suggests that the CCHS is seen as an effective vehicle for supporting research collaboration and dialogue both within the academic community and the broader (both domestic and international) human security research community.

Both key informants and CCHS constituents are of the opinion that the CCHS has positively contributed to Canada's image as a leader in human security on the international stage.

The CCHS has supported the increase in the number of Highly Qualified People (HQP) working in the field of human security.

The CCHS has a number of programs/services that contribute to the development of Highly Qualified People (HQP) who work in the field of human security. Based on comparisons of outcomes of funded vs. non-funded fellowship applicants, it appears that CCHS fellowships have contributed to a positive net impact in terms of HQP and/or research related to human security. Similarly, teaching resources assembled by the CCHS are viewed as having contributed to the expansion of the number of people who have an interest in the area of human security. Overall, most stakeholders were of the opinion that the work done by the CCHS had contributed to increased human resource capacity in the area of human security.

The inability of the CCHS to secure timely approvals and funding commitments from DFAIT has affected the ability of the CCHS to seamlessly deliver program and services.

The financial relationship between CCHS and DFAIT has been, and continues to be, a major impediment to pro-active planning and program delivery by the CCHS. The one-year funding cycle, and delays in project funding, has had several detrimental effects on the operations of CCHS, including:

  • staff turnover, as research/administrative assistants cannot be hired for more than one year at a time, and funding lapses have resulted in times where UBC has provided "bridge funding" for key staff pending project funding from DFAIT;
  • an inability to award research/scholarship funding for more than one year; and
  • an inability to complete all planned activities (i.e., workshops, conferences) due to delays in funding approval.

The delays and uncertainties associated with project funding have generally resulted in the CCHS under-spending in terms of actual vs. planned expenditures. In fact, during the period from December 2001 to March 31, 2005, the CCHS spent, on average, only 94% of planned expenditures.

Program Efficiency and Cost Effectiveness

While the CCHS has adequate financial reporting systems, the CCHS/DFAIT have yet to implement a performance measurement framework.

Analysis of the administrative systems established by the CCHS suggest that the Consortium can easily track a number of financial and/or program activity indicators. As part of the University of British Columbia, the CCHS does benefit from strong support in the area of financial management and reporting. DFAIT program officers further confirmed the adequacy and comprehensiveness of CCHS financial reporting.

While formal and administrative systems are in place to monitor program activities and/or expenditures, both CCHS and DFAIT have yet to establish systems to measure the effectiveness of program activities or outputs. For example, until this evaluation was undertaken, the CCHS had not completed any formal internal analysis of client satisfaction with products and services (i.e., exit surveys with conference participants, follow-up surveys with fellowship recipients, online surveys for users of various online publications/products).

Key informants are of the opinion that the CCHS represents a cost-effective vehicle to support human security research/policy development in Canada's academic community.

Given the considerable level of activity achieved by the CCHS, most key informants felt that the CCHS represented a cost-effective mechanism to increase dialogue, research and policy development capacity in the academic community with respect to human security. DFAIT staff noted that other alternatives (i.e., DFAIT in-house processes to house/manage policy/research consultation) would likely be more costly relative to the CCHS model. However, that there may exist some scope for the CCHS to reduce its reliance on DFAIT funding through the use of other cost recovery mechanisms such as user fees and/or subscription fees for CCHS products and services.

It should be noted that other models may in fact be much more resource intensive than was the CCHS. For example, prior to 2005, the Natural Sciences and Engineering Research Council (NSERC) and Social Sciences and Humanities Research Council (SSHRC) provided funding for National Centres of Excellence (NCEs). While budgets for each NCE varied, funding for a typical NCE (to support both networking and research) ranged between $3 to $4 million per year per NCE (much higher than DFAIT's contribution of approximately $313,000 per year). It should be noted, however, that since 2005, NSERC/SSHRC have established "new" centres of excellence which are intended to support networking activities only, and such centres receive funding in the range of approximately $400,000/year.

Top of page

Conclusions and Recommendations

The overall conclusion of the Summative Evaluation of the Canadian Consortium on Human Security is that the CCHS has proven good value for money. It has succeeded in broadening awareness about human security, increased dialogue about and strengthened capacity in human security, particularly across the academic community, and did so at a modest cost.

Analysis of the data suggests that among both key informants and constituents surveyed, the CCHS has been an effective vehicle to enhance interaction and dialogue across the academic community about human security. The CCHS is also seen as being an important contributor to the development and support of Highly Qualified People (HQP) who are active in human security and are contributing to the human security agenda.

In general, analysis of the outputs/activities and constituent satisfaction with CCHS products and services suggests that CCHS serves as a cost-effective model to support DFAIT's objective to improve the coordination of research and activities with partner institutions, both in Canada and internationally. In addition to DFAIT's direct contribution of approximately $300,000 per year, the CCHS obtains in-kind and other expenditures of approximately $100,000 annually. In this context, each $1 expenditure by DFAIT "leverages" approximately $0.29 in non-DFAIT expenditure or in-kind support. DFAIT staff acknowledges that it would be unlikely that DFAIT could undertake the tasks and activities done by CCHS using fewer resources than currently provided to the CCHS.

Although the CCHS appears to be meeting the broad needs of its constituent community, the results of the evaluation suggest that there are some aspects of the CCHS which have not met DFAIT expectations, including:

  • the ability of CCHS to mobilize a broad range of academic experts quickly to provide timely advice regarding emergent policy/programs issues as identified by the Department;
  • the ability of CCHS to coordinate research dialogue effectively among Francophone researchers; and
  • the extent to which CCHS activities align to DFAIT expectations/requirements (i.e., preference for complimentary activities rather than scholarships).

It should be noted, however, that these shortcomings are to some extent the result of the apparent absence of an effective mechanism to convey DFAIT policy/program needs to the CCHS (poor communications structure) as well as a function of a disjuncture in the production cycles between DFAIT and the academic research community; the policy development cycle for DFAIT can be measured in weeks and months, while academic research may take years, culminating in publication.

Findings and Recommendations

Based on the results of the evaluation, the report provides several recommendations under the themes of program relevance, success and efficiency and cost effectiveness.

Relevance

Finding: 1

The results of the evaluation suggest that the CCHS has achieved several positive outcomes in supporting academic research/networking in the area of human security as well as supporting capacity building with respect to Highly Qualified Personnel (HQP) who are engaged in research related to human security issues. The sustainability of these positive outcomes, however, is placed in jeopardy by continued uncertainties over funding, which has frustrated long term planning.

Recommendation 1:

If DFAIT determines that the CCHS fills a valuable role in coordinating and promoting Canada's Research Agenda with respect to human security, the Department should enter into a longer term and predictable funding arrangement with the CCHS.

Success

Findings: 2

While CCHS has met the broad objectives set for it, the CCHS is perceived to be less responsive to emergent DFAIT policy needs. Improved communication mechanisms, in the form of joint planning exercises, would help ensure that the CCHS receive explicit direction as to DFAIT policy/research/consultation requirements.

Recommendation 2:

Establish more formal communication mechanisms to support the collaboration between DFAIT and the CCHS.

Findings: 3

Work done by the CCHS has been seen to support the development of Highly Qualified Personnel, both through the Fellowship program as well as through the provision of teaching resources.

Recommendation 3:

That DFAIT be supportive of the Fellowship program offered by CCHS, and where possible, DFAIT may wish to specify "theme areas" or topics that could be supported through the Fellowship program.

Findings 4:

While a performance measurement framework has been developed for the CCHS, reporting on the key elements of the framework has not occurred. Performance measurement information would provide the CCHS/Department with timely information as to results achieved and outcomes associated with DFAIT's investment in the CCHS.

Recommendation 4:

Implement a Performance Measurement Framework for the CCHS.

Efficiency and Cost Effectiveness

Findings 5:

The CCHS has made progress in diversifying its funding sources, but has never been formally directed to achieve financial self sufficiency. DFAIT, in consultation with the CCHS, should identify the proportion of total funding that the Department is expected to provide. Similarity, the CCHS should explore the feasibility of alternative funding sources (e.g., fee for service) to lessen its dependence on Departmental funding.

Recommendation 5:

Establish program targets in terms of cost sharing of the CCHS.


Top of page

Glossary of Terms

CCHS
- Canadian Consortium on Human Security
CHS
- Centre for Human Security
CICP
- Cambodia Institute for Cooperation and Peace
CIDA
- Canadian International Development Agency
CIR
- Center of International Relations (UBC)
CMS
- Content Management System has an on-line sign-up form for subscriptions to CCHS publications
CMS
- Contact Management System facilitates the collection, sorting and searching of names and contact details of organizations and individuals relevant to the work of the CCHS
CPCC
- Canadian Peacebuilding Coordinating Committee
DFAIT
- Department of Foreign Affairs and International Relations
DND
- Department of National Defence
FAC
- Foreign Affairs Canada
HSF
- Human Security Fellowship
HSG
- Human Security Gateway
IDRC
- International Development Research Centre
NAB
- National Advisory Board
NIRA
- National Institute for Research Advancement
RRU
- Royal Roads University
THS
- Teaching Human Security
UBC
- University of British Columbia
UQAM
- Université de Québec à Montréal
UVIC
- University of Victoria
YCISS
- York Centre for International and Security Studies

Top of page

1.0 Introduction

1.1 Overview of the Canadian Consortium on Human Security

Established in December 2001, the Canadian Consortium on Human Security (CCHS) is an academic-based network supporting policy-relevant research on human security. CCHS's primary mission is to facilitate the exchange of information, analysis and dialogue on human security issues between non-governmental organizations (NGOs), academics, research institutes, "think tanks" (domestic and international), government and trans-national organizations involved in human security studies and policy development. The CCHS is funded by the Human Security Program of Canada's Department of Foreign Affairs and International Trade (DFAIT).

Situated at the Centre of International Relations (CIR) at UBC, the Consortium operates in cooperation with the Centre for International and Security Studies at York University (CISSYU) to deliver its programs and publications. CCHS is funded through a grant from the Human Security Program (HSP) of Foreign Affairs Canada (FAC). Over 65% of the funds supporting CCHS activities and services come from DFAIT, with the balance coming from in-kind contributions from other sources. With the end of Phase III and the beginning IV, DFAIT has determined that an evaluation of the HSP contribution to the CCHS is timely.

The following is a list of activities and services through which the CCHS fulfills its mandate:

  • CCHS Website: The CCHS hosts a comprehensive website that lists all the CCHS activities, online publications, and reports and research from the Complementary Activities program. The site is available in both English and French. In the period from September to December 2005 alone, there were as many as 246,716 hits on this website.
  • Human Security Bulletin: The Bulletin is the CCHS's principal publication. Published online regularly, it presents policy-relevant human security research currently being pursued in Canada and internationally. The CCHS also provides annotated resources, such as article and publication abstracts, conferences, seminars, announcements and proceedings.
  • Human Security Fellowship (HSF) Program: Through fellowship awards, the CCHS funds innovative research and policy development, which reflect the Canadian government's human security agenda: encompassing projects that address conflict prevention, humanitarian and peace support operations, security-related governance and accountability, as well as international norms and institutions relevant to this agenda. During the past five years, the CCHS has provided approximately 29 fellowships (typically one year in duration).
  • Complementary Activities: As part of its "Complementary Activities" program, the Consortium sponsors human security-related workshops and conferences, such as the Annual Human Security and Peacebuilding Consultations.
  • Mailing List: The CCHS has compiled and maintains a confidential e-mail list of people interested in/undertaking human security research. As of December 2006, the list includes more than 9,700 contacts in academia, government, NGOs and international organisation representatives. It should be noted, however, that many of the contacts that receive CCHS publications are listed in different databases such that the number of "unique" individuals who receive various CCHS publication is less than 9,700.
  • Human Security Gateway: In collaboration with the Human Security Centre at the University of British Columbia, the CCHS develops the Human Security Gateway, a free online portal and database of human security related resources. The Gateway has thousands of resources including articles, reports, facts and figures, audio-visual resources, bibliographies and organizations.
  • Human Security and Cities Agenda: The most recent and comprehensive product of the CCHS, this online publication draws on the work of 40 external contributors who apply an urban lens to human security including children and armed conflict, security system reform, small arms and light weapons, stabilization and reconstruction, peacebuilding and democracy promotion.
  • Teaching Human Security Database: In addition to a teaching resource page on the CCHS website, the CCHS has developed a new database targeting students and instructors of human security who strive to remain informed about academic developments in teaching human security in universities and colleges across Canada. The searchable course database, research materials, and sample human security course syllabus provide a comprehensive overview of human security as currently taught in Canadian classrooms.

1.2 Purpose of the Evaluation

In order to assess the CCHS initiative since its inception in December 2001, DFAIT implemented an evaluation of the initiative in late fall 2006 - early winter 2007. The evaluation was summative in nature and focussed on issues relating to the overall relevance, success, and cost-effectiveness of the initiative to assist in the determination of whether or not continued support to the CCHS is warranted. More specifically, the objectives of this evaluation were:

  • To determine the extent to which the activities of the CCHS are consistent with the human security agenda as defined by DFAIT and reflect the terms and conditions of the CCHS' various contribution agreements;
  • To determine the extent to which the activities of the CCHS have mobilized an academic-based network to serve as a credible civil society partner for dialogue with government on human security;
  • To determine the extent to which the activities of the CCHS are responsive to, and meet the needs of, its government partners;
  • To determine the extent to which performance information being collected, captured, and used is in accordance with the terms and conditions of the contribution agreement and are appropriate to ensure results-based and performance oriented program planning, monitoring, reporting, and management;
  • To determine the extent to which the CCHS is meeting its objectives, within budget, and without undesirable outcomes;
  • To determine the extent to which the CCHS can diversify its funding base, and thus ensure its continued viability; and
  • Based on the foregoing, determine what level of support the CCHS should receive and what, if any, modifications to the CCHS activity profile, its administrative practices, and methods of reporting, may be necessary to achieve desired results and support DFAIT's policy development agenda.(1)

Contained in the remainder of this document are the key findings and general observations drawn from the Summative Evaluation of the Canadian Consortium on Human Security.


Top of page

2.0 Methodology

Highlighted in the following section are the research activities undertaken by the evaluation team as part of the Department of Foreign Affairs and International Trade (DFAIT) Canadian Consortium on Human Security (CCHS) evaluation project.

2.1 Scope of Work

As part of this study, a mixture of quantitative and qualitative research methods were used to provide a comprehensive overview of the strengths and weaknesses of the CCHS, including:

  • Document/literature and administrative data review. As part of the evaluation, the evaluation team reviewed various data including contribution agreements, CCHS annual reports (narrative reports from Phase I and Phase II) and review of various databases provided by the CCHS.
  • Key informant interviews. Ten key informant interviews were completed with current/former DFAIT staff (4) and other stakeholders (6).
  • Constituent survey. An online survey was administered to 109 individuals who had some affiliation to the CCHS as either a bulletin subscriber, conference attendee, and/or as individuals who had applied for a CCHS fellowship (both funded and non-funded fellowship applicants).

2.2 Survey Administration - Constituent Survey

Survey Administration

Upon approval of the survey instrument by the Project Authority, the survey was programmed into the Consultant's CallWeb system and subject to extensive testing to ensure its proper functioning. Full survey administration began on February 13, 2007 and was completed on March 2, 2007. A total of 109 completions with CCHS constituents was obtained during administration of the Constituents Survey, which is within range of the 100-150 completions initially proposed for this project. While a census approach was taken, not all available contacts participated in the study, as such, a sampling error of ±8.4% (at the 95% confidence level) can be calculated based on the number of completions obtained. Survey completions by constituent group is illustrated in Table 2-1.

Table 2-1: Survey Completions
Constituent Group (1a)Survey Completions by Constituent GroupApproximate PopulationResponse Rate

1a Constituents are grouped according to responses to question A1 of the survey.

2a Note that this figure represents the total number of survey respondents. The column does not sum to 109 due to respondents' membership in multiple constituent categories.

Funded Fellows21n=3757%
Non-Funded Applicants4n=2020%
Complementary Activities Provider10n=1663%
Conference Attendees/Complimentary Activities Participant71n=52230%
Bulletin Subscriber53
Other35
Total Respondents109(2a)n/an/a

The largest proportion (49.5%) of constituents who responded to the survey were academic researchers/professors, followed by policy consultants or NGO representatives (28.4%), and program officers or officials involved in the design/delivery of HS initiatives (22%). The affiliation of respondents is shown in Table 2-2.

Table 2-2: Survey Completions by Respondent Affiliation
Constituent Group# of Survey Participants% of Total Respondents
Academic Researcher5449.5%
Policy Consultant/NGO Representatives/Other3128.4%
Program Officer/Official2422.0%
Total109100%

Note: Totals may not sum to 100% due to rounding.

Of the 595 respondents to whom an email was sent, approximately 150 were returned due to delivery failure (e.g., incorrect or expired address), and approximately 47 were returned out-of-office. Among the latter group, where an alternate email address was provided the survey link was re-sent.

The majority of completions were obtained within the first 4 days of surveying. In order to maximize the amount of data collected from the survey, the proposed survey end date was extended by a week for the following reasons:

  • to accommodate the numerous out-of-office replies among academic constituents due to reading break; and
  • to allow for some follow-up contact to be undertaken by telephone (where contact information was available) with some of the "hard-to-reach" constituents, in particular, non-funded fellowship applicants.

The extension resulted in an additional 30 survey completions (28%).

Top of page

Respondent Tracking Activities

The Consultant undertook significant respondent tracking activities in an effort to obtain alternate contact information for constituents for whom email addresses were incorrect or expired. In addition, all of the constituent databases were cross-referenced to determine whether or not individuals were present in multiple constituent lists and, if so, whether or not alternate contact information was available.

Special emphasis was placed on tracking non-funded fellowship applicants, as their perspectives of, and experience with, the CCHS are unique. In addition to the tracking activities described above, where telephone numbers were available, follow-up calls were made directly to individuals to encourage them to complete the survey. It should be noted, however, that telephone numbers for only 6 of the 20 non-funded fellowship applicants were provided by the CCHS.

Finally, data monitoring and quality control measures were undertaken throughout survey administration to maximize the quality of the data collected. In particular, responses to question A1 of the survey were monitored, as this question determined the subsequent sections of the survey to which respondents were directed. This monitoring activity resulted in approximately six individuals being redirected to the appropriate section of the survey, however, in the remaining cases, follow-up contact could not be made or no telephone information was available.

2.3 Key Informant Interviews

An interview guide was designed that served as the basis for discussions with key informants such as representatives of the CCHS, DFAIT and members of the CCHS Executive Council. The guide was developed with the key research objectives in mind and involved a combination of approximately 30 open- and closed-ended questions pertaining to the overall relevance, success, efficiency and cost effectiveness of the consortium. Ten key informant interviews were completed with stakeholders.

2.4 Administrative Data Review

Print and electronic documentation describing the CCHS were reviewed to provide context for the development of data collection instruments and interpretation of research results. Similarly, a review of relevant literature (e.g., research/discussion papers) provided insight into specific issues pertaining to existing human security research and policy development and the issue of (continued) program relevance, among others. Examined in the document review component of the evaluation (e.g., CCHS narrative and financial reports, contribution agreements) are activities undertaken by the CCHS in relation to the funding received from DFAIT and/or generated from other sources.

Administrative information/data(2) maintained by CCHS were also reviewed in the context of performance-based planning, monitoring, reporting and management. Where data of sufficient quality exist, administrative files were used to measure/assess certain aspects of the CCHS (e.g., usage). Administrative information about operations provided basic information regarding program inputs and outputs and some information about program success in terms of usage, accessibility and reach.

2.5 Data Analysis and Interpretation

Upon completion of survey administration activities, data entry of all quantitative data obtained was undertaken. A datafile of all survey responses was developed, and data were cleaned and prepared for analysis. Extensive analysis was undertaken in the writing of this document.

Because of the multiplicity of data collection methods/data sources used, the variety of stakeholder groups interviewed, and the number of surveys administered, many of the observations and findings were not easily quantified, combined nor translatable into statistics for analytical purposes. Therefore, readers will note that while some sections of the report rely primarily on qualitative information, other sections of the report rely on quantitative results.

Qualitative data received through interviews was reviewed, synthesized and analyzed. Issues raised by interviewees were examined in relation to key evaluation components (e.g. CCHS success, relevance, efficiency and cost effectiveness) and in relation to the findings of the survey of CCHS constituents.

In this respect, the findings described in this report should be considered to represent a synthesis of qualitative and quantitative information gathered from information provided by more than 120 individuals who participated in this evaluation.

2.6 Challenges and Potential Limitations of the Project

It should be emphasized that, while the study was intended to be a summative evaluation, there was only limited ability to detect net or incremental impacts using either a longitudinal (i.e., pre/post) methodology or a cross-sectional (participant vs. comparison group) approach. While some incremental impacts were identified using a comparison groups approach (i.e., funded vs. non-funded fellowship applicants), much of the evaluation is based on information obtained through document review, key informant interviews and the constituent survey.

The results of the constituent survey should also be viewed with caution as the individuals surveyed by definition would be those who had some level of involvement with the CCHS.


Top of page

3.0 Survey Respondent Profile – Constituent Survey

As described in the methodology section of this report, survey respondents were categorized according to the nature of their involvement with the CCHS and subsequently directed to specific sections of the survey for additional questions of relevance to each constituent group. This section provides a brief profile of respondents based on survey responses.

3.1 General Characteristics of Survey Respondents

In total, 21 CCHS Fellows responded to the survey, representing more than one-half (57%) of the individuals who received a CCHS fellowship. Among them, the largest proportion (33%) had received funding in Phase 1 of the fellowship award process (i.e., between 2001-2003).

Despite extensive respondent tracking and sample management activities, survey completions were obtained from only 20% of non-funded fellowship applicants (n=4). Two of the four non-funded applicants applied for the award between 2001 and 2003, and the other two respondents applied in 2004 or in 2005.

Ten out of 16 (63%) Complementary Activities(3) providers responded to the survey, representing nearly 10% of all survey respondents. One-half of these received funding to provide a workshop/seminar on a specific topic, while 20% received funding to serve as a guest speaker at a CCHS conference or event. The remaining 30% reportedly undertook another role, such as:

  • providing chapter material for the Human Security Report and Bulletin;
  • organizing a conference on Failing, Failed and Fragile States and Canada's policy in relation to them; or
  • attending the Human Security and Peace Building Consultations in Ottawa in 2005.

Almost two-thirds (65%) of the survey respondents had attended a CCHS-sponsored conference or Complementary Activity(4). As shown in Table 3-1, the Annual Peacebuilding and Human Security Dialogue was the most highly attended event among survey respondents.

Table 3-1: Conferences Attended
ConferencePercent
The Human Security Colloquium34.6%
The Canada-Norway Peace Prize Symposium17.3%
The Annual Peacebuilding and Human Security Dialogue76.9%
The Human Security in Urban Spaces Conference28.8%
Others13.5%

Source: Constituent Survey
n=52
Note: Totals may exceed 100% due to multiple response.

As highlighted in Chart 3-1, the majority of survey respondents were academic researchers/professors (49.5%) or were a program officer or official involved in the design and/or delivery of human security initiatives (22.0%). About one in ten respondents represented an NGO (10.1%) or were a non-academic policy consultant or contractor (9.2%).(5)

Chart 3-1: Composition of CCHS Constituents

Composition of CCHS Constituents graph

Source: Constituent Survey
n=109


4.0 Key Findings

The following section highlights the key findings drawn from the synthesis and extensive analysis of informant and survey data sources. Findings are grouped and reported according to the major issues of interest to the evaluation: Relevance, success, and efficiency and cost effectiveness.

Readers are encouraged to consult the Evaluation Matrix (Appendix A) for a detailed description of the evaluation issues and indicators for each evaluation topic area.

4.1 Relevance

The issue of relevance relates to the continued rationale for support to the CCHS and its activities. Rationale issues address whether or not the objectives and guiding principles of the CCHS remain relevant. These questions are explored in the remainder of this section.

Evaluation results suggest that the CCHS is consistent with and supportive of the Government of Canada's Human Security Agenda (R1, R8).

The term human security can be defined as a people-centered approach to foreign policy, which recognizes that lasting stability cannot be achieved until people are protected from violent threats to their rights, safety or lives.(6) This definition is at the core of Canada's human security agenda which highlights Canada's "responsibility to protect."

The Canadian Consortium on Human Security (CCHS) was established as a mechanism through which the Government of Canada (GOC) could better engage Canada's academic community with respect to research/dialogue on human security issues. It was in many respects seen as a parallel institution to the Canadian Peacebuilding Coordinating Committee (CPCC), which was structured to provide DFAIT with a mechanism to access research and dialogue with the Non-Government Organization (NGO) community involved in human security issues.

The majority of stakeholders interviewed (90%) felt strongly that the activities and services administered by the CCHS are consistent with, and supportive of, the GOC's human security agenda, and 63.3% of constituents felt that the CCHS has made a positive contribution to shaping the human security research agenda in Canada.

While stakeholders generally agreed that the CCHS serves as an effective vehicle to facilitate dialogue and research related to human security, there is some debate among stakeholders about the extent to which the CCHS is responsive to the relevant policy needs of the Department and the extent to which it has contributed to program development (R2).

The majority of stakeholders interviewed felt strongly that the CCHS serves as an effective vehicle to facilitate dialogue, research, and to generate new ideas and approaches to address the challenges of human security that are relevant to policy development and programming.

Survey results show that there was only a marginal difference in opinion as to the overall effectiveness of the range of CCHS activities and services. While the conferences were thought to be effective or very effective by the largest proportion of constituents, all of the activities and services were considered highly effective by at least eight out of ten respondents.

Chart 4-1: Overall Effectiveness of CCHS Activities and Services in Terms of Supporting HS Research Dialogue

Overall Effectiveness of CCHS Activites graph

Source: Constituent Survey
n=74 to 94
Note: Included valid responses only

Although the broader constituent groups were very supportive of the work done by the Consortium in facilitating dialogue about and providing policy advice on human security, there was a lower level of satisfaction expressed by DFAIT stakeholders with the extent to which the CCHS has contributed to relevant policy needs and program development for the Department, for example:

  • In comparison to the CPCC, DFAIT stakeholders felt that the CCHS was less responsive to Departmental needs regarding policy consultation and/or policy development than was the case for the CPCC;
  • Some stakeholders felt that the CCHS did not reach all academic institutions, and was initially weak in engaging the Francophone academic community; and
  • DFAIT staff felt that the CCHS could not easily identify and/or mobilize a wide range of academic experts who could comment on pressing and/or newly identified policy issues that were of priority for the Department.

A sample of comments illustrate such concerns:

"CCHS works well as a "dialogue vehicle", but has not performed up to expectations in terms of providing timely policy advice to the Department."

"Initially what we found is that the CCHS got high engagement from only a limited number of academic institutions and a limited number of academics. There was a low level of engagement among the Francophone community."

"We found the CCHS to be of only limited value in terms of quickly responding to emerging policy/program needs of the Department."

In defence of the CCHS, it should be noted that the CCHS has received limited policy direction from DFAIT. While the CCHS and DFAIT staff convene on an annual basis, much of the discussion in these meetings revolves around administrative and/or funding issues. In addition, DFAIT generally adopted a "hands off" approach when reviewing CCHS activities, and did not actively communicate expectations about or advice on desired direction of fellowships, complementary activities and/or general policy direction.

It should also be noted that not all Departmental staff were of the opinion that the CCHS should serve as a mechanism to solicit academic policy advice regarding emergent DFAIT policy/program issues. For example, one staff member commented that the CCHS should serve as a "networking facilitator" and should have a goal of supporting long term research and/or the development of Highly Qualified Personnel (HQP) rather than trying to meet emergent DFAIT policy issues.

The results of the evaluation suggest that the CCHS has achieved several positive outcomes in supporting academic research/networking in the area of human security as well as supporting capacity building with respect to Highly Qualified Personnel (HQP) who are engaged in research related to human security issues. What is not clear, however, is the extent to which the CCHS has met Departmental needs in providing timely policy advise with respect to emergent issues. In this context, it is critical that the Department more clearly specify what roles/activities should be undertaken by the Consortium to reduce/minimize the ambiguity as to what role the CCHS fills in meeting specific DFAIT needs as opposed to broader human security objectives.

Notwithstanding concerns as to the role of the CCHS meeting emergent policy issues, several mechanisms identified in the evaluation that could better support the CCHS meeting Departmental objectives included:

  • formal planning meetings at which time emergent issues and/or expected DFAIT policy issues are shared with the CCHS;
  • involvement of DFAIT in reviewing and approving projects/applications for fellowships and/or complementary activities; and
  • enhanced/more proactive communication between DFAIT and CCHS at the senior manager levels.

In terms of reach, CCHS clients currently include academics, consultants, NGOs, and government officials from across the country. While the engagement of Francophone academics and practitioners involved in human security activities was slow at the beginning, the CCHS has made significant progress in reaching this community. Notwithstanding these achievements, it is unclear at this juncture whether the CCHS is reaching all those who could potentially make a contribution to Canada's human security agenda. There is a need to establish goals and action plans with respect to reach to ensure that CCHS activities are responsive to Canadian geographic and linguistic needs.

It should also be emphasized that the ability of the CCHS to mobilize the academic community quickly for research/policy advice runs counter to typical university/ academic research cycles, in which research typically occurs over a longer period of time (1-2 years) and generally culminates in publication in a research journal and/or other publication. In this context, it may be the research processes in academia, rather than the functions of the CCHS that may have contributed to the limited ability of the CCHS to respond rapidly to DFAIT requests for policy/program advice.

The CCHS has developed a wide range of products and services that meet the needs of constituents and add value to the human security agenda in Canada (R3, R5).

The vast majority (90%) of the individuals interviewed(7) held the opinion that the activities and services administered by the CCHS meet the needs of its members, academics, government and the larger international community. For example, the Consortium manages four separate fully bilingual websites that list all CCHS activities, online publications, conference announcements, and contain a large number of reports and research from the Complementary Activities program including conferences, seminars and workshops held across the country. The websites require regular updating, maintenance, trouble-shooting and development on both language interfaces. These sites have consistently attracted thousands of visitors monthly.

In analyzing the extensive use of the CCHS products and services, it would appear that the CCHS is utilized as a resource by the broader human security community. As detailed below, the products/services provided through the CCHS have generated considerable attention/usage among persons within the human security community.

Table 4-1: HS Bulletin Usage and % Change Over Time
Timeframe# of Hits# of Visits# of Unique Visitors
June 200433,0003,0092,046
September 200430,9723,0892,240
December 200440,0003,9972,760
March 200572,0006,9345,491
% of Change March 05 compared to June 041.1821.3041.684

Source: CCHS Final Narrative Reports, various phases. Data unavailable prior to June 2004.

The total traffic for the reporting period April 2005 - August 2006 was approximately 94,000 visitors for the main CCHS website and more than 246,000 hits for the Human Security Bulletin website, representing an all-time high for any four-month period since the inception of the CCHS. The CCHS website currently has more than 100 pages of content, while the Bulletin site has approximately 150 content pages and has received over 1,300 online requests for subscription. In addition, a search query for the term "human security" places the Bulletin as the fifth search result and the CCHS website as the eighth search result (based on a six search engine average), making the sites highly accessible to any interested parties.(8)

The Consortium also developed the Human Security Gateway (HSG), a research and information database that provides electronic and bibliographic resources for policy-makers, academics, and non-governmental organizations on human security issues. Content for the HSG is drawn from the websites of several hundred research institutes, journals, IGOs, NGOs and government sources. The HSG received a total of 103,659 visits and 3,431,234 hits during the 2005 fiscal year (April 2005-March 2006).(9) As of June 30, 2006, the English-language list numbered 761 sources, and the French language list numbered approximately 176. Academic articles and reports currently account for more than 62% of the total.(10)

A new Content Management System (CMS) was designed and developed in Phase III with an online sign-up form for subscription to CCHS publications, including the Human Security Bulletin. Launched in early 2006, by July 2006, 1,300 users subscribed to the Bulletin using the online subscription function.

The CCHS also created a Teaching Human Security Canadian Course Database, which provides an updated collection of all courses and workshops taught across the ten provinces and three territories of Canada. The user-friendly interface allows anyone with an online connection to search the database easily based on course type, topics, location, language or institution.

In collaboration with the Human Security Program of Foreign Affairs Canada, the Consortium has in addition to the above developed a new website for the Human Security and Cities Agenda. This entailed communicating with Human Security Partners staff, liaising with a Toronto-based design and development company, coordinating with the Vancouver-based development company and drafting suggestions for website features, etc.

It should be emphasized that, while key informants interviewed for this project felt that the CCHS has been an effective vehicle to engage the academic community, not all were sure of the extent to which the CCHS has engaged government and/or the policy research community. The CCHS, however, has managed to attract a mix of participants to conferences, including a high proportion of government officials and NGO/policy experts, etc. For example, the 2006 Peacebuilding and Human Security Dialogue brought together over 300 people to engage in dialogue, debate and information sharing on the Drivers of Conflict and Drivers of Peace. This conference included 103 Canadian and international NGO representatives, 102 government officials from a wide range of departments and agencies (including 62 from DFAIT, 15 from CIDA, 3 from DND), 71 academics and students, 17 Embassy representatives, 5 parliamentary staff and others.

When asked to identify additional potential structures to support this research dialogue, DFAIT staff felt that other mechanisms such as an in-house secretariat function and/or the development of direct links with academics would be more cumbersome and costly than would services provided by the CCHS.

While DFAIT staff may have questioned the ability of CCHS to meet emergent on-going human security policy issues, DFAIT stakeholders agreed that when provided explicit direction, CCHS was an effective vehicle to encourage policy dialogue among academics, government and the policy research community. Examples of projects that the CCHS met or exceeded expectations in this area include:

  • Human Security and Cities Agenda (website launch in Jan. 2006)
  • UN World Urban Forum (June 19-23, 2006)
  • Canada-Norway Peace Prize Symposium (Feb 3-5, 2006)

Evaluation findings suggest that more could be done by the CCHS to complement the programs and activities of other HSP partners (R6).

The results of the evaluation suggest that the CCHS may not have fully met expectations in this area, for example:

  • Only one-half of key informants interviewed held the opinion that the activities and services provided by the CCHS support, complement or otherwise productively intersect with the programs and activities of other Human Security Program (HSP) partners.
  • Stakeholders noted that, while the CCHS had a good "reach" into the academic community, mechanisms had not been established which would facilitate more active dialogue/policy research coordination between DFAIT, the NGO community and the CCHS.

There is evidence to suggest that the CCHS has contributed to Canada's reputation on the international stage and that, without it, Canada's commitment to human security may be called into question. (R4, R7)

Two-thirds (66%) of constituents and almost nine out of ten (88%) key stakeholders felt that the CCHS has contributed to Canada's reputation and credibility on the international stage as a leader and innovator in the area of human security policy development. The Canada-Norway Peace Prize Symposium was cited as an example of this international exposure.

It was felt that the CCHS provides a low cost research network of individuals and countries with human security as a common interest that otherwise would not exist, such as the relationship that has been established with Norway. Without the CCHS, other mechanisms would have to be put in place to locate and liaise with individuals associated with human security both nationally and internationally.

In reviewing other similar initiatives, it was determined that few other countries have established a similar consortium and/or institution with a comparable suite of products and services that supports information exchange among and across the academic community(11). This is not to imply that such institutions are not important, but rather, emphasizes Canada's commitment and leadership in the area of human security.

Moreover, without the activities and services provided by the CCHS, it was suggested by several respondents that Canada's reputation would suffer both domestically and internationally, since it would appear that Canada is not fully committed to the issue of human security. From that perspective, Canada's presence as a leader in the field of human security must be sustained in order for Canada to maintain its international standing. As noted by a DFAIT key informant:

"Eliminating and/or reducing our commitment to CCHS would result in damage to our credibility as a world leader in the area of human security."

Summary of Findings - Relevance

  • The findings associated with the key informant interviews and the constituent survey underscores the need and relevance of the CCHS in meeting the Government of Canada's broad objectives in terms of human security.
  • The relevance of the CCHS is demonstrated by the high proportion (in excess of 60%) of constituents who agree that the CCHS has positively contributed to the human security agenda in Canada, and that the CCHS has contributed to Canada's reputation and credibility on the international stage as a leader and innovator in the area of human security policy development (88% agreement).
  • The relevance of the CCHS is further supported by the high, and growing, level of usage for the products and services developed by the CCHS. As detailed previously, there has been a significant increase in usage in almost all products/services provided by CCHS.
  • While DFAIT is supportive of the concept of the CCHS, the Consortium has not fully met DFAIT expectations in providing timely policy advice/research assistance with respect to emergent human security issues. However, this deficiency reflects several factors including a lack of formal mechanisms by which DFAIT can inform the CCHS as to upcoming policy/research requirements, as well as the academic structure of the CCHS which is not fully suited to the timely response to DFAIT issues (i.e., use of call for nominations, review and selection process for fellowships/complementary activities generally limits the extent to which CCHS can use some program/services to meet emergent policy needs). Furthermore, it is unclear as to whether DFAIT expects to have the CCHS act as a policy "think tank" or whether DFAIT is comfortable with the role of the CCHS as a "networking facilitator" as well as building research capacity inthe area of human security.

Top of page

4.2 Success

Another of the key objectives of the evaluation was to assess the extent to which the CCHS has been successful in the development and delivery of its products and services. As distinct from the issue of relevance, questions relating to the success of the CCHS focus more narrowly on the generation of outputs and their contribution to the achievement of outcomes as defined within the project's Evaluation Framework.

Constituents felt that the CCHS enhances dialogue and/or interaction between the human security academic and policy worlds, however, some held that there continues to be room for improvement (S9, S12, S16).

Almost three-quarters (73.3%) of CCHS constituents and 80% of stakeholders expressed that the CCHS had been effective in enhancing the dialogue and/or interaction between the human security academic and policy worlds. It was suggested that the CCHS helps build capacity in the Canadian human security research/academic community by creating a common pool of people who share an interest in the subject. Seminars and conferences aimed at engaging participation and input from academic/non-academic, NGO and GOC were thought to be particularly effective.

Analysis of the information obtained from the constituent survey suggests that across all groups (i.e., academics NGO/policy consultants and/or program officers) there is a perception that the CCHS has broadened/enhanced respondents' knowledge and understanding of Government of Canada needs. In contrast, only a low proportion (1%) of respondents felt that the CCHS had limited their ability to understand GOC policy needs.

Chart 4-2: Constituent Perceptions of the CCHS in Terms of Understanding GOC Policy Needs

Constituent Perceptions of the CCHS graph

Source: Constituent Survey
Academics n=54; NGO/Policy Consultants n=21; Program Officers n=24
Total= 109 (includes 10 "other" individuals)

Notwithstanding the high level of support voiced by CCHS constituents about the extent to which the Consortium facilitates information sharing, key informants did note that there was room for improvement in terms of the CCHS' promotion of information sharing and dialogue both within the academic communities and across the other communities of interest (i.e., policy researchers, NGOs, etc.), for example:

  • Stakeholders noted that, until recently, the CCHS has not built effective relationships with Francophone academics/Francophone institutions;
  • There remains limited interaction between the NGO/policy community and academics; although not all stakeholders were of the opinion that this should be an explicit role of the CCHS; and
  • The CCHS has not made effective use of available tools (i.e., fellowships, complementary activities) to align CCHS-sponsored research with DFAIT policy needs more closely.

There is evidence to suggest that the CCHS has been effective in terms of contributing to the number of Highly Qualified People (HQP) in the area of human security in Canada (S10, S11).

Evaluation findings suggest that it is generally agreed among constituents and stakeholders that the CCHS has contributed to the development of highly qualified people (HQP) in the area of human security (60% of constituents and 100% of stakeholders agreed or strongly agreed with the statement). Among the CCHS services thought to promote quality research and researchers in human security are the Human Security Fellowship (HSF) Program, the Human Security Bulletin and the Teaching Human Security Database, which includes a syllabus and material to support instruction in human security.

For example, since implementation of the Human Security Fellowship (HSF) Program in 2001, 38 awards have been offered to researchers specializing in a diverse range of disciplines and, in the case of academic researchers, representing a variety of different universities across Canada. The HSF program has fostered innovative research by academic and non-academic researchers on a range of human security issues, including (but not limited to):

  • the purpose of sanctions and shifting international beliefs about their employment;
  • war-affected children;
  • post-genocide Rwanda;
  • assessing military intervention for humanitarian purposes;
  • the roles of women's organizations in peace-building and post-conflict reconstruction;
  • educating for inter-group conflict or reconciliation;
  • impacts of environmental stress and demographic change; and
  • human rights and peacebuilding in Afghanistan.

To assess the net incremental impact of the fellowships, the outcomes/ perceptions of those individuals who received funding were compared to the outcomes/perceptions of the non-funded applicants. The low number of non-funded applicants demands that caution be exercised in interpreting the results, but several inferences can be drawn from the data. Analysis of the data indicates that:

  • Only one-quarter of the non-funded fellowship applicants were able to undertake their planned research in the absence of CCHS funding. In contrast, one-half reported that they did not complete any of their research, and a further one-quarter noted that the scope of their research had been modestly reduced.
  • Among fellowship recipients, more than 90% reported that their research would not have occurred (42.9%) or would have been substantially scaled back (47.6%) in the absence of CCHS funding.

These findings are illustrated in Chart 4-3.

Chart 4-3: Impact of Not Receiving Award

Impact of Not Receiving Award graph

Source: Constituent Survey
Funded Fellows n=21; Non-funded Applicants n=4

Analysis of the outcomes of fellowship recipients as compared to non-funded applicants suggests that the CCHS has contributed to the development of highly qualified people (HQP) in the area of human security and that these individuals are currently working in capacities that contribute to Canada's Human Security Agenda. For example, a higher proportion of fellowship recipients as compared to non-funded applicants reported: speaking/lecturing on the topic of human security, that they were teaching in the area of human security, and/or that they had published work on the topic of human security.

Chart 4-4: Professional Accomplishments Since Applying for Award

Professional Accomplishment since Applying for the Award graph

Source: Constituent Survey Funded Fellows n=21; Non-funded Applicants n=4

Note that Fellows were asked to state whether any of these occurrences had taken place as a direct result of the award. Non-funded applicants were asked to state whether any of these occurrences had taken place since applying for the award.

In addition, more funded researchers (86%) were working in the field of human security at the time of the survey than non-funded researchers (75%), and more Fellows had secured an academic posting on the basis of research supported by the CCHS than had individuals who were not awarded a Fellowship. These data further suggest that the award may positively affect the career opportunities that are open to researchers and, as a result, the pool of Canadian researchers in the field of human security.

In addition to the CCHS Fellowship, individuals who had received funding from the CCHS to undertake a complementary activity noted that the funding had allowed them to conduct research in human security, for example:

  • 70% of recipients noted that the funding was important (10%) or very important (60%) in terms of assisting their research;
  • 30% were neutral in their response or had no opinion regarding the importance of the funding; and
  • no recipients felt that the funding was not at all important or not very important in enabling them to conduct research in the field of human security.

The Teaching Human Security Canadian Course database contains approximately 1,027 human security-related courses, including 186 graduate level courses and 159 course descriptions with links to course syllabi for 37 human security-related courses offered across Canada. CCHS staff thought that the availability of quality teaching materials enhanced the overall utility and effectiveness of the Consortium, for example:

  • after the introduction of the Human Security Information database (which included access to the teaching materials) in March 2005, the CCHS website witnessed an 11% increase in the number of new visitors per day relative to February 2005 usage statistics; and
  • CCHS staff also indicated that the increase in the number of website "hits" also increased as a result of the introduction of the Human Security database.

The Human Security Bulletin was also thought to contribute to the quality of human security research produced in Canada. For example, the research completed by CCHS Fellows, including abstracts and summaries, is available to the public via the Human Security Bulletin. Fellows, themselves, are also profiled (e.g., past and present research history, area of expertise, publications, etc.) on the HS Bulletin, making this site an important platform for showcasing innovative research on a range of issues related to human security, facilitating information sharing and further establishing a network of researchers in this field. Moreover, several stakeholders indicated that the HS Bulletin was a CCHS service that exceeded their expectations.

Three-quarters (75%) of stakeholders and 60% of constituents agreed/strongly agreed that a network of researchers has been developed as a result of the programs and services administered by the CCHS, however, there was some concern about its sustainability. For example, delays in funding allocations, especially in the last phases of the Consortium's lifespan, have resulted in staff lay-offs, uncertainty about the CCHS' ability to run the fellowship program, as well as hesitation about the Consortium's ability to meet its program requirements or its future in general. In addition, a high rate of turnover at DFAIT has also contributed to insufficient communication between DFAIT and the CCHS and has rendered a consistent relationship between the two difficult to maintain.

Results show that constituents and stakeholders feel that the CCHS has enhanced Canada's stature in the field of human security and increased public attention to the Canadian HS agenda nationally and internationally (S14, S15).

The results of the key informant interviews and constituent survey suggest that there is a high level of agreement that the CCHS has both improved the coordination of research and has increased public and international attention as to Canada's role in terms of the Human Security Agenda. As highlighted in Chart 4-5, among individuals who had an opinion, the majority felt that the CCHS had improved the coordination of research and that the Consortium had contributed to Canada's reputation as a leader in the field of human security policy development.

Chart 4-5: Constituent Opinion as to CCHS' Role in Research Coordination/Promotion of Canada's Role in Human Security Policy Development

Constituent Opnion as to CCHS' Role in Research Coordination graph

Source: Constituent Survey
n=83
Note: Includes valid responses only

Among key informants, all (100%) of the individuals interviewed felt that the CCHS has increased national and international attention on the Canadian Human Security Agenda. It was further agreed by most interviewees that the CCHS has improved the coordination of research and activities with partner institutions, both in Canada and internationally (88% agreed/strongly agreed).

Numerous key informants provided several examples as to how the CCHS had enhanced Canada's stature in the field of human security. For example, the 2005 Peace Prize Symposium was organized by the Canadian Consortium on Human Security at the Centre of International Relations, UBC and was held February 3-5, 2005. The goal was to generate debate, increase focus on and public awareness about human security issues related to peace, such as intervention, conflict prevention and peacebuilding. The event was attended by representatives of governments and academia from numerous countries, including Canada, Norway, Sweden, and the US.

In an effort to generate attention to the common agenda of Canada and Norway, the Royal Norwegian Embassy in Ottawa and the CCHS are striving to establish the Canada-Norway Peace Prize Symposium as an annual, high profile event. The CCHS planned to hold the second annual event in January 2006 at York University, however, due to the timing of the federal elections, it was postponed in Phase III and due to funding issues, the date has yet to be determined.

In discussions with the CCHS, it was noted that a number of jurisdictions, other than Norway, had expressed interest in the role/objectives supported by the CCHS. For example, the CCHS noted that several countries including Chile, Thailand, Japan and Slovenia had expressed an interest in developing a similar model/institution in their own country based on their interaction with the CCHS.

It was generally felt that the CCHS had made strides to sponsor quality research and other initiatives in both official languages, but that more could be done in this area (S13).

During Phase II of operations, the CCHS turned its attention to increasing participation of the Francophone community in CCHS activities(12). The Human Security Bulletin, CCHS website and high-profile human security consultations in Montréal were the principal methods utilized to increase the participation of the Francophone community in CCHS. At the beginning of Phase II, CCHS began preliminary discussions with the Canadian Research Chair in Canadian Foreign and Defense at the Institut d'Études Internationales de Montréal at the Université du Québec à Montréal (UQAM) to hire a part-time Francophone Research Coordinator.

A Francophone Research Coordinator was hired in May 2004 to provide French language research support for the activities and publications of Canadian Consortium on Human Security (CCHS), including the Human Security Bulletin and the Human Security Gateway. The Coordinator was also partly responsible for disseminating information about CCHS to interested parties, as well as assisting in performing outreach to the francophone community, research, editing and other organizational tasks.

As part of the CCHS' Francophone Outreach Strategy, CCHS funded and organized a Francophone Consultation: (Human Security: ten years later (1994-2004)/ La sécurité humaine: dix ans plus tard (1994-2004)) held at Université du Québec à Montréal (UQAM) on October 28-29, 2004. This conference represented the first effort by the CCHS and the Canada Research Chair in Canadian Foreign and Defence Policy to engage the Francophone community of experts working on human security issues. Among the primary objectives of the conference were to assess how the concept resonates in Francophone Canada, particularly among three key stakeholders: academics, NGOs and civil servants. As well, the conference was intended to develop policy recommendations on human security and its application and to create a network of Francophone experts on human security.

In addition to the Francophone Outreach strategy, the CCHS has undertaken an number of other bilingual activities. For example, all of the websites developed and maintained by the Consortium (the CCHS website, the HS Bulletin site, the HS and Cities Agenda website, the Gateway) are available in and showcase research in both official languages which represent approximately 19% of the total number of publications on the website. As of June 2006, the HS Gateway included 176 French language sources, and the Teaching Human Security Database features more than 115 HS-related graduate and undergraduate courses at universities across the country that are delivered in French.

While stakeholders were in general agreement that an improvement in the rate of engagement of Francophone scholars and higher outputs in the French language could be made, overall 70% of key informants felt that the CCHS had successfully sponsored research and other initiatives in both official languages.

Analysis of the actual program activities undertaken by the CCHS underscores the increasing attention given to providing services in both official languages. For example, as highlighted in Chart 4-6, based on a review of complementary activities, there has been a significant increase in the number of French language activities sponsored by CCHS.

Chart 4-6: Proportion of French Language Complementary Activities Supported by CCHS by Phase

Proportion of French Language Complementary Activities supported by CCHS by Phase graph

Source: Final Narrative Reports, various phases

It should also be noted that among stakeholders interviewed, there was some concern that the "counting" of French language publications would not be an effective way to measure the interaction of Francophone researchers as it was noted that many Francophone researchers may choose to publish their research in English due to the wider potential audience/readership of English language articles and/or publications.

The CCHS has delivered its products and services in timely way and within, and sometimes below, budget (S17).

Based on information obtained from key informant interviews and analysis of CCHS financial reports, it appears that the CCHS has delivered its products and services within budget, and in some instances, actual expenditures were below projected expenditures.

Of importance is that the financial relationship between the CCHS and DFAIT has been, and continues to be, a major impediment to pro-active planning and program delivery by the CCHS. The one-year funding cycle, and delays in project funding, has had several detrimental effects on the operations of the CCHS, including:

  • staff turnover, as research/administrative assistants cannot be hired for more than one year at a time, and funding lapses have resulted in times where UBC has provided "bridge funding" for key staff pending project funding from DFAIT;
  • an inability to award research/scholarship funding for more than one year; and
  • an inability to complete all planned activities (i.e., workshops, conferences) due to delays in funding approval.

As detailed in Table 4-2, the delays and uncertainties associated with project funding have generally resulted in the CCHS under-spending in terms of actual vs. planned expenditures. In fact, during the period from December 2001 to March 31, 2005, the CCHS spent on average, only 94% of planned expenditures.

Table 4-2: Planned vs. Actual Expenditures - CCHS Selected Period ($000s)
PeriodPlanned Expenditures (1b)Actual Expenditures (1bb)Actuals as % of Planned

Source: CCHS Financial Reports. Funding data was not available for 2006-2007.

(1b), (1bb) includes DFAIT and other sources of funds

(2b) no funding provided for this period from September 1, 2003, to March 30, 2004

Dec. 18, 2001 - Mar. 31, 2002132.2114.886.8%
April 1 - June, 2002n/an/an/a
July 1 - Sept. 30, 200275.576.1100.8%
Oct. 1 - Dec. 31, 2002166.8156.093.5%
Jan. 1 - Mar. 31, 200398.193.595.3%
Apr. 1 - June 30, 2003137.8180.9131.3%
July 1 - Aug. 21, 2003 (2b)190.6198.1103.9%
Apr. 2004 - Mar. 31, 2005538.1441.782.1%
Total1,339.11,261.194.2%

The CCHS appears to have succeeded in providing its constituency with quality information which strengthens research and policy development capacity (S16).

Analysis of the information provided in the constituent survey suggests that CCHS' key constituent groups are of the opinion that the Consortium is facilitating information sharing, enhancing dialogue and supports the development of a sustainable network of researchers that facilitates information sharing between constituent groups.

Highlighted in Chart 4-7 are the proportion of respondents who agreed or disagreed with selected statements with respect to the role of the CCHS in terms of facilitating collaboration and information sharing. It should be noted that totals will not add to 100% as neutral responses are not displayed.

Chart 4-7: Constituent Opinion as to the Role of CCHS in Terms of Facilitating Collaboration and Information Sharing

Constituent Opinion Role of CCHS Facilitating Collaboration and Information Sharing

Source: Constituent Survey
n=83 to 98
Note: includes valid responses only.

Notwithstanding concerns among DFAIT staff that the CCHS has yet to fully meet DFAIT expectations with respect to Departmental policy requirements, it appears that the results of the constituent survey support the finding that the CCHS has made a positive contribution in the area of influencing research on human security and positive contribution to the shaping of the HS research agenda in Canada.

As highlighted in Chart 4-8, it appears that the majority of all constituents who had an opinion agree or strongly agree (80%) that the CCHS has positively contributed to shaping the research agenda in human security in Canada. In contrast, a lower proportion of respondents (64%) noted that the work done by CCHS had influenced their own human security-related research. It should be noted that less than 15% of respondents were of the opinion that the CCHS did not influence research in the field of human security.

Chart 4-8: Respondent Opinion as to Whether CCHS Positively Influences Research Related to Human Security

Respondent Opinion Whether CCHS Positively Influences Research graph

Source: Constituent Survey
Academics n=86 to103
Note: includes valid responses only

Summary of Findings - Success

Based on a review of multiple lines of evidence, the following observations can be made with respect to the success of the CCHS in meeting various objectives.

Information obtained from the key informant interviews, constituent survey and a review of administrative data suggests that the CCHS is seen as an effective vehicle for supporting research collaboration and dialogue both within the academic community and the broader (both domestic and international) human security research community.

  • Among its constituents, the CCHS is seen as more of a "window" into the policy needs of government rather than as a gatekeeper which may restrict/limit the contact between the government and the academic community.
  • The CCHS has a number of programs/services that contribute to the development of Highly Qualified People (HQP) who work in the field of human security. Based on comparisons of outcomes of funded vs. non-funded fellowship applicants, it appears that CCHS fellowships have contributed to a modest positive net impact in terms of HQP and/or research related to human security.
  • While there have been concerns voiced as to the extent to which the CCHS meets the needs of the Francophone research community, the CCHS has increased its level of French language resources although it is possible that more attention should be given to enhancing the input/contribution of Francophone researchers/institutions.
  • Both key informants and CCHS constituents are of the opinion that CCHS has positively contributed to Canada's image as a leader in human security on the international stage. This report has highlighted several examples of relationships that have been developed by the CCHS.
  • Notwithstanding the general high level of support for the CCHS, the uncertainty regarding funding has contributed to delays in the provision of programs/services and has often resulted in the CCHS not being able to expend allocated resources within the fiscal timeframe.

Top of page

4.3 Efficiency and Cost Effectiveness

A key issue examined in the evaluation is the extent to which the CCHS represents an efficient and cost effective mechanism through which DFAIT can strengthen the research and policy development capacity in the area of human security in Canada. Also included in this section is an analysis of the performance measurement system in place at both CCHS and DFAIT to facilitate ongoing assessment of program strengths or weaknesses.

Has the CCHS/DFAIT established appropriate planning, budgeting and performance measurement systems to ensure initiatives are managed efficiently and effectively? (E18, E19)

Analysis of the administrative systems established by the CCHS suggest that the Consortium can easily track a number of financial and/or program activity indicators. As part of the University of British Columbia, the CCHS does benefit from strong support in the area of financial management and reporting. DFAIT program officers further confirmed the adequacy and comprehensiveness of CCHS financial reporting.

The CCHS has considerable statistics regarding program activity. For example, CCHS Annual Reports include detailed information as to the number of fellowships awarded, number of complementary activities funded, number of publication and website hits. In this context, the CCHS (and DFAIT) have the mechanisms to monitor the activity level of CCHS programs and services.

While formal and administrative systems are in place to monitor program activities and/or expenditures, both CCHS and DFAIT have yet to establish systems to measure the effectiveness of program activities or outputs. For example, until this evaluation was undertaken, the CCHS had not completed any formal internal analysis of client satisfaction with products and services (i.e., exit surveys with conference participants, follow-up surveys with fellowship recipients, online surveys for users of various online publications/products).

Of note is that while performance measurement related outcomes/indicators were included in the 2005-2006 contribution agreement, the CCHS has used its own template for results reporting purposes. It is unclear as to whether the CCHS has actually collected the required data that would provide information as to program effectiveness.(13)

It should also be noted that DFAIT has not required the CCHS to provide information as to the effectiveness of CCHS programs/services. Although DFAIT receives program documentation from the CCHS, it is unclear as to what analysis, if any, is completed with respect to the internal tracking of activities and/or outputs. Furthermore, review of the Contribution Agreement shows little direction in terms of evaluation.(14)

Clearly, both DFAIT and the CCHS need to establish appropriate performance measurement benchmarks that include an assessment of client satisfaction with the CCHS and measures to establish the extent to which the CCHS has met DFAIT objectives. From DFAIT's perspective, future contribution agreements should also include DFAIT expectation in the area of policy research coordination, which is an "implicit" DFAIT objective but is not currently clearly defined in the contribution agreements examined by the Consultant.

Are the resources and activities being directed to initiatives and individuals/groups most likely to yield the best results?

Analysis of the CCHS budget for the period 2001-2003 and 2005/06 highlights the changes in relative emphasis across the different programs and services. For example:

  • Whereas expenditures on fellowships represent more than one-half of the total CCHS expenditures (50.3%) in 2001-2003, by 2005/06, fellowship expenditures represented only 11.8% of total CCHS expenditures;
  • Expenditures on the Bulletin/Gateway increased from 5.0% of total expenditures in 2001-2003, to 15.9% in 2005/06; and
  • There has been a modest increase in the proportion of expenditures allocated to CCHS secretariat/overhead functions (29.2% to 35.9%).

Analysis and comparison of the CCHS budget for FY01/02 and FY05/06 show close to a 10% increase in the amount of the budget spent on secretarial support functions between 01/02 and 05/06, and a slight decrease in the cost of overhead over the same timeframe. That is, secretariat expenses rose from just over 20% in 01/02 to approximately one-third (30.4%) of the budget in 05/06. This change, however, can be attributed to the increase in the number of programs/services supported by the CCHS over this same time period. The amount of budget allocated to programs decreased slightly from 61.1% in 01/02 to 57.5% in 05/06. These figures are shown in Table 4-3 below.

Table 4-3: Expenditure Profile - CCHS (Selected Periods)
Area2001 - 20032005 - 2006 (planned)

Source: Contribution Agreement, April 19, 2005. Proposed Budget, 19 April 2005 - 31 March 2006
Contribution Agreement Year 1 and 2.

(1c), (1cc) Includes DFAIT contributions plus Other Funds

(2c) Fellowships estimated to be $63,500 not $103,500 as listed in the financial report

$ Amount (1c)% of Program Area% of Total$ Amount (1cc)% of Program Area% of Total
Administration
CCHS Secretariat193,499n/a21.4%164,124n/a30.4%
Overhead70,340n/a7.8%29,740n/a5.5%
Sub-total - Administration263,839n/a29.2%193,864n/a35.9%
Programs
Bulletin/Gateway45,2008.2%5.0%85,90027.6%15.9%
Fellowships454,00082.2%50.3%63,500 (2c)20.5%11.8%
Complementary Activities/Other52,9509.6%5.9%161,00051.9%29.8%
Sub-total - Programs552,150100.0%61.1%310,400100.0%57.5%
HS Colloquium87,030n/a9.6%35,276n/a6.5%
Total903,019n/a100.0%539,540n/a100.0%

In order to provide perspective as to whether the CCHS allocates appropriate resources to the various program areas, the Consultant has compared current program expenditures against the "effectiveness" ratings provided by survey respondents. As detailed in Table 4-4, it appears that while all programs/services provided by the CCHS are valued by CCHS constituents, given the high level of effectiveness assigned to both Gateway resources and Fellowships, some reallocation of funds should be considered in terms of the current expenditure profile that has the majority of program expenditures spent on complementary activities.

Table 4-4: Comparative Rankings of Programs/Services Provided by CCHS Relative to CCHS Expenditures FY05/06
AreaMean Score (1d)% Citing Effective / Very Effective% of CCHS Program Budget (2d)
2005/06

Source: Constituent Survey

(1d) 1=Very Ineffective, 3=Neutral, 5=Very EffectiveM

(2d) See previous Table 4-3

Human Security Gateway4.1984.7%27.6%
Human Security Bulletin4.0984.0%
Fellowships4.1186.5%20.5%
Complementary Activities4.0183.8%51.9%

It should be emphasized that there was no agreement across all stakeholder groups as to whether funds were being directed to the appropriate program areas. For example, academic stakeholders viewed the fellowships programs as being important and an area that could benefit from increased funding. In contrast, DFAIT officials felt that the CCHS should limit expenditures directed to fellowships and increase resources targeted toward maintenance/expansion of the Gateway as well as more funding for complementary activities. Notwithstanding the above, there was consensus between DFAIT and the CCHS that within the Fellowship Program, PhD fellowships were yielding better results than the MA or post-doctorate fellowships.

What is the feasibility of the CCHS securing/diversifying alternative sources in order to ensure its long-term visibility (E23)?

Analysis of the original CCHS proposal suggests that initially the Consortium was envisioned that it would become a self-sustaining entity as described in the proposal. For example, the proposal authors expected that the CCHS would:

"Evolve into a bottom-up one [organization] featuring a self-sustaining, national network, representative of the spectrum of institutions across the country and distribution of faculty and student resources."(15)

It should be noted that, while the project proponents did envision the CCHS becoming less dependent on DFAIT and/or other direct government funding, analysis of the contribution agreements signed between DFAIT and UBC/CCHS suggests that no formal goal or target of revenue diversification was explicitly included in the operating agreement.

Notwithstanding the lack of direction/understanding as to whether the CCHS was intended to become financially self-sufficient, analysis of financial data for the period from 2002 to 2004/2005 suggests that the CCHS has significantly increased its proportion of non-DFAIT contributions (in-kind and/or other contributions) from 20% of total expenditures in 2002 to 29% of total expenditures in 2004/05.

Table 4-5: CCHS Expenditures by Funding Source Selected Years
PeriodDFAIT Grant FundsIn-Kind/Other SourcesTotalOther Sources as % of Total
2002 (from Dec. 18, 2001)$353,258.75$87,702.81$440,961.5620%
2003 (from January to August 2003)$356,682.28$106,836.84$472,519.1223%
1 April 2004 - 31 March 2005$312,790.48$128,906.30$441,696.7829%

Source: CCHS Financial Reports, various phases.
Note: no funding was provided for the period from September 2003 to March 31, 2004.

Key informants interviewed noted that there was only limited opportunity for CCHS to diversify its funding base and remain viable should DFAIT reduce and/or cease funding the Consortium. DFAIT stakeholders noted that other similar organizations (i.e., CPCC) were also reliant on DFAIT funding although the CPCC had made some progress in terms of securing alternative funding from other agencies/organizations (i.e., CIDA). However, it was unknown as to whether the CPCC had been able to secure a similar level of in-kind/alternative funding that had been the case for the CCHS (i.e., 2004/05, almost 30% of CCHS funding was in the form of in-kind or other (non-DFAIT) contributions). In addition, few stakeholders were of the opinion that the CCHS could derive sufficient funds using a "fee for service" model that would enable the Consortium to operate effectively in the absence of DFAIT funding.

It should be noted, however, that there exists some scope for the CCHS to potentially diversify its funding base through the use of "user fees" (cost recovery for some products/services) for on-line products and/or the use of subscription fees for publications. It is unclear, however, as to whether the fees generated for such items would discourage interest/participation in the Consortium and hence indirectly undermine the goal of networking and capacity building in the area of human security.

What best practices, insights, or lesson learned have been drawn from the CCHS experience as well as the experience of other agencies engaged in like activities and with similar resources, and how well has the CCHS [and/or DFAIT] learned from these experiences?

Key informants provide several suggestions and/or lessons learned in terms of the future implementation of a similar initiative. Given that the CCHS is a relatively unique organization, and that other models that deal with human security do not generally exist and/or provide the same range of products and services, it was difficult for stakeholders to compare the CCHS experience to that of other similar organizations. However, based on a review of information provided by stakeholders (CCHS, DFAIT, other), it was identified that practices/policies would reflect the "lessons learned" as part of this initiative. Those "lessons learned" have been grouped under the following headings:

  • Funding
  • Program Governance/Communication
  • Policy/Program Issues
CCHS LearningsDFAIT Learnings
1. Funding
  • To support long-term planning, and better overall project management, multi-year funding is essential.
  • Recognizing the challenges of FY funding model, would explore different funding models in the future.
  • Need to get explicit direction from DFAIT as to the need for secondary funding.
  • DFAIT should define whether the CCHS will sunset. If it is an important initiative for DFAIT, then the Department should be prepared to fully fund the organization.
2. Program Governance/ Communications
  • CCHS would welcome more/better communication with DFAIT.
  • DFAIT should establish a formal planning cycle whereby priorities/direction of the Department can be communicated to the CCHS.
  • DFAIT expectations need to be clearly defined in terms of planning/program delivery.
  • DFAIT may consider more active participation in review/selection but may be constrained by legal requirements of the contribution agreement to be more "hands off."
  • DFAIT should take a more active role in terms of assisting in selection of Fellows/ Complementary Activities.
  • DFAIT should establish formal processes to measure the extent to which the CCHS has met Departmental objectives (i.e., performance measurement framework).
 
  • DFAIT should be more pro-active in terms of its communication with the CCHS.
3. Policy/Program Issues
  • CCHS sees itself as primarily serving the academic community using standard academic tools - this may not be the same vision as DFAIT.
  • DFAIT has expectations that the CCHS would be more responsive to Departmental priorities rather than the priorities of the academic community - future structure should incorporate process through which CCHS could better meet/ accommodate DFAIT program objectives.

Does the sponsorship of the CCHS and its activities constitute the most effective means of strengthening the research and policy development capacity in the area of human security within Canada?

Despite some reservations as to the extent that the CCHS had explicitly met DFAIT goals and objectives, almost all key informants were of the opinion that the CCHS represented an effective means of strengthening research and policy development capacity in the area of human security in Canada.

Analysis of administrative data would suggest that the CCHS is contributing to such capacity, as the CCHS has witnessed a considerable increase in the volume of activity associated with the products and services that it supports.

Many key informants felt that the CCHS provided good value for money for the Department, for example:

"The CCHS does provide value, it provides a way to create a low cost research network which is better and more cost-effective than previous DFAIT models in which DFAIT sponsored fellowships/other research directly."

"The CCHS has done great work in terms of getting the message out and building an inventory of resources related to human security."

Analysis of the data from the constituent survey suggests that respondents among the various constituent groups were uncertain whether the CCHS represents the most effective way to strengthen the research and policy development capacity in the area of human security in Canada. For example:

  • Overall, just over one-half (50.5%) of constituents surveyed felt it was the most effective mechanism to achieve the stated objectives;
  • Just over 13.8% felt that it was not the most effective mechanism; and
  • More than one-third (35.0%) were unsure or had no opinion with respect to the effectiveness of the CCHS in terms of meeting this objective.

Chart 4-9: Extent to Which Respondent Believes that CCHS/CCHS Activities are Most Effective Means of Strengthening the Research and Policy Development Capacity in the Area of Human Security

Extent Respondent Believes CCHS Activities are Most Effective graph

Source: Constituent Survey
Academics n=109

Survey participants offered several comments as to their views on whether the CCHS was the most effective means of strengthening human security research and policy development capacity. A sample of comments is provided below:

"There is no other mechanism that is specifically tasked with promoting the research agenda for human security issues. CCHS provides a flexible, rapid and cost-effective avenue for academics and others to pursue urgent applied research to breaking human security issues."

"It's unfair to say that it is the most effective, but it is certainly part of a larger suite of expertise that the government of Canada can draw upon. It is unique in that it is Canada-wide and a good mix of academics and policy experts."

"It has helped to shape the policy research agenda of the department of foreign affairs, as well as the types of individual action projects funded."

"The CCHS provides a forum where human security issues (and emerging issues such as the Cities Agenda) can be publicized and shared."

The Consultant also examined other potential models to fund research/networking in an academic environment. Detailed below is a brief discussion of possible alternative models.

Network of Centres of Excellence (NCE)

Natural Sciences and Engineering Research Council (NSERC), Social Sciences and Humanities Research Council (SSHRC). The Network of Centres of Excellence program was developed in order to foster research collobaration across Canadian universities. In 2006, there were approximately 20 NCEs in operation in Canada. Prior to 2006, a NCE received funding in the range of $4 million to $5 million to support both research and networking functions. In 2006, NSERC/SSHRC launched a new program (NCE - New Initiative) which included funding of a NCE for two years ($800,000 or $400,000 per year) to support networking activities only. While NCE-NI may represent an alternative model to the CCHS, analysis of the activities funded by NCE-NI suggests that such operations do not include Fellowships nor do they generally include the development of a on-line database containing relevant materials/information.

NSERC/SSRHC Joint Initiative Program

The Joint Initaitve (JI) program represents an mechanism whereby different departments with a common issue can pool resources and support multi-year research projects that are managed by NSERC/SSHRC. While such initiatives are not generally for the purposes of networking, in many instances a portion of the funding will be used to support information sharing/networking among the principle/related research teams.

Summary of Findings - Efficiency and Cost Effectiveness

  • While the CCHS benefits from a strong administrative and financial accounting system, the CCHS has not established a viable set of performance measure that can assist in the on-going measurement as to the extent to which the CCHS has met program objectives.
  • Analysis of the activities and outputs of the CCHS suggest that the Consortium is completing a variety of activities, and has been able to attract significant non-DFAIT support (29%) of the program budget consist of non-DFAIT contributions in the form of in-kind contributions and/or other support. There may, however, be additional opportunities to further diversify funding through the use of user fees, although more study should be undertaken to understand potential negative impacts associated with such cost recovery.
  • Based on a review of the constituent survey results, there is a high level of support for the various programs and services provided by the CCHS. While, DFAIT staff feel that there may be a disproportionate emphasis on the Fellowship Program, a comparison of the 01/02 and 05/06 budget shows a significant decrease in the proportion of program funds allocated to fellowships over that timeframe (50.3% in 01/02 compared to 20.5% in 05/06) and a corresponding increase in the amount of funds dedicated to complementary activities (5.9% in 01/02 compared to 51.9% in 05/06).
  • Given the considerable level of activity achieved by the CCHS, most key informants felt that the CCHS represented a cost-effective mechanism to increase dialogue and policy development capacity in the academic community with respect to human security. DFAIT staff noted that other alternatives (i.e., DFAIT in-house processes to house/manage policy/research consultation) would likely be more costly relative to the CCHS model.
  • Relative to other potential models (e.g., NCE, Joint Initiatives Program), it appears that the CCHS has represented a cost-effective mechanism to support networking, HQP development and development/maintenance of informational database(s).

Top of page

5.0 Conclusions and Recommendations

5.1 Conclusions

The overall conclusion of the Summative Evaluation of the Canadian Consortium on Human Security is that the CCHS has succeeded in broadening awareness about human security and has, moreover, increased dialogue about and strengthened capacity in human security, particularly across the academic community.

Analysis of the data suggests that among both key informants and constituents surveyed, the CCHS has been an effective vehicle to enhance interaction and dialogue across the academic community about human security. The CCHS is also seen as being an important contributor to the development and support of Highly Qualified People (HQP) who are active in human security and are contributing to the human security agenda.

In general, analysis of the outputs/activities and constituent satisfaction with CCHS products and services suggests that CCHS serves as a cost-effective model to support DFAIT's objective to improve the coordination of research and activities with partner institutions, both in Canada and internationally. In addition to DFAIT's direct contribution of approximately $300,000 per year, the CCHS obtains in-kind and other expenditures of approximately $100,000 annually. In this context, each $1 expenditure by DFAIT "leverages" approximately $0.29 in non-DFAIT expenditure or in-kind support. DFAIT staff acknowledges that it would be unlikely that DFAIT could undertake the tasks and activities done by CCHS using fewer resources than currently provided to the CCHS.

Although the CCHS appears to be meeting the broad needs of its constituent community, the results of the evaluation suggest that there are some aspects of the CCHS which have not met DFAIT expectations, including:

  • the ability of CCHS to mobilize a broad range of academic experts quickly to provide timely advice regarding emergent policy/programs issues as identified by the Department;
  • the ability of CCHS to coordinate research dialogue effectively among Francophone researchers; and
  • the extent to which CCHS activities align to DFAIT expectations/requirements (i.e., preference for complimentary activities rather than scholarships).

It should be noted, however, that these shortcomings are to some extent the result of the apparent absence of an effective mechanism to convey DFAIT policy/program needs to the CCHS (poor communications structure) as well as a function of a disjuncture in the production cycles between DFAIT and the academic research community; the policy development cycle for DFAIT can be measured in weeks and months, while academic research may take years, culminating in publication.

5.2 Recommendations

Based on the results of the evaluation, the Consultant has provided several recommendations under the themes of program relevance, success and efficiency and cost effectiveness.

Relevance
Finding: 1

Evaluation results demonstrate that CCHS has been effective in meeting broader GOC/DFAIT objectives regarding the human security agenda. However, the ability of the CCHS to provide programs/services has been impaired by funding uncertainties, which have characterized CCHS operations since 2003.

Recommendation 1:

As the CCHS fills a valuable role in coordinating and promoting Canada's Research Agenda with respect to human security, the Department should establish more permanent funding for the CCHS.

Success
Findings: 2

While CCHS has met the broad objectives set for it, the CCHS is perceived to be less responsive to emergent DFAIT policy needs. Improved communication mechanisms, in the form of joint planning meetings, would help ensure that the CCHS receive explicit direction as to DFAIT policy/research/consultation requirements.

Recommendation 2:

Establish more formal communication mechanisms to support the collaboration between DFAIT and the CCHS.

Findings: 3

Work done by the CCHS has been seen to support the development of Highly Qualified Personnel, both through the Fellowship program as well as through the provision of teaching resources.

Recommendation 3:

That DFAIT be supportive of the Fellowship program offered by CCHS, and where possible, DFAIT may wish to specify "theme areas" or topics that could be supported through the Fellowship program.

Findings 4:

While a performance measurement framework has been developed for the CCHS, reporting on the key elements of the framework has not occurred. Performance measurement information would provide the CCHS/Department with timely information as to results achieved and outcomes associated with DFAIT's investment in the CCHS.

Recommendation 4:

Implement a Performance Measurement Framework for the CCHS.

Efficiency and Cost Effectiveness

Findings 5:

The CCHS has made progress in diversifying its funding sources, but has never been formally directed to achieve financial self sufficiency. DFAIT, in consultation with the CCHS, should identify the proportion of total funding that the Department is expected to provide.

Recommendation 5:

Establish program targets in terms of cost sharing of the CCHS.

DFAIT, in consultation with the CCHS, should identify the proportion of total funding that the Department will expect to provide in the future. Currently, the CCHS is unaware as to whether the Consortium is expected to achieve self-sufficiency in the medium to long term.


Top of page

Annex A: Management Response

The Human Security Policy Division (GHS) has read with interest the Summative Evaluation of the Canadian Consortium on Human Security (CCHS). GHS is pleased that the evaluation gave the Consortium good marks on relevance, success, program efficiency and cost effectiveness. This notwithstanding, the evaluation did identify areas of improvement for both parties, such as clarified expectations/objectives, improved policy direction from DFAIT, and enhanced performance and results reporting capacity by the CCHS. Management's response and action plan on these issues is detailed below.

DFAIT-CCHS Background and Context

The CCHS was established in December 2001 with funding provided by DFAIT's Human Security Program (now Glyn Berry Program). The Consortium was one component of the 'Partnerships' sub-envelope, which sought to establish Canadian academic- and NGO-based networks to generate and promote policy-relevant human security research; facilitate dialogue with government, NGO, and private sector partners; and exercise a leadership role in the international academic and think-tank communities on human security. The Consortium spearheaded the academic network, while the Canadian Peacebuilding Coordinating Committee (CPCC) directed NGO efforts. Each organization received up to $360,000 per year.

As part of the most recent phase of funding (fiscal year 2006-07), IRD recommended to GHS that an evaluation of the CCHS be conducted to determine the latter's utility and cost effectiveness. ZIE agreed to lead the evaluation process, which began in the fall of 2006.

Evaluation Findings

The CCHS scored well on the measures of relevance, success and efficiency and cost effectiveness. In terms of relevance, the evaluation notes that 90% of stakeholders interviewed felt strongly that the activities and services administered by the CCHS are consistent with, and supportive of, Canada's human security agenda. Overall, the broader constituent groups were very supportive of the work done by the Consortium in facilitating dialogue and policy advice regarding human security. The evaluation also concludes that the products and services developed by the CCHS are valuable resources for research and dialogue in human security, as viewed by stakeholders and constituents. In terms of efficiency and cost effectiveness, the majority of stakeholders and constituents felt that the CCHS model represented the most efficient way to achieve the stated objectives. While no single research model is without problems, the evaluators noted that a similar model for policy dialogue and research generation, the NSERC/SSHRC-supported Network of Centres of Excellence, cost about $4 to $5 million each per year, as compared to the $360,000 for the CCHS' activities.

Management Response and Action Plan

The evaluation details five areas for improvement, and provides recommendations for action in these cases. The findings/recommendations and GHS' response to these issues are:

Finding 1:

Evaluation results demonstrate that CCHS has been effective in meeting broader GOC/DFAIT objectives regarding the human security agenda. However, the ability of the CCHS to provide programs/services has been impaired by funding uncertainties, which have characterized CCHS operations since 2003.

Recommendation 1:

As the CCHS fills a valuable role in coordinating and promoting Canada's Research Agenda with respect to human security, the Department should establish more permanent funding for the CCHS.

GHS Response and Action Plan:

We agree that a longer-term funding arrangement would be beneficial but it may not be possible due to the terms and conditions of the Glyn Berry Program (GBP) and its parent, the Global Peace and Security Fund.

Nonetheless, two-year funding may become a reality in April 2008 (broader terms and conditions expire in 2010 and may be extended). GHS will work with IRD to determine if a multi-year funding arrangement would be possible for the CCHS. If not, GHS will seek to improve its funding approvals cycle to avoid gaps in support. Currently, there is a break every April as new agreements await the approval of a Partnerships envelope concept paper. GHS will explore drafting concept papers for an upcoming cycle in the current year to avoid these delays.

Finding 2:

While CCHS has met the broad objectives set for it, the CCHS is perceived to be less responsive to emergent DFAIT policy needs. Improved communication mechanisms, in the form of joint planning meetings, would help ensure that the CCHS receive explicit direction as to DFAIT policy/research/consultation requirements.

Recommendation 2:

Establish more formal communication mechanisms to support the collaboration between DFAIT and the CCHS.

GHS Response and Action Plan:

GHS has, for the most part, not conceived of the Consortium as a 'responsive' organism. This is a function of two things: 1) the fact that policymakers work on an accelerated cycle when compared to academics, who often have long-term research priorities; and 2) the CCHS Secretariat was never resourced to be an interlocutor between the policy and academic worlds, in the sense that it would mobilize an academic response to individual DFAIT issues. Nevertheless, CCHS has performed admirably in the few instances where DFAIT has requested more active participation of the academic world (e.g. on Human Security and Cities).

That said, there is clearly room to communicate more clearly and frequently DFAIT's current and future priorities. GHS will aim to convene the Executive Committee of the CCHS on a 12-18 month basis to signal priorities and discuss agendas. On a more ad hoc basis, GHS will hold Fast Talks (a rapid expert consultation mechanism) with interested CCHS constituents on a new or emerging issue. This will allow GHS to reach into the pool of experts and test new concepts. In combination, both these elements may encourage greater coordination between academic research agendas and government policy development, which should improve the quality of both.

Finding 3:

Work done by the CCHS has been seen to support the development of Highly Qualified Personnel, both through the Fellowship program as well as through the provision of teaching resources.

Recommendation 3:

That DFAIT be supportive of the Fellowship program offered by CCHS, and where possible, DFAIT may wish to specify "theme areas" or topics that could be supported through the Fellowship program.

GHS Response and Action Plan:

GHS is wholly supportive of this idea, as it would offer us the opportunity of tapping into a pool of young academics to develop new research. GHS will seek to develop "theme areas" in advance of each new fiscal year - beginning 1 April 2008 - and will consult other human security divisions for ideas.

Finding 4:

While a performance measurement framework has been developed for the CCHS, reporting on the key elements of the framework has not occurred. Performance measurement information would provide the CCHS/Department with timely information as to results achieved and outcomes associated with DFAIT's investment in the CCHS.

Recommendation 4:

Implement a Performance Measurement Framework for the CCHS.

GHS Response and Action Plan:

GHS agrees. A new and much improved version of the Results-based Performance Framework (RBPF) was released last year. However, we were disappointed that the CCHS' reporting, while voluminous (over 100 pages), did not follow the standard format. GHS will ensure that the CCHS is aware of the need to report in full accordance with the RBPF.

Finding 5:

The CCHS has made progress in diversifying its funding sources, but has never been formally directed to achieve financial self sufficiency. DFAIT, in consultation with the CCHS, should identify the proportion of total funding that the Department is expected to provide.

Recommendation 5:

Establish program targets in terms of cost sharing of the CCHS.

GHS Response and Action Plan:

GHS has suggested that CCHS actively explore other options for funding within Canada. This would be in keeping with our practice of entering into contribution agreements (i.e., contributing to a larger project), not grants. Historically, the CCHS has provided between 20 and 30% of project costs in-kind, which is a target we will continue to strive for. Additionally, GHS will encourage the Consortium to approach other Canadian donors, with a view to 10% of project costs being recouped in this fashion.

International funding would also be welcome, but it is unlikely that a foreign donor will want to support the CanadianConsortium on Human Security. The onus falls on Canada to support indigenous actors, especially in light of the fact that we specifically created the CCHS to work in tandem with DFAIT in advancing the human security agenda.

Conclusion

The Consortium's proposal for the current fiscal year is expected shortly. GHS will ensure that these recommendations are reflected in this proposal, where appropriate, and that we implement ZIE's recommendations to the extent feasible.

Top of page


1 Summative Evaluation of the Canadian Consortium on Human Security - Terms of Reference.

2 For example, number of and attendance at CCHS-sponsored events, constituency, fellowship recipients, non-funded fellowship applicants, etc.

3 Note that this constituent group is not mutually exclusive from the other groups.

4 Note that this constituent group is not mutually exclusive from the other groups.

5 It should be noted that approximately 10% reported having some "Other" involvement and were too varied to be categorized.

6 Department of Foreign Affairs and International Trade website: http://www.humansecurity.gc.ca/

7 Note that the interview guide included several scale questions, the findings of which are reported quantitatively throughout the report.

8 Canadian Consortium on Human Security (CCHS). Final Narrative Report for Phase III, 19 April 2005 - 20 August 2006.

9 Ibid.

10 Ibid.

11 There are, however, some examples: The Commission on Human Security (CHS) was established in the United Nations Secretariat at the Office for the Coordination of Humanitarian Affairs (OCHA). The overall objective of the CHS is to place human security in the mainstream of UN activities. http://www.humansecurity-chs.org/

12 CCHS Phase II: Final Narrative Report for the period 1st September - 31st March 2005.

13 At the time of the evaluation, the Narrative Report for 2005-2006 was not available to the Consultant.

14 Section 11 of the Contribution Agreement states that the results of all evaluations carried out will be communicated to DFAIT, but fails to specify who, when and what would be evaluated.

15 Proposal for the Development and Implementation of a Canadian Consortium on Human Security, Part A, Technical and Managerial Proposal, p.2

Office of the Inspector General

* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Footer

Date Modified:
2012-11-04