Foreign Affairs, Trade and Development Canada
Symbol of the Government of Canada

Foreign Affairs, Trade and Development Canada

international.gc.ca

Summative Evaluation of the International Education and Youth Programs

(May 2010)

(PDF Version, 799 KB) *


Abbreviations, Acronyms and Symbols

ACCCAssociation of Canadian Community Colleges
AIESECAssociation Internationale des Étudiants en Sciences Économiques et Commerciales
APECAsia-Pacific Economic Cooperation
ARAFAccountability, Risk and Audit Framework
AUCCAssociation of Universities and Colleges of Canada
CBIECanada Bureau for International Education
CECNCanadian Education Centre Network
CICCitizenship and Immigration Canada
CICICCanadian Information Centre for International Credentials
CIDACanadian International Development Agency
CMECCouncil of Ministers of Education Canada
DECDepartmental Evaluation Committee
DFAITDepartment of Foreign Affairs
ELAPEmerging Leaders in the Americas Program
FEPFaculty Enrichment Program
FGAForeign Government Awards
FPCCERIAFederal-Provincial Consultative Committee on Education-Related International Activities
FRPFaculty Research Program
GoCGovernment of Canada
G&CGrants & Contributions
HRSDCHuman Resources and Skills Development Canada
IASTEInternational Association for the Exchange of Students for Technical Experience
ICCSInternational Council for Canadian Studies
IECInternational Experience Canada
IEYPInternational Education and Youth Programs
ISPInternational Scholarship Program
MOUMemorandum of Understanding
OASOrganization of American States
ODAOfficial Development Assistance
OECDOrganisation for Economic Co-operation and Development
PAAProgram Activity Architecture
PCHCanadian Heritage
PDRFPost Doctorate Research Fellowships
PRDIntergovernmental Relations and Public Outreach Bureau
PREInternational Education and Youth Division
SEPStudent Exchange Program
SSHRCSocial Sciences and Humanities Research Council
TBSTreasury Board Secretariat
UNESCOUnited Nations Educational, Scientific and Cultural Organization
VTCVirtual Trade Commissioner
ZIDOffice of the Inspector General
ZIEEvaluation Division, Office of the Inspector General

Executive Summary

This document presents the results of the conduct of the evaluation of the International Education and Youth Programs which include the following:

  • Understanding Canada: Canadian Studies Program;
  • International Scholarships Program;
  • International Education Policy;
  • Edu-Canada Program International Promotion of Education in Canada;
  • International Experience Canada.

Project Background and Context

A summative evaluation approach was applied to the three Grants and Contributions programs (International Scholarships, Understanding Canada, and the International Education Policy) following the requirements of the Transfer Payment Policy and Treasury Board and in view of the forthcoming application for renewed funding. A formative (implementation) evaluation approach was used for the evaluation of the International Experience Canada and Edu-Canada.

Program2008-2009
O&M
2008-2009
G&C
Understanding Canada$0.24M$5.4M
International Scholarships Program$0.33M$9.6M
International Education Policy$0.25M$0.4M
Edu-Canada$1.30M$3.5M
International Experience Canada$7.1M (user fees)

Since the target audience for this evaluation is the International Education and Youth Division (PRE), an overall assessment of the management of the five programs and the extent to which they jointly support the Advantage Canada Agenda of the government was conducted as well. The focus was on how the five programs complement each other and leverage their strengths to support Canada's foreign policy in the areas of knowledge and to promote Canada as a study and research destination.

Project Delivery

The PRE programs are planned and controlled centrally. A significant portion of the activities is delivered by missions, third parties and stakeholders such as provinces, Canadian educational associations and educational institutions, foreign academics and associations, and contracted delivery agents.

Evaluation Objectives, Issues and Approach

The evaluation focused on the programming activities carried out between 2005, when the last evaluation was conducted, and 2009. Due to the specific nature of each program, including their different funding bases and arrangements, as well as their current status (e.g., well established or in the process of transition and major reorganization), the emphasis placed on each program varied. A major focus of the evaluation was the performance, and in particular the economy (efficiency and effectiveness) and value for money of each program, and of the Division as a whole.

The evaluation is based on the following lines of evidence:

  • comprehensive review of documentation, files and reports;
  • in-depth interviews with the DFAIT managers and staff;
  • in-depth interviews with 17 partners and stakeholders;
  • field visits to missions in the United Kingdom, France, Germany, China, India, Brazil, and Columbia where approximately 100 people were interviewed;
  • self-administered Web survey of missions to which 37 missions participated.

Key Findings

The key findings of the study are as follows.

  • Concerning PRE:
    • Understanding Canada, the International Scholarships Program, International Experience Canada and, to a lesser degree, the International Education Policy are rooted in the logic of public diplomacy whereas Edu-Canada is founded on an economic logic.
    • PRE programs are connected to DFAIT's program activity architecture.
    • PRE has not formally planned strategies and ways to cross-fertilize its programs.
    • PRE has not formally planned its strategic communications with Canadians, Canadian institutions and DFAIT missions.
  • Concerning Understanding Canada:
    • The demand for Understanding Canada is pressing in some missions and much less so in others; the demand for funding by foreign academics has decreased over the past five years for two of the key sub-programs - the Program Development Grant and the Faculty Enrichment Program.
    • DFAIT's responsibility for Understanding Canada is justified by the importance of factors other than academic excellence in grant adjudication.
    • Understanding Canada contributes to substantial intellectual activity about Canada and may influence the position of foreign countries about Canada but there is limited factual demonstration of the latter.
    • Understanding Canada is a complex program that has become bureaucratic and burdensome on applicants and mission staff. This trend is explained by the application of a stricter due diligence process.
    • Understanding Canada is a highly centralized program that would benefit from additional mission input.
    • Understanding Canada has not yet capitalized on new technology.
    • Understanding Canada has limited performance reporting capabilities.
  • Concerning the International Scholarship Program:
    • While scholarships can be conceived as attractors of foreign students or as complements to other action such as education marketing, the missions do not generally articulate a pressing need for them.
    • International scholarships could potentially be managed by federal granting agencies.
    • A case for the performance of the International Scholarship Program, as a stand-alone intervention, could not be made because of a lack of information on scholarship beneficiaries.
    • The performance of International Scholarships could be improved if they were integrated in a wider PRE strategy.
    • ISP has very limited performance reporting capabilities.
  • Concerning the International Educational Policy:
    • The need for attention given to education matters on the international scene is undisputed. However, the name of the program may announce more than the program plans to deliver.
    • A case for the performance of the International Education Policy could not be made because of the lack of performance information.
  • Concerning Edu-Canada:
    • The economic attractiveness of the international education market is unquestioned, as is the need to face strong competition while trying to capture a piece of this market.
    • Edu-Canada has achieved important milestones; it still has to document performance in the production of outcomes.
    • The expectations created by the program may exceed its level of funding.
    • The performance of Edu-Canada is dependent upon the involvement of many stakeholders over whom the program has no authority.
    • Edu-Canada has very limited performance reporting capabilities.
  • Concerning International Experience Canada:
    • Young people value the wide opportunities to travel internationally and IEC is a mechanism to offer such opportunities.
    • IEC has strengthened its management processes; current performance priorities include the promotion of the program among young Canadians and performance measurement.
    • The 2008 IEC transition has been difficult but the program is regaining control over operations.
    • IEC was traditionally a low-risk program but upcoming bilateral agreements with higher risk countries will require stronger controls.
    • CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured.
    • A centralized management information system and an on-line application process would significantly reduce the overall cost of managing IEC.

Conclusions

Program Logic and Relevance

  • The IEY programs are a relevant government instrument for furthering public diplomacy and increasing the awareness of the international community about Canada as a preferred travel and study destination. The evaluation found that the demand for, and the interest in these programs among academia, policy makers, researchers and students is growing, and so are the commercial benefits and revenues for Canada.

IEY programs are aligned with and contribute to two of the main Government of Canada priorities: Priority #1: Greater economic Opportunity for Canada, with a focus on growing /emerging markets; and Priority #2: United States and the Americas, as well as to the Advantage Canada Agenda.

Program Management, Coordination and Delivery

  • The overall operational funding and resourcing at PRE and DFAIT missions is not commensurate with the growing demand for, and interest in the programs, nor with the daunting complexity of the programs and their delivery environment involving OGDs and agencies, provinces and territories, Canadian and foreign partners and stakeholders.

PRE programs are complex and possess limited resources. This may explain why PRE has not formally planned strategies and ways to cross-fertilize its programs, nor has it formally planned strategic communications with Canadians, Canadian institutions and DFAIT missions, which ultimately impairs the overall performance of the Division.

Efficiency and Economy

There is room for improvement in PRE on the efficiency and economy front.

  • Understanding Canada: The limited human and financial resources at DFAIT and the lack of designated Academic Relations Officers in some missions impact the effective implementation of the Understanding Canada programs.

While changes introduced to the Understanding Canada program in 2008 aimed at enhanced transparency and improved quality of the applications through new content priorities, according to mission staff and program participants, these changes ultimately resulted in increased program complexity and less transparency, which are compounded by the very limited resources available at missions to deliver the programs adequately.

  • The efficiency and effectiveness of International Scholarships could be improved if they were integrated in a wider PRE strategy.

The program has very limited performance reporting capabilities.

  • The expectations created by Edu-Canada may exceed its level of funding. Program performance is dependent upon the involvement of many stakeholders over whom the program has little or no authority.

Edu-Canada has been successful in achieving its immediate outcomes, including the approval and promotion of the new "Imagine" Brand; however, modest funding levels and insufficient resources at missions threaten to jeopardize some of the major achievements of the Program.

  • The 2008 transition has been difficult for International Experience Canada but the Program is regaining control over its operations.

Traditionally, IEC has been a low-risk program, but the growing number of bilateral agreements will require stronger controls. CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured. A centralized management information system and an on-line application system would significantly reduce the overall cost of managing IEC.

Recommendations

Regarding the International Education and Youth Division:

Recommendation #1
That DFAIT and the International Education and Youth Division (PRE) continue to deliver the five programs while developing and implementing a sound Divisional Strategic Plan that capitalises on the strengths of each individual program and identifies opportunities for leveraging of resources and cross-fertilizing programs' results. This Plan should also clarify the specific role and contribution of the International Education Policy to the division strategy.
Recommendation #2
That PRE develop and implement an integrated Communication Strategy targeted at Canadians, Canadian institutions and foreign stakeholders and supported with respective funding for outreach activities carried out by missions.
Recommendation #3
That PRE set aside resources to implement a performance monitoring plan and use performance information in making planning and management decisions.

Regarding Understanding Canada:

Recommendation #4
That the delivery of Understanding Canada be better coordinated between PRE, geographic divisions and missions, with adequate resources being granted to missions based on country and/or regional trends and priorities.
Recommendation #5
That a new review of the complexity of Understanding Canada, the variety of the sub-programs and the respective application and adjudication procedures be conducted in consultation with missions to identify feasible ways to decrease the administrative burden on missions while increasing the transparency and the overall program efficiency and effectiveness.

Regarding the International Scholarships Program:

Recommendation #6
That PRE respond to the need for a more effective promotion of the variety of scholarships among Canadian universities and colleges, as well as potential foreign participants, along with identifying the right balance between planned quotas and the capacity of Canadian universities to accept foreign scholars.

Regarding Edu-Canada:

Recommendation #7
That DFAIT, and PREP in particular, continue to advocate for a significant increase in the funding level for Edu-Canada and start exploring the options for establishing an independent organisation to promote education in Canada in order to sustain current achievements and increase Canada's visibility and ultimate benefits in a highly competitive international environment.

Regarding International Experience Canada:

Recommendation #8
That International Experience Canada continue to strengthen its management processes, including policies, management information systems, on-line application system, performance measurement, and risk management.

 


1.0 Introduction

The Evaluation Division (ZIE) of the Department of Foreign Affairs (DFAIT) in the Office of the Inspector General (ZID) is mandated by the Treasury Board Secretariat (TBS) through its new Evaluation Policy to conduct evaluations of all direct program spending of the Department (including Grants & Contributions). All evaluation reports are presented for approval to the Departmental Evaluation Committee (DEC) chaired by the Deputy Ministers.

This document presents the results of the conduct of the IEYP evaluation.

1.1 Background and Context

This evaluation covers the five programs managed by the International Education and Youth Division. The evaluation approach to each program differed based on a number of factors. A summative evaluation approach was applied to the three Gs&Cs programs (Scholarships, Understanding Canada, and the International Education Policy) following the requirements of the Transfer Payment Policy and Treasury Board and in view of the forthcoming application for renewed funding. A formative (implementation) evaluation approach was used for the evaluation of the International Experience Canada and Edu-Canada.

Since the target audience for this evaluation is the International Education and Youth Division (PRE), an overall assessment of the management of the five programs and the extent to which they jointly support the Advantage Canada Agenda of the government was conducted as well. The focus was on how the five programs complement each other and leverage their strengths to support Canada's foreign policy in the areas of knowledge and to promote Canada as a study and research destination.

The evaluation also reviewed the extent to which Understanding Canada and the Scholarships program have acted upon the recommendations of the 2005 Evaluation of the International Academic Programs.

1.2 Programs

The International Education and Youth Programs (IEYP) are part of DFAIT's strategy to advance Canada's interests worldwide in the areas of knowledge and learning, to promote Canada as a study and research destination and to provide services to Canadians to facilitate international mobility for travel and work.

The programs contribute to Canada's international agenda under program activities #2 (Diplomacy and Advocacy) and #3 (International Commerce). They aim to build Canadian influence through dialogue, profile-raising, collaborative research, and the creation of networks of contacts and understanding between people in partner countries, especially in those presenting a priority for Canada's foreign policy objectives. The programs aim to contribute to the Canadian economy through the promotion of education service exports, especially the recruitment of foreign students which brings more than $6.5 billion to the Canadian economy annually,(1) the International Experience Canada which brings significant amounts to Canadian economy each year(2) from the expenses of participants, and through the Understanding Canada program which generates more than $55 million in spending in Canada.(3)

IEYP programs facilitate collaboration with the provinces/territories on issues related to international education and engage their participation in international fora and partnerships. They support the Advantage Canada agenda, especially the Knowledge Advantage agenda in contributing to the development of Canada as a knowledge economy with a highly skilled labour force.

These programs operate pursuant to a series of international agreements and other reciprocal efforts to promote education exchange and youth mobility. They also operate in coordination with other GoC departments, agencies and international programs, including CIDA, CIC, IDRC, CBSA, the granting councils and HRSDC, as well as provincial programs. Consultations are facilitated through the Federal Provincial Consultative Committee on Education-Related International Activities, the National Education Marketing Roundtable, the Advisory Committee on International Students and Immigration, MOUs with the Council of Ministers of Education, Canada (CMEC) and CIC.

1.2.1 Programs and Governance

International Education and Youth Programs are managed by the International Education and Youth Division and consist of a number of programs which assemble a range of tools to advance Canadian objectives in the area of education and youth. The programs within PRE are:

  • Understanding Canada: Canadian Studies Program;
  • International Scholarships Program;
  • International Education Policy;
  • Edu-Canada Program International Promotion of Education in Canada;
  • International Experience Canada.

Flow Chart of Governance and Reporting Structure

The Director General, Intergovernmental Relations and Public Outreach Bureau (PRD) is responsible for the management of the International Education and Youth Division. The Director, International Education and Youth Division (PRE) is responsible for the Division's operations and the delivery of the five components of the International Education and Youth Programs. He is supported by four Deputy Directors. The breakdown of resources per program is as follows:

  • The Deputy Director for the International Academic Programs is responsible for Understanding Canada: Canadian Studies Program and was supported by three FTEs and one Grants & Contributions administrative assistant at the beginning of this evaluation in September 2009. One FTE has since been removed. A significant part of the work is carried out by the International Council for Canadian Studies (ICCS) under a contract. This section is also responsible for the Canada US Fulbright program which includes the foundation established under a Canada-USA Treaty on Higher Education cooperation.
  • The Deputy Director for Policy and Planning is responsible for the International Scholarships Program, International Education Policy, divisional business planning, as well as the supervision of the e-communications and websites manager. This Deputy Director is supported by four FTEs. The International Scholarships program is delivered primarily through a partnership with an outside organization, currently the Canadian Bureau of International Education (CBIE). Scholarship programs are also governed by a number of international arrangements. International Education Policy is managed in close cooperation with the Council of Ministers of Education, Canada (CMEC), through consultations with stakeholders representing major education associations and with other government departments.
  • The Deputy Director for Edu-Canada is responsible for the international promotion of education in Canada. The Deputy Director is supported by six FTEs. The Edu-Canada initiative is part of the Global Commerce Strategy. The program is responsible for marketing Canadian educational institutions to foreign markets, contributes directly to the International Commerce program activity and falls under the Deputy Minister of International Trade, while the other divisional programs fall under the Diplomacy and Advocacy program activity of DFAIT's Program Activity Architecture (PAA).
  • The Deputy Director and Head for the International Experience Canada (IEC) is responsible for this program which promotes youth mobility between Canada and abroad. The Deputy Director is supported by eight FTEs. The program also directly funds 13 FTEs abroad. The IEC currently administers 21 international arrangements governing the issuance of work permits and facilitating youth mobility and cultural exchange. Several other bilateral agreements are being negotiated. IEC is a cost recovery program with a revenue fund.

1.2.2 Understanding Canada

Foster a greater knowledge and understanding of Canada among scholars and other influential groups abroad.

Understanding Canada is designed to develop greater knowledge and better understanding of Canada, its values and its culture among scholars and other influential groups abroad. The program primarily targets foreign academics (known as Canadianists) who are interested in studying or conducting research on Canada, promoting teaching and publications about Canada, and organizing conferences. The Program also reaches out to influential groups abroad including Canadian studies associations whose membership consists of academics, politicians, senior public servants, business people, students, administrators, publishing house representatives' professionals and journalists.

The Program's expenditures in 2008/2009 were $5.489 M in Gs&Cs funding and $.237 O&M funding, excluding salaries.

Understanding Canada is a complex mesh of programs managed in part locally, in part globally, by DFAIT staff as well as by the International Council for Canadian on a contract basis. Exhibit 1.1 offers a birds-eye view of the number of grants awarded under each program over the years.

Exhibit 1.1

Number of Understanding Canada Program Grants, 2004-2008
Programs managed by DFAIT alone20042005200620072008
TOTAL344553562509513
Canadian Leadership000131
Conference Grant Program (only offered in the USA until April 1, 2008)669810
Canada-Mexico Facilitation Exchange (called Bank of Missions before 2009)21118149
Governor General's Award11110
Graduate Student Fellowship Program (only offered in the USA)5131164
Graduate Student Research Program00100
North American Research Linkages00030
Program Development Grants0172165201128
Strategic Initiatives (initiated in 2008)000013
Student Mobility (initiated in 2008)000015
Support to Associations049524033
Programs managed with the assistance of ICCS20042005200620072008
Canadian Prime Minister's Award (only for Japan)33220
Canada Asia Pacific Awards20332
Canada Europe Awards30422
Canada Latin American Awards30223
Canadian Publishing Awards00001
Doctoral Student Research Program664231
Faculty Enrichment Program1181131199880
Faculty Research Program180189174126133
International Research Linkages80778
University Library Support Program     
Book Display Program     
Conference Travel Assistance Awards     
IRL807710

Source: ICCS and Program files; data for 2009 were not available.

Key Understanding Canada programs include the following:

  • Faculty Enrichment Program: FEP grants assist foreign academics to develop or teach courses about Canada by funding travel to Canada to gather information and materials to develop or modify a course.
  • Faculty Research Program: FRP grants assist foreign academics to undertake short-term research about Canada, leading to publication of articles in scholarly journals.
  • Program Development Grants: This program encourages scholarly inquiry of foreign higher learning institutions or organizations that contribute to the development or expansion of a program dedicated to Canada studies or Canadian foreign relations.
  • Support to Associations: This program provides financial support to national associations dedicated to Canada studies or Canadian foreign relations
  • Conference Grants: This program supports multidisciplinary conferences on Canada or thematic with a Canada content to assist a foreign higher learning institution or organization in holding a conference and publishing the resulting papers and proceedings in a scholarly fashion.
  • Doctoral Student Research Program: This program assists full-time PhD students to undertake doctoral research in Canada; the dissertation must be related to Canada.
  • Canadian Leadership: These awards allow Canadian missions to invite Canadian experts to attend strategic events hosted by higher-learning institutions to speak on topics relevant to foreign policy objectives.
  • Strategic Initiatives: These awards allow Canadian missions to propose projects on Canada, related to important and timely issues about Canada, its relationship with a specific country or region, and its international affairs. Projects must be in support of the national or regional strategies of the local Canadian mission.

Ultimately, the Program is intended to have a positive influence on the promotion of Canada's interests abroad, especially among foreign academics, with the following expected results:

Understanding Canada's immediate outcomes are expected to be:(4)

  • Canadian studies programs are better aligned with the government of Canada's priorities;
  • Conferences, Research, teachings and publications on Canada are developed and supported by local sources; and
  • New collaborative linkages are in place.

Understanding Canada Program's intermediate outcomes include:

  • Foreign scholars and opinion leaders have broadened and deepened their knowledge of Canada and Canadian issues;
  • Sustained, stable, and growing network of foreign professionals and leaders with a well-disposed and sustained interest in Canada; and,
  • Increased linkages between foreign and Canadian academic institutions.

1.2.3 International Scholarships Programs

Providing scholarships to foreign graduate students who wish to study in Canada.

The International Scholarship Program (ISP) is intended for students of merit who want to pursue post-secondary studies or research programs in Canada, and for Canadians to take advantage of reciprocal awards offered by foreign governments. The purpose is to enable these individuals to contribute to their country's knowledge and understanding of Canada upon their return, and to support internationalization agendas of Canadian institutions.

The International Scholarships Program has the largest Gs&Cs budget within the Division, reaching $9.574 M in 2009/2010. A separate O&M budget has been established for the Emerging Leaders in the Americas Program of $.326 M.

Exhibit 1.2 details the number of scholarships awarded between 2005 and 2009 under each of the programs. There was a substantial increase in the number of scholarships awarded in 2009 with the advent of the Emerging Leaders in the Americas Program (ELAP). Adding ELAP awards to Canada-CARICOM Leadership Awards and Canada-Chile Leadership Exchange Program awards which are of the same nature and target the same region, 369 of 582 (63%) 2009 awards were short-term scholarships (four to six months) given to this region.

Exhibit 1.2

Number of Scholarship Awards, 2005-2009
Program20052006200720082009
International Students Total87152270319582
Competitions for Canadians(5)     
Commonwealth Scholarship Plan     
Foreign Government Awards     
Organization of American States     
Competitions for International Students     
Canada - CARICOM College    4
Canada - CARICOM Leadership Awards   5042
CARICOM Virtual    47
Canada-Chile Leadership Exchange Program    33
College and Undergraduate Student Exchange Program / Graduate Student Exchange Program42613914680
Emerging Leaders in the Americas Program    294
Post-Doctoral Research Fellowships  325851
Canada Commonwealth Scholarship Program / Government of Canada Awards83126996531
Canada China Scholars Exchange Program   3244
Canada-USA Fulbright Program (incl. Killam awards)948482109108

Source: CBIE, program

The Canada USA Fulbright Program operates pursuant to the Canada-USA Treaty on Education Cooperation. This is a bi-national program managed by the Foundation for Educational Exchange between Canada and the US and is co-funded by DFAIT and the US State Department. Canadian funding in the last few fiscal years has been at least $600,000.

The Canada China Scholars Exchange Program and the Canada Chile Equal Opportunity Scholarship Program operate pursuant to memoranda of understanding between Canada and the respective countries and are delivered through the Canadian Bureau for International Education (CBIE) as are the other scholarship programs.

These programs are predicated on two scholarship formats: a short-term exchange and a post-doctoral research fellowship. These new formats were implemented in 2007 on a permanent basis and represented a move away from the traditional degree-granting format.

  • The Student Exchange Program (SEP) provides short term scholarships to students from developing countries tenable for up to 6 months at Canadian colleges and universities to pursue studies or research in a priority field of study while remaining students of their own institutions. Areas of study and research include but are not limited to good governance, rule of law, democratic development, economic and trade policy issues, teaching and commerce. The scholarships are valued at $7,500 for 4 months and $10,000 for six months.
  • Post Doctorate Research Fellowships (PDRF), designed to provide further research opportunities to recent doctoral graduates in social and natural sciences, humanity and engineering fields. They are valued at $32,000 CDN and are one-year research awards tenable at recognized public Canadian universities and affiliated research institutes.

CBIE also administers the Foreign Government Awards Program which consists of the promotion and pre-selection of Canadian students for scholarships offered by foreign governments. These and other foreign government scholarships are offered in reciprocity to Canadian awards.

The ISP aims at achieving the following immediate outcomes:(6)

  • Increased collaboration by institutions and government;
  • Increased awareness of Canada as a study destination; and,
  • Increased mobility and optimized demand of highly talented Canadians and foreign nationals

The expected intermediate outcomes are:

  • Increased, stable and self-sustainable inter-institutional linkages and collaboration;
  • Increased understanding of and collaboration with Canada by foreign alumni;
  • Increase in the number of graduate students from ODA countries who return to their countries to contribute to its development; and
  • Increased contribution to Canada's innovation by Canadian and foreign scholars.
1.2.4 International Education Policy

Coordinates Canadian international activities related to higher education.

The International Education Policy is aimed at enhancing and promoting Canada's international agenda (diplomacy and advocacy) through benefits to higher education in Canada and increased impact of Canadian education policies on the international community. It coordinates and conducts studies and analysis of issues connected to international cooperation in higher education as they relate to DFAIT's priorities. The program also provides leadership in Canada's bilateral and multilateral relations in international education, and coordinates the negotiations, the signing and ratification of agreements on education related issues with different states (Canada EU Framework Agreement on Cooperation in Higher Education, Training and Youth) and international organizations, such as APEC, the Commonwealth, OAS, the OECD, and UNESCO. It also ensures that Canada complies with the obligations it has assumed through such agreements.

To ensure a common national position on Canadian education, the International Education Policy group works closely with the Council of Ministers of Education Canada (CMEC), other federal government departments and non-governmental organizations. The relationship with CMEC is governed by an MOU that was mutually approved in 1986.

The expected immediate outcomes from the International Education Policy are:(7)

  • Enhanced representation, coordination, and understanding of international Canadian priorities as it relates to education;
  • Enhanced international cooperation through implementation of policies reflecting Canadian interests;
  • Increased awareness of Canadian policies and experience in education in the international education community; and
  • Compliance in all areas covered in the terms of agreements.

The intermediate outcomes have been summarized as follows:

  • Increased impact of Canadian policies on the international community; and
  • Increased provincial engagement in a pan-Canadian approach to international education issues.

In 2008/2009, the program received $0.41M Gs&Cs funding and an operating budget of $0.25M.

1.2.5 Edu-Canada

The Edu-Canada initiative is part of the Global Commerce Strategy and builds on DFAIT education marketing efforts under PAA (#3 International Commerce). To promote Canada's Knowledge Advantage, Budget 2007 committed $1 million a year to launch a new marketing campaign, for international education including the development of a Brand for Education in Canada to attract talented international students to Canada in priority markets.

Develops the branding of Canadian education and coordinates the marketing strategy.

The campaign, led by DFAIT in collaboration with the Council of Ministers of Education Canada is aimed at creating a strong, distinctive image of the education system in Canada and its advantages. The brand with its logo presenting a stylized Canadian Maple Leaf and a tagline "Imagine: Education au/in Canada," encourages cooperation between all levels of government while recognizing the exclusive jurisdiction of provinces and territories in education and the interest of the federal government in promoting education abroad.

The Brand was approved by DFAIT, CIC, Heritage Canada and all ten provincial governments in June 2008 and was launched in September 2008. Since then, it has been used in all priority markets except in France and Mexico.

A secondary objective of this campaign was to offer better services and market intelligence to education-sector clients (Canadian education institutions). The Edu-Canada PRO web-portal was launched to leverage the Virtual Trade Commissioner (VTC) with a dedicated education-specific portal for clients.

The main activities of Edu-Canada include:

  • developing and managing a marketing campaign to promote Canadian education;
  • supporting promotional activities with Canada's missions abroad;
  • providing services to Canadian educational institutions in support of their international promotion activities;
  • coordinating partner involvement in marketing Canadian education, including other federal departments and provincial governments.

A number of immediate and ultimate outcomes have been identified for this program in the 2009 ARAF, namely:

Expected immediate outcomes include:(8)

  • enhanced international profile of Canadian education;
  • clients and stakeholders use Edu-Canada as a preferred vehicle for obtaining market information and promote their interests; and
  • Canadian educational institutions have greater buy-in.

Edu-Canada's expected ultimate outcomes include:

  • Canada's knowledge advantage is recognized around the world and serves to strengthen Canadian interests and economic prosperity; and
  • the International agenda is shaped to Canada's benefit and advantage in accordance with Canadian interests and values.
1.2.6 International Experience Canada

Supports Canadian and foreign youth who wish to travel and work abroad.

International Experience Canada was established to promote reciprocal international cultural exchanges through travel, life and work experience between Canadian and international youth based on bilateral agreements. The program facilitates the issuance of work permits for youth, typically aged 18-35. There are currently 21 such bilateral arrangements with negotiations under way or planned with 30 more countries. The program also works with Canadian partners who offer exchange programs for young Canadians and international youth. Young Canadians gain international work experience, develop their inter-cultural skills and knowledge of other countries and thereby contributes to the Advantage Canada agenda. International youth gain knowledge of Canada and develop links with Canada which influences their perceptions of Canada, their careers or eventual migration to Canada. The program supports country strategies to promote people-to-people contacts. It also contributes to the flexibility of Canada's labour market and helps meet short-term labour market needs representing 20-30% of all temporary foreign workers admitted to Canada.

In 2000, Treasury Board authorized the International Experience Canada to charge international participants a participation fee of $150 in order to cover program expenses. The fee, initially charged only to participants from certain countries, was extended globally in 2008 following consultation with Parliament. Extension of implementation of the User Fee Act to the program entails increased regulatory compliance and accountability.

As a cost recovery program, it is funded by a Revenue Fund, expected to reach $ 9.3 M in 2009/10.

Exhibit 1.3 presents the total number of Canadians and foreigners who used the International Experience Canada program between 2004 and 2008 (2009 data not yet available).

Exhibit 1.3

Number of International Experience Canada Program Participants, 2004-2008
Outgoing Canadians with Scholarship Awards
Outgoing Canadians200420052006200720082009
All Regions20,83621,65621,06721,46219,51521,139
Americas5166143931,670643453
Asia-Pacific9,7839,0769,68510,26910,78412,185
Europe10,53711,96610,9899,5238,0888,418
Middle-East0000026
Africa0000057

 

Outgoing Canadians with Scholarship Awards
Incoming Canadians200420052006200720082009
All Regions23,86930,46732,91334,28244,89553,102
Americas4007405395385121,068
Asia-Pacific12,91315,75215,74516,10624,05023,994
Europe10,55613,97516,62917,63820,33327,931
Middle-East000005
Africa00000104

Source: Program files

IEC's expected immediate outcomes include:(9)

  • Mutual understanding of other cultures through a travel, life and work experience abroad.
  • Increased awareness of IEC;
  • Ensure consistency of IEC in terms of reporting and accountability; and
  • Effective collaboration with Program partners.

The expected intermediate outcomes are:

  • Foster close bilateral relations between Canada and other countries;
  • Increase of the favourable perception of Canada internationally;
  • Support the implementation of other government initiatives (i.e., Advantage Canada, Education au-in Canada, economic development, temporary or permanent immigration to Canada, labour shortage, etc);
  • Enhance participants' personal and professional development through an experience abroad giving them a competitive edge in the job market.
1.2.7 Integrated Program Logic

In 2008-09, the International Education and Youth Division developed a logic model as part of its ARAF. In the context of the Department's Program Activity for Diplomacy and Advocacy, the long-term objective of the Division is to support the achievement of one of its key results of shaping the international agenda to Canada's benefit and advantage in accordance with Canadian interests and values. In the short-term, the division has the following objectives:

  • to broaden and deepen the knowledge that foreign scholars and opinion leaders have about Canada and Canadian issues;
  • to enhance the international profile of Canadian Education;
  • to increase international mobility; and,
  • to increase awareness of Canadian policies and experience in education.

The medium-term outcomes address the following:

  • increased intercultural awareness and positive perceptions of Canada;
  • increased number of foreign students in Canada;
  • increased inter-institutional linkages and cooperation; and,
  • increased provincial engagement in a pan-Canadian approach to international education issues.

In the context of the Department's Program Activity for International Commerce, the division supports the Canadian education industry in the recruitment of international students. The Edu-Canada initiative of the Global Commerce Strategy provides support for the:

  • development and management of the branding of Canadian education excellence;
  • implementation of strategies in priority markets; and,
  • increased client service and promotion of institutional linkages.

All programs require the active participation of DFAIT missions in their implementation. In effect, the Division offers tools that may or may not be adopted by missions as part of their own activity planning. The ultimate performance of PRE programs is often contingent upon the capacity of the missions to use these tools and deliver the programs.

1.3 Evaluation Governance

The evaluation was managed by DFAIT's Evaluation Division in the Office of the Inspector General, further quoted as the Project Authority. The evaluation team consisted of two ZIE evaluation managers, one junior officer, and an external evaluation consultant.

The conduct of the evaluation was overseen by an Evaluation Advisory Committee comprised of representatives from International Education and Youth Division and other DFAIT divisions. The Final Report with a Management Response to the recommendations of the evaluation and an Action plan will be published on the Departmental Website upon approval from the Departmental Evaluation Committee (DEC).

1.4 Evaluation Objectives and Scope

The objectives of this Evaluation were:

  • To evaluate the relevance of the International Education and Youth (IEY) programs and their alignment with both federal government priorities and DFAIT's strategic outcomes.
  • To assess the level of performance and success of each program towards achieving their stated objectives.
  • To assess the capacity of the PRE Division and Canada's missions to manage and deliver the IEY programs by examining the current resource base, identifying eventual gaps, and indicating areas for improvements and/or more efficient coordination, synergies, leveraging and streamlining of resources within and outside DFAIT.

The evaluation was designed to generate findings and recommendations, as well as identify lessons-learned based on performance to date. The evaluation focused on the programming activities carried out between 2005, when the last evaluation was conducted, and 2009. Due to the specific nature of each program, including their different funding bases and arrangements, as well as their current status (e.g., well established or in the process of transition and major reorganization), the emphasis placed on each program varied. A major focus of the evaluation was the performance, and in particular the economy (efficiency and effectiveness) and value for money of each program, and of the Division as a whole.


2.0 Evaluation Methodology

This evaluation was conducted in accordance with the project outlined in the evaluation work plan which was approved by the Evaluation Advisory Committee. The evaluation is concerned with program realities for the period from 2005 to 2009.

2.1 Evaluation Design

The design of this evaluation is essentially descriptive. Facts concerning program demand, usage, and management as well as opinions concerning rationale, effectiveness and efficiency are portrayed in this report.

2.2 Data Collection

2.2.1 Evaluation Focus

This evaluation assessed the extent to which the five programs continue to be aligned with the needs and priorities of the Government of Canada and DFAIT. It also assessed the extent to which the activities undertaken by these programs are contributing to the achievement of expected outcomes. The evaluation determined the impact these programs had on the beneficiaries and stakeholders.

An evaluation matrix formed the foundation for addressing the findings to the issues, key questions, indicators, and sources of data.

2.2.2 Lines of Evidence

Document Review

A review of documentation, files and reports was conducted to develop an understanding of the programs and of any changes these programs have undergone in recent years. The International Education and Youth Division (PRE) was responsible for providing the evaluation team with the necessary documentations. Several evaluation indicators, found in the evaluation matrix, were drawn from the recently produced Accountability, Risk and Audit Framework for each program, but much of the anticipated data were gathered with regard to new indicators. The information collected was complemented by interviews with program managers and staff, stakeholders, partners and beneficiaries.

DFAIT Interviews

In-person interviews were conducted with the Director of the International Education and Youth Division, with the Deputy Directors of the programs, and with several staff members. The purpose of these interviews was to gather information on the context in which the programs operate, on current issues and areas for potential improvement and further development. Many evaluation issues were addressed in one way or another as part of these interviews. DFAIT representatives were asked for factual information pertaining to each issue, for their analysis of the situation as it relates to the issues and for additional documents that can shed light on the evaluation issues.

External Interviews

Interviews were conducted with selected partners and stakeholders. The purpose of these interviews was to gather a wide range of information from multiple perspectives about the programs, the context in which they operate, current issues and areas for potential improvement and further development. Interview topics included the need for the programs, the contribution of the programs to the government priorities and the achievement of objectives. Key informants included 17 representatives from a wide array of organizations:

  • Association Internationale des Étudiants en Sciences Économiques et Commerciales (AIESEC)
  • Association of Canadian Community Colleges (ACCC)
  • Association of Universities and Colleges of Canada (AUCC)
  • Canadian Bureau for International Education (CBIE)
  • Canadian Information Centre for International Credentials (CICIC)
  • Citizenship and Immigration Canada (CIC)
  • Government of Manitoba
  • International Association for the Exchange of Students for Technical Experience (IASTE)
  • International Council for Canadian Studies (ICCS)
  • Social Sciences and Humanities Research Council (SSHRC)
  • SWAP
Field Visits

Field visits to missions abroad were conducted for in-person interviews with mission staff responsible for the delivery of the PRE programs and key local partners. Field visits are justified in cases where specific problems cannot be captured with general surveys and when one mission deals with most or all of the International Education and Youth Programs.

A list of relevant missions was informally discussed with program management and further refined based on program use and the presence of facilitating or inhibiting factors relative to program success. Opportunities also arose in the form of mission visits already occurring to support other evaluation studies. Ultimately, missions in the following countries were visited:

  • United Kingdom
  • France
  • Germany
  • China
  • India
  • Brazil
  • Colombia

Up to three days were spent at each location to interview key personnel, partners and associations' representatives, and to review documentation maintained locally. Between 10 and 20 individuals were interviewed at each mission for a total of approximately 100 individuals. Field visits served to collect information regarding the actual demand for the programs, relationships with beneficiaries, benefits to Canada, program impacts, and capacity of mission staff to deliver the programs.

Survey of missions

A survey of missions was conducted. Issues similar to those addressed in field visits were touched upon in the context of an electronic survey of missions. While the information gathered from a survey is not as in-depth as that collected through individual interviews, it was important to obtain the perspective of several missions about the positioning of the IEYP programs in the context of the overall service offering.

Out of the 103 missions asked to fill out the on-line questionnaire, 37 completed it between February 23 and March 18, 2010. Those who make no use or little use of the program were asked for the reasons why this is the case. Users were asked for the contribution of the programs to their success.

Reliability, Verification, Validation

The evaluation team had abided by the evaluation standards of the TBS Evaluation Policy with regard to producing information and findings that are reliable, validated and verifiable. Qualitative information collected through interviews was triangulated with information from other sources, such as other interviews, documents, files, studies and reports. Confidentiality of the answers of each informant is strictly preserved and respected.

2.3 Limitations

This evaluation touches upon five programs, three of which are subject to the Grants and Contribution policy (the International Scholarships Program, the International Education Policy, and Understanding Canada), and two which are not (Edu-Canada and International Experience Canada). The breadth of program areas and intervention tools and the necessary constraints to evaluation resources limit the depth of the analysis.

In particular, the evaluation team did not have the resources to conduct formal surveys of program beneficiaries, such as scholarship recipients and participants in any of the programs (although selected in-depth interviews were carried out).


3.0 Relevance

3.1 IEYP As a Public Diplomacy Tool

Finding #1
Understanding Canada, the International Scholarships Program, International Experience Canada and, to a lesser degree, the International Education Policy are rooted in the logic of public diplomacy whereas Edu-Canada is founded on an economic logic.

Four out of five International Education and Youth Programs find their relevance in the logic of public diplomacy. The exception is Edu-Canada which is generally rooted in an economic logic.

There is much debate in the literature on public diplomacy(10) about the very definition of the concept and on the strategies that it includes. Taken at its widest, public diplomacy is "the relationship between diplomats and the foreign publics with whom they work"(11) or "the process by which direct relations with people in a country are pursued to advance the interests and extend the values of those being represented"(12) or "a deliberate act designed to communicate with the public in foreign countries."(13)

The literature suggests that public diplomacy is structured according to three time frames to which correspond three types of action.(14)

  • To face short-term events and pressures, public diplomacy may use media relations and other information activity in a reactive manner. This consists largely of messages under the direct control of government.
  • Working in the medium-term, proactive public diplomacy or "strategic communication" is based on a strategy that covers several months or years. It is often referred to as issue management or advocacy campaigning. Such communication is designed to argue in favour of a cause, idea or policy. Messages and other activities are still under the control of government.
  • Long-term unmediated relationship building is the third approach to public diplomacy. It focuses on developing in-depth relationships over the years with individuals who may influence the analysis of situations in foreign countries in a way that favours government priorities and policies. These influential nationals may be called upon to intervene in conflicts or to provide rooted analysis of situations that are better reflections of reality.

This third approach is characteristic of the IEYP programs (except Edu-Canada). As Potter puts it:(15)

This network of actors extends well beyond government. Much of what public diplomacy needs to be successful actually occurs outside of government. A government could never have enough resources to have a significant impact on image management and country branding. Public diplomacy, therefore, must be seen as a catalyst. While a government plays a relatively small role in overall image management, it must be able to do the following:

  • articulate a cohesive national position, taking into account the history of an issue, all the facts, players, and local sensitivities;
  • intervene to address misinformation or a lack of information; and
  • leverage resources in support of particular policy interests (whether from the private or not-for-profit sectors or simply by mobilizing expatriate Canadians).

Writers concerned with public diplomacy ascribe a number of potential objectives to it:

  • mutual understanding and respect;(16)
  • more receptive environment for further information;(17)
  • enhanced national reputation or prestige;(18)
  • increased sales by national companies;(19)
  • improved performance of foreign policies;(20)

However, there is essentially no literature on empirically assessing the results of public diplomacy efforts. This is understandable. Public diplomacy efforts aim to place seed that may bring fruits years down the road, with little to no ability to connect the efforts to their results. That's why "traditionally, public diplomacy evaluation has been output-oriented" rather than outcome-oriented.(21) Assessment of the performance and relevance of public diplomacy has requested a leap of faith and an assumption that short-term observable outcomes (e.g., research being undertaken on Canada, young foreigners visiting Canada) would eventually translate into long term positive effects.

3.2 Continued Need for Programs

Assessing the need for the programs requires first identifying whose need is addressed.

  • Understanding Canada: the long-term need is that of the Government of Canada, and particularly the missions abroad, which are looking for knowledgeable interprets of the Canadian situation to inform foreign media and governments; the short-term need is represented by the demand for grants from foreign academics;
  • International Scholarships Program: on the whole, like Understanding Canada, scholarships are a response to the needs of the Government of Canada for knowledgeable interprets of the Canadian situation; the short-term need is represented by the demand for scholarships among targeted groups;
  • International Education Policy: the need for a Canadian international education policy is that of the Government of Canada and the provinces and territories to develop a common front regarding education on the international scene;
  • Edu-Canada: the short- and long-term need is that of Canadian educational institutions seeking paying students at all levels and high caliber graduate students;
  • International Experience Canada: the long-term need is that of the Government of Canada in general, and DFAIT missions in particular, which are looking for allies in their diplomatic work; the short-term need is represented by the demand from youth for travel and work documents as part of a travel and work experience.

3.2.1 Understanding Canada

Finding #2
The demand for Understanding Canada is pressing in some missions and much less so in others; the demand for funding by foreign academia has decreased over the past five years for two key sub-programs - the Program Development Grant and the Faculty Enrichment Program.

Twenty-two of 37 missions who took part in the mission survey indicated that they are involved in the implementation of Understanding Canada. The main reason for non-involvement by the missions appears to be lack of knowledge about the programs.

Understanding Canada enjoys a high or extremely high level of priority in thirteen missions out of 22 providing a rating. The number of work days devoted to Understanding Canada in missions varied widely among the ones that indicated involvement: from fewer than ten to 200.

Qualitative information gathered through site visits confirm the importance given to this program by several missions. Some high-ranking officials described Understanding Canada as one of the most important tools at their disposal and as the most efficient mechanism to support Canadian diplomacy. There is a definite need for activities like those supported by Understanding Canada in several missions.

The demand for these programs can be expressed as the ratio of applications to awards. Based on data supplied by ICCS and by the program, it is observed that this ratio changed as follows from 2004 to 2008: 2.4, 1.8, 1.6 1.7 and 1.5. Thus, there were 2.4 applications for each award in 2004 compared to 1.5 for 1 in 2008, and there was a steady decline over the period for the Program Development Grants and the Faculty Enrichment Program, two key program elements. This decline could be attributed to changes in academe worldwide and to an aging population of Canadianists, and not the Program itself, as indicated by program management, but the fact of the decline is undisputed.

3.2.2 International Scholarships Program

Finding #3
While scholarships can be conceived as attractors of foreign students or as complements to other action such as education marketing, the missions do not generally articulate a pressing need for them.

Like Understanding Canada, international scholarships are meant to respond to the need of Canada for allies in foreign countries. Twenty missions out of 37 survey participants indicated that they are involved in this program. Seven of them rated scholarships as a high or an extremely high priority of theirs.

Stakeholders and mission representatives indicated that they considered scholarships important to their action. However, this program was rarely if at all a crucial component of strategies probably in good part because there are few scholarships available. An exception to this observation is the Vanier Canada Graduate Scholarships program which was identified as having the potential to address the need identified above. However, the Vanier Scholarship program is not a DFAIT program but rather a Government of Canada program managed by the three federal granting councils (CIHR, NSERC, and SSHRC).

In general, the student demand for scholarships is substantial.(22) For example, France reported that applications for two available doctoral scholarships increased more than four times. It is expected that similar levels of demand would be found across the board if appropriate statistics were maintained.

3.2.3 International Education Policy

Finding #4
The need for attention given to education matters on the international scene is undisputed. However, the name of the program implies more expectations than the program plans to deliver.

The Government of Canada is constitutionally responsible for matters of international resonance. Education is an important public policy issue, and it is fully justified for the Government of Canada to get involved in education matters on the international scene. This includes the negotiation and ratification of international treaties on education, the enhanced representation of Canadian international priorities related to education, support to provincial engagement in a pan-Canadian approach to international education issues and participation in international events.

However, the name of the program ("International Education Policy") may announce more to some than the program expects to deliver. Stakeholders reacted to questions concerning this policy with surprise: they were not aware of the existence of such policy. Most informants indicated that, pragmatically, the need that DFAIT should be able to address is that of a common marketing strategy for Canadian education (handled by Edu-Canada). To them, aiming for a Canadian policy on education is bound to raise considerable federal-provincial-territorial issues.

It was suggested that there is a need for an "international knowledge policy" that would be wider than education and that could incorporate the science and technology agenda and provincial strategies with regard to international education.

3.2.4 Edu-Canada

Finding #5
The economic attractiveness of the international education market is unquestioned, as is the need to face strong competition while trying to capture a piece of this market.

The rationale behind Edu-Canada is economic: through educational institutions, Canada can export education services by getting foreign nationals to attend school in Canada. The market for international education is very strong. A recent report cites an OECD publication stating that "In 2002 international education represented $30 billion or 3% of OECD's total trade services;"(23) the same report cites several other statistics suggesting that the international market for education services has grown considerably since then.

It is, however, a very crowded market. According to a Conference Board report, in 2000, Canada exported $796 million (US) in educational services.(24) That compared to $2.1 billion (US) for Australia, the leader in per capita exports of educational services which has a population one-half that of Canada. The same report indicates that a decade earlier, in 1989, Canada and Australia had the same levels of exports in this area. Other important suppliers include the United States ($10.4 billion US in 2000) and the United Kingdom ($3.8 billion US).

Stakeholders consulted were of the view that federal action on the international marketing of education front is necessary to ensure that Canada gets the portion of this market that it is able to service. Strong marketing action is necessary because competitor countries are deploying vast efforts to acquire the market. According to stakeholders, without concerted action, Canada is bound to lose international education to the likes of Australia, the United States and the United Kingdom. Moreover, competitors have centralized promotional efforts whereas Canada is characterized by a decentralized education system.

3.2.5 International Experience Canada

Finding #6
Young people value the wide opportunities to travel, live and work in a foreign country and the International Experience Canada (IEC) program is a mechanism to offer such opportunities.

Some 17 of the 37 missions which took part in the survey conducted as part of this evaluation indicated that they were involved with IEC. The number of working days devoted to IEC by missions is extremely variable, of course, ranging from a few days in the US where 161 permits were issued in 2008 to hundreds of days in France where upwards of 9000 permits were processed in 2008.

Asked how much of a priority IEC is at their mission, 9 of 17 missions providing an answer indicated that it was an extremely high or a high priority; 8 rated it a moderate or low priority.

During visits, missions involved with IEC indicated that they considered it an important opportunity for young Canadians to get experience outside Canada and an important source of goodwill for the Government of Canada because, in the experience of mission staff, having been exposed to Canada makes foreigners more positively inclined towards it.

The main frustration of larger users of IEC (e.g., France, United Kingdom) is that the quotas are filled as fast as they are announced. In their experience, the demand for the program could exceed two to three times current allocations.

In some site visits, tensions between diplomatic objectives and labour supply objectives were quite apparent. DFAIT staff emphasized the diplomatic purposes of the program (as outlined above), whereas Citizens and Immigration Canada (CIC) staff noted that the large numbers of issued Working Holiday permits compared to the Young Professional and Internship permits. CIC tended to justify the need for the program on the basis of the Canadian demand for labour rather than the demand for mutual goodwill.

3.3 Alignment with Government Priorities

Finding #7
The IEY programs are aligned with and contribute to two of the main Government of Canada priorities: Priority #1: Greater economic Opportunity for Canada, with a focus on growing/emerging markets; and Priority #2: United States and the Americas, as well as to the Advantage Canada Agenda.

These programs support directly the Advantage Canada Agenda of the Government of Canada, focused on improving the quality of life of Canadians and Canada's success on the international stage. By facilitating youth mobility, funding student exchange programs, granting scholarships for Canadians and foreign students, and supporting academic research in priority fields for Canada, these programs help the government fulfill its "KnowIedge Advantage" commitments, and in particular the creation of a well educated, skilled and flexible workforce.(25)

3.3.1 Understanding Canada

Understanding Canada was often described by Heads of Mission as a program that provides seed money for future economic prosperity, increased international visibility and awareness of Canada. Even though the immediate economic impacts of this program are difficult to measure, recent empirical evidence suggests that it brings benefits to Canada in terms of economy, research and development, science and technology, and international cooperation.(26) By fostering collective research and the creation of networks of scholars, policy and decision makers, Understanding Canada advances Canada's foreign policy objectives in a most efficient way.

3.3.2 International Scholarships Program

By providing scholarships to foreign students who wish to study in Canada and creating opportunities for Canadians to take advantage of reciprocal awards, this program raises the profile of Canada as a preferred study destination and supports the development of skilled and educated young generation of Canadians who will be shaping the future of the country.

The institution of a considerable number of scholarships for emerging markets (e.g., Canada-China Scholarship Exchange Program; the Canada Commonwealth Scholarship program) is fully aligned with Priority #1 of GoC.

Programs, such as the Emerging Leaders in the Americas, the Canada-Chile Leadership Exchange, the Canada - CARICOM Leadership Awards, and Canada-US Fulbright Scholarships directly support Priority # 2 of the government, and in particular, Canada's comprehensive strategy for relations with US, and the advancement of the Americas Strategy.

3.3.3 International Education Policy

The ultimate outcomes of the International Education Policy, as described in the 2009 ARAF, support both the Advantage Canada Agenda and GoC Priority #1:

  • Canada's knowledge advantage is recognized around the world and serves to strengthen Canadian interests and economic prosperity; and
  • The International Agenda is shaped to Canada's benefit and advantage in accordance with Canadian interests and values.

3.3.4 Edu-Canada

As part of the Global Commerce Strategy, Edu-Canada, and in particular the promotion of Canada as a preferred study destination support the GoC priority to "Create Greater Economic Opportunity for Canada, with a focus on growing /emerging markets." The recent study by Roslyn Kunin & Associates, Inc (July 2009), indicates that the annual contribution from international students to Canada's economy is in the range of $6.5 billion dollars, a fact that unequivocally demonstrates the value of Edu-Canada for the economy.

3.3.5 International Experience Canada

By fostering close bilateral relations between Canada and other countries through cultural exchanges, the International Experience Canada contributes to Canada's Knowledge Advantage and enhances the personal and professional development of the participants - both Canadian and International Youth. The program also supports, albeit indirectly, GoC Priority #1.

3.4 Consistency with Federal Roles and Responsibilities

This section analyzes whether the programs correspond to an identifiable federal role and whether DFAIT is the appropriate implementing department.

Finding #8
The International Education and Youth programs are connected to DFAIT's program activity architecture.

International Experience Canada, Understanding Canada, the International Scholarships Program and the International Education Policy support DFAIT's second program activity "Diplomacy and Advocacy."(27) The associated expected result is "Canada's diplomatic efforts and programs engage and influence international players to protect and build support for Canadian interests and values." This result ties up to Strategic Outcome 1: "The international agenda is shaped to Canada's benefit and advantage in accordance with Canadian interests and values."

There is little doubt that DFAIT is the appropriate delivery department for the International Education Policy: in the absence of a federal entity responsible for education (a provincial responsibility in the constitution of Canada), it is logical to resort to the entity responsible for international relations to handle this question.

IEC is based on bilateral agreements between Canada and other participating countries and its objectives are diplomatic rather than economic. These are the two justifications for DFAIT's leadership in a program whose day to day operations could possibly be under the responsibility of CIC.

Finding #9
DFAIT's responsibility for Understanding Canada is justified by the importance of factors other than academic excellence in grant adjudication.

Understanding Canada aims for diplomatic objectives through the support for academic interest in Canada. The program has repeatedly stated that the prime criterion for support decisions is academic excellence. If that was the only criterion in use, then Understanding Canada could possibly be delivered in collaboration with federal granting agencies which already handle a large number and types of grant applications in all possible subject-matter areas. If criteria other than academic excellence are at play, the management of the program by DFAIT is justified since the program is designed and administered by people who are knowledgeable of the foreign policy priorities of the Government of Canada.

Finding #10
International scholarships could potentially be managed by federal granting agencies.

The International Scholarships Program also aims for diplomatic objectives. However, once the specific programs are in place (possibly established through bilateral agreements), their management is focussed on the academic excellence of the candidates, very much like any other scholarship program managed by the federal granting agencies (e.g., the Vanier Scholarships). In that sense, the justification for DFAIT's direct involvement in the implementation of the scholarship programs is thinner than in the case of other PRE programs.

Edu-Canada is a component of Program Activity #3: "International commerce." This activity includes three expected results:

  • International commercial activity performed by Canadian business clients, such as exporting, direct investment abroad and technology commercialization, has increased.
  • Foreign direct investment in proactive sectors and from key markets to which DFAIT contributes has increased in number and value.
  • Overall high-quality international commerce services are delivered to Canadian exporters, importers, investors and innovators.

Edu-Canada is delivered by DFAIT because this department is the entity responsible for representing Canada internationally. In that sense, Edu-Canada acts as the focal point for the interests of provinces and territories in this area as well as for educational associations and institutions, each of which has limited resources and capacity to develop and implement an international education marketing strategy.

From a legal standpoint, it can probably be argued that, while provinces have exclusive legislative jurisdiction over education, pursuant to section 93 of the Constitution Act, 1867, the promotion of educational services is a proper matter for international trade which is a DFAIT responsibility.(28)

A report commissioned by DFAIT(29) has recommended the creation of: "a new, stand-alone international education marketing agency (IEMA), owned by the Canadian Government. IEMA should be: small, agile, and performance-oriented; expertise-driven; ready to grow and evolve; and uniquely Canadian. IEMA will face the challenge of integrating itself into a fragmented, complex, and currently shaken-up landscape. It is important to keep in mind that in order to ensure IEMA's success under these conditions, everyone must agree to compromises; there will be no room for dogmatic positions." This evaluation is not in a position to determine the feasibility of this option.


4.0 Performance

This chapter addresses three performance issues: have the programs achieved the outcomes expected of them; have the programs been managed efficiently and economically; and, have program governance and management structures contributed to program success?

4.1 Achievement of Expected Outcomes

The achievement of expected outcomes was assessed in relation to the reach of the programs to their audience, the program achievement of objectives, and the use of programs by posts.

The original evaluation matrix contained several indicators collected from the program ARAFs. However, it was found in the implementation of the evaluation study that a large number of these indicators are not yet measured by programs, leaving this evaluation with mostly qualitative evidence collected through mission site visits and the survey of missions.

4.1.1 Understanding Canada

Finding #11
Understanding Canada contributes to substantial intellectual activity about Canada and may influence the position of foreign countries toward Canada but there is limited factual demonstration of the latter.

 

According to the program ARAF, the immediate outcomes of Understanding Canada include the following.
OutcomeObservations
Canadian studies programs are better aligned with the government of Canada's priorities.With the advent of the new Understanding Canada program (2008), applications that are rated equal in academic excellence are re-weighted in relation to their fit to the content priorities of the program, which are: democracy, rule of law, human rights; economic prosperity and competitiveness;environment and energy;managing diversity;North-American partnership; and,peace and security.

According to a compilation of information prepared at PRE, "In this first year, more than 84% of the grants awarded on a competitive basis where directly linked to one of the six above mentioned priority themes."(30)

In interviews and site visits, several informants indicated that the inclusion of the priority areas was badly received by stakeholders in 2008 because they appeared to encroach on the academic freedom that grant recipients had enjoyed for many years. In some countries, this led to a decrease in the number of applications to the program. According to informants, in other countries, it led to a diversification of the pool of researchers attracted by the program.

Several informants indicated that, now that applicants have some experience working with the themes, they realise that the themes are wide enough to accommodate most research interests.

Conferences, research, teachings and publications on Canada are developed and supported by local sources.Beneficiaries of Understanding Canada reported a significant level of activity in a survey conducted by ICCS on behalf of DFAIT.(31) This activity included at least the following: 103 newsletters; 30 learned journals about Canada and Canadian Studies;12 annual, 11 biennial, and 2 triennial conferences held by associations;17 student seminars;32 off-campus programs;63 community or school outreach activities undertaken by Canadian studies centres;40 initiatives to develop curriculum for use in schools;79 conferences and seminars undertaken by the centres;54 special publications prepared by the centres;541 articles published annually;94 books published annually;150,000 students taking courses with some component, or in their entirety, on Canada;415 doctoral theses; and,1570 Master's theses worked on annually.
New collaborative linkages are in place.Collaboration could occur among researchers, among study centres and associations, among stakeholders and missions, among missions and PRE. From interviews and site visits, the conclusion is rather that the new program has exacerbated competition and rivalry among stakeholders and reduced communications among them. Several representatives from associations have expressed concern that these organizations are cut out of the application selection process and that no one has the knowledge or ability to foster synergies as they claim associations once did. PRE files show that missions have been consulted and invited to provide comments and recommendation on all applications. However, missions are critical of the role they are given, the burden it represents and the little influence they feel they have over award decisions.

 

The intermediate outcomes of Understanding Canada are the following.
OutcomeObservations
Foreign scholars and opinion leaders have broadened and deepened their knowledge of Canada and Canadian issues.Direct evidence of foreign scholars and opinion leaders' knowledge was available to the evaluation team only though the limited number of interviews conducted with award recipients during the field trips. It is reasonable to believe that knowledge was produced via the work performed using the 500 plus grants and contributions distributed by the program in 2008 (and previous and following years). It must also be factored into this analysis that the Understanding Canada funds are often used as seed money to initiate or validate a project that will get substantial additional funding from other sources (the argument being, if Canada finds the academic activity about Canada worthwhile, it must be).

Exhibit 4.1 (extracted from Exhibit 1.1) depicts a disturbing trend, however: over the years, the number of Faculty Enrichment Program grants (to develop course material) and the number of Faculty Research Program grants (to collect data in Canada) have decreased. This is a concern since these programs are considered important in the availability of courses (with student interest snowball effects) and of publications on Canada.

Sustained, stable, and growing network of foreign professionals and leaders with a well-disposed and sustained interest in Canada.The evaluation team has not seen scientific data on the size of the network of professionals and leaders called "Canadianists". The evaluation team was informed that ICCS maintains a database of international Canadianists but there is no further information on how many of them are still active. While, list of Centres for Canadian Studies exist as well, not all of them seem to be active. It was therefore not possible within the framework of this evaluation to assess whether these networks are sustained, stable or growing.

Opinions collected in interviews and field visits were to the effect that the addition of the priority themes in 2008 resulted in specialists of non traditional disciplines showing interest in the program and in Canada (literature and history were traditionally heavily represented among beneficiaries of the program).

Some changes made to the program in 2008 were not well received (such as the new application forms and process, as discussed in the next section) and may have alienated some members of the network.

Increased linkages between foreign and Canadian academic institutions.There is anecdotal evidence that the program has contributed to the establishment of linkages between foreign and Canadian academic institutions: some researchers met during site visits named Canadian counterparts with whom they worked or whom they visited during an FEP/FRP trip. No quantitative evidence is available to that effect, nor evidence that would distinguish between established relationships that were continued versus new linkages that could be established.

Bar Chart - Number of FEP and FRP Awards, 2004-2008

In response to the survey of mission, 14 missions (out of 22 who answered the question) indicated that they agreed somewhat or totally that "Clearly, in the context of this mission, Understanding Canada has achieved its objectives" and 16 (out of 22) indicated that "In the context of this mission, Understanding Canada has made a significant contribution to Canada's foreign policy in the areas of knowledge and research."

Respondents mostly equated achieved objectives with the program's contribution to promoting Canadian foreign policies. Several means were identified as facilitating progress toward this objective including developing and maintaining relationships with local organizations and institutions as well as the organization of conferences and other events made possible by the program. Survey participants have also indicated the opportunities provided through Understanding Canada funding for partnering with local think-thanks and the organization of events on priority issues for Canadian foreign policy. These events are seen as unique opportunities to promote Canada's foreign policy and stimulate dialogue between Canadian and foreign policy-makers and thinkers. Other respondents identified an increased interest in Canada as a demonstration of meeting objectives. Some of these respondents specifically indicated the increase in high quality applicants pursuing research in priority areas, and the growing number of Canadian Studies programs enabling the participation of high level key speakers at events.

Six missions were not positive about the achievement of Understanding Canada objectives. They pointed to a lack of human resources and lack of funding, as well as a shortage in interest and successful candidates. Other comments regarded the need to simplify program processes, to better align the program with priorities and for PRE to provide better feedback.

4.1.2 International Scholarships Program

Finding #12
A case for the performance of the International Scholarship Program, as a stand-alone intervention, could not be made because of a lack of information on scholarship beneficiaries.

According to the program ARAF, the immediate outcomes of the International Scholarship Program include the following:

  • increased collaboration by institutions and government;
  • increased awareness of Canada as a study destination; and,
  • increased mobility and optimized demand of highly talented Canadians and foreign nationals.

The intermediate outcomes of International Scholarship Program are the following:

  • increase in stable and self-sustainable inter-institutional linkages and collaboration;
  • increased understanding of and collaboration with Canada by foreign alumni;
  • increase in graduate students from ODA countries who return to their countries to contribute to its development; and,
  • increase in contribution to Canada's innovation by Canadian and foreign scholars.

Notwithstanding the priority given to scholarships by the Government of Canada (in the form of the recent creation of Emerging Leaders in the Americas Program), scholarships were not discussed extensively during the interviews and site visits because informants had little to say about them. In most of the cases the missions play the role of a mail box from where applications are re-directed to the respective institutions in Canada. In parallel, very little performance information exists on the scholarships and on their beneficiaries.

Take-up data were presented in Exhibit 1.2. As Exhibit 4.2 depicts, the progression in the number of scholarships in the last few years has been fast.

Bar Chart - Number of Scholarships, 2005-2009

Regarding the largest program, ELAP, the main constraint to program success identified in site visits was the slow pace of adoption by Canadian universities (which need to accept the students and co-sign the scholarship application) consequent to what is seen as insufficient promotion of the program in Canada.

In response to the missions' survey, 10 missions (out of 21 who answered the question) indicated that the International Scholarship Program achieved their objectives and 11 stated that the program made a significant contribution to Canada's foreign policy in the areas of knowledge and research. Respondents made reference to an increase in the quality of candidates applying and receiving awards as well as an increase in interest and quantity of applications received. One respondent indicated that, in particular, the PDRF was only in its third year, but that applications had doubled in the past year and forecasted a substantial increase for the next year. Other indicators of meeting objectives were the increase in the variety of disciplines represented and important research being conducted in priority areas: "The variety, depth, and importance of work being done by post doctoral researchers is a strong testament to the effectiveness of the program."

The criticisms from missions on ISP were different for each respondent. They included: lack of applicants of the calibre sought; insufficient criteria for optimal selection; and, the absence of scholarships in the mission's country.

4.1.3 International Education Policy

Finding #13
A case for the performance of the International Education Policy could not be made because of the lack of performance information.

The ARAF of the International Education Policy indicates that it is associated to the following activities:

  • building and maintaining relationships with federal and provincial governments regarding to engage them in Canada's international education related policies and activities;
  • organizing, funding and coordinating Canada's representation in international education events;
  • coordinating, conducting and analyzing issues connected to international cooperation in education;
  • assessing and reporting on national and international trends in education;
  • leading and facilitating the process leading to signing and ratification of agreements on education related issues; and,
  • monitoring the compliance of stakeholders to terms of agreements.

Outputs of the Policy include:

  • Canadian participation and representation at international fora;
  • conferences advocating international policies consistent with Canadian interests;
  • reports on issues and trends in the international education field;
  • agreements on education related issues;
  • strategy reports on bilateral and regional relationships; and
  • assessment reports completed by delegates.

Identified immediate outcomes are:

  • enhanced representation, coordination, and understanding of international Canadian priorities as it relates to education;
  • enhanced international cooperation through implementation of policies reflecting Canadian interests;
  • increased awareness of Canadian policies and experience in education in the international education community; and,
  • compliance in all areas covered in the terms of agreements.

Intermediate outcomes are:

  • increased impact of Canadian policies on the international community; and
  • increased provincial engagement in a pan-Canadian approach to international education issues.

While little evidence was available on the performance and achieved results of the International Education Policy, information was found on activities and outputs such as agreements with China on a Scholars' Exchange Program and on Equal Opportunities Scholarships; with Brazil on Academic Mobility and Scientific Cooperation; and with India on Cooperation in Higher Education. Other agreements negotiated in part or in full by the International Education Policy include: the Canada-USA Agreement for the Establishment of a Bi-national Educational Exchange Foundation, the Canada-EU Framework Agreement for Cooperation in Higher Education, Training and Youth, the Convention on the Recognition of Qualifications concerning Higher Education in the European Area, and the Canada-Mexico Cultural Agreement.

The evaluation found that key informants were not aware of the existence of this Program and its ultimate mandate. In fact, some interviewees expressed surprise at the title "International Education Policy," indicating that while "no such thing exists," they would welcome one.

The 2008-2009 expense report submitted by CMEC signals that approximately $99,000 were expended in 27 events using the federal grant whereas CMEC itself invested about $54,000 in 11 events.

4.1.4 Edu-Canada

Finding #14

Edu-Canada has achieved important milestones; it still has to document performance in the production of outcomes.

 

According to the program ARAF, Edu-Canada's immediate outcomes include the following.
OutcomeObservations
Raised awareness among targeted audience.This outcome could have two meanings: raised awareness of Canadian opportunities by foreign students and raised awareness of foreign opportunities by Canadian educational institutions.

Edu-Canada being a marketing program aimed at increasing the inflow of foreign students in Canadian institutions, the former meaning has great importance. At this point, there is no empirical demonstration that Edu-Canada efforts have produced this effect. It may be too early to observe it, but the program is not currently equipped to measure a change attributable to its action anyway.

Regarding the awareness of Canadian institutions, key informants have indicated that several among the largest ones already had an international marketing strategy. However, the process of developing the "Education in Canada" Brand appears to have had the side effect of better informing institutions on the potential of international education. Informants indicated that there is still a lot of work to do to educate Canadian educational institutions about the benefits of exports and to convince them that a whole-of-Canada promotion approach is more effective for everyone than efforts at selling individual institutions.

Canadian missions in priority markets are more active in the promotion of Canada as a desired destination for study.Thirty-four of the 37 missions which completed the mission questionnaire indicated that they were involved in the promotion of Canadian education services. The level of involvement varied considerably, from an estimate of one day per year to 550 person-days. The promotion of education services was considered a high or extremely high priority in 20 out of 37 missions.

There are disagreements between Edu-Canada and some missions on what constitutes a priority market. In some instances, Edu-Canada requested actions on the part of missions which considered that education marketing was not a priority and was already performed adequately by Canadian institutions themselves.

Have more active missions overall and strengthen brokerage relationships.No baseline information exists to determine whether more promotional activity has taken place under Edu-Canada compared to before its advent.

Since the Trade sector information system is focussed on results rather than activities, it is doubtful whether it could be used to provide performance information on mission activity in education marketing.

Key informants indicated that the development of the Canada trademark was a significant accomplishment in and by itself -in terms of both the success of the multi-party process it required and the usefulness of presenting a coherent Canadian image on international markets.

Increased inter-institutional linkages.This outcome is not measured quantitatively. Key informants did not identify this as a likely result of the program. In mission visits, comments were more to the effect that institutions tend to try to conduct their business in an isolated fashion rather than in collaboration with others. Collaboration between Canadian institutions and foreign institutions pre-date the existence of Edu-Canada.
Increased number of foreign students attending Canadian institutions.The program is not currently equipped to reliably measure whether its action has produced an increase in the number of foreign nationals studying in Canada. Such a result would likely occur only a few years after program action anyway. Nonetheless, the program has started to track trends in enrolment which suggests that change may be taking place in the expected direction.

 

The intermediate outcomes of Edu-Canada are the following.
OutcomeObservations
Enhanced international profile of Canadian education.While several key informants and mission representatives praised the work done by Edu-Canada, few were of the view that it has yet succeeded at enhancing the international profile of Canadian education. It might well be too early to tell. Views were that the work completed to date included important building blocks toward achieving this objective - in particular, the establishment of the Canada trademark.
Clients and stakeholders use Edu-Canada as a preferred vehicle for obtaining market information and promote their interests.Based on the qualitative evidence gathered in in-depth interviews, institutions have not yet adopted the "Imagine Education au/in Canada" Brand as their preferred vehicle to perform their international promotion. Here again, it might be a question of time now that the trademark is available.
Canadian educational institutions have greater buy-in.There are no data available to support a conclusion on this outcome. Educational institutions have not been surveyed, either by the program, or in the context of this evaluation. However, strictly anecdotal evidence collected during field visits suggests that Canadian institutions are still reluctant to move away from their established isolated ways of promotional activities.

In response to the survey of mission, 23 mission representatives (out of 34 who answered the question) agreed that "Clearly, in the context of this mission, promotion of education services offered by Canadian education establishments has achieved its objectives." Most of these positive comments concerned the ability for missions to take part in numerous events such as fairs, trade shows and conferences. According to respondents, presence at such events aims mainly to increase the profile of Canada as a study destination, but one respondent also indicated other benefits: "This provided the opportunity to network with key players in the international education industry, to observe developments in international education and to provide valuable market intelligence to Canadian stakeholders." Several others mentioned an increase in trade activity, shown by the growing number of student visas issued, the attendance of students at fairs and conferences and their expressed interest in studying in Canada.

Some mission representatives were critical of Edu-Canada. Reasons for not achieving objectives varied, but a common theme was found in the disconnect between the focus of Edu-Canada and the local realities of some countries. Other impediments to the success of educational promotion included the lack of scholarships offered by Canadian universities for graduate students (unlike the US) and issues with the recognition, in Canada, of undergraduate degrees from some countries (e.g., EU countries as a result of the Bologna Process and the modular system in Europe, India).

Other reasons provided for not reaching objectives included: education not being a priority; lack of human resources; lack of strategic approach; and, lack of communication from HQ. One respondent noted the difficulty in measuring outcomes: "Our activities are not targeted in specific fields that would be relevant to our priorities but rather to the general public. It is impossible for us to measure in any real way whether our efforts are having an impact or not in this market."

4.1.5 International Experience Canada

Finding #15
IEC has strengthened its management processes; current performance priorities include the promotion of the program among young Canadians and performance measurement.

 

According to the program ARAF, IEC's immediate outcomes include the following.

OutcomeObservations
Mutual understanding of other cultures through a travel, life and work experience abroad.There is no empirical demonstration of the realisation of this outcome. Several mission representatives cited this outcome as a strong justification of the program but their conclusion was based on anecdotal evidence and common sense.
Increased awareness of IEC.There is very limited empirical evidence of the realisation of this outcome. The ARAF suggests the use of Web site visits as an indicator of awareness. The monthly total of Web site visits has progressed in a very limited fashion from about 8,000(32) between August 2008 and June 2009 to 8,500(33) between July 2009 and February 2010.
Ensure consistency of IEC in terms of reporting and accountability.According to key informants, IEC suffered from a severe lack of structure and policies in 2007 when it was converted into a user fee program. Since then, considerable efforts have been expanded to develop standardized practices and interpretation guidelines. A Privacy Impact Assessment, an ARAF, a Standard Operating Procedures Guide and Service sSandards were developed along with a reporting architecture to measure performance. A number of areas still await standardization, such as overpayments. The program possesses no integrated data base allowing it to report on implementation and to analyze performance.
Effective collaboration with Program partnersCanadian delivery partners are generally satisfied with the administration of the program. An issue has arisen with the legal request from the Canadian government to open application processing in the United Kingdom (and possibly elsewhere) for bids to find a service provider to process IEC applications. Since the ensuing contract would be for the single service of application processing, it would possibly reduce the market for partners such as SWAP who are in a monopoly situation in some countries.

CIC is a key IEC partner. Relationships with CIC are generally positive and productive. Irritants include the absence of a clear communication strategy within and between the organizations and the imprecision of annual targets supplied by DFAIT and used by CIC for HR panning purposes.

 

ICE's intermediate outcomes are the following.

OutcomeObservations
Foster close bilateral relations between Canada and other countries.Translated in the IEC context, bilateral relations mean the existence of bilateral agreements and the two-way flow of youth. There are currently 21 bilateral agreements in place; 3 more are awaiting ratification; negotiations are underway with 15 more countries. The countries with which Canada has IEC agreements are typically low-risk countries (i.e., countries where participants are most likely to abide by the rules of the program). Some stakeholders are concerned that some of the countries with which agreements are contemplated present higher risks (e.g., not returning to the home country at the expiration of the permit or claiming refugee status upon arriving in Canada) and that more resources will be required to screen applications from these countries and to monitor participants.

The reciprocal nature of the agreements can take two forms: reciprocity in opportunities (e.g., France) and reciprocity in permits (e.g., United Kingdom). In effect, as shown in Exhibit 4.3 which is based on Exhibit 1.3, the proportion of outgoing Canadians as part of all IEC incoming and outgoing participants has decreased between 2004 and 2009 compared to the growing number of foreign participants.

Increase of the favourable perception of Canada internationally.As stated earlier with regard to the outcome of mutual understanding, there is no empirical demonstration of the realisation of this outcome. Several mission representatives cited it as a strong justification for the program but their conclusions were based on anecdotal evidence and common sense.
Support the implementation of other government initiatives.Other government initiatives in question could include Advantage Canada, Education au-in Canada, economic development, temporary or permanent immigration to Canada, labour shortage. During site visits and key informant interviews, there were several mentions of such benefits. For example, IEC was said to be useful for Canadian employers looking for temporary workers. However, there is a resistance among some mission staff to divert IEC from its diplomatic objectives, in particular because the agreements with other countries are negotiated on the basis of the mutual understanding objective, not labour-market objectives. There is a risk that increased expectations from IEC with regard to objectives other than diplomatic could jeopardize relationships with some governments. However, such objectives have been formalized by an exchange of letters between the Deputy Minister of CIC and the Deputy Minister of DFAIT in which CIC has asked DFAIT for in their goal to fix short-term labour market challenges.
Enhance participants' personal and professional development through an experience abroad giving them a competitive edge in the job market.The evaluation Team has not seen empirical evidence demonstrating the realisation of this outcome. Stakeholders and mission representatives have cited it as a justification of the program but their conclusions were based on anecdotal evidence and common sense.

% of Canadians in All IEC Participants, 2004-2008

Seventeen mission representatives rated their agreement with the following statement: "Clearly, in the context of this mission, International Experience Canada has achieved its objectives." Eight out of 17 agreed somewhat or totally with this statement. Those who did mostly emphasized the very high interest and participation in the program and consequent economic benefits for Canada as well as an expected impact on future relations with Canada. Specifically, the high participation rate would lead to an increased understanding of Canadian culture and society: "It means there will be tens or hundreds of thousands of adult citizens in government, private sector, or wherever, who will have spent a formative year of their life in Canada." This understanding was said to further bilateral cooperation and to provide a "positive messaging opportunity" in communication with foreign officials as well as the public.

Mission representatives who were critical of IEC success indicated most commonly that there was a lack of human resources to manage the program and a lack of structures or mechanisms to maximise the benefits of the program. However, ten missions out of 17 who answered chose the following response: "Clearly, in the context of this mission, International Experience Canada has made a significant contribution to Canada's foreign policy in the areas of knowledge and research." This view is also supported by the Prime Minister's Office and the Minister's Office.

4.2 Demonstration of Efficiency and Economy

Evaluation questions related to efficiency and economy relate to program management issues and their impact on program performance, the possibility of delivering programs at lower cost and the capacity of the programs to monitor their progress and performance.

4.2.1 Understanding Canada

In missions, Understanding Canada garners substantial support and criticism. Support goes to the principles of the program, its target and its results; criticisms are around program processes and level of funding.

Program management issues and their impact on performance

Understanding Canada comprises a large collection of funding tools, some targeted at national associations, others at study centres, yet others at individuals and again at missions. Even though the 2008 revisions to the Program included opening it to non-academic organizations, the complex programming environment is still reported to favour individuals and organizations already in the know, while discouraging new applicants with its lengthy application procedures and requirements that are not seen as commensurate with the low amounts of the respective grants. This complexity, coupled with the limited resources available at missions, makes difficult the development of any kind of strategy for countries or regions. Thus, the variety of tools appears to translate into complexity and rigidity rather than flexibility and adaptability.

Finding #16
Understanding Canada is an overly complex program that has become bureaucratic and burdensome on applicants and mission staff.

The application process put in place with the new Understanding Canada Program as a result of the identified need for stricter due diligence (2005 Evaluation of the Canadian Studies Program) was severely criticised by many key informants. Applicants considered it cumbersome and costly, especially in view of the award amounts available. The evaluation team heard from several sources that potential applicants have seriously questioned whether they would take the time to fill out the new application kit.

  • The financial information section of the application forms is considered particularly demanding.
  • Comparisons were quoted with other international programs (e.g., the German Academic Exchange Service - DAAD), where the main emphasis is on the content and not on logistics (such as cost of board rooms, electricity, etc.).
  • The application process is seen as overly bureaucratic by many potential applicants, particularly the budget section of the application form.
  • Complex application processes tend to deter new applicants and benefit existing beneficiaries.
  • Changes in the program had more negative impacts than they could have had, had there been a communication strategy in place.
  • All programs are managed on an annual planning horizon making it close to impossible to support a series of activities spanning over a longer period. Researchers are brought to work more precipitously and in a less thoughtful manner.

The selection process was also criticized.

  • It often involves reviews by missions, by ICCS, by geographic groups and by the Program at headquarters - a process that is considered too bureaucratic and opaque by many.
  • Applications for three programs receiving many applications (FEP, FRP, DSRA) are reviewed by an ICCS committee of two individuals. Some question how these two individuals are able to discern academic excellence in a wide variety of disciplines. Also, these reviews are not independent: examples indicated that notes from one reviewer made reference to comments by the other reviewer. This review process, while low cost, could be less than effective at properly assessing proposal quality.
  • Some missions are critical of the expertise of ICCS proposal evaluators and of their qualifications to override ratings made by missions and national experts. While this could be a reflection of a selection process perceived by missions as unclear, unsubstantiated and lacking in transparency, PRE management indicates that such changes are either administrative or intended to add additional awards to the already pre-selected ones. However, no information was provided to the Evaluation Team on the criteria supporting such discretionary decisions.
  • The final decisions rest with the PRE Management; some key informants indicated that these decisions may diverge from recommendations of missions and ICCS without clear justification and feedback. It is not clear whose plan these decisions reflect: that of the program at headquarters (which has no formal plan), or that of the missions (which cannot currently have a plan), or no plan at all.
  • From the point of view of applicants, past the mission, the process is a black box: they are not informed of the process and analyses, and no justification accompanies the decision.
  • Because of the complexity of the program tools, because of researchers being expected to initiate projects, and because funding decisions are made at headquarters, some missions consider it is impossible to draw up a plan of what the optimal network of Canadianists would be in their country and to strategically place resources to implement this plan.

Formally, the selection process is based primarily on the academic excellence of the proposals; where applications are of equal academic quality, those within priority areas are favoured. However, the objectives of Understanding Canada find meaning at the national level rather than at the global level.(34) In fact, the real beneficiaries of the program are the missions since they are at the forefront of the diplomatic battleground. Formal recognition of the importance of missions in this program could take the form of a new criterion, that of "merit," that would combine academic excellence and potential impact of the funded work.(35)

Finding #17
Understanding Canada is a highly centralized program that would benefit from better communications to ensure additional mission input.

For some, missions are best located to determine which types of supports are best suited for the national network of Canadianists. In some countries, there may be enough Canadian studies programs in place, such that Program Development Grants may be unnecessary. Elsewhere, course development may present the best potential for impact and Faculty Enrichment Program grants may be the best investment. Where the mission has a clear concept of what the optimal development of the network of Canadianists should be and which levers will produce the most significant impact, Canadian Leadership funds may be the preferred vehicle. The question is: where should these decisions be made: in missions or at Program headquarters? For some in the missions, the contexts are so different from country to country that it would be more effective and efficient to plan and manage Understanding Canada at missions; headquarters could play an advisory role and be responsible for program oversight. Short of this drastic change, suggestions included to give the ratings of missions a substantial weight in funding decisions and to ensure a much more agile administration at headquarters to seize opportunities.

Some key informants were critical of the management of the program. For example, tardiness in decisions on application deadlines has meant that some applicants had less than desired time to prepare applications, e.g., the 2010 deadlines for the Program Development Grant (March 7) were announced to missions only on January 22. Delays in funding decisions have also been noted and quoted as factors affecting the ability of applicants to plan efficiently and ahead of time their visit to Canada. Application deadlines for some countries were also quoted to be ill-adapted to university semesters. Centralized program management is viewed as further exacerbating such issues.

Not surprisingly, researchers were not keen on the imposition of priority themes. Academics cherish their independence and could easily perceive the priorities as an encroachment. Some researchers argued that the imposition of priorities is counter-productive because the solutions to the problems of today are found in the research conducted years if not decades ago. In their view, supporting research focussed on today's problems is short-sighted. Some mission staff indicated that imposing Canada's priority themes on foreign researchers is not the best way to initiate a dialogue.

The program deals with dozens of countries; it counts on ICCS and PRE staff to act as integrative mechanisms. Other possible tools for integration and support could include communities of practice for mission officers and performance measurement, as well as the Departmental WIKI system, but little is done in this regard.

Understanding Canada is open to university professors with permanent status. Some missions indicated that, because of the demography of university professors, as a group, young academics may have to wait for several years before they get this status. They suggested giving them access to grants in order to support the renewal of the Canadianist ranks.

There were several mentions of a need to better promote Understanding Canada with target groups in selected countries. According to informants, this would improve the number, and eventually the quality of proposals. However, considering the limited resources available to the program, a realistic analysis of the consequences of such a promotional campaign would have to be conducted first.

Finding #18
Understanding Canada has not yet capitalized on new technology.

Innovation may be dampened by the long history of the program. For decades, Understanding Canada has been based on grants to academic researchers and organizations. This is a model that may need to be updated in view of possibilities offered by electronic communications, electronic publishing and social networking. For example, for a while, the Paris embassy maintained a list of Canadian researchers residing in France that was considered extremely effective and efficient by French researchers and the embassy itself; maintenance was discontinued due to budget cutbacks. Example: there is no tool available to build networks of Canadianists that would cross national boundaries (with the possible exception of Europe); in fact, program applicants indicated that the structure of the program tends to make cross-national work essentially impossible; yet, freely available social networking applications could be leveraged to support such exchanges. ICCS has initiated a project using Facebook. Example: ICCS maintains a list of 6,000 Canadianists on its Web site. Another example: a survey conducted by ICCS on behalf of DFAIT(36) indicates that "30.5 learned journals about Canada and Canadian Studies are produced by the centres and associations."(37) Many of these are supported financially via grant agreements with national associations. There is also an International Journal of Canadian Studies (IJCS) managed by ICCS; in this world of ubiquitous electronic connection, the program could concentrate its funding of an electronic-only IJCS.

Some key informants wondered whether the program has too little funding to really expect to make an impression. Many conclude that the program has produced remarkable results in view of its resources but that much more funding would be required to exert the type of effects that would be beneficial for Canada. Alternatively, they suggested that funding could be focussed on fewer countries so as to initiate a dynamic that may self-sustain later on.

In particular, key informants indicated that the University Library Support Program and the Book Display Program were hardly worth the trouble considering the maximum amounts of support they offer.

Delivering the program at lower cost

On a contract basis, ICCS receives about $300,000 for the management of approximately $1.7 million.(38) Thus, administration represents about 17% of total costs within the ICCS contract. These estimates do not include the costs incurred by Understanding Canada at headquarters or in the missions. Administration costs in the vicinity of 15% to 20% are common in grant programs.(39)

ICCS is working on a Web-based system to receive and process applications. Key informants expressed some frustration at the lack of access to such electronic applications.

Performance monitoring

Finding #19
Understanding Canada has limited performance reporting capabilities.

ICCS produces an annual report which includes lists of activities conducted by their members.(40) However, since ICCS is an independent organization and runs its own programs, such a report does not distinguish results attributable to Understanding Canada funding and other funding, and it does not include beneficiaries of the program that are not ICCS members. Much of the information concerns the number of awards in each program.

Missions indicated that there are no formal reporting requirements related to Understanding Canada. When reports on events and success stories are occasionally prepared, there is a prevailing perception that such reports are neither read, nor considered at Headquarters.

In the course of the conduct of this evaluation, ICCS and PRE staff were asked to supply basic counts of the number of applications and awards, and the value of awards by program and country for the most recent five years. ICCS delivered this information rapidly but Understanding Canada headquarters had to deploy considerable effort over several weeks to produce the data with a caveat as to their reliability. This experience suggests that the program does not have up to date, reliable information on which to base program management decisions.

Recommendation Number 6 of the 2005 Evaluation of the Academic Relations Program (the previous incarnation of PRE) reads: "PCE Management needs to place more emphasis and importance on the implementation of the RMAF."(41) There is little evidence that much work was performed in this area over the past five years. An ARAF was developed (dated February 2009) but systems don't appear to be in place to support reporting on performance. A survey of foreign researchers, study centres and Canadian studies associations was conducted in 2009 but no final report of scientific caliber was available at the time of the evaluation.

4.2.2 International Scholarships Program

Finding #20
The performance of International Scholarships could be improved if they were integrated in a wider PRE strategy.
Program management issues and their impact on performance

Key informants noted that there is a concern that scholarships may produce brain drain for the developing countries sending students among their best to study in Canada. They suggested that more reciprocity could be built into the relationships by providing study, teaching or research opportunities to Canadians in these foreign countries.

The evaluation team is not aware of a document that presents Canada's overall strategy in relation to international scholarships. As for the Emerging Leaders in the Americas Program (ELAP), Canada-Chile Leadership Scholarships and Canada-CARICOM scholarships, they were announced by the Prime Minister as part of DFAIT's trade goals and priority commitments in the Americas Strategy. The rationale is that scholarships promote democratic governance, prosperity and security in the region through capacity development contributions to the countries of Latin America and the Caribbean.

The large variety of scholarships may be a cause for confusion. According to key informants, some scholarships share the same platform and there would be an advantage in bringing them all under one strong brand: this is the case for the Canada/CARICOM Leadership Awards, the Canada/Chile Leadership Exchange Program and the Emerging Leaders in the Americas Program which could all be subsumed under the third label.

DFAIT does not appear to leverage the goodwill developed by scholarships and to systematically build its network of support and influence based on scholarship winners.

The processing of postdoctoral awards by missions is problematic. First, since this is a competition open to all disciplines, it is difficult to find experts to cover the entire field. Second, the demand for these awards is considerable in some countries and missions have to engage large amounts of resources to screen for very few awards. For example, Paris received 70 applications in 2010-2011 for an allocation of two awards. Some missions have decided to not perform pre-selection and to relay all applications to Ottawa.

Some key informants consider that there is too much paperwork involved in applications and management of the scholarships. In their view, Web solutions should be developed soon.

Scholarship programs demand that candidates identify the receptive Canadian institution in their application. This is a significant challenge due the lack of centralised information on placement capacity and possible programs of study at Canadian universities. It is also a challenge because many Canadian universities are not well informed of the existence of the scholarships and are slow in signing agreements with universities in Latin America in order to comly with the specific requirements of the scholarships. Nonetheless, a total of 28 institutional agreements have been signed between delegates from the Americas and representatives of Canadian universities and colleges in the context of ELAP. Out of an initial allocation of 255 ELAP awards, 294 were handed out in 2009. Another issue was that many universities refused to consider ELAP applications sent after the university February application deadline, even though the official deadline to submit an ELAP application was April.

Communication strategies about the program, or the lack of such were a recurring theme. According to responses to the mission survey, more communication is needed, particularly at the mission level. Several respondents mentioned about not having enough information on the ISP and the awards available. Additional communication needs at the student level were also mentioned. Specifically, many respondents said that clear information regarding deadlines for applications was needed, as well as funding to increase promotional capabilities. Two respondents also mentioned that follow up with previous recipients of scholarships would be beneficial for obtaining better outcomes information.

Brazilian partners were puzzled by how few Canadian students come to Brazil whereas there are a lot of American, European and Chinese students visiting Brazil. Studying in Brazil was perceived as an important means of increasing Canada's awareness of Brazil its culture, traditions and trade relations, especially at a time when this country is a priority for Canada. Partners in Brazil, China and India considered that in Canada, and especially among Canadian Universities, the awareness about PRE programs, and especially about ISP seemed very low, which in turn resulted in some international students being denied registration at Canadian universities within the framework of the scholarship agreements.

Performance monitoring
Finding #21
ISP has very limited performance reporting capabilities.

At present, there is minimal performance monitoring of the International Scholarship program. It takes the form of maintaining counts of awards of scholarships to foreign students, applications by Canadian students and term-based monitoring reports for long-term scholarships. No measurement of performance is done in other areas of program activity (e.g., processing time) and in the area of program results.

4.2.3 International Education Policy

No formal reporting or performance monitoring information was provided by PRE with regard to International Education Policy.

4.2.4 Edu-Canada

Edu-Canada is considered a success by many because it delivered an Education in Canada Brand that can be used by all involved in the marketing of international education. The process leading to this result is, itself, considered an achievement. Along the way, though, various management issues have surfaced.

Program management issues and their impact on performance
Finding #22
The expectations created by the program may exceed its level of funding.

Most key informants consider that the program is vastly underfunded in view of the objectives it strives for: "There is a real risk that this whole thing will be a dead duck (pétard mouillé) because expectations are high but resources are insufficient."

Another risk is that missions do not support Edu-Canada on the ground. The program has severely limited resources to ensure a concrete presence outside Canada. Much of this must be taken on by missions and missions are under severe resource constraints as well. Moreover, not all missions share the program's sense of urgency: for example, mission personnel in Paris and London indicated that the marketing of Canadian education services was not a priority of the Trade Sections, which are mostly reactive to requests from industry and have not received such requests from educational institutions beyond assistance with arranging venues for meetings.

In other countries, such as India and China, international education promotion is seen as a priority within the activities of the Trade Commissioner Service. Senior Trade Commissioners in these countries noted that the benefits for Canada from Edu-Canada are often commensurate with, if not higher than the revenues brought by the automotive sector. The success of international education promotion in these countries can also be attributed to the personal commitments of the trade officers to deliver on this program, as well as to the close cooperation with CIC in ensuring higher rates of visa issuance and considerable reduction in the numbers of rejected visas. These examples prove that concerted efforts of the Trade Commissioner Service and the Immigration Sections, as well as strong support from the respective Heads of Mission can contribute to increased visibility of Canada in highly competitive markets.

At missions where Edu-Canada is delivered by the PA section along with Understanding Canada and the Scholarships Programs, opportunities to leverage the strengths of each program and complement their roles in promoting Canada as a study destination are better optimized.

Without continued presence and effort, such as the services offered by the Canadian Education Centres Network (CECN) before its collapse, some informants remain pessimistic that Edu-Canada can make a lasting impression, especially in large countries like India, China and Brazil, which are subjected to intense marketing by other nations. Some critics conclude that missions do not have the necessary resources and the capacity to focus on what is necessary to be successful at education marketing. One risk of the current model (which involves many loosely coordinated organizations) is that it may leave an impression of competing Canadian interests each looking to grab a piece of the education market pie. No one currently conveys an integrated "Canada story."

Edu-Canada efforts can be put at risk by the behavior and choices of others. For example, slow application processing times by universities are said to adversely affect the performance in international education because competitors respond much more rapidly. Another example: Canadian universities use only their own marketing material when promoting themselves abroad (instead of integrating Imagine Education au/in Canada material); this may leave the target audience confused with the image of Canada.

Some informants noted that Edu-Canada has missed some opportunities due to its unpredictable funding, or because of being caught in a slow-moving organizational vehicle.

Finding #23
The performance of Edu-Canada is dependent upon the involvement of many stakeholders over whom the program has no authority.

CIC has been the target of criticism for the time taken to process student visa applications. In response, CIC representatives indicated the need and responsibility to ensure that applicants are bona fide students. According to CIC, there would be less problems if marketing efforts were more successful at conveying immigration requirements, if they were better targeted at students, and if stakeholders were more careful in their analyses. For example, a problem solution was developed for fraudulent applications from India through a joint pilot project with the Association of Canadian Community Colleges (ACCC). ACCC agreed to perform a more diligent review (pre-screening) of applications before these are sent to CIC (Student Partnership Program).

Lessons from countries that have been and are most successful at international education marketing are summarized as follows: "Common to all competitor countries are clearly articulated policies which emphasise (sic) the importance of international student recruitment."(42) This raises the issue of the ability of Edu-Canada and its partners to build and deliver coherent plans without the support of an agency or an organization, similar to the CEC Network. It also points to the integration of immigration and education marketing policies. In the absence of organizational integration, other coordination mechanisms are available: for example, coordinator positions that exist to bridge separate groups, temporary task groups, permanent committees, matrix management structures.(43) Each provides a different level of integration.

In certain countries, including China and Brazil, the Maisons du Québec orchestrate strong education promotion campaigns often in conjunction with the Quebec Immigration Service, providing guidance on when and how students can apply for a permanent immigration status to Canada.

Contrary to successful competitors, Canada accepts that non-regulated education establishments host international students. This makes the monitoring of international students more difficult because government entities cannot request reports on student attendance from non-regulated establishments.

The marketing material prepared by Edu-Canada to date targets undergraduate students. Those who are working with graduate students find these materials of little to no use. In parallel, educational institutions visiting foreign countries rarely use the Imagine brand, preferring to use their own marketing material. Conversely, in some countries (e.g., Germany, India, China), Edu-Canada campaigns are still not effectively targeting high schools as the main source of potential undergraduate candidates for studying in Canada. This is where countries such as Australia and the United Kingdom are focussing their marketing campaigns.

Edu-Canada has not prepared an overall strategy, with identified operational objectives, strategies, action plans. Most of the time, actions are taken in a reactive or opportunistic way based on little information about partners' preferences.

The program has not prepared a compendium of best practices for its headquarters staff and for mission personnel. Without such a reference, each person involved does what they consider to be the best thing to do, with little coordination and no performance measurement.

The contracting processes to which the program is subjected (being housed within the federal public service) are slow, complex and ill-adapted to the international context. Some have identified missed opportunities (such as shows that could not be attended) that they trace back to cumbersome contractual processes.

Thus far, Edu-Canada has focussed its implementation strategy on physical presence at events. This is considered a costly and not necessarily very effective approach by several key informants. Some argue that promotional efforts should make better use of the Internet in general (the Edu-Canada Web site was criticized as a disorganized potpourri) and of social networking, in particular. Edu-Canada does not use sophisticated technologies. There have also been dissenting voices concluding that individual contacts are important to increase awareness of Canada as a potential study destination.

According to key informants, an important part of the achievements of Edu-Canada is attributable to the quality of the work of the Division. Concerns have been expressed that the program could suffer from lack of succession and sustainability planning.

Delivering the program at lower cost

The general opinion heard from key informants and during field visits was that Edu-Canada is managed as efficiently as it possibly can. Its team is small, committed and productive, and its expenses are kept at a minimum. The only contrary view points were to the utilization of a generally costly promotional strategy through educational shows, as opposed to more virtual (and cheaper) methods such as social networking.

Performance monitoring
Finding #24
Edu-Canada has very limited performance reporting capabilities.

Edu-Canada does not maintain information on the immediate impacts of the events attended, such as number of attendees, number of contacts made, information packages distributed, satisfaction surveys of attendees, etc. Edu-Canada staff have developed a template for missions to report on events but there are no resources at headquarters to aggregate and analyze their contents.

4.2.5 International Experience Canada

Program management issues and their impact on performance

IEC is globally considered a simple, well managed program. Nonetheless, a number of program management issues were raised.

Finding #25
The 2008 IEC transition has been difficult but the program is regaining control over operations.

According to the information gathered through key informant interviews and site visits, the conversion of IEC into a user-fee program in 2007 was quite hectic. Missions were advised of the changes to the program and of their new fee-collecting responsibilities some two months before the 2008 quotas were to open. Whereas CIC essentially delivered the program before, missions were left to themselves to develop their program delivery mechanisms with limited and generic guidelines to ensure consistency globally.(44) This situation caused confusion and duplication of efforts. For example, computer systems were created by missions that could have been developed centrally. Also, the short lead time in getting ready to implement the program meant that coordination could not take place among missions with similar needs (e.g., in Europe). Each mission had to build a new bureaucracy for the management of this single program. According to some, more lead time may have allowed for better use of existing CIC systems for case management.

The situation has improved since with the commissioning of a Privacy Impact assessment, Standard Operating Procedures and service standards, as well as the development of respective policies and measuring architecture. According to mission representatives, more coordination is still needed, for example in the area of financial management, information systems, unpaid internships and the use of personnel during slow times.

Finding #26
Traditionally, IEC has been a low-risk program but upcoming bilateral agreements will require stronger controls.

Traditionally, the countries involved in bilateral agreements for youth mobility presented low risk from an immigration point of view and the program could be managed accordingly. Some countries being considered for new bilateral agreements present more immigration-related risks which will require different program policies and procedures.

According to CIC data, in 2009, IEC represented 62,000 of the planned 150,000 temporary residents. The program has grown rapidly in recent years, from 34,000 in 2007 to 45,000 in 2008, and approximately 53,000 in 2009. Following the request of DFAIT's and CIC's deputy ministers, IEC program management is expected to double the quota by 2011 using the data of 2007 as a baseline. Some of the most active missions interviewed (e.g., United Kingdom, France) indicated that their current quotas are much smaller than the existing demand which reduces the positive effects the program could have.(45) CIC has concerns that this rapid growth will exceed its case management capacity and that it may inadequately redefine the temporary worker program by putting an undue emphasis on diplomatic and cultural considerations. CIC is also concerned by the security risks associated with the addition of new participant countries. Interviewees at CIC indicated that isolated cased of abuses have already been reported, such as Canadian companies or schools promoting IEC as a means to bring workers to Canada, small companies using IEC to avoid HRSDC paperwork and a lengthy process, and IEC being used to bring sex workers into Canada.(46)

The quality of communications between IEC and CIC vary. Some missions enjoy good communications, while in others communications have been strained. At headquarters, CIC perceives that the department is not as well informed and included in the decision making process as it should be considering the importance of its role in program delivery. The evaluation team is not aware of the existence of an IEC communication strategy regarding partners or clients.

The maximum age to be eligible for IEC is 35 (30 in some countries like the United Kingdom). While the motivations of younger applicants are fairly clear and the risk they represent is low, motivations of older applicants are not as transparent and they may bear a higher risk. To our knowledge, the program has not studied how program use and risk varies according to the age of the applicant with a view to eventually review the upper age limit.

The program does not maintain a data base of past participants and has not yet developed an official system to build a community around them. Some informants suggested that the program objectives of contributing to the reputation, positioning and influence of Canada abroad would be well served by such a network of past participants; some missions are currently experimenting in this area.

Some bilateral agreements demand that applicants reside in the country of nationality. This is somewhat paradoxical in today's Europe where it is easy to settle in several countries. In a sense, the most mobile youth are punished by this rule of a program that encourages and values mobility. Some missions have circumvented the problem by requesting a postal address in the country of nationality rather than a proof of residence. A Standard Operating Procedure issued in February 2010 now clarifies the concept of residence.(47)

Delivering the program at lower cost
Finding #27
CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured.

IEC transferred 42% of its revenues to CIC in 2009, however, financial management and volume management seem to remain problematic for CIC. According to CIC, the IEC Program provides annual targets for each mission, based on which CIC plans its resource allocations and acquisitions. IEC often sends fewer applications to CIC than originally planned and expects to get part of the funding back on that basis. However, CIC may well have engaged the totality of the funds in order to face the expected demand and may not be in a position to transfer back resources as expected by IEC.

Where feasible, CIC is moving towards on-line applications for work permits (including C21 IEC Work Permits) and in-Canada processing. It is not yet clear how this will affect IEC. IEC program management is working towards an on-line application system that will tie into CIC's system. IEC and CIC are evaluating the possibility of using an integrated on-line applications system where all case processing could be performed from Canada, which could significantly reduce costs of the program. It is also possible that the complexities of twenty different bilateral agreements (soon to be more) would make online applications system unwieldy.(48) Until a conclusion is reached on this issue, it may be inefficient to make sweeping changes to mission program delivery models.(49)

Finding #28
A centralized management information system and on-line application system could significantly reduce the overall cost of managing IEC.

The issue of a centralized management information system and on-line appliaction system must be disentangled from that of centralized case management operations. Faced with the challenge of managing several thousand applications, the mission in Paris started to develop a system for on-line applications (the only mode for IEC application in France), applications management, financial management and program monitoring. This system has already proved to be successful and was offered to other European missions. Several missions are now using the Paris application with more expected to get integrated in the near future. This means that fewer resources need to be used collectively to develop more efficient and integrated systems in support of IEC administration, such as a centralized management information system. However, program management would remain a responsibility of each mission, since each bilateral agreement is different from the others. The experience of France has shown that having a centralized information system is possible and more efficient than the absence of one.

While the Paris on-line application system has proven effective, it is limited by the inability to accept credit card payments. Credit card payments would be advantageous because the entire application could be completed on-line (with the exception of external documents, such as medical exams which are the responsibility of CIC). Introducing a credit card payment would solve the exchange rate issue.(50) IEC management is working towards this method of payment even if a general DFAIT (or PWGSC) payment policy solution must be approved before it can implement a change.

Performance monitoring

IEC has developed service standards. For example, processing by IEC must be completed within three weeks and processing by CIC must be conducted within five weeks. Some missions have computer systems in place to monitor their performance with regard to measurable service standards; for example, the Paris management information system supplies real time information on the state of cases and identifies cases that approach the limit of the standard processing time.

Some missions indicated that they receive no data on the number of Canadians who have used a youth mobility permit to reside in their mission country; program management stated that such data are delivered in May for the previous calendar year.

IEC currently has no means to monitor the experience of participants, e.g., a system to re-contact them part-way through and at the end of the stay.

The program currently has no centralized information management system to store, analyze and report information on performance. The ARAF is vague regarding reporting commitments, and responsibilities of the program in this regard are not identified.

4.3 Program Governance

Program governance concerns include four dimensions:

  • the contribution of current governance and management structures of PRE to the achievement of the stated objectives for each program;
  • the contribution of communication and feedback strategies to informed management decisions;
  • the establishment of clear roles, responsibilities, and reporting relationships with key partners and stakeholders by each program; and,
  • the effectiveness of the coordination among PRE programs.
Finding #29
PRE has not formally planned strategies and ways to cross-fertilize its programs.

While the evaluation team found anecdotal evidence of cross-fertilization among the PRE programs, no information became available regarding a concerted strategy of the Division to deploy the programs so as to build on the success of each program and leverage the achievements of all. Several mission representatives indicated that PRE programs would be more effective overall if their action was planned and executed in a joint manner.

That said, there is a possible disjoint in that 26 missions (out of 36 which were asked) agreed with the statement that "This mission uses PRE programs in an integrated and planned manner to maximize efficiencies." Missions would appear to integrate the PRE programs to a larger extent than PRE itself. However, missions are also opportunistic in their use of PRE programs: 28 indicated that "This mission uses PRE programs as opportunities arise to maximise benefits." Most missions also stated that they have developed formal communication strategies to implement PRE programs, that roles and responsibilities of the mission, PRE and partners are well defined, and that relationships with PRE are very effective.

To some degree, IEC is a seasonal program in missions with high peaks in demand when the allocations are made available and slow periods later in the year. Some of this variation can be absorbed using contract employees(51) but it is probable that there are times when the IEC resources are underutilized. Some informants have suggested that they should then be allowed to be redirected to other PRE programs and that their knowledge could be used to build synergies. However, the User Fees Act may prevent this.

Finding #30
PRE has not formally planned its strategic communications with Canadians, Canadian institutions and DFAIT missions.

Communications with Canadians and Canadian institutions

With IEC, Canada has been more successful at attracting foreign youth to Canada than at sending Canadian youth to foreign countries despite substantial investments in promotional activities. Yet, this Canadian presence abroad could generate positive impacts on Canadians' understanding of the situation in other countries and this could facilitate, in the long run, the implementation of other programs, such as Edu-Canada. More generally, informants questioned the level of effort invested by PRE in the promotion of its programs with Canadian stakeholders.

Communications with missions

The level of importance attributed to the PRE programs varied considerably from mission to mission. Some indicated that PRE programs were of forefront importance in their strategic thinking (e.g., Understanding Canada and IEC in the United Kingdom and France, Edu-Canada in India and China). In other countries, these programs did not enjoy much attention. PRE did not appear to have developed a strategy to explain to all Heads of Mission the importance and benefits of the IEY programs in helping missions achieve their own objectives.

In fact, mission survey respondents expressed the need for more rigorous communications and training from PRE, detailed information on the programs, regular updates and clear expectations in order to maintain consistent program delivery, particularly at missions that have been identified as a priority. ("We believe that a semi-annual distribution of a summary of all available programs and updates that could be forwarded to our contacts would be beneficial as a reminder of the existing opportunities.") Obviously, the training sessions provided since 2008 have not covered all mission staff, especially the LES officers involved in the delivery of the program.


5.0 Conclusions

Program Logic and Relevance

The IEY programs are a relevant government instrument for furthering public diplomacy and increasing the awareness of the international community about Canada as a preferred study destination. The evaluation found that the demand for, and the interest in these programs among academia, policy makers, researchers and students is growing, and so are the commercial benefits and revenues for Canada.

IEY programs are aligned with and contribute to two of the main Government of Canada priorities: Priority #1: Greater economic Opportunity for Canada, with a focus on growing /emerging markets; and Priority #2: United States and the Americas, as well as to the Advantage Canada Agenda.

The evaluation also found sufficient evidence demonstrating the actual need for these programs; however, the communication of this need required a sophisticated discourse and the acceptance that some logical links between activities and outcomes were difficult to demonstrate empirically. Understanding Canada, the International Scholarships Program, International Experience Canada and, to a lesser degree, the International Education Policy are rooted in the logic of public diplomacy, whereas Edu-Canada is founded on an economic logic.

Demand varies from program to program. The demand for Understanding Canada is pressing in some missions and much less so in others; however, the overall demand for funding from foreign academics has decreased over the past five years for the Program Development Grant and the Faculty Enrichment Program. While scholarships can be conceived as attractors of foreign students or as complements to other action such as education marketing, the missions do not generally articulate a pressing need for them.

The Government of Canada is constitutionally justified to get involved in education matters on the international scene. For example, this includes the negotiation and ratification of international treaties on education. However, the name of the International Education Policy program may announce more to some than the program expects to deliver.

The economic attractiveness of the international education market is unquestioned, as is the need to face strong competition while trying to capture a piece of this market. Young people value the wide opportunities to travel internationally and IEC is a mechanism to offer such opportunities.

Program Management, Coordination and Delivery

The overall operational funding and resourcing at PRE and DFAIT missions is not commensurate with the growing demand for, and interest in the programs, nor with the daunting complexity of the programs and their delivery environment involving OGDs and agencies, provinces and territories, Canadian and foreign partners and stakeholders.

PRE programs are complex and possess limited resources. This may explain why PRE has not formally planned strategies and ways to cross-fertilize its programs, nor has it formally planned strategic communications with Canadians, Canadian institutions and DFAIT missions, which ultimately impairs the overall performance of the Division.

The evaluation has shown that each program taken individually brings benefits. The complexity of the program environment is daunting, with a multitude of stakeholders, Canadian and foreign, affecting program performance in various ways. PRE has not planned its actions with consideration for a possible cross-fertilization if programs are taken as a group rather than as five individual pieces.

Understanding Canada contributes to a substantial intellectual activity related to Canada and may influence the position of foreign countries on Canada; however, there is limited factual demonstration of the latter. A strong case for the performance of the International Scholarships Program and of the International Education Policy, as stand-alone interventions, could not be constructed due to the lack of information on scholarship beneficiaries. Edu-Canada has achieved important milestones; however it still has to document performance in the achievement of outcomes. IEC has succeeded in strengthening its management processes: current performance priorities include the promotion of the program among young Canadians and the implementation of performance measurement.

The International Education and Youth Division as a whole may benefit from more decentralized planning and decision-making approaches, so that its actions make the best of each program and national environment. This would, however, be possible only if missions are sufficiently resourced and willing to take on the challenge and if PRE reshapes itself more as a coach for missions rather than as a front-line program manager.

Efficiency and Economy

There is considerable room for improvement in PRE on the efficiency and economy front.

1.

Understanding Canada: The limited human and financial resources at DFAIT and the lack of designated Academic Relations Officers in some missions impact the effective implementation of the Understanding Canada programs.

While changes introduced to the Understanding Canada program in 2008 aimed at improved quality and relevance of applications through new content priorities and diversification of the available sub-programs, not much effort was placed on the streamlining of the planning, distribution, adjudication and approval processes. This has ultimately resulted in the increased complexity of the program delivery complexity and in a growing confusion among program participants. This is compounded by the very limited resources available at missions to deliver the programs adequately. Its highly centralized governance model may benefit from additional mission input. Understanding Canada has not yet capitalized on new technology and has very limited performance reporting capabilities.

2.

The efficiency and effectiveness of International Scholarships could be improved if they were integrated in a wider PRE strategy.

The program has very limited performance reporting capabilities.

3.

The expectations created by Edu-Canada may exceed its level of funding. Program performance is dependent upon the involvement of many stakeholders over whom the program has little or no authority.

Edu-Canada has been successful in achieving its immediate outcomes, including the approval and promotion of the new "Imagine" Brand; however, modest funding levels and insufficient resources at missions threaten to jeopardise some of the major achievements of the Program. Due to the considerable autonomy of Canadian universities and the ultimate provincial responsibility for education in Canada, the program is experiencing major difficulties in projecting a unified image of the advantages of Canada as a study destination. The individual promotional campaigns led by major Canadian universities and the lack of a designated organization to promote Canada's education system as a whole, threaten the sustainability of the Program achievements to date, and may easily lead to losing a significant momentum and missing opportunities in a highly competitive market.

4.

The 2008 transition has been difficult for International Experience Canada but the Program is regaining control over its operations.

Traditionally, IEC has been a low-risk program, but the growing number of bilateral agreements will require stronger controls. CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured. A centralized management information system and an on-line application system would reduce the overall cost of managing IEC.

Performance Measurement

The management of IEY programs does not reflect the Government of Canada's focus on measuring results and improving performance.

The evaluation found that all PRE programs have little to no performance measurement and reporting capabilities. Following the recommendations of previous evaluations, PRE has developed ARAFs for each program; however, no evidence was presented to the evaluation team on how these ARAFs are being operationalised and used for performance measurement and reporting purposes. No specific information on risk management was available either.

While some of the main delivery organizations (e.g., ICCS, CIBC, AUCC) issue annual reports on the number of awards and granted scholarships, no information is collected to reflect trends in the number of applications on a national and/or regional level. PRE (with the exception of EIC in the missions where it is delivered) has not placed any specific requirements on missions to report on results and trends, and there is no evidence that incoming information from delivering partners is being analyzed and used for strategic planning purposes at PRE.


6.0 Recommendations

The following recommendations stem from the conclusions in the previous chapter and are made with the view that the IEYP will continue to support DFAIT's priorities and strategic advocacy goals of advancing Canada's Knowledge Agenda and promoting Canada as a preferred study destination. Three of the recommendations refer to the overall management of the IEY programs within the PRE Division, while the rest are specific for each program. It is recommended for:

The International Education and Youth Division

1.

That DFAIT and the International Education and Youth Division (PRE) continue to deliver the five programs while developing and implementing a sound Divisional Strategic Plan that capitalises on the strengths of each individual program and identifies opportunities for leveraging of resources and cross-fertilizing programs' results. This Plan should also clarify the contribution of the International Education Policy to the Divisional Strategy and consider re-naming the program to better reflect its specific role.

The five PRE programs target the same population of mobile youth. They also share fundamental objectives even though they contribute to two program activities in the PAA: #2 (Diplomacy and Advocacy) and #3 (International Commerce). The five programs are managed as distinct entities; however, indications are that a more concerted and coordinated approach would produce more results than an isolated one. Also, a clear Strategic Plan for PRE would be an opportunity to define the contribution of the International Education Policy to the achievement of the collective goal.

2.

That PRE develop and implement an integrated Communication Strategy targeted at Canadians, Canadian institutions and foreign stakeholders and supported with respective funding for outreach activities carried out by missions.

The program environment is extremely complex. It includes provinces (constitutionally responsible for education), OGDs (e.g., CIC), Canadian associations (e.g., associations of educational institutions), Canadian educational institutions (e.g., colleges), Canadian companies (e.g., offering internships), Canadian not-for-profits (e.g., involved in student exchanges), Canadian youth (as students, as travellers), DFAIT missions (as partners in delivery and as beneficiaries of results), foreign governments (e.g., as partners in bilateral agreements), foreign associations (e.g., associations of Canadian studies), foreign educational institutions (e.g., universities and colleges in support of scholarships), foreign companies (e.g., for internships), foreign not-for-profits (e.g., handling open work permits), foreign youth (as beneficiaries or targets of most programs), etc. Yet, while PRE managers are aware of the individual challenges raised by the management of communications for their own program, PRE has not developed a corporate communications strategy to handle messages to all of these partners, stakeholders and clients.

3.

That PRE set aside resources to implement a performance monitoring plan and use performance information in making planning and management decisions.

While ARAFs were developed for each program and for PRE in 2009, there is no evidence from most programs that they have geared up to implement on-going performance measurement. In fact, programs have very limited performance information. For example, significant efforts were involved by Understanding Canada in producing basic information on the program for the purposes of this evaluation, such as applications received and grants approved. Performance monitoring and efficiency in administration would be improved by on-line applications and on-line application management.

Understanding Canada

4.

That the delivery of Understanding Canada be better coordinated between PRE, geographic divisions and missions, with adequate resources being granted to missions based on country/regional trends and priorities.

Understanding Canada is a complex program that has become bureaucratic and burdensome on applicants and mission staff (this trend is explained by the application of a stricter due diligence process). Yet, the demand for Understanding Canada is pressing in some missions (but much less so in others) who see the program as an important bridge-builder with influential nationals. To a degree, Understanding Canada suffers from the ambivalence between its traditional perspective as a granting program supporting academic excellence in Canadian studies and its new personality brought about by priority areas introduced in 2008. PRE management see Understanding Canada as a long-term relationship building effort while missions want to use the program in a more medium-term utilitarian way. Refocusing the program on mission as beneficiaries would give it new life.(52)

5.

That a review of the complexity of Understanding Canada, the variety of the sub-programs and the respective application and adjudication procedures be conducted in consultation with missions to identify feasible ways to decrease the administrative burden on missions and improve overall program efficiency and effectiveness.

Some missions have indicated that they would like to contribute to improving Understanding Canada. For a start, the program could target the simplification of bureaucratic processes through effective consultations with missions. A feasible option would be the use of the on-line departmental WIKI to stimulate debate and consolidate best practices across DFAIT/ missions abroad.

International Scholarships Program

6.

That PRE respond to the need for a more effective promotion of the variety of scholarships among Canadian universities and colleges, as well as potential foreign participants, along with identifying the right balance between planned quotas and the capacity of Canadian universities to accept foreign scholars.

While a considerable number of scholarships have been designed to support Canada's engagement with priority countries and regions and respective announcements have been made in these countries (e.g., the CARICOM and the Emerging Leaders of the Americas (ELAP) scholarships), the message does not seem to have reached all Canadian universities. The evaluation found that the lack of relevant information in Canada and at Canadian universities is one of the stumbling blocks in the successful implementation of these scholarship programs. On the other hand, interest in some scholarships (e.g., post-doctoral studies) seem to exceed missions' capacity to review applications (e.g., 70 applications for 2 scholarships) and DFAIT's capacity to ensure funding commensurate with this interest. PRE needs to seek options for a more efficient management of the International Scholarships Program, especially in view of the high delivery costs of this program.

Edu-Canada

7.

That DFAIT, and PREP in particular, continue to advocate for a significant increase in the funding level for Edu-Canada and start exploring the options for establishing an independent organisation to promote education in Canada in order to sustain current achievements and increase Canada's visibility and ultimate benefits in a highly competitive international environment.

The economic attractiveness of the international education market is unquestioned, as is the need to face strong competition while trying to capture a piece of this market. Edu-Canada has achieved important milestones recently but its performance is dependent upon availability of resources, both at DFAIT and missions abroad. Program funding is limited and expectations from the program may exceed the current level of funding.

The involvement of many stakeholders over which the program has limited or no authority place additional challenges on sustaining Edu-Canada's achievements. DFAIT could take a leadership role in exploring the feasibility of establishing a new mechanism to coordinate educational promotion abroad and conveying a single but powerful image of Canada as a preferred study destination. Best practices of Australia, the UK and the US, as well as lessons learned from the CEC Network experience could serve as a starting point. Failure to do so, may result in Canada continuing to miss opportunities in a highly beneficial but also competitive market.

International Experience Canada

8.

That International Experience Canada continue to strengthen its management processes, including policies, management information systems, on-line application system, performance measurement, and risk management.

Transition has been difficult but the program is regaining control over operations. IEC has strengthened its management processes by creating a Privacy Impact Assessment, ARAF, standard operating procedures guide and service standards. Currently, performance priorities should include a reassessment of promotional activities with young Canadians to improve reciprocity, and performance measurement. IEC was traditionally a low-risk program but upcoming bilateral agreements will require stronger controls and possibly additional resources in this area. CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured. Finally, a centralized management information system would reduce the overall cost of managing IEC.


7.0 Management Response and Action Plan

Recommendation 1

That DFAIT and the International Education and Youth Division (PRE) continue to deliver the five programs while developing and implementing a sound Divisional Strategic Plan that capitalises on the strengths of each individual program and identifies opportunities for leveraging of resources and cross-fertilizing programs' results. This Plan should also clarify the contribution of the International Education Policy to the Divisional Strategy and consider re-naming the program to better reflect its specific role.

The five PRE programs target the same population of mobile youth. They also share fundamental objectives even though they contribute to two program activities in the PAA: #2 (Diplomacy and Advocacy) and #3 (International Commerce). The five programs are managed as distinct entities; however, indications are that a more concerted and coordinated approach would produce more results than an isolated one. Also, a clear Strategic Plan for PRE would be an opportunity to define the contribution of the International Education Policy to the achievement of the collective goal.

Associated Findings: 1, 8, 20, 22, 29
Management Response & Action PlanResponsibility CentreTime Frame
As part of the rebalancing and SR-400 exercises, PRE is seeking to place additional positions abroad to strengthen program coordination with posts at the regional level.  
PRE programming has been transformed over the past 5 years to better align with departmental priorities and strategies, for example the Americas Strategy.  
The New Business Model and PRE's transfer to GLD will further reinforce the integration of divisional strategies with strategic planning in the Geo Group.As part of GLD, PRE will contribute to geographic strategies. 
Programming is driven by regional strategies, most recently by the India Strategy. PRE coordinates to maximize program synergies while contributing to national strategies such as Advantage Canada and the Global Commerce Strategy. The division will reinforce its ARAF to better highlight outputs and results related to departmental and government strategies.  
PRE agrees with the recommendation to change the title of the International Education Policy activity. September 2010

 

Recommendation 2

That PRE develop and implement an integrated Communication Strategy targeted at Canadians, Canadian institutions and foreign stakeholders and supported with respective funding for outreach activities carried out by missions.

The program environment is extremely complex. It includes provinces (constitutionally responsible for education), OGDs (e.g., CIC), Canadian associations (e.g., associations of educational institutions), Canadian educational institutions (e.g., colleges), Canadian companies (e.g., offering internships), Canadian not-for-profits (e.g., involved in student exchanges), Canadian youth (as students, as travellers), DFAIT missions (as partners in delivery and as beneficiaries of results), foreign governments (e.g., as partners in bilateral agreements), foreign associations (e.g., associations of Canadian studies), foreign educational institutions (e.g., universities and colleges in support of scholarships), foreign companies (e.g., for internships), foreign not-for-profits (e.g., handling open work permits), foreign youth (as beneficiaries or targets of most programs), etc. Yet, while PRE managers are aware of the individual challenges raised by the management of communications for their own program, PRE has not developed a corporate communications strategy to handle messages to all of these partners, stakeholders and clients.

Associated Findings: 11, 30
Management Response & Action PlanResponsibility CentreTime Frame
The New Business Model and PRE's integration into the Geo Group will allow PRE to benefit from the Geo Group's strategic oversight. The potential synergy afforded by this reorganisation will also be explored in the area of web.Through GLDIntegration into GLD,
June 2010

Recommendation 3

That PRE set aside resources to implement a performance monitoring plan and use performance information in making planning and management decisions.

While ARAFs were developed for each program and for PRE in 2009, there is no evidence from most programs that they have geared up to implement on-going performance measurement. In fact, programs have very limited performance information. For example, significant efforts were involved by Understanding Canada in producing basic information on the program for the purposes of this evaluation, such as applications received and grants approved. Performance monitoring and efficiency in administration would be improved by on-line applications and on-line application management.

Associated Findings: 10, 12, 13, 14, 19, 21, 23, 24, 26
Management Response & Action PlanResponsibility CentreTime Frame
PRE will continue to contribute to the GCS performance management process. Integration into GLD will also allow PRE to benefit from the performance management structure under development there. As indicated above, these together with the drawing on the CEP Business Plan process will serve to reinforce the Divisional ARAF.

Understanding Canada has tried on-line applications between 2001-2002 and 2007-2008 but it became too heavy to handle internally. On-line applications are available only for the part of the program administered through ICCS. PRE is currently working on the implementation of on-line applications for all the units.

With respect to better monitoring, PRE does not have the resources. In recent years, budgets, both operational and for program, have been constantly reduced. In order to better monitor performances as well as grants, and to support program delivery (see recommendation #4), 250-300K per year in Vote 1 would be needed. This will be considered in the context of the New Business Model.

GLD / PREApril 1, 2011

Recommendation 4

That the delivery of Understanding Canada be better coordinated between PRE, geographic divisions and missions, with adequate resources being granted to missions based on country/regional trends and priorities.

Understanding Canada is a complex program that has become bureaucratic and burdensome on applicants and mission staff (this trend is explained by the application of a stricter due diligence process). Yet, the demand for Understanding Canada is pressing in some missions (but much less so in others) who see the program as an important bridge-builder with influential nationals. To a degree, Understanding Canada suffers from the ambivalence between its traditional perspective as a granting program supporting academic excellence in Canadian studies and its new personality brought about by priority areas introduced in 2008. PRE management see Understanding Canada as a long-term relationship building effort while missions want to use the program in a more medium-term utilitarian way. Refocusing the program on mission as beneficiaries would give it new life.

Associated Findings: 2, 10, 15, 16, 17, 18, 30
Management Response & Action PlanResponsibility CentreTime Frame
Integration into GLD will greatly enhance the link between grant approval/administration and program delivery by posts in the context of the country strategy. The Geo Group expects to restore the resource base of posts over time which may allow better program delivery in the context of post priorities.

Due to the lack of resources, PRE can now only consult through e-mails or at the rare occasions of field trips. However, despite this limited capacity, PRE has implemented a system under which all grant requests coming from abroad are transiting through missions as well as provided them with recommendations and report forms so they can provide input on each request. In addition, they can also provide input after final results are communicated in order to either profit of a better synergy between similar activities happening in the region or even recommend additional awards.

PRE will explore the possibility of consulting virtually (Wiki).

PRE / part of GLD with GLBOn going

Recommendation 5

That a review of the complexity of Understanding Canada, the variety of the sub-programs and the respective application and adjudication procedures be conducted in consultation with missions to identify feasible ways to decrease the administrative burden on missions and improve overall program efficiency and effectiveness.

Some missions have indicated that they would like to contribute to improving Understanding Canada. For a start, the program could target the simplification of bureaucratic processes through effective consultations with missions. A feasible option would be the use of the on-line departmental WIKI to stimulate debate and consolidate best practices across DFAIT/ missions abroad.

Associated Findings: 2, 16, 17
Management Response & Action PlanResponsibility CentreTime Frame
Since the renewal of the program in 2008, it is planned to have a second forum to discuss the future of the program once PRE's Ts & Cs are renewed (March 2011). As previously, this review will involve the program's principal stakeholders and Missions. Virtual consultations (Wiki) will also be considered.

Although PRE will aim at a less bureaucratic process, the very reasons that rendered the program complex (necessity to comply with the new Policy on Transfer Payments, the due diligence process, and DFAIT's internal rules) may be an impediment to reaching this goal.

With respect to the variety of components (sub-programs) PRE expects the coming review to demonstrate if some should be reconsidered. Up to now, the existing components are considered to respond to the stakeholders' needs.

PREConsultations with missions: January-April 2011;

Forum with stakeholders: May 2011;

Implementation of new program: April 1st, 2012

Recommendation 6

That PRE respond to the need for a more effective promotion of the variety of scholarships among Canadian universities and colleges, as well as potential foreign participants, along with identifying the right balance between planned quotas and the capacity of Canadian universities to accept foreign scholars.

While a considerable number of scholarships have been designed to support Canada's engagement with priority countries and regions and respective announcements have been made in these countries (e.g., the CARICOM and the Emerging Leaders of the Americas (ELAP) scholarships), the message does not seem to have reached all Canadian universities. The evaluation found that the lack of relevant information in Canada and at Canadian universities is one of the stumbling blocks in the successful implementation of these scholarship programs. On the other hand, interest in some scholarships (e.g., post-doctoral studies) seem to exceed missions' capacity to review applications (e.g., 100 applications for 2 scholarships) and DFAIT's capacity to ensure funding commensurate with this interest. PRE needs to seek options for a more efficient management of the International Scholarships Program, especially in view of the high delivery costs of this program.

Associated Findings: 20
Management Response & Action PlanResponsibility CentreTime Frame
PRE agrees with the need for a more effective promotion of the variety of scholarships among Canadian universities and colleges along with identifying the right balance between planned quotas and the capacity of Canadian universities to accept foreign scholars. Consultations with Canadian Bureau of International Education, Association of Canadian Universities and Colleges and Associations of Canadian Community Colleges will help identify a promotional strategy and its implementation.GLD / PREJuly-September 2010
PRE agrees with the need for a more effective promotion of the variety of scholarships among potential foreign participants. PRE will continue it's program of collaborative missions, which started in 2009. Up to 20 academics from focus countries in the Americas regions are invited on a tour of Canadian institutions to establish institutional linkages, gain appreciation for Canada's education system and act as champions for the scholarships. Communications with the missions will be reviewed and adapted to ensure we meet our promotional aspect. PRE Policy and Planning has already joined hands with Edu-Canada to increase promotional imprint abroad in focus countries. September 2010
PRE programming has grown over the past 5 years. Over 700 scholarships have been awarded to non-Canadian students studying or conducting research in Canada and the creation of new opportunities for Canadians to study abroad.  
In terms of more effective promotion of the variety of Scholarships: There has been 100% uptake of all scholarship programs with demand far exceeding supply. In some priority regions, such as Brazil and Mexico, demand is more than double the supply. PRE will continue to collaborate with missions to attain a more ideal distribution in our priority partners in the context of this FY's additional ELAP resources and possible additional funding.  
Specific promotion among Canadian universities and colleges: Of the various scholarship programs, 500 awards are dependant on the participation of Canadian universities which apply on behalf of foreign scholars. While promotion to new institutions and staff is always a consideration, current uptake of scholarship opportunities by Canadian universities is also over 100%. October 2010
As demand for all scholarship programs currently exceeds supply, distribution of awards is based on DFAIT policy and priority regions. We will continue to readjust award distribution as per government priorities and will do a better job of informing stakeholders of policy based selection.  
PRE is currently undertaking a further study of federally funded scholarship/award offerings, as well as among key stakeholders. The results of the study will assist PRE in making management decisions regarding programming and supply and demand.  
PRE's integration in GLD will also strengthen linkages to Geo Group's regional and country -focus strategies and domestic outreach. The establishment of the Canada Bureau and additional resources in ROs will further facilitate outreach and program communications.  

Recommendation 7

That DFAIT, and PREP in particular, continue to advocate for a significant increase in the funding level for Edu-Canada and start exploring the options for establishing an independent organisation to promote education in Canada in order to sustain current achievements and increase Canada's visibility and ultimate benefits in a highly competitive international environment.

MINT's release of the significant economic impact of international studies together with the release of the best practices study has reinforced the case for adequate resources as the existing GCS. Additional resources in the field will also be sought as part of the SR 400 exercise

Associated Findings: 22, 23, 24
Management Response & Action PlanResponsibility CentreTime Frame
PRE agrees with this recommendation. Expectations created among stakeholders and clients by our successes, far exceed our present level of funding. Additional resources were sought in the last budgetary cycle backed by a well-developed business case and with strong support of our stakeholder community.

PRE recognises the need for independent organizations with which to partner related to service delivery more appropriately out-sourced. In summer 2009, DFAIT commissioned a report from the Illuminate Consulting Group (ICG) to study best practices in Education marketing for Australia, New Zealand, France, Germany, the UK and the US, as well as lessons learned from the CEC Network experience, and present DFAIT recommendations as to what Canada should do.

The ICG report was tabled at the National Education Marketing Roundtable in November 2009 and is available on our website. It recommended the establishment of an Independent Education Marketing Agency, the increased use of Web 2.0 tools in attracting International students to Canada and better streamlining and marketing of DFAIT scholarships for international students. This report formed the basis for the resource ask in the last budgetary circle.

Senior Management in the context of the possible renewal of the sunsetting2011-12 budget cycle

Recommendation 8

That International Experience Canada continue to strengthen its management processes, including policies, management information systems, on-line application system, performance measurement, and risk management.

The 2008 IEC transition has been difficult but the program is regaining control over operations. IEC has strengthened its management processes by creating a Privacy Impact Assessment, ARAF, standard operating procedures guide and service standards. Currently, performance priorities should include a reassessment of promotional activities with young Canadians to improve reciprocity, and performance measurement. IEC was traditionally a low-risk program but upcoming bilateral agreements will require stronger controls and possibly additional resources in this area. CIC is IEC's main partner and the relationship between the two organizations needs to be carefully nurtured. Finally, a centralized management information system would reduce the overall cost of managing IEC.

Associated Findings: 6, 7, 15, 25, 26, 27, 28, 29, 30
Management Response & Action PlanResponsibility CentreTime Frame
IEC agrees with this recommendation. Management processes have been strengthened with the creation of the ARAF, PIA, a standard operating procedures guide and service standards.IEC in conjunction with DFAIT Senior management, with the collaboration of our stakeholders and clients 
A reassessment of promotional activities with young Canadians and performance measurement are two performance priorities in 2010. IEC will develop strategic approach to promotional events geared on greater impact on Canadian participants and a more efficient use of resources. The promotional material geared at Canadian audiences is now being redesigned and we are exploring a strategic approach to promotional event focused on communities while being able to measure the exact impact of promotional efforts. We will reassess promotional activities in 2010 and look to implement a new strategy for the next promotional campaign.
IEC agrees that upcoming bilateral agreements will require the entire program to have stronger controls. Following the completed Privacy Impact Assessment, a Threat and Risk Assessment have been conducted June 7 and 8 of 2010 with PWGSC to clearly identify risks and mitigation strategies associated with operational aspects of the IEC. A Threat and Risk Assessment will be conducted June 7 and 8 of 2010
The relation with CIC will continue to be carefully nurtured following the MoU signed between the two departments. Various meetings and constant communications with CIC will ensure that this relationship remains productive and mutually beneficial.  
IEC agrees that a centralized management information system is needed. A consultant has been hired June 3rd 2010 for a period of one year to manage the design, development and implementation of an interface design for new online tools including the implementation of a system to manage IEC programs delivered at missions abroad and at DFAIT consisting of online application forms with an online payment system to be integrated on a global scope. A consultant was hired June 3, 2010 to develop a centralized management system in the next year

1. According to Roslyn Kunin & Associates, Inc., Economic Impact of International Education in Canada, presented to the Department of Foreign Affairs and International Trade, July 2009.

2. According to "Youth Travel Matters: Understanding the global phenomenon of Youth travel" Report published in 2007 by the World Tourism Organization, United Nations and Canadian Tourism Commission Statistic Reports.

3. According to a 2009 survey conducted by ICCS. No final report on this study was available when this evaluation report was prepared. Estimates of economic impacts are based on self-reports and include direct expenditures in Canada, payments to Canadians, and orders placed for items such as like books and other pedagogical materials.

4. These outcomes have been identified and formulated in the 2009 ARAF for the program.

5. CBIE and DFAIT do not maintain information on awards regarding competitions for Canadians; only data on applications are available.

6. These outcomes have been identified and formulated in the 2009 ARAF for the program.

7. These outcomes have been identified and formulated in the 2009 ARAF for the program.

8. These outcomes have been identified and formulated in the 2009 ARAF for the program.

9. These outcomes have been identified and formulated in the 2009 ARAF for the program.

10. Among others: Evan H. Potter, Branding Canada: Projecting Canada's Soft Power through Public Diplomacy, Montreal/Kingston, McGill-Queen's University Press, 2009; Philip Seib, Toward a New Public Diplomacy: Redirecting U.S. Foreign Policy, New York, Palgrave MacMillan, 2009; Jan Melissen, The New Public Diplomacy: Soft Power In International Relations, New York, Palgrave MacMillan, 2005; Taylor, Maureen, "Toward a Relational Theory of Public Diplomacy," Paper presented at the annual meeting of the NCA 94th Annual Convention, TBA, San Diego, CA, November 20, 2008 http://www.allacademic.com/meta/p256423_index.html; Lee, Hyung Min, "Public Diplomacy as International Public Relations: Speculation on National Determinants of World Governments' Web Public Diplomacy Interactivity," Paper presented at the annual meeting of the International Communication Association, TBA, San Francisco, CA, May 23, 2007 http://www.allacademic.com/meta/p170558_index.html; United States Government Accountability Office, State Department Efforts to Engage Muslim Audiences Lack Certain Communication Elements and Face Significant Challenges, Document GAO-06-535, May 2006; United States Government Accountability Office, State Department Expands Efforts but Faces Significant Challenges, Document GAO-03-951, September 2003; Stacy Michelle Glassgold, Public Diplomacy: The Evolution of Literature, January 12, 2004;

11. Jan Melissen, op. cit., p. xix.

12. Jan Melissen, op. cit., p. 11.

13. William A. Rugh, "The Case of Soft Power," in Philip Seib, op. cit., p. 12.

14. Evan H. Potter, op. cit., pages 44-45.

15. Evan H. Potter, op. cit., p. 70.

16. Shawn Powers and Ahmed El Gody, "Lessons from Al Hurra Television," in Philip Seib, op.cit., p. 63.

17. Philip Seib, op.cit., p. 244.

18. William A. Rugh, loc .cit., p. 13.

19. William A. Rugh, loc. cit., p. 13.

20. William A. Rugh, loc. cit., p. 19.

21. Evan H. Potter, op. cit., p. 221.

22. This is a conclusion based on anecdotal evidence since CBIE and DFAIT do not maintain statistics on applications for incoming scholarships or on awards for outgoing scholarships.

23. Roslyn Kunin & Associates, Inc., Economic Impact of International Education in Canada presented to Department of Foreign Affairs and International Trade, July 2009, page 7.

24. Conference Board of Canada, Opportunity Begins at Home. Enhancing Canadian Commercial Services Exports, 2006, p. 11.

25. Making Canada a World Leader for Today and Future Generations, Advantage Canada, November 2006. http://www.fin.gc.ca/ec2006/plan/ecpam-eng.asp

26. According to a 2009 survey conducted by ICCS. No final report on this study was available when this evaluation report was prepared.

27. Department of Foreign Affairs and International Trade, Report on Plans and Priorities 2009-2010, pages 3 and 4.

28. This assertion is not a legal opinion.

29. The Illuminate Consulting Group, Best Practices on Managing the Delivery of Canadian Education Marketing, prepared for Foreign Affairs And International Trade Canada, September 2009.

30. Report prepared at PRE for Understanding Canada on the basis of 2008-2009 data. The classification of awards was not verified.

31. Unpublished and unfinished report prepared by ICCS on behalf of DFAIT. Elements of this unaudited study were released by the deputy director of Understanding Canada.

32. Based on an analysis of server logs performed by PublicInsite for DFAIT, delivered in August 2009. The average of 8,000 visits per month excludes the month of March 2009 where a punctual promotional campaign took place. If that month is included, the average rises to 17,000.

33. Based on monthly traffic reports produced by PRE.

34. The introduction of the program Terms and Conditions read as follows: "Academic Relations is an integral part of DFAIT's strategy to advance Canada's interests worldwide. It builds influence through dialogue, profile-raising, cooperative research, and the creation of networks of contacts and understanding between people in partner countries, especially in those that represent a priority for Canadian foreign policy objectives."

35. Such a criterion is used by the Canadian Health Services Research Foundation in its review of grant applications.

36. Unpublished and unfinished report prepared by ICCS on behalf of DFAIT. Elements of this unaudited study were released by the deputy director of Understanding Canada.

37. A brief Internet search uncovered the following titles: British Journal of Canadian Studies, Études Canadiennes (Revue française d'études canadiennes), Canadian Studies in Europe, Central European Journal of Canadian Studies, Literatura: Canadian Studies, Nordic Association of Canadian Studies Text Series, The American Review of Canadian Studies, Australian-Canadian Studies.

38. Interview with ICCS staff, December 10, 2009.

39. See, for example, page 66 of the report entitled Evaluation of the Networks of Centres of Excellence Program prepared by Circum Network Inc. and R.A. Malatest and Associates Ltd. for the Interagency Evaluation Steering Committee (CIHR, NSERC, SSHRC), dated October 19, 2007. http://www.nce-rce.gc.ca/pubs/reports/2007/evaluation/NCEEvaluationReport2007-eng.pdf

40. International Council for Canadian Studies, Annual Report 2007-2008, http://www.iccs-ciec.ca/annrep_en.asp?shownav=3

41. Department of Foreign Affairs Canada, Evaluation of the International Academic Relations Program, July 2005, page 56.

42. The Illuminate Consulting Group, Best Practices on Managing the Delivery of Canadian Education Marketing, prepared for Foreign Affairs and International Trade Canada, September 2009, page 14.

43. More on this in Henry Mintzberg, The structuring of organizations: a synthesis of the research, Englewood Cliffs, Prentice-Hall, 1979, chapter 10.

44. Some missions questioned whether DFAIT should get involved in actual program delivery because it has limited such experience. IEC program management insisted that the program was a diplomatic program, not an immigration program, and had to be delivered as such.

45. Note that, in the UK where the unwritten bilateral agreement demands strict reciprocity of beneficiaries, increasing the quotas in one direction necessitates increasing it in the other direction. In this case, increasing the number of foreign participants may require promotional activities in Canada to increase the volume of Canadian youth attracted to the UK.

46. Since 1986, three problematic cases (sex worker or refugee) have been reported to IEC management (all in 2009 and from one high risk country).

47. See IEC Standard Operating Procedures, Chapter II Section 2.6.3, PREE Archives, Ottawa

48. In the management of its programs, CIC applies the same set of rules world-wide. IEC must apply the terms of agreements that are negotiated one by one with each foreign country.

49. For example, the United Kingdom system, which is currently based on service providers, appears stable and acceptable nationally; it may not be efficient or effective to change this system and to introduce a no-frills visa processing-only supplier now, if substantial changes to the CIC case processing delivery model are forthcoming in the short- or medium-term. The no-frills visa processing-only supplier project aims at introducing a more efficient solution. In the United Kingdom and Ireland, IEC permit applications are traditionally handled by not-for-profit intermediaries (BUNAC and USIT respectively). These organizations offer general accompaniment, including program application and pre-processing. They charge a fee for their service, including the IEC program fees which are integrally transferred to the mission. In essence, the application processing is performed without charge to IEC. In comparison, in the no-frills visa processing-only supplier scenario, IEC would pay the supplier a fee for the processing of the application, leaving the program with reduced net revenues for the same service.

50. The fee for the program is established and expressed in Canadian dollars but applicants pay in their own currency. With fluctuating exchange rates, the exact amount due upon application varies regularly. Imprecise payments due to fluctuating exchange rates are a burden on program management. Note that there is no consistent policy on how to deal with imprecise payments.

51. Note: one mission indicated that hiring on an emergency basis was cumbersome and that people responsible for finances, PREE and the embassy were not aligned on this issue.

52. Program management considers that the policy on transfer payments makes blocks this avenue and that previous attempts have proven unsuccessful.

Office of the Inspector General


* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Footer

Date Modified:
2013-01-02