Foreign Affairs, Trade and Development Canada
Symbol of the Government of Canada

Foreign Affairs, Trade and Development Canada

international.gc.ca

Evaluation of the International Science and Technology Partnership Program (ISTPP)

(June 2008)

(PDF Version, 449 KB) *


Acronyms and Abbreviations

ARAF
Accountability, Risk and Audit Framework
BRIC
Brazil, Russia, India and China
CIIRDF
Canada-Israel Research and Development Foundation
DFAIT
Foreign Affairs and International Trade Canada
EOI
Expression of Interest
GITA
Global Innovation Technology Alliance (India)
HQP
Highly qualified personnel
IP
Intellectual Property
IRAP
Industrial Research Assistance Program
ISTPP
International Science and Technology Partnership Program
MOST
Ministry of Science and Technology (China)
MOU
Memorandum of Understanding
MRRS
Management, Resources and Results Structure
NRC
National Research Council
NSERC
Natural Sciences and Engineering Research Council of Canada
PAA
Program Activity Architecture
R D
Research and Development
SDTC
Sustainable Development Technology Canada
SME
Small and medium-sized enterprises
S T
Science and Technology
TBS
Treasury Board Secretariat
WED
Western Economic Diversification Canada

Exectutive Summary

Introduction

This Executive Summary presents an overview of the conclusions and recommendations of the Formative Evaluation of the International Science and Technology Partnership Program (ISTPP). The evaluation was undertaken between January and April 2008.

Program Description

Canada recently1 signed bilateral science and technology (S T) agreements with China and India to encourage, develop and facilitate cooperative activities in S T.

The purpose of the S T agreements is to:

  • Promote collaborative research between Canadian and foreign scientists and technologists in a variety of areas, such as the use of cleaner and more efficient forms of energy, biotechnology, information and communication technology;
  • Foster and support bilateral research projects between Canada and these selected partner countries that have been proposed by companies, universities/colleges and other private sector research and development institutes; and
  • Stimulate bilateral S T networking and matchmaking activities to further new partnerships and accelerate the commercialization of research and development.2

The International Science and Technology Partnership Program (ISTPP) was established in 2005 to support these bilateral S T Agreements and to provide funding to four selected countries (China, India, Brazil and Israel). The ISTPP was allocated $20 million (over 5 years).

Evaluation Purpose

The purpose of the evaluation was to contribute to the decision-making process for the ISTPP regarding the future directions and activities of the program and to respond to the requirement for an evaluation as articulated by official program documentation. The evaluation focuses only on China and India, since there is no contribution agreement signed with Brazil and a summative evaluation for Israel was conducted in 2004. The overall objectives of the evaluation of the ISTPP were to:

  • Assess the relevance of the program;
  • Review the appropriateness of the design and delivery of the ISTPP; and
  • Assess the extent to which the ISTPP is in line to achieve its outcomes.

Evaluation Methods

The evaluation study was conducted between January and April 2008. The evaluation conclusions and recommendations outlined below were derived from findings gathered from three main lines of evidence for the evaluation: interviews, document review and file review.

Conclusions and Recommendations

Continued Relevance of the ISTPP

The evaluation concluded that the objectives of ISTPP continue to be relevant but there are varying perceptions of the focus of the program.

The evaluation found that the program objectives are well aligned with DFAIT's departmental objectives and that they are considered appropriate by the stakeholders. However, there are varying perceptions of the primary focus of the program (e.g. commercialization, bilateral linkages, or strengthening R D).

During the evaluation, some concerns were raised as to whether the ISTPP Accountability, Risk and Audit Framework (ARAF) prepared by DFAIT adequately reflects the program as it currently exists.

Recommendation 1:

There is a need to better communicate the main focus of the program to all stakeholders to ensure that they have a good understanding of the program. This can be done by creating a hierarchy of objectives, which describes the main goal of the program as well as the associated objectives.

Recommendation 2:

At the time of the next evaluation, a review of the ISTPP ARAF structure should be undertaken.

The evaluation concluded that China and India are relevant partner countries.

Most of the key informants agreed that China and India are relevant partner countries for S T collaboration because they are fast-growth emerging economies with appropriate technological capacity. A few concerns were raised, particularly with respect to weak institutional linkages between government, academia and the private sector in India and Intellectual Property (IP) protection in China.

The value-added of the ISTPP is that it is a mechanism to foster international S T relationships and leverage support, and it supports applicants through a rigorous vetting of prospective projects.

Because ISTPP is delivered by an external delivery agent, at arm's length from government, potential political pressures are reduced. ISTPCanada is perceived to provide faster and less bureaucratic program delivery and it can facilitate leverage through contributions by the provinces or other entities.

As projects have not yet been funded it was not possible to directly assess whether the program resulted in successful international S T collaborations. However, the evaluation found evidence that the ISTPP provides a valuable mechanism in support of international collaboration.

At this stage it is not possible to assess whether projects would move forward without ISTPP support. However, ISTPP has a number of strengths, including its ability to foster and support international S T relationships and the support it provides to applicants through a rigorous vetting process for prospective projects.

Recommendation 3:

ISTPP should assess the nature of the incrementality of the program to determine: a) whether the program is fostering international S T partnerships and collaborative research in the supported sectors (with an emphasis on industry-industry and university-university partnerships); and b) the extent to which the program is contributing to increased R D in a commercial context.

Outputs of the ISTPP

At the time of the evaluation, ISTPP was on track to achieving its main program outputs.

ISTPP is on track to achieve or has already achieved the successful implementation of program outputs under the collaborative R D Investment and the R D Matchmaking program components.

Design and delivery of the ISTPP

The intent of the collaboration between ISTPCanada and partnering countries is clear, and the bilateral structure of the program facilitates the achievement of program objectives.

Based on key informant interviews and the documentation review, our analysis indicated that the intent of the collaboration between ISTPCanada and partnering countries is clear. Responsibilities for ISTPP activities are clearly defined in the documentation, and, overall, stakeholders and program applicants are satisfied with the program design. The bilateral structure of the program facilitates the achievement of program objectives because the linkage to a counterpart agency in the partner countries helps to facilitate the linkages between counterpart organizations and private sector firms.

The program is well structured and defined

The overall structure of the program is well designed and well defined in the program documentation. DFAIT has put in place a robust governance structure with the establishment of a Secretariat and a Steering Committee, and the selection of ISTPCanada (overseen by a Board) as the delivery organization. However, the role of expert coordinators in the program needs to be clarified.

Recommendation 4:

The roles and responsibilities of the export coordinators is an area where clarification is needed. It should be determined whether their role in providing support to program proponents at the proposal stage would conflict with their role as evaluators of these proposals.

Clarify the roles of expert coordinators.

The evaluation found that although promotional activities have been undertaken, and the program is targeted towards a relatively specific community, the program could benefit from a communications plan to ensure that all potential applicants are aware of the program.

Many promotional activities are currently undertaken by DFAIT and ISTPCanada, such as trade delegations for targeted technology sectors in partner countries, promotion of the program through IRAP, NSERC, and Industry Canada, and through the ISTPCanada website. However, a communication plan is required to ensure equity of access, particularly for SMEs, federal communities and academic organizations.

Recommendation 5:

Develop a communication plan to promote the program more broadly to the Canadian S T community, targeting SMEs, federal communities and academic organizations.3

It is too early to fully assess the adequacy of the selection process at the time of this evaluation (as project assessments are currently underway), however the evaluation found that the process seems to be adequate thus far.

The evaluation found based on comments from stakeholders and potential recipients interviewed, that the selection process was satisfactory in terms of transparency and fairness.

Cost-effectiveness of the ISTPP

The study found that ISTPCanada is a cost effective organization in spite of the relatively small size of the contribution budget. This is because of the fact that the organization is able to leverage provincial resources to augment its contribution budget and to help offset its administrative costs. As well, ISTPCanada receives assistance in the evaluation of projects from its partner federal agencies and departments, at virtually no cost. Furthermore, several project proponents will likely leverage the ISTPP support for their projects with other sources of funding.

The evaluation team observed that the current structure of the delivery organization can be sustained and can accommodate a "scaling up" in overall project funding. Under this scenario the costs of delivering the program should not increase significantly and economies of scale should be achievable. However, monitoring and financial management will be a challenge if the program is enlarged and Precarn Inc. would need to increase its capacity. If the program "scales up," formal agreements with other federal departments may be required, to ensure their support for project assessments and in terms of support for ISTPP activities.

The evaluation assessed four different models for delivery of the program: program delivery though government (DFAIT), delivery through a not-for-profit organization or foundation (such as ISTPCanada), delivery through IRAP, and contracting out to a private-sector consulting organization. It was determined that the current delivery model is the most cost-effective, mainly because ISTPCanada is able to leverage support from other sources and has successfully engaged the majority of project assessment reviewers at virtually no charge.

Furthermore, several project proponents expect to lever support for their projects through Canadian governmental programs and resources and private investment.

Progress and/or success of the ISTPP

The overall impact of the program is constrained by the limited size of the budget. However, the program will likely contribute to the competitive position of Canadian R D organizations and the attraction and retention of highly qualified personnel.

The competitive position of Canadian R D organizations supported by the program is likely to improve as a result of participation in the program. Furthermore, evidence already exists of an impact on the attraction and retention of highly qualified personnel occurring through partnership development and the proposal preparation process.

Top of Page


1.0 Introduction

This report presents the findings of the evaluation of the International Science and Technology Partnership Program (ISTPP).4 The evaluation was undertaken between January and April 2008. The evaluation was overseen by an Evaluation Advisory Committee comprised of representatives from ISTPCanada, DFAIT, Industry Canada, Natural Resources Canada, and NSERC.

The report is organized as follows:

  • Section 1 presents a description of the program and the overall context and purpose of the evaluation;
  • Section 2 presents the evaluation issues and questions;
  • Section 3 presents the methodology for the evaluation, including the limitations;
  • Section 4 presents the key findings of the evaluation;
  • Section 5 presents the overall conclusions and recommendations.

1.1 Program Description

1.1.1 Context and objectives

Canada recently5 signed bilateral science and technology (S T) agreements with China and India to encourage, develop and facilitate cooperative activities in science and technology for peaceful purposes in fields of common interest and on the basis of equality and mutual benefit.6

The International Science and Technology Partnership Program (ISTPP) was established in 2005 to support bilateral S T Agreements and to provide funding to four selected countries (China, India, Brazil and Israel).7 The overall responsibility and accountability for the ISTPP rests with the Minister of International Trade. A Steering Committee made up of senior-level government representatives jointly appointed and co-chaired by Foreign Affairs and International Trade Canada (DFAIT) and Industry Canada, was established to provide direction to and oversight for the program delivery organizations, and to advise the Minister on changes and additions to program priorities.

The objectives of the ISTPP, as described in the program documents, are to:

  • Encourage domestic competitiveness through the transfer of technology and knowledge resulting from international S T partnerships;
  • Foster international S T partnerships and collaborative research in all sectors, with an emphasis on industry-industry partnerships and university-industry partnerships;
  • Accelerate the commercialization of research and development that would benefit Canada through international partnerships with a focus on small and medium-sized enterprises;
  • Access international technologies for Canadian enterprises;
  • Promote Canadian research and development (R D) capacity and Canada as a destination for foreign technology-based investments;
  • Encourage the mobility of researchers and promote Canada as a career destination for foreign researchers and highly qualified personnel; and
  • Strengthen overall bilateral S T relations.

The program logic model describes how the program activities and outputs lead to the immediate, intermediate and long-term outcomes, as detailed in the ISTPP Accountability, Risk and Audit Framework.

1.1.2 Delivery Organization for the ISTPP

ISTPCanada, a non-governmental delivery organization formed through a partnership with Canada-Israel Industrial Research and Development Foundation (CIIRDF) and Precarn Incorporated for the purpose of delivering the ISTPP, was selected by the Government of Canada through a tendering process to provide funding and other services in Canada to program proponents.8 The ISTPCanada Board of Directors was formed to oversee the delivery of the program. The Board is formed of sub-committees that are responsible for the delivery of each bilateral program (i.e. one for Canada-Israel, one for Canada-India, etc.). The Board of Directors is responsible for all major funding decisions for the bilateral partnering activities. The Program Office manages the Program activities for the delivery organization. ISTPCanada subcontracts to Precarn Incorporated, a range of corporate services, the most important of which are those involving project and financial management.

1.1.3 Program Activities and Expected Outcomes

The ISTPP has two main activity components that are supported through the ISTPP delivery organization (ISTPCanada): R D Matchmaking and Collaborative R D Investment.

R D Matchmaking:

Provides support for participation in bilateral S T networking activities and organizes and delivers a number of matchmaking events designed to bring together Canadian and partner country companies, universities/colleges and other research and development institutes to explore partnership opportunities. The partnering activities supported may include, but are not limited to, technical and scientific seminars, conferences, workshops and other events designed to foster partnerships with the selected countries.

Collaborative R D Investment:

Provides support for bilateral research projects between Canadian and partner country companies, universities/colleges and other research and development institutes by funding joint research projects and activities.

In general, supporting R D collaboration through matchmaking and investing in bilateral projects leads to the potential development of commercial technology, which provides business opportunities for Canadian organizations and hence economic benefits (success in the global marketplace).

Program documentation describes the following expected outcomes of the ISTPP:

The program expects the immediate outcomes of R D Matchmaking to be:

  • Increased awareness among researchers of program's potential benefits;
  • Increased networking among researchers globally;
  • Increased numbers of joint programs with partners; and
  • More R D projects in identified targeted areas.

The intermediate expected outcomes of R D Matchmaking are:

  • Better collaboration between R D organizations;
  • More technology consortia involving international partners; and
  • Increased investment in targeted technology areas.

The Program expects the immediate outcomes of R D investment to be:

  • Increased investment in R D;
  • Increased revenues/royalties and/or reduced costs through technologies, products and services (improved commercialization);
  • More highly qualified personnel (HQP) involved in R D projects; and
  • Access to the world's best facilities equipment, talent and knowledge.

The intermediate expected outcomes of R D investment are:

  • More competitive Canadian R D organizations at the international level;
  • Improved economic position for R D organizations (wealth creation); and
  • More permanent HQP created and maintained.

The Program long-term results are to enable Canadian R D organizations to succeed in the global marketplace by enhancing the quality and speed of Canadian research activities and the potential for timely commercialization.

1.1.4 Governance Structure and Accountabilities

The overall responsibility and accountability for the ISTPP rests with the Minister of International Trade.9 The Minister of International Trade is responsible for the program, including its development, implementation and administration. The Minister of Industry, while not responsible for the program, is consulted and plays a key role in the development and implementation of the program. Responsibility for these functions at an operational level has been delegated to the ISTPP Steering Committee and the ISTPP Secretariat. The senior-level Steering Committee is a Government of Canada senior-level committee, which is jointly appointed and co-chaired by the Department of Foreign Affairs and International Trade (International Trade) and Industry Canada and supported by a secretariat.

The responsibilities of the Steering Committee include:

  • Providing advice on the allocation of ISTPP resources to the identified program delivery organizations, initially for China, India, Brazil and Israel;
  • With the approval of Cabinet and based on Government priorities, modifying country priorities, as necessary, as the program evolves or is enhanced;
  • Reviewing the annual activity conducted by each program delivery organization;
  • Reviewing the evaluation of the chosen delivery mechanisms which will be conducted on a regular basis (at least once prior to the consideration of renewal of funding);
  • Providing advice to Ministers on changes or additions to program priorities; and
  • Providing an annual progress report to the Minister of International Trade.

The responsibilities of the Secretariat, which is made up of staff from the Science and Technology Division of the Department of Foreign Affairs and International Trade (international Trade), include but are not limited to:

  • Providing support to the Steering Committee;
  • Assisting in the negotiation of the bilateral Science and Technology Agreements with the various countries;
  • Assisting in the negotiations of Implementing Arrangements outlining how the program is delivered under each bilateral Science and Technology Agreement;
  • Negotiating and managing the financial agreements between Canada and the delivery organization;
  • Drafting Treasury Board submissions and any follow-up action required;
  • Collecting reports and audits from the delivery organizations; and
  • Supporting program evaluations and audits as requested by the Office of the Inspector General.

1.1.5 Resources

The funding profile of the Program, as detailed in the ISTPP Accountability, Risk and Audit Framework (ARAF) is displayed below. The Tables below represent the Vote 1 expenditures (covering DFAIT operating costs (Table 1), and Vote 5 expenditures covering Grants and Contributions (Table 2). As well we have included reprofiling of the Vote 5 expenditures due to delays in startup (Table 3).

Table 1: Operating Budget for Foreign Affairs ISTPP Secretariat (Vote 1)
 2005/20062006/20072007/20082008/20092009/2010Total
Salaries$85,714$129,000$129,000$129,000$129,000$601,714
EBP$17,143$26,000$26,000$26,000$26,000$121,143
Other Operating*$126,000$118,000$148,000$68,000$238,000$698,000
Vote 1 Totals$228,857$273,000$303,000$223,000$393,000$1,420,857

*Note: Other operating includes the budget for evaluation activities and the activities conducted by the ISTPP secretariat.

Table 2: Contribution Budget (Vote 5)
Countries2005/20062006/20072007/20082008/20092009/2010Total
China* $500,000$1,500,00$1,500,00$1,750,00$5,250,00
India**$500,000$1,500,00$1,500,00$1,500,00$1,750,00$6,250,00
Israel$1,000,00$1,000,00$1,000,00$1,000,00$1,000,00$5,000,00
Brazil   $750,000$750,000$1,500,00
Vote 5 Totals$1,500,000$3,000,000$4,000,000$4,750,000$5,250,000$18,500,000

*Note: An amendment to the Contribution Agreement with ISTPCanada was made in 2007 to reprofile the $1,750,000 allocated to China in 2009-2010.  An amount of $1,000,000 would be moved to fiscal year 2006-2007.

*Note: An amendment to the Contribution Agreement with ISTPCanada was made in 2008 to reprofile the $750,000 allocated to China in 2009-2010.  An amount of $435,000 would be moved to fiscal years 2007-2008.

**Note: A request was made to TBS to re-profile the $500,000 allocated to India in 2005-2006. An amount of $ 250,000 would be moved to fiscal years 2006/2007 and 2007/2008.

Table 3: Departmental Total for Vote 1 and Vote 5 before re-profiling
 2005/20062006/20072007/20082008/20092009/2010Total
Departmental total$1,728,857$3,273,000$4,303,000$4,973,000$5,643,000$19,920,857
Plus Accommodation$11,143$17,000$17,000$17,000$17,000$79,143
Grand Total$1,740,000$3,290,000$4,320,000$4,990,000$5,660,000$20,000,000

As the selection of ISTP Canada was delayed until fiscal year 2006/07, DFAIT re-profiled the funding to meet the cash flow needs of the program, as follows:

Table 4: Reprofiled Contributions (Vote 5)
Countries2006/20072007/20082008/20092009/2010
China$1,750,000$1,750,000$1,500,000$1,750,000
India$1,500,000$1,935,000$1,500,000$315,000

Top of Page


2.0 Purpose of the evaluation

The purpose of the evaluation was to contribute to the decision-making process for the ISTPP regarding the future directions and activities of the program and to respond to the requirement for an evaluation as articulated in official program documentation.

2.1 Evaluation Objectives and Scope

The overall objectives of the evaluation of the ISTPP were to:

  • Assess the relevance of the program;
  • Review the appropriateness of the design and delivery of the ISTPP; and
  • Assess the extent to which the ISTPP is in line to achieving its outcomes.

2.2 Evaluation Issues and Questions

This section describes the main evaluation issues that were examined for the evaluation. Evaluation issues are the broad areas examined within an evaluation, while evaluation questions are the more specific questions that need to be answered to be able to address each evaluation issue.

The evaluation issues covered are described below:

  • Relevance: Are the objectives reasonable and adequately defined? Are the objectives of ISTPP relevant to Canadian priorities and partner countries' needs? Are the selected partner countries relevant? What is the likelihood that proposed projects would be undertaken without ISTPP support? What are ISTPP's strengths and weakness? How does ISTPP add value to the interaction currently taking place between Canada and partner countries? Etc.
  • Outputs: What key outputs have there been from ISTPP?
  • Design/Delivery: Is it clear what the intent of the collaboration between ISTPP and partnering countries is? Is the bilateral structure of the program appropriate? Are ISTPP's activities appropriately designed to achieve its objectives? What is being done to promote ISTPP (beyond calling for proposals)? Is there an adequate balance of activity (e.g., partnerships vs. funded projects) in support of ISTPP mandates? Are the roles, responsibilities and activities of delivery organization staff appropriately defined? Is ISTPP well managed? Etc.
  • Cost-effectiveness: Is the funding of the ISTPP the most effective way for the Government of Canada to achieve the stated ISTPP objectives? Does ISTPP have a performance system to track progress and achievements?
  • Program Success: Is ISTPP on track to yield increased investment in R D? Is ISTPP in line to increase the competitiveness of Canadian R D organizations at the international level? Etc.

2.3 Time Frame for the Evaluation Study

The evaluation study was conducted between January and April 2008. Planning and instrument development took place between January and February 2008, followed by the data collection for the three lines of evidence that took place between February and April 2007. The draft data analysis and report writing were completed in April 2008.

Top of Page


3.0 Methodology

The following sections describe the methodology for the evaluation.

3.1 Evaluation Design

The evaluation approach was designed to assess the delivery mechanisms and, through interviews, a review of documentation and a file review (review of project proposals), identify any gaps and future needs. The evaluation assessed the various aspects of the program's activities, processes, and preliminary results according to the evaluation issues and questions and the extent to which the program is on track to achieving the desired outcomes. Finally, the proposed approach provides recommendations regarding possible corrections and emerging priority areas for the ISTPP.

3.2 Lines of Evidence

There are three main lines of evidence for the evaluation findings:

  1. Interviews;
  2. Document review; and
  3. File review (ISTPP project proposals).

3.2.1 Interviews

Interviews were carried out to inform most evaluation questions (e.g., provide feedback on achievement of outcomes). In all, 43 interviews were conducted with the following types of respondents:

 Key InformantsSampleTargetCompleted interviews
Program PersonnelISTPP Secretariat222
ISTPCanada Board Members
(and Precarn representative)
155-712
Steering committee members
(including co-chairs members)
75-76
StakeholdersTrade commissioners332
Government department representatives65-63
Partners in counterpart countries442
Expert coordinators (Canada-China)442
Expert coordinators (Canada-India)646
Potential Program RecipientsCanada-China833
Canada-India835
Total6338-4343

Key informants were identified purposefully according to selection criteria outlined in the evaluation work plan. Participants sampled using this approach were chosen deliberately due to their particular knowledge and/or experience with the program itself. Additional interviews were completed with individuals corresponding to an alternative key informant group (i.e. ISTPCanada Board Members) to compensate for fewer interviews completed from another group (i.e. partners in counterpart countries) due to difficulties reaching and/or scheduling interviews. The total target number of interviews (43) was achieved.

3.2.2 Document review

Documents were reviewed in order to better understand the activities undertaken to date by the ISTPP and the related outputs produced. There were two main categories of documents that were consulted: program documents and other documents. Program documents are those generated by the program in the implementation and delivery of the ISTPP. Other documents are those prepared by organizations outside of the program.

3.2.3 File Review

In all, 20 program files (ISTPP project proposals) were reviewed. The purpose of this review was to obtain a better understanding of the what type of projects will be potentially undertaken by program recipients, the rationale for project funding, and potential benefits of collaborative R D, including the commercialization potential and the establishment of partnerships.

3.3 Limitations/Challenges

The evaluation was based on an assessment of the extent to which outputs, cost-effectiveness and success of the program are on track. It must be noted, however, that, due to the fact that the program has barely started, the ability to assess success and achievement of the planned outcomes was very limited.

One of the overall limitations of the evaluation design is the heavy reliance on qualitative data from interviews. Interviews can only offer anecdotal and opinion-based evidence against the evaluation issues. Thus, the utilization of multiple lines of evidence was critical to confirm (or refute) findings from the interviews. As well, interview-based evidence has been triangulated with available documented evidence and a review of program files.

3.4 Approach to Analysis

In preparing the draft report, all lines of evidence have been triangulated and synthesized by each main evaluation issue and question. The evaluation team sought to identify a pattern of findings, or themes, across all lines of evidence for each question. Where a discrepancy was noted, efforts were made to present the diverse findings and seek a rationale for the opposing views/findings.

Top of Page


4.0 Key Findings

The following section presents the findings of this evaluation study by evaluation issue, i.e. program relevance, outputs, design and delivery, cost effectiveness and progress/success.

4.1 Relevance

Issues 1 and 2:

Adequacy of ISTPP objectives and relevance to Canada and partner countries

Finding:
The program objectives are well aligned with DFAIT's departmental objectives and considered appropriate by stakeholders. However, the evaluation team noted that there are varying perceptions regarding the primary focus of the program (e.g. commercialization, bilateral linkages, or strengthening R D).

According to program documentation, ISTPP has seven major objectives:

  • Encourage domestic competitiveness through the transfer of technology knowledge resulting from international S T partnerships;
  • Foster international S T partnerships and collaborative research in all sectors, with an emphasis on industry-industry partnerships and university-industry partnerships;
  • Accelerate the commercialization of R D that would benefit Canada through international partnerships with a focus on small and medium-sized enterprises;
  • Access international technologies for Canadian enterprises;
  • Promote Canadian research and development capacity and Canada as a destination for foreign technology-based investments;
  • Encourage the mobility of researchers and promote Canada as a career destination for foreign researchers and highly qualified personnel; and
  • Strengthen overall bilateral S T relations.

Official program documents also note that the results of the program will include, but not be limited to:

  • Fostering the global networks critical to improving commercialization;
  • Access to the world's best facilities, equipment, talent and knowledge; and
  • Wealth creation resulting from increased Canadian production of innovative goods and services through international linkages.

The outcome of meeting these objectives, as outlined in these documents, will be the fostering and support of bilateral research projects between Canada and partner countries that have been proposed by companies, universities and colleges. It will also stimulate bilateral S T networking and matchmaking activities to further new partnerships and accelerate the commercialization of R D.

Our analysis, based on interviews and documents reviewed, indicated that the above objectives of fostering international S T bilateral partnerships between Canada and India and Canada and China remains highly relevant to current Canadian priorities.

According to program documentation, Canada has not placed a high priority on participation in international S T, as evidenced by a lack of a coherent policy framework, the lack of an efficient mechanism for coordination, and the lack of dedicated funding. However, significant developments occurred [in 2005 and 2007] to indicate shifts in the Government's priority. The first was the appointment of a National Science Advisor10 to the Prime Minister to engage the public and private sector in a national S T agenda. The National Science Advisor also expressed a strong interest in linking domestic and international S T policies and programs. The second is that the Government signaled its stated priority for international S T in Budget 2005, which provided $20 million over five years to support new international S T initiatives such as those contained in the Prime Minister's Joint Declarations with India and China in January 2005. Budget 2007 provided a further $60 million over two years to the Global Commerce Strategy, which is designed to strengthen Canada's competitiveness in global markets.11 The budget features three core elements: supporting an expansion of Canada's bilateral trade network, strengthening Canada's competitive position in the U.S. market, and extending Canada's reach to new markets, starting with Asia.

Our analyses also found ISTPP objectives are well aligned with key DFAIT documentation, including DFAIT's Management, Resources and Results Structure (MRRS) and Program Activity Architecture (PAA). Most notably, the program supports DFAIT's MRRS Strategic Outcome 2: Enabling Canada to Succeed in the Global Marketplace and the PAA program activity: Internal Business Development (p. 6). A good fit was also found with DFAIT's Global Innovation Strategy, which clearly overlaps with both the Global Commerce Strategy and Federal S T Strategy, as illustrated in Exhibit 1.1 below:

Exhibit 1.1: Overlap of the Global Commerce Strategy and Federal S T Strategy 12

Image depicting the Overlap of the Global Commerce and Federal S T Strategies

Overall, the majority of stakeholders who were interviewed found the objectives of the program to be appropriate, though there were different understandings of the program's main focus: some highlighted the commercialization aspect, while others mentioned the collaborative aspect (bilateral relations, partnerships, etc.) or the enhancement of Canadian competitiveness on the international scene. A few key informants felt the program budget is insufficient to cover the broad range of technology sectors13 and argued that the technology sectors should be more focused to reflect the areas with the best growth potential over time.

Complementarity studies were conducted in the scope of the Canada and China and Canada and India S T Agreements. The Canada-China S T Complementarity Study was conducted in 2006 in response to the commitment to enter into an S T Agreement. The Institutional Linkages: Academic/Government/Private Partnerships In the Canadian and Indian Science and Technology Sector study conducted in 2004 provided recommendations for the S T Agreement, which was signed with India in 2005.

The Complementarity studies have slightly different objectives for Canada-China and for Canada-India, as follows:

  • Canada-China study objective: To identify mutually advantageous S T priority areas between Canada and China, and the modalities and mechanisms for achieving mutually beneficial results through the framework of an S T Agreement.
  • Canada-India study objective: To identify Canada and India's different strengths and weaknesses in S T to provide a tool for assessing the potential for Canada-India relations in these fields.

The Canada-China S T Complementarity Study presents three pillars for Canada-China S T complementarity: economic environment and challenges, S T capacity and performance, and the longstanding cultural and institutional ties between the two countries. The study determined that the priority areas of the two national governments under the agreement should:

  • Hold Significant social and economic benefit for the citizenry of both countries, as well as the broader global community;
  • Demonstrate opportunity for S T cooperation, particularly in pre-competitive research areas where governments can play an important role in promoting and enabling S T; and
  • Allow for S T cooperation across the three major S T performing sectors (business, higher education, and government).

Four areas were deemed to meet the criteria for implementing the Canada-China S T Agreement: Energy, Environment, Health Life Sciences / Biotechnology, and Agricultural Foods and Bioproducts.

The Institutional Linkages: Academic/Government/Private Partnerships In the Canadian and Indian Science and Technology Sector study provides an overview of the S T innovation agenda as it relates to Canadian and Indian institutional linkages with potential for partnerships between various S T sectors themselves and with the private sector in both countries. The main observations of the study were:

  • There are mutual needs in both countries that can be met through a comprehensive bilateral S T linkage;
  • India is doubling its investment in the S T sector in the next 10 years to $8.25 billion. Substantial investments are being made in its R D, academic and laboratory facilities (such as 20 new national research centers, 220 universities, and 200 laboratories). This is a huge opportunity for participation by Canadian S T institutions;
  • There are no substantial S T linkages between Canada and India. Other countries of the G8 have many with India. More broadly, India has S T agreements with 57 countries and Canada is not on their list;
  • The current Canada-India S T "linkages" are small and fragmented along government departmental or sectoral lines. Even these modest links have been structured only in recent years;
  • In consideration of a conservative employment growth rate of 4% to 5%, Canada would need about 16,000 to 20,000 new highly qualified professionals per annum;
  • Universities are facing shortages. By 2011, almost 40,000 new professors are required, half to replace retirees. There is no national education strategy to address this issue and its impact on Canadian R D; and
  • There are fewer foreign students enrolling in Canadian universities at the graduate or PhD levels than comparable advanced educational jurisdictions, despite fee waivers and other personal support.

Several key informants agreed that ISTPP objectives are relevant to both Canada and partner countries and referenced the Complementarity studies. Furthermore, many of these key informants stated that the objectives are also relevant to the needs of organizations that conduct joint R D projects, mainly because they enable these organizations to engage in international partnerships, and provide them with opportunities, such as accessing technologies and increasing their revenues.

Issue 3:

Relevance of selected partner countries

Finding:
China and India are relevant partner countries for S T collaboration. A few concerns were raised, particularly with respect to weak institutional linkages between government, academia and the private sector in India and Intellectual Property (IP) protection in China.

The February 2004 Speech from the Throne stated that more attention would be focused on newly emerged economic giants such as China, India and Brazil. Several key informants pointed out that China and India were specifically identified in the 2004 Throne Speech. Some ISTPCanada board members and ISTPP Steering Committee members provided additional reasons why those countries are relevant:

  • They are emerging countries with leading economies and large populations;
  • They have good industrial strategies and S T potential; and
  • They both tend to invest a lot in R D.

As stated in the program documentation explaining the choice of China as a target for the program: "Modern China is developing at an impressive rate and is investing heavily in large-scale science and technology development. Funding in research and development has increased from about 0.7% of GDP in 1998 to 1.23% in 2002. China is well on its way to developing its own sophisticated science and technology capabilities which would make it ripe for collaboration with Canadian firms. There are currently some 30 science and technology related Memorandum of Understanding (MOU) in place by various Canadian government departments and agencies. Currently China has science and technology agreements with 96 other countries." Official documentation also mentions that: "India has made a significant commitment to increasing its capability in science and technology, doubling its investment to over 1.4% of GDP from 1995 to 2003. Currently India has science and technology agreements with 57 other countries."

The majority of key informants in all of the respondent groups believe that China and India are highly relevant countries with which to partner. The most common rationale given is that they are fast-growth, emerging economies with appropriate technological capacity. However, some concerns were raised by a few key informants with respect to weaknesses in the links between government, academia and the private sector in India, and IP protection issues in China.

A number of other countries were proposed by key informants as relevant countries with which to partner on S T Agreements. These are listed below along with the rationale provided by the key informants interviewed:

  • European countries (Ireland, Spain, etc) or European Union: These countries are already collaborating with Canada, and it is believed that an S T Agreement would be relatively easy to implement.
  • United States: The U.S. is a relatively easy environment for Canadian firms given the similarity in U.S business practices, transparency of IP management, lack of language barriers, and the attractiveness of the US from a technological point of view.
  • Other BRIC countries (Brazil, Russia): These countries have a strong science and technology base and have good research facilities.

Issue 4:

Need for ISTPP support

Finding:
As projects have not yet been funded it was not possible to directly assess whether the program resulted in successful international S T collaborations. However, the evaluation found evidence that the ISTPP provides a valuable mechanism in support of international collaboration.

Our analysis indicated that it is the entire ISTPP "package" that enables projects to proceed. That is, it is the combination of support mechanisms provided to program recipients that enables them to develop and proceed with their projects, including:

  • Matchmaking activities;
  • Funding activities (including leveraging); and
  • Validation through ISTPCanada assessment.

Furthermore, in general Canadian small and medium-sized enterprises (SMEs) have limited resources to establish and maintain international linkages, and many of them are not accustomed to global marketplace undertakings; therefore, the program is particularly useful in serving as a "broker" and a catalyst for new partnership development.

Several key informants pointed out that the incremental value of the ISTPP should be fostering international S T partnerships and collaborative research, permitting firms to accelerate the commercialization process - specifically by accessing international technologies - and strengthening overall bilateral S T relations. The mechanism the program has put in place facilitates the process of establishing partnerships and moving forward from R D to commercialization. Furthermore, the matchmaking program activities (and the leverage aspect) are considered to be the cornerstone of the program by many key informants:

  • The program established consortia composed of small and medium-sized organizations (SMEs) from partnering countries that can leverage and make alliances;
  • Since research and development projects tend to be more risky, leveraging support is key as it diversifies the risk;
  • Even if projects are not approved, some proponents may still benefit from the matchmaking process, which may lead to the launch of their projects; and
  • The program is playing a catalytic role by linking with (i.e. levering) other sources in order to grow the size and scope of the projects.

The evaluation study did not find evidence that current S T collaborative activities (including those independent of the program) are being documented; therefore, it is challenging to establish the degree to which collaboration is occurring independently of the program and the degree to which the program is supporting new collaborations. It is of interest to point out that, in the Canada-China S T Complementarity Study (2006), it was mentioned that it would be important to ensure that the existing baseline of S T cooperative activities between Canada and China is adequately documented and recognized in developing new initiatives. On this matter, the Expert Team (authors of the study) recommended that the existing range of S T collaborations between Canada and China be catalogued.

A few program and Stakeholder key informants (including board members, Steering Committee members, Trade Commissioners and Government Department representatives) believe it possible for projects to move forward without ISTPP support. Some point to other sources of support, such as IRAP and NRC, as alternatives to ISTPP (especially for SMEs).

Issue 5   6:

ISTPP strengths and weaknesses; value-added of the ISTPP

Finding:

ISTPP has a number of strengths. Its value-added is mainly that it is a mechanism to foster international S T relationships and leverage support, and it supports applicants through a rigorous vetting process for prospective projects.

The fact that the ISTPP is delivered by an external delivery agent, at arm's length from government, is perceived by key informants to reduce potential political pressures, and provide for faster and less bureaucratic program delivery. The evaluation found that ISTPCanada facilitates leverage through contributions by the provinces or other entities.

Our analysis, based on key informant interviews and the documentation review, indicated the following strengths:

  • Consolidation of S T collaborations. One of the strengths of the ISTPP is that it provides funding for S T Agreements that can be used by partner organizations in each country for joint R D projects. Without programs such as ISTPP, the S T Agreements are more difficult to actualize as there is limited funding to support collaboration and support of projects. ISTPP (in the four program countries) consolidates S T collaborations under one national umbrella; It is a way to operationalize S T agreements;
  • The ISTPP is one of the few programs available that provides funding to S T intensive firms for R D collaborations overseas. An advantage of this is that it allows companies to gain first hand experience with technological progress in partner countries;
  • Another strength of the program is the DFAIT backing. In countries such as India and China, government involvement provides for better access to high-level officials in the respective governments. As well, the fact that there is Canadian government involvement provides greater credibility to the Canadian partner organizations;
  • A perceive strength of ISTPCanada is that it has well-established networks for evaluating projects. Furthermore, the network of Canadian government and university evaluators has largely provided this service at no cost.
  • A weakness that was identified was that the ISTPCanada has relatively limited funding. This is because the relative impact of the program is perceived to be related to the funding level and the number of collaborations that can be supported. Another issue related to the limited funding is that countries such as India and China have collaborative arrangements with other countries, which are better funded.

Most key informants felt that they could not compare ISTPP to similar R D collaborative programs since the program is unique, given its international focus, specificity with respect to targeted technology sectors partner countries, and the involvement of private- and public-sector organizations. Nevertheless, a few comparisons were made:

  • Other similar R D programs (i.e. IRAP) have field organizations that ISTPP doesn't have;
  • CIIRDF has an infrastructure in place that is well established. The language/culture barriers are also less of an issue with Israel as they are with China, as English is a common working language in Israel and Israeli firms are experienced in North America; and
  • In comparison to similar R D collaborative programs managed by the USA, the program is lacking in terms of resources (staff and funding).

Although many key informants feel that it is still too early to determine the value added by the program, many believe that the program is on track with respect to enhancing certain aspects of the bilateral relationships between Canada and India and Canada and China. Most importantly, the program has the ability to foster international S T relationships, leverage support, and add value through the vetting process. For example:

  • It allows government organizations to pool their resources;
  • It is a federal program that can leverage support from the provinces; and
  • At the proposal preparation stage, the rigor of project criteria and feedback provided to program proponents can strengthen project proposals.

4.2 Outputs

Issue 7:

Key outputs anticipated

Finding:
Despite it being early in the funding cycle, our analysis indicates that ISTPP is on track to achieve or has already achieved the implementation of planned program outputs.

The expected outputs of the R D matchmaking component include: i) communication tools and events; ii) introductions between R D organizations; iii) identified consortia; iv) joint meetings/task forces; and v) technology forecasts. The evaluation found the following evidence of achievement of the planned outputs:

  1. Communication tools and events:
    • A call for proposals was issued for both China and India; and
    • An ISTPP web site was developed and implemented.
  2. Introductions between R D organizations
    • Trade delegations to India/China have taken place.
  3. Identified consortia
    • This has occurred through the proposal process.
  4. Joint meetings/task forces
    • The ISTPP Steering Committee was established.
  5. Technology forecasts
    • The Canada-China S T Complementary study (2006) and the Canada-India S T Mapping Study (2004) were completed. The expected outputs of the Collaborative R D Investment component include: i) collaborative R D projects;
    • Products, services and processes; and
    • Project commercialization plans. The evaluation found the following evidence of achievement of the planned outputs:
      1. Collaborative R D projects

        Following the reception of Expressions of Interest (EOI) from potential program recipients, ISTPCanada selected 47 proposals under the Canada-China component and 28 proposals under the Canada-India component for the proposal evaluation stage. The following exhibits present these results by province (Exhibit 1.2) and by targeted technology area (Exhibits 1.3 and 1.4).

      2. Products, services and processes

        All proposals submitted included a presentation of the expected benefits of projects, and attested to the products, services and processes that would be produced as a result of the projects (i.e. lower costs to end users, improvement of safety standards, eco-friendly technology, compliance with Kyoto protocols, etc.).

      3. Project commercialization plans

Exhibit 1.2: Number of Proposals by Province, India and China

Number of proposals by province, India and China

* Atlantics: includes New Brunswick (3), Nova Scotia (2) and Newfoundland (1)

Exhibit 1.3: Distribution by Technology Sector, China Proposals

Distribution by Technology Sector, China Proposals

* Misc: includes forestry (3), construction, nanotechnology and material

Exhibit 1.4: Distribution by Technology Sector, India Proposals

Distribution by Technology Sector, India Proposals

* Misc: includes agriculture, aerospace, materials, etc.

  • All proponents provided project commercialization plans in project proposals.

4.3 Design/Delivery

Issue 8:

Clarity of the intent of the collaboration between ISTPCanada and partner countries

Finding:
The intent of the collaboration between ISTPCanada and partnering countries is clear.

Based on key informant interviews and the documentation review, our analysis indicated that the intent of the collaboration between ISTPCanada and partnering countries is clear. Responsibilities for ISTPP activities are clearly defined in the documentation, and, overall, stakeholders and program applicants are satisfied with the program design. A few noted that cultural differences with partner countries may be a challenge, and that ongoing consultations with country counterparts is critical to ensure there is adequate comprehension of the program objectives and processes on both ends.

Issues 9   10:

Appropriateness of the bilateral structure of the ISTPP and appropriateness of design of ISTPP activities

Finding:
For the most part, the bilateral structure of the program facilitates the achievement of program objectives - particularly enhancing R D collaboration. This is mainly due to the program officials and sub-committee board members who act as liaisons between Canada and the partner countries.

Our analysis found that the bilateral structure of the ISTPP is a significant contributor to the achievement of program objectives, particularly with respect to enhancing R D collaboration between Canadian organizations and international partners. This is because there are complementary organizations in each country specifically focused on generating partnership projects that can be funded. The existence of China/India sub-committees of the ISTPP Board to act as the liaison between each country's delivery organization (ISTPCanada, MOST and GITA) is perceived to be of particular value by stakeholders in Canada and in partner countries because it facilitates contacts, as each sub-committee becomes more familiar with the complementary organizations and proponents in the opposite country.

The work of country managers is also considerably valued by stakeholders, as they are perceived to be well positioned to designate expert reviewers for the project proposals, and undertake promotional and matchmaking activities (including the call for proposals).

Only a few potential difficulties have been identified with respect to the bilateral approach. The first is to clarify the extent to which delivery organizations can utilize standard approaches. For instance, in the call for proposal and evaluation processes, it is important to consider the different budgetary cycles of partnering countries and the implications for both applicants and financial management of the projects by delivery organizations. Furthermore, since each country conducts its own project evaluations, it will be important to establish whether partnering countries can utilize the same evaluation criteria (i.e. currently China is not using the same evaluation criteria as Canada and India). Lastly, some key informants who were consulted as part of the evaluation expressed concern over a potential mismatch between partnering countries' and Canadian objectives and priorities. Although S T complementarity studies were conducted for both China and India that established key complementary technology sectors, it is possible that there could be changes in perceptions as to what those sectors should be as a result of new developments in technology sectors or changes in government (shifting priorities).

Our analysis, based on key informant interviews and a review of documents, indicates that - to date -the program is adequately designed to meet the needs of program recipients. It was pointed out by potential program recipients that the Partnership Development Activities (PDA), in particular, are well designed to meet their needs. For example, the targeted technology trade missions to partner countries are felt to be successful in establishing potential partnerships between partner country and Canadian organizations. Whether ISTPP acts as a catalyst in the formation of new partnerships cannot be assessed at the time of this evaluation, but could be considered for the gathering of baseline information for future evaluations.

Issue 11:

Promotion of the ISTPP

Finding:
Although DFAIT and ISTPCanada currently undertake promotional activities, the program is not systematically advertised. Although many organizations that are potential applicants are aware of the program, there is a need for a more systematic approach to awareness building to ensure that all potential applicants are equally aware of the program.

Many different promotional activities are undertaken by both DFAIT and ISTPCanada to promote the program.

DFAIT promotional activities include:

  • Program promotion through the DFAIT website (S T);
  • Interdepartmental meetings (i.e. the Interdepartmental Network for S T (INST);
  • Promotion through the S T network abroad (i.e. DFAIT trade commissioners); and
  • Promotion through regional offices.

ISTPCanada promotional activities include:

  • Presentations, meetings and information sessions with interested organization representatives (in Canadian provinces);
  • ISTPCanada website;
  • Attendance at trade shows;
  • Promotion through sector associations, IRAP regional meetings, NSERC, Industry Canada, etc.;
  • Promotion of the program through personal contacts of ISTPCanada board members and ISTPP Steering Committee members;
  • Trade delegations for target technology sectors in partner countries; and
  • Sessions with chambers of commerce in China and India.

Despite the numerous promotional activities undertaken, a number of interviewees indicated that the program was not well enough publicized, and that small firms (SMEs) federal communities and academic organizations that could have applied may not have been aware of the program. Given this situation, access to the program by SMEs, federal communities and academic organizations may not have been equitable, as not all possible applicants may have known of the program. The evaluation concluded that the program would benefit from a communication plan to ensure equity of access, particularly for SMEs, federal communities and academic organizations. It was suggested by key informants that making better use of the expert coordinators to promote the program (i.e. provide them with strategy, guidelines and resources) is advisable.

In support of the above, when asked in interviews how applicants found out about the program, the evaluation found that there were no systematic promotional activities to raise awareness about the program. Applicants indicated that they found out often by happenstance. Typical ways that they learned about the program included:

  • Trade mission to China, accompanied by ISTPP;
  • Announcement of the program obtained through a listserv;
  • Informed by university department director;
  • Through attendance at Indo-Canadian workshop on nanotechnology; and
  • Through informal networks.

Issue 12:

Adequacy of the selection process

Finding:
Although it is too early to fully assess the adequacy of the selection process at the time of this evaluation (project assessments being currently underway), the process seems to be adequate thus far.

The assessments of ISTPP project proposals was underway at the time of the evaluation, therefore it was too early to assess the adequacy of the selection process. Nevertheless, many positive comments were received from stakeholders and potential recipients interviewed, particularly with respect to the selection process and the level of transparency and fairness.

ISTPCanada utilizes a similar process for the evaluation of projects as has been used for the CIIRDF program. As ISTPCanada was a new program, with a number of technologies and scientific areas not previously encountered in CIIRDF, it was necessary at first to establish a list of external reviewers. These came from:

  • CIIRDF's database of reviewers;
  • The National Research Council institutes;
  • The Industrial Research Assistance Program (IRAP)14; and
  • The Ontario Centers of Excellence.

In addition, consistent with established practice for evaluating R D grant applications, program applicants are often asked to submit suggestions for potential expert reviewers. This procedure helps ISTPCanada increase its database of reviewers. In using reviewers recommended by the applicants, ISTPCanada assesses whether there is a potential conflict of interest before utilizing the reviewer.

Overall, the process for evaluation begins with an initial call for expressions of interest. These are then vetted by ISTPCanada and those meeting the ISTPCanada Guidelines are asked to submit full proposals. The full proposals are then sent out for external review.

It is worth mentioning that, with one exception, the designated program expert coordinators (representing various government departments, in specific technology areas) are not currently called upon to participate in the selection and evaluation process. Some key informants felt that the expert coordinators should be involved in this process, given their expertise in given technology areas. However, it remains to be resolved whether their role in providing support to program proponents at the proposal preparation stage (i.e. facilitation of new S T partnerships) might conflict with their potential or proposed role as evaluators.

Issue 13   14:

Reporting on program achievements; and roles, responsibilities and activities of delivery organization staff.

Finding:
The reporting requirements, roles and responsibilities, and activities are well defined in the program documentation; however, clarification is needed regarding the roles and responsibilities of expert coordinators.

According to program documentation, projects are required to submit technical, financial and commercialization reports to the delivery organization. Commercialization is to be reported to the delivery organization on a regular basis and forms the basis for repayments.15 The delivery organization is required to submit annual audited financial statements and activity reports to the Steering Committee.

The majority of key informants feel that the roles and responsibilities of delivery organization staff are appropriately defined. Several mentioned that, since they are a small team, they communicate well with each other and they have a good governance structure.

As mentioned in the previous section, the roles and responsibilities of the expert coordinators is an area where clarification is needed. It should be determined whether their role in providing support to program proponents at the proposal stage would conflict with their role as evaluators of these proposals.

Issue 15   16:

Management of the ISTPP; balance of activity in support of ISTPP mandates.

Finding:
Overall, both ISTPP and ISTPCanada appear to be well managed to date. The balance of activity (matchmaking versus project funding) also seems to be appropriate at this stage. However, as no projects have been approved, the evaluation team could not assess the effectiveness of procedures for project and financial monitoring.

A detailed description of the governance structure for ISTPP is provided in section 1.1 Program Description of this report. As for the management responsibilities of the delivery organization, the ISTPCanada Board of Directors is formed for each bilateral program for which the delivery organization is responsible (i.e. one for Canada-Israel, one for Canada-India etc.). ISTPCanada's President reports directly to the ISTPCanada Board of Directors, which is composed of China and India sub-committee representatives, and other private sector and academic representatives. The Precarn Inc. Vice President, External Relations, assumes the duties of Managing Director at ISTPCanada, reporting to the President. In addition, the Precarn senior manager for partnerships serves as a senior advisor to both the president and program manager in light of his specific experience and knowledge with technology partnerships in the Pacific Rim countries. ISTPCanada subcontracts to Precarn a range of corporate services, including project and financial management. Country managers work in the program office and oversee program activities such as matchmaking and the selection of external reviewers for project proposal assessments.

The evaluation team examined program documentation and conducted interviews with stakeholders to assess the adequacy of program management, but since the ISTPP was only at the project proposal assessment stage at the time of the evaluation, only the assessment process could be evaluated. Thus, processes such as monitoring procedures and financial assessments could not be assessed at this time.

With this in mind, our evidence suggests that, to date, ISTPP management is satisfactory. Many strengths were identified by key informants with respect to the management of ISTPP, including:

  • The high number of linkages between ISTPP and other agencies (i.e. IRAP, NSERC, the provinces, etc.); and
  • The experience of the individuals on the ISTPP Steering committee of.

A few key informants, however, questioned the expertise DFAIT, as a foreign-service organization, could bring to the program (i.e. trade commissioners may not have experience in science and technology management). Furthermore, many pointed to a significant amount of turnover in the department and the corresponding instability this engenders.

A particular strength of program management is that ISTPCanada as an arms-length agency has been contracted as the delivery organization for the program. The majority of key informants were positive about ISTPCanada's board members and country sub-committees, as well as the program activities they manage.

Most key informants feel that the balance of activity (collaborative R D investment: 80% vs. R D Matchmaking: 20%) is adequate, though it is still early to tell if adjustments will be required at a later date.

Issue 17:

ISTPP performance system

Finding:
DFAIT prepared an ARAF to track progress and achievements (which incorporates both a formative and summative evaluation of the ISTPP).

An ARAF has been prepared by DFAIT, which provides the basis for ongoing monitoring. It provides an integrated view of the ISTPP's risks and planned results. This information is intended to be used to manage the program and to monitor and report progress toward achieving the planned results. According to the ARAF, the key performance issues for ISTPP are:

  • Collaboration between companies;
  • Development of consortia in targeted technology areas;
  • Economic benefits; and
  • Workforce impacts.

During the evaluation, some concerns were raised whether the ISTPP Accountability, Risk and Audit Framework (AFAF) prepared by DFAIT adequately reflects the program as it currently exists. The evaluation team reviewed the ARAF (and logic model) and found that some clarification is needed to more adequately reflect the program as it currently exists.

4.4 Cost-Effectiveness

Issues 18, 19, 20   21:

Cost-effectiveness

Finding:
Overall, the current funding of the ISTPP and delivery through ISTPCanada appears to be the most cost effective way to deliver the program. It was noted that several project proponents expect to lever support for their projects.

The cost effectiveness analysis completed was based on assessing the operating costs of ISTPCanada, as the ISTP Secretariat is an internal DFAIT organization charged with responsibilities additional to monitoring ISTPCanada.

The operating costs for ISTPCanada are set at a rate not to exceed 15%. Therefore the operating budget for ISTPCanada encompasses 15% of the contribution amounts for China, India, Israel, and Brazil. The total estimated expenditure for all four countries amounts to $18,500,000 and the operating costs can reach $2.8 million over 5 years. Our review of the detailed cost breakout indicated that the cost of the president, and four-program managers part time as well as the real estate and travel and administration costs are reasonable. The operating cost will drop as a percentage of the contribution costs if the contribution levels increase. This is due to the economies of scale that can be achieved.

ISTPCanada has also arranged for additional contributions from IRAP, and provincial governments in British Columbia, Alberta, Quebec, and Ontario. Each of these provinces will also contribute to the operating costs - which will also help realize economies of scale. ISTPCanada receives a 15% administrative fee from these sources. Finally, it should be noted that ISTPCanada has successfully engaged the majority of project assessment reviewers at virtually no charge.

Furthermore, the evaluation team observed that the current structure of the delivery organization can be sustained and can accommodate a "scale up" in overall project funding. With a scaling up, the costs of delivering the program should not increase significantly and economies of scale will be achieved.

However, some potential challenges of "scaling up" the program have been identified. One likely challenge could be in terms of monitoring and financial management. To meet this challenge, Precarn Inc. would need to increase its capacity. Another challenge that was identified is that ISTPP will have to clarify its relationship with other federal departments. If the program "scales up," formal agreements with other federal departments may be required, to ensure their support for project assessments and in terms of support for ISTPP activities.

In terms of alternative delivery models, four models were considered as part of this evaluation: program delivery though government (DFAIT), delivery through a not-for-profit organization or foundation (such as ISTPCanada), delivery through IRAP, and contracting out to a private-sector consulting organization.

Strengths and weaknesses are presented for each of these delivery models, based on observations made by program managers and stakeholders, as well as on our own observations:

Delivery modelsStrengthsWeaknesses
Government (DFAIT)
  • Sustainability
  • Easier to keep program delivery aligned with government changes
  • Subject to political pressures
  • Staff turnover
  • Salaries for some positions may not be cost-competitive with the private sector
  • Leveraging support more difficult (especially from Canadian provinces)
Not-for-profit organization or Foundation (such as ISTPCanada)
  • More efficient (i.e. salaries can be adjusted)
  • Outside government (not subject to political pressures)
  • Not subject to one-year appropriations
  • Can accommodate financial contributions from other sources (and other levels of government)
  • Need to establish network for project assessment process
  • Potential of "goal displacement"
  • Tendency of arm's length organization to shift from their original focus and undertake other objectives
IRAP
  • Can use existing networks and/or NRC labs to support project assessments
  • Internal technical capacity (i.e. Industrial Technology Advisors)
  • IRAP has significant budget to also fund projects
  • Leveraging support more difficult (especially from Canadian provinces)
  • Is under one-year appropriation government rule
  • SME-focused
  • Not focused on international area
Private sector consulting organization
  • Not constrained by government salary structures
  • Outside government (not subject to political pressures)
  • Not subject to one-year appropriations
  • Can accommodate financial contributions from other sources (and other levels of government)
  • Need to establish network for project assessment process
  • Need to establish governance structure
  • Need to hire appropriate managers

Another measure to assess the cost effectiveness is the anticipated capacity for both the delivery agent and individual program proponents to leverage DFAIT funding. Both private sector and not-for-profit organizations permit the delivery agent to seek other sources of funding (i.e. from the provinces), which leverages DFAIT funding. This would be more challenging if the program was delivered by IRAP or DFAIT.

Through examination of the project proposals submitted to ISTPCanada, it was determined that many proponents anticipate leveraging support for their projects. For example, a Canadian project proponent expects to receive support from Canadian governmental programs and resources such as IRAP, Sustainable Development Technology Canada (SDTC), FedNor, and the Ontario Centers of Excellence, as well as in-kind support from the National Research Council. Several proponents expect to receive private investment for their projects.

4.5 Progress/Success

Issue 22   23:

Potential to yield increased investment in R D and competitiveness; potential to impact ability to attract and retain HQPs

Finding:
The overall impact of the program is constrained by the limited size of the budget. However, the program will likely contribute to the competitive position of Canadian R D organizations and the attraction and retention of highly qualified personnel.

It is early at this stage to assess impacts on increased investment in R D and increases in the competitiveness of Canadian R D organizations internationally. Our analysis was based on key informant interviews supplemented by interviews with 8 project applicants and a review of 20 project proposals.

The review of the sample of 20 project files indicated that in each case the project is leading to an increase in R D investments by the Canadian firms. Additionally the linkages to the partners in India and China should increase their technical competency and hence competitiveness of these firms and provide the firms with innovative products that are not currently available. It is expected that not all of the project R D will lead to successful innovations.

Several of the ISTPP's features support the programs ability to enhance the competitive position of Canadian R D organizations, in particular:

  • ISTPP's matchmaking activities, which can lead to partnerships;
  • Proposal requirements for detailed commercialization, plans which can lead to the further development and speedy commercialization of R D projects; and
  • The experience of SMEs with the process, which can lead to further developments on international markets.

These observations are for the most part shared by the stakeholders who were interviewed as part of the evaluation.

A few concerns were raised by stakeholders who were interviewed regarding international partnerships. Some expressed concern that it will be challenging to ensure collaboration with partner countries on a long-term basis; some were concerned about IP protection (with Chinese partners); and some highlighted the need for a global perspective. Still other key informants mentioned that there have been several instances where countries proposed partnerships with Canada that had to be turned down because Canada could not match the offer. This fact highlights the interest of other countries in developing and commercializing technology-based R D projects.

Although it was not specifically required in the project proposals, many program proponents included a description of the potential of the program to impact the attraction and retention of Highly Qualified Personnel (HQPs). For example, one proponent explained that capacity building is a secondary objective of the proposed project; an exchange program between Canada and the partner country will be established to encourage education among young researchers in the development of the system. Another explained that the organization has invested significant capital into development and is willing to financially support graduate students at both Canadian and partner country universities; upon their graduation, the organization is willing to hire these individuals as regular employees to continue their development work.

While some key informants believe the attraction and retention of HQPs is a secondary priority of the program, others believe it is one of the key goals of the program. Likewise, some believe attracting and retaining HQPs can be directly attributed to the program, while others believe it can only be indirectly attributed. However, several who think the program will be successful in this regard believe greater international competitiveness will render Canadian organizations more attractive to HQPs. Furthermore, as explained by one key informant, R D organizations that become involved in competitive funding processes may be encouraged to increase their staff to build their in-house research capability.

Top of Page


5.0 Conclusions

5.1 Overall Assessment

The ISTPP was only established in January 2007, and therefore, an evaluation at this stage was focused on whether the program has been well designed and whether the governance structures appear to be adequate to oversee the program. The evaluation indicated that to date the ISTPP has met its objective of building on the success of the CIIRDF program (Canada Israel Research and Development Foundation) in support of bilateral Science and Technology Agreements, specifically in China and India.

Program management has selected a delivery organization with an established reputation in managing the CIIRDF program (ISTPCanada). ISTPCanada has put in place a management and governance structure to manage the two components, with sufficient Board oversight. Reciprocal arrangements have been struck with India and China to establish counterpart organizations in each country, which will support bilateral research partnerships between organizations in each partner country and Canada respectively.

In Canada, India and China, calls for proposals have been issued and at the time of the evaluation the awarding of individual project contributions was underway.

The evaluation concluded that the ISTPP program is on track to meet its objectives. The evaluation also ascertained that ISTPCanada has achieved the short term outputs that it set out to achieve.

The evaluation assessed the cost effectiveness of the operation of the program, and determined that it is cost-effective, based on a review of the operating costs of the program, the ability to leverage additional contribution dollars from IRAP and a number of the provinces, as well as the ability to obtain additional funds for operations from the provinces. The added contributions should reduce the operating cost ratio for the federal government as the contribution budget increases and the federal contribution to operating costs becomes a lower percentage of the total.

Initial impressions from the project proponent applications supplemented by interviews indicate that if the R D projects are successful, there are plans in most cases to commercialize the results. Applicants interviewed indicated that many had pre-existing relationships with firms in China or India, but had limited funds to engage in joint R D.

Finally, the size of the ISTPP is relatively small given the demand experienced so far and potential for R D collaborations in other countries. However, it is too early to assess the impacts of the program, and the potential for scaling up the program.

5.1.1 Program Findings

The section present the findings by evaluation issue, including program relevance, outputs, design and delivery, cost-effectiveness and program success.

Program Relevance

The program objectives were found to be well aligned with DFAIT's departmental objectives and considered appropriate by stakeholders. However, there are varying perceptions of the primary focus of the program (e.g. commercialization, bilateral linkages, or strengthening R D).

During the evaluation, some concerns were raised whether the ISTPP Accountability, Risk and Audit Framework (AFAF) prepared by DFAIT adequately reflects the program as it currently exists.

Most key informants agree that China and India are relevant partner countries for S T collaboration because they are fast-growth, emerging economies with appropriate technological capacity. A few concerns were raised, particularly with respect to weak institutional linkages between government, academia and the private sector in India and Intellectual Property (IP) protection in China.

The value-added of the ISTPP is that it is a mechanism to foster international S T relationships and leverage support, and it supports applicants through a rigorous vetting of prospective projects. Because ISTPP is delivered by an external delivery agent, at arm's length from government, potential political pressures are reduced. ISTPCanada is perceived to provide for faster and less bureaucratic program delivery and it can facilitate leverage through contributions by the provinces or other entities.

Although it is not possible to directly attribute successful international S T collaborations to the program at this stage, the ISTPP provides a valuable mechanism in support of international collaboration. At this stage it is not possible to assess whether projects would move forward without ISTPP support. However, ISTPP has a number of strengths, including its ability to foster international S T relationships and leverage support, and the support it provides to applicants through a rigorous vetting process for prospective projects.

Program Outputs

ISTPP is on track to achieve or has already achieved the successful implementation of program outputs under the collaborative R D Investment and the R D Matchmaking program components.

Program Design Delivery

The intent of the collaboration between ISTPCanada and partnering countries is clear and both ISTPP and ISTPCanada appear to be well managed to date. The bilateral structure of the program facilitates the achievement of program objectives.

The overall structure of the program is well designed and well defined in the program documentation. DFAIT has put in place a robust governance structure with the establishment of a Secretariat and a Steering Committee, and the selection of ISTPCanada (overseen by a Board) as the delivery organization. However, the role of expert coordinators in the program needs to be clarified.

Many promotional activities are currently undertaken by DFAIT and ISTPCanada, such as trade delegations for target technology sectors in partner countries, promotion of the program through IRAP regional meetings, NSERC, and Industry Canada, and ISTPCanada's website. However, a communications plan is required to ensure equity of access, particularly for SMEs, some federal communities16 and academic organizations.

Program Cost-Effectiveness

The evaluation team observed that the current structure of the delivery organization can be sustained and can accommodate a "scale up" in overall project funding. The costs of delivering the program should not increase significantly and economies of scale could be achieved. However, some potential challenges of "scaling up" the program have been identified. One likely challenge could be in terms of monitoring and financial management. To meet this challenge, Precarn Inc. would need to increase its capacity. Another challenge that was identified is that ISTPP will have to clarify its relationship with other federal departments. If the program "scales up," formal agreements with other federal departments may be required, to ensure their support for project assessments and in terms of support for ISTPP activities.

The evaluation assessed four different models for delivery of the program: program delivery though government (DFAIT), delivery through a not-for-profit organization or foundation (such as ISTPCanada), delivery through IRAP, and contracting out to a private-sector consulting organization. It was determined that the current delivery model is the most cost-effective, mainly because ISTPCanada is able to leverage support from other sources and has successfully engaged the majority of project assessment reviewers at virtually no charge.

Furthermore, several project proponents expect to lever support for their projects through Canadian governmental programs and resources and private investment.

Program Success

The competitive position of Canadian R D organizations supported by the program is likely to improve as a result of participation in the program. Furthermore, evidence already exists of an impact on the attraction and retention of highly qualified personnel occurring through partnership development and the proposal preparation process.

5.2 Recommendations

Recommendation 1:

There is a need to better communicate the main focus of the program to all stakeholders to ensure that they have a good understanding of the program. This can be done by creating a hierarchy of objectives, which describes the main goal of the program as well as the associated objectives.

Recommendation 2:

At the time of the next evaluation, a review of the ISTPP ARAF structure should be undertaken.

Recommendation 3:

ISTPP should assess the nature of the incrementality of the program to determine: a) whether the program is fostering international S T partnerships and collaborative research in the supported sectors (with an emphasis on industry-industry and university-university partnerships); and b) the extent to which the program is contributing to increased R D in a commercial context.

Recommendation 4:

Clarify the roles of expert coordinators.

Recommendation 5:

Develop a communication plan to promote the program more broadly to the Canadian S T community, targeting SMEs, federal communities and academic organizations.

Top of Page


Annex A: Manamgement Response

RecommendationsCommitments/ActionsResponsibility CentreTarget Completion DateStatus
Recommendation #1:
There is a need to better communicate the focus of the program to all stakeholders to ensure that they have a good understanding of the program. This can be done by creating a hierarchy of objectives, which describes the main goal of the program as well as the associated objectives.
Recommendation #5:

Develop a communication plan to promote the program more broadly to the Canadian S T community, targeting SMEs, federal community and academic organizations.

Recommendations #1 and #5 are seen to be linked and therefore our response addresses the two together for a more efficient action plan.

Develop a "Communications Plan" which would establish the hierarchy of objectives of the ISTPP and how best to communicate them as well as the role and functions of ISTPCanada. Once the overall program's objectives are well understood by the S T community, the next logical step is in promoting the call for proposals and services of the delivery organization, ISTPCanada.

Identification of most appropriate S T community members needs to be undertaken in the development of the Communications Plan. This would include Federal and provincial government departments and agencies, various committees including the ISTPP Steering Committee, industry associations, community of networks, etc. These groups would receive the Call for Proposals immediately once announced, regular updates for conferences and workshops, and other information as deemed appropriate.

ISTPCanada's web site will need to be updated on a regular basis and be the "go to point" for announcements and updates. This site should be linked to as many other S T web sites as possible.

IIS in consultation with ISTPCanadaJanuary 2009 
Recommendation #2:

At the time of the next evaluation, a review of the ISTPP ARAF structure should also be undertaken.

IIS agrees with this recommendation and will address relevant issues during the Summative Evaluation planned for the Fall of 2009.IISDecember 2009

(TBC)

 
Recommendation #3:
In monitoring and evaluation of the program, ISTPP should assess the nature of the incrimentality of the program to determine: a) whether the program is fostering international S T partnerships and collaborative research in the supported sector (with emphasis on industry-industry and university-university partnerships); and b) the extent to which the program is contributing to increased R D in a commercial context.
At the time the Formative Evaluation was undertaken, no research projects had been selected and funded.

a) The international partnerships and sector linkages will be clear once research projects have been identified. This will facilitate the assessment of the incrimentality of the program.

b) Again, this will be examined more thoroughly after research projects have commenced and even, upon completion, as some projects may not reach the commercialization stage until many years into the future.

It is noted that these issues are identified in the ARAF and will be measured during the final evaluation, however to prepare and have data readily available, we will explore an opportunity to bring some successful project participants together for: exchange of information; benefit of funding; increase in international partners; lessons learned or best practices, etc. This could be accomplished by having a tri-country workshop, Canada-India-China which would also provide other S T researchers and stakeholders to interact first-hand and understand the benefit of the program.

IIS in consultation with ISTPCanadaFall 2009 (Mid term through projects) 
Recommendation #4:

Clarify the roles of expert coordinators.

The role of the Expert Coordinators is an important function which increases the quality of the S T bilateral relationships by helping to foster partnerships and contributing to the development and implementation of a S T action plan, in addition to the promotion of the ISTPP. To ensure all Expert Coordinators have a clear understanding of their role and responsibilities, an "Expert Coordinator Guide" will be created following a workshop which will be organized to bring together the Expert Coordinators from both the China and India files. The Guide would then be used in additional S T bilateral relationships. This would build on established practices used in other jurisdictions, such as the European Union's Framework Program's network of National Contact Points.IIS in partnership with Expert Coordinators and ISTPCanadaNovember 2008 

Top of Page


1 Signed in January 16, 2007 with China, and November 18, 2005 with India. The agreement with the Canada-Israel Research and Development Foundation (CIIRDF) was renewed on March 16, 2006.

2 Formative Evaluation of the ISTPP Terms of Reference, DFAIT (2008).

3 Although federal scientists are not eligible, as opposed to business and university researchers, each department in the federal government works with private sector organizations that may have an interest in ISTPP.

4 The evaluation covers only the China and India components of the ISTPP; it excludes Brazil, as a contribution agreement has not yet been signed and excludes Israel, as a summative evaluation of CIIRDF was conducted in September, 2004.

5 Signed in January 16, 2007 with China, and November 18, 2005 with India. The agreement with the Canada-Israel Research and Development Foundation (CIIRDF) was renewed on March 16, 2006. A contribution agreement with Brazil has not yet been signed.

6 ISTP program overview documents for Canada-China and Canada-India

7 DFAIT Accountability, Risk and Audit Framework (ARAF)

8  For China, funding and other services are provided through the Ministry of Science and Technology (MOST); for India, they are provided through the Global Innovation Technology Alliance (GITA), a non-governmental organization, engaged by the Department of Science and Technology (DST), Government of India.

9 As detailed in the ISTPP Accountability, Risk and Audit Framework.

10 The National Science Advisor position and his Office were phased out by the government on March 31, 2008.

11 http://www.budget.gc.ca/2007/bp/bpc5ce.html

12 From PowerPoint deck "A Proactive DFAIT Innovation Strategy," Innovation and Partnerships Bureau (February, 2008).

13 Technology sectors for China include: Hydrogen and Fuel Cells Energy, Health and Life Sciences/Biotechnology, Agricultural Foods and Bio-products, Environment. Technology sectors for India include: Biotechnology, Health Research Medical Devices, Earth Sciences Disaster Management, Information Communications Technology (Infotainment Multimedia), Nanoscience and Nanomedicine, Sustainable Alternate Energy Environmental Technologies, Photonics, Synchroton Science and Aerospace.

14 At the time of the evaluation, IRAP had stated its intent to participate in the ISTPP more actively in the future. ISTPCanada recently entered into an agreement with IRAP to provide Canadian SMEs with the following three services: provision of information on similar or complementary technology under development by SMEs or institutes in key countries/markets; assessment of and linkages to partners abroad, which the SMEs will have identified. This will be achieved through ISTPCanada's working relationship with counterparts in other countries; and pro-actively promoting and assisting in the design of the collaborative R D project, including the organization of reciprocal visits.

15 Or if so stipulated in the bilateral arrangement with the partner country.

16 Although federal scientists are not eligible, as opposed to businesses and university researcher, each department in the federal government works with private sector organizations that may have an interest in ISTPP.

Office of the Inspector General


* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Footer

Date Modified:
2013-01-21