Foreign Affairs and International Trade Canada

international.gc.ca

Common menu bar

• Français
• Home
• Contact Us
• Help
• Search
• canada.gc.ca

Breadcrumb

1. Home
2. >
3. About the Department
4. >
5. Accountability

Summative Evaluation - Going Global Science and Technology Program

(June 2008 )

(PDF Version, 425 KB) ^*

• List of Acronyms
• Executive Summary
  • 1.0 Program Profile
    • 1.1 Context
    • 1.2 Objectives and Expected Outcomes
    • 1.3 Stakeholders and Beneficiaries
    • 1.4 Resources
    • 1.5 Application Process
    • 1.6 Context of the Evaluation
  • 2.0 Evaluation Issues and Methodology
    • 2.1 Evaluation Objectives
    • 2.2 Approach and Methodology
    • 2.3 Evaluation Limitations
  • 3.0 Relevance
    • 3.1 Assessment
    • 3.2 Conclusion on the Relevance
  • 4.0 Success
    • 4.1 Assessment
    • 4.2 Conclusion on the Success
  • 5.0 Efficiency and Cost-effectiveness
    • 5.1 Assessment
    • 5.2 Conclusion on the Efficiency and Cost-Effectiveness
  • 6.0 Lessons
  • 7.0 Conclusions and Recommendations
• Appendix A: Management Response

List of Acronyms

ARAF

Accountability, Risk and Audit Framework

CISP

Community Investment Support Program

DFAIT

Department of Foreign Affairs and International Trade

GCS

Global Commerce Strategy

GG

Going Global Program

IIS

Innovation, Science and Technology Division

IIT

Investment, Innovation and Sectors Branch

ISOP

International Strategic Opportunities Program

JSTF

Japan Science and Technology Fund

PEMD-A

Program for Export Market Development - Associations

PWGSC

Public Works and Government Services Canada

R D

Research and development

S T

Science and technology

STEP

Science and Technology with European Partners

TCS

Trade Commissioner Service

Executive Summary

Introduction

The Evaluation Division of the Office of the Inspector General at the Department of Foreign Affairs and International Trade conducted this summative evaluation of the Going Global Program (GG) beginning in January 2007. The evaluation was intended to inform a Treasury Board submission seeking a joint renewal of the Going Global Program, the Community Investment Support Program and the Program for Export Market Development - Associations as components of DFAIT's Global Commerce Strategy.

The Going Global Program is a federal contribution program with an annual budget of $390,000. The objective of GG is to promote and enhance Canada's international science and technology efforts by supporting Canadian researchers in accessing international R D collaborations through partnerships with counterparts in other countries. GG is managed and coordinated by a small team in Ottawa with some support from the regional offices of the Trade Commissioner Service in Canada. The Program funds up to 50 percent of total eligible project costs, with a minimum eligible project cost of $16,000. The maximum contribution payable to any organization for an approved project is $50,000.

The objective of this evaluation was to assess the relevance, success, efficiency and cost-effectiveness of GG. It covers the period from 2001 to 2007.

The evaluation methodology included collaboration with a small evaluation group, document review, and seven semi-structured interviews with DFAIT staff at headquarters and regional offices as well as the administration of an evidence-based questionnaire.

Key Findings

Relevance

GG is relevant to the Government of Canada's priorities in S T and within the realm of international trade. It is the only Program that meets a strong demand from the Canadian research community for flexible financial support in the early stages of solidifying potential international partnerships or collaborations for future R D initiatives. It is well aligned with elements of the Global Commerce Strategy. GG is also relevant to the work of the Trade Commissioner Service both in learning about activity in research sectors and in offering other government services for R D and commercialization.

Success

Given the shortcomings of GG's performance measurement system, complete information about the achievement of results is not available from Program documentation. Nonetheless, the data gathered during the course of the evaluation show results being achieved, especially immediate level outcomes. Chief among these are enhanced networking for international R D opportunities and new collaborative projects. Most funding recipients noted the important contribution that GG had made to the formation of partnerships. This is a prerequisite to launching new collaborative projects. Many funding recipients also noted how GG's financial support had enabled their organizations to enhance their profile among the leading researchers in their field globally.

There is also some evidence of intermediate level outcomes (i.e. inflow of new knowledge to Canada) and more limited evidence of final level outcomes (i.e. increased foreign market penetration for Canadian products and services). The Program has contributed to positive unexpected results and shows signs of sustainability. Barriers to further success include a lack of federal funding mechanisms for other stages in the R D process, the restrictions of GG with respect to eligible project costs, and a lack of awareness about GG.

Efficiency and cost-effectiveness

The incomplete performance reporting framework and the lack of monitoring of results are the major shortcomings of the Program in relation to its effectiveness. Although the Program's Accountability, Risk and Audit Framework has a partial performance monitoring system, it fails to link results statements to measurable indicators. The lack of a post-project mechanism for regular measurement of results means that Program staff have no clear picture of how well the Program is performing and what results are being achieved, especially at the upper levels of expected results.

The staffing structure is unsustainable and leaves the Program vulnerable to disruption, although it has not hampered GG's effectiveness to date. The Program's moderately high administration cost is an issue that requires ongoing monitoring, especially with a planned amalgamation with other programs. With respect to the Program's value compared to its administration cost, the costs must be weighed against the strong relevance of the Program and the significant results achieved. The GG application process needs to be examined to ensure that private companies are not discouraged from applying since their participation can increase the Program's effectiveness in achieving results. The Program is expeditious in processing applications and has high levels of client satisfaction.

Lessons

• Developing international business opportunities is essentially about cultivating relationships. The relationship element is an intangible but necessary step in transforming opportunities into actual research and business collaborations.
• Relatively small amounts (as little as $10,000 or less) of seed money invested in the early stages of R D collaborations can lead to larger initiatives, attract other investments, and produce downstream results.
• Cultivating relationships is a non-linear process and not without risks.
• Since GG supports non-linear processes of relationship building, it poses challenges in applying a linear performance measurement system, such as results-based management.
• Due diligence for GG projects is complex.
• Program administration is too demanding for a single staff person.

Conclusions and Recommendations

GG is a relevant Program that responds to Government of Canada priorities in S T and international trade as well as the needs of the Canadian research community. Data gathered for this evaluation point to significant achievements, including positive unexpected results. There is also some evidence of the Program's sustainability. The Program is generally well managed and delivered, with the important exception of its incomplete performance reporting system. In order to address this and other challenges, the evaluators recommend:

1. That DFAIT proceed to expand funding for GG, given the relevance of the Program and the growing demand.
2. That program management take steps to strengthen performance measurement (primarily by revising the performance monitoring system to link results statements to measurable indicators, and by establishing formal mechanisms for regular follow-up with funding recipients to identify post-project results).
3. That program management examine the staffing structure to reduce administrative costs and resolve disruption vulnerabilities.
4. That program management take steps to make the best use of regional TCS staff in program delivery.
5. That program management modify the application form and criteria in order to encourage applications from research groups in the private sector.

Top of Page

1.0 Program Profile

1.1 Context

The Going Global Program (GG) will continue to support Canadian researchers in accessing international R D collaborative opportunities by providing funding for the development of partnerships with key players in other countries/economies.

GG fulfills a unique niche by being the only federal contribution program that provides early-stage financing to Canadian researchers for the specific purpose of assisting them in solidifying partnerships with key players in other countries. As stated in the evaluation report of July 2003, "The program's relevance and contribution to international collaboration in R D is undeniable. Interviews conducted with fund recipients, trade commissioners and representatives from other science-based departments and agencies and granting councils underscore the uniqueness of the fund as a start-up fund to facilitate access for Canadian researchers to international research network, and as a platform to promote exploration of international R D collaborative opportunities. GG allows initial contact among researchers and/or companies to gain access to cutting-edge research and technologies that are not available in Canada."

1.2 Objectives and Expected Outcomes

The principal objective GG is:

To promote and enhance Canada's international science and technology efforts by supporting Canadian researchers in accessing international R D collaborative opportunities through the development of partnerships with key players in other countries/economies.

In addition, GG promotes Canada's productivity and prosperity through innovative policies, programs and partnerships designed to increase the international competitiveness of Canadian researchers and of Canadian business.

The Program's expected results are expressed at three levels:

1. Short term outcome:
   1. enhanced networking related to international R opportunities
   2. new collaborative partnerships established between Canadian and foreign researchers
2. Medium term outcome:
   1. Launch collaborative international R D initiatives which:
      1. provide access to the world's best facilities, equipment, talent, knowledge and cost sharing mechanisms
      2. facilitate the acquisition and dissemination of innovative technologies and technological information
      3. increase Canada's competitive standing and enhance its profile as world leader in R D
      4. advance the priorities of Canada with regards to international relationships and field/sector specific interests for advancement of Canadian R D
3. Final outcome: Enable Canadian researchers to succeed in the global marketplace by enhancing the quality and speed of Canadian research activities and the potential for timely innovation and commercialization.

GG supports the strategies outlined in Canada's International Policy Statement (IPS) of April 2005. The IPS clearly reinforces the need for Canada to position itself advantageously within the new global economy by making Canada a magnet for talent and investment and advancing our relationships with select partners. It states that "The Government will strengthen innovation by providing access to early-stage financing for innovative firms, and by actively encouraging university research and the commercialization of new technologies by Canadian companies and research institutions; and we will support international science and technology partnerships between Canadian firms and research institutes and their counterparts in other countries..."

1.3 Stakeholders and Beneficiaries:

Overall responsibility and accountability for GG rests with the Minister of International Trade. Responsibility for program development, implementation and administration at an operational level has been delegated to the Innovation, Science and Technology Division (IIS) of DFAIT who administers the Program.

Recipients/ Beneficiaries: GG supports projects designed to bring together Canadian researchers to foster identified opportunities for collaborative partnerships with foreign researchers. Eligible beneficiaries are Canadian researchers from private companies, universities, and non-government research centres, that require support to solidify partnership opportunities for collaborative R D initiatives, through visits abroad, workshops, or other non-research activities associated with the establishment of collaborative partnerships. Researchers at government research centres (federal, provincial, municipal, Crown Corporations) are not eligible as recipients of the funding, however, they may participate in projects funded by GG provided their expenses are covered by other sources (i.e. they pay for their own expenses).

Exhibit 1: Number of Approved Projects, 2000-01 to 2007-08

Exhibit 2: Number of Approved Projects by Region, 2000-01 to 2007-08

1.4 Resources

The annual budget for contributions under GG is $390,000. Actual disbursements during the year will depend on the individual contributions approved and then the further approval of the specific expenditures submitted by the recipients. Any annual budget that is not used is returned to general revenue and is not carried forward to the following year. No budget is reserved for operations or management costs which are covered by IIS who allocate staff time and other resources, as appropriate, from the Divisional budget.

1.5 Application Process

Canadian researchers from private companies, universities and non-government research centres are invited to submit applications for project funding at any time during the year, but at least six weeks in advance of the beginning of the proposed project. Applicants must complete a form detailing the project they propose to undertake and submit it to the program administrator in IIS at DFAIT headquarters. The application must include:

1. the identification of a project leader;
2. the names and positions of the other Canadian participants and the foreign participants;
3. an executive summary that summarizes the objective of the proposed project;
4. an explanation of how the proposed collaborative activity will solidify a joint R D partnership with the key players in foreign countries;
5. a description of the potential economic benefits to be gained as a result of the project;
6. a description of the complementary knowledge, expertise or technology brought to the collaboration by foreign participants;
7. a description of potential downstream activities and the funding sources being planned for subsequent international collaborative R D initiatives that may result from the partnership; and
8. a budget indicating planned costs and all funding sources.

An initial review of the project application is completed by the program administrator. If there is sufficient information for a formal review, the application is sent to the relevant officers in the division, at post, the geographic divisions, and the regional office for their recommendation on whether to support the project. If the initial review shows that additional information is required, the project applicant is asked to provide the additional information. Once the review process has been completed, the application is sent to the relevant management in IIS for their review and approval on the project. If approved, the Program Manager has the final sign off for project approval. Applicants are then notified (via email or phone) of the decision on the project and, in the case of successful applications, the amount of funding support.

Eligible costs

A Contribution Agreement for successful projects states the general conditions of relationships as well as the obligations and responsibilities of all parties.

The Program operates on a cost-shared basis with approved contributions expected to help defray costs associated with international activity by Canadian researchers. Three funding conditions apply for recipients.

1. Contributions are:
   • expected to serve as seed money to solidify collaborative R D initiatives; and,
   • funds up to 50 percent of total eligible costs,
   • a minimum eligible project cost of $16,000. The maximum contribution payable to any organization for an approved project is $50,000.
2. Eligible expenses include all non-research costs in Canada and abroad related to the establishment of collaborative R D, for example: travel, accommodations, meals (as per Treasury Board guidelines) workshops costs, translation, hospitality, etc.
3. Costs that are not eligible include:
   • actual research costs (stipends to students for R D, technicians, samples, equipment acquisition, etc.);
   • costs of transportation and warehousing of samples and equipment;
   • technology or equipment development costs;
   • salaries and honoraria of participating applicants; and,
   • activities not directed at establishing specific partnerships.

1.6 Context of the Evaluation

This evaluation has been conducted in parallel with a summative evaluation of the Community Investment Support Program (CISP). While GG's Terms and Conditions from Treasury Board are valid until March 2009, this evaluation is intended to feed into a Treasury Board submission that will seek joint renewal for CISP, GG and the Program for Export Market Development - Associations (PEMD-A) as components of the Global Commerce Strategy (GCS), which was approved in 2006 in support of Advantage Canada. This summative evaluation focused on providing management with information to make decisions about the amalgamation of CISP, GG and PEMD-A expected in 2008-09.

Top of Page

2.0 Evaluation Issues and Methodology

2.1 Evaluation Objectives

The summative evaluation focused on the three issues as outlined in the Treasury Board "Evaluation Policy" and "Guide for the Review of Evaluation Reports." These issues are relevance, success, efficiency and cost-effectiveness.

2.2 Approach and Methodology

The evaluation followed a utilization-focused approach⁽¹⁾ that has proven successful in other evaluations undertaken by the evaluators at DFAIT and elsewhere. It is aimed at fostering ownership and effective use of the evaluation by the intended users, primarily DFAIT program staff and management. The evaluation focused on results achieved in relation to performance objectives and endeavoured to provide useful information to assist DFAIT with its planning and decision-making about the future direction of GG.

Toward these ends, the evaluation team worked collaboratively with a small working group composed of the Program Manager, Evaluation Manager and professional evaluators. The working group met twice during the evaluation. A first meeting clarified expectations of stakeholders and allowed the evaluation team to focus the study. The second meeting gave the evaluators an opportunity to present their preliminary findings, discuss the implications and identify any information gaps. The evaluators submitted the draft report with conclusion and recommendations to an Evaluation Advisory Committee for review.

The evaluation team used a combination of the following primary methods to gather data:

• document review including Treasury Board submission, Accountability, Risk and Audit Framework (ARAF), annual reports, audit reports, 2003 evaluation report, Going Global Program web site and promotional materials, application and reporting templates, and documentation on the contribution programs amalgamation initiative;
• semi-structured interviews (either face-to-face or by telephone) with key stakeholders, including the Program Manager and administrator, one former officer, and four regional officers; and
• an evidence-based questionnaire administered to funding recipients from past years to help assess relevance, success, efficiency and cost-effectiveness. The questionnaire was sent to 76 funding recipients since 2001-02. Thirty-two responses were received, for a response rate of 42 percent.⁽²⁾ Ten respondents indicated their willingness to share opinions directly with the evaluators, and some follow-up questions to elicit more depth and detail on certain issues were asked of these respondents.

2.3 Evaluation Limitations

The evaluation team faced no major impediments and benefited from the complete cooperation of headquarters staff and regional Trade Commissioner Service (TCS) offices. Given that this was not a full-scale evaluation, fewer interviews were conducted than would have been the case in a more extensive study. However, the evaluators do not believe that this imposed any limitation in addressing the key evaluation issues.

Top of Page

3.0 Relevance

3.1 Assessment

Specifically:

3.1.1 - Are the GG objectives consistent with the current International Trade priorities?
3.1.2 - Is there a continued need and demand for the Program?

3.1.1 Consistency with international trade priorities

As Exhibit 3 shows, given the outcomes that the GG seeks to achieve, the Program supports three elements of the GCS.

Exhibit 3: Relevance of Going Global Program to Global Commerce Strategy

With respect to the DFAIT Report on Plans and Priorities 2007-08, GG is aligned with the International Trade priority - Greater economic competitiveness for Canada through enhanced commercial engagement, secure market access and targeted support for Canadian business.

GG is relevant to the operations of the TCS in several ways. Trade commissioners located in Canada find the GG a useful tool for approaching the R D community in their region since it offers the possibility of funding to researchers. Whether or not local researchers actually apply to the Program, GG becomes an entry point for client acquisition by regional trade commissioners and for learning about different R D sectors. GG also contributes to the integration function of the TCS since it allows researchers to learn about the range of federal and provincial programs on offer that support R D and international business collaboration.

3.1.2 Continued need and demand for the program

Strength in the demand for GG is demonstrated by the applications received. In 2007 the Program received 78 applications - the highest since the Program started. In comparison, in 2005 the Program received 59 applications. In 2004 there were 36 applications and in 2001 there were 18 applications. The number of application confirms that demand for the Program within the Canadian research community has grown steadily.

Funding recipients say that demand is strong because the Program fills a unique niche in the cycle of R D and product development. There are no other sources of funding for those very early stages of R D collaboration when Canadian researchers are exploring potential partnerships. No other program will help cover the costs of travel to meetings and conferences to scout out the terrain for business collaborations. Moreover, the Program is completely flexible with regard to sectors and countries, whereas other programs provide funding only for initiatives in specific sectors or countries. Responses to the evidence-based questionnaire indicate that GG is one of the few, if not the only, resource for business and relationship development overseas. The response also indicated the importance of GG in supporting small focused international meetings in science. The Program continues to play a critical role in research and development by bringing together potential collaborators under circumstances favouring extensive discussion.

According to interviews conducted for the evaluation, another indicator of the demand for GG is the fact that provinces such as Ontario have designed their own complementary programs. For example, Ontario's International Strategic Opportunities Program (ISOP) is specifically targeted towards new strategic international collaboration that succeeds the initial meetings of the type supported by Going Global Program. ISOP funds successive meetings as well as the cost of setting up a secretariat to manage the international collaboration.

Demand is likely to remain strong, according to respondents to the evidence-based questionnaire. Eighty-six percent said they are certain or likely to seek GG funding in the future.

Among the respondents to the evidence-based questionnaire, 79 percent said that GG is very effective or effective at creating new international R D opportunities. Program staff and regional trade commissioners agree with this assessment. They explained GG's effectiveness by saying that it allows Canadian researchers to gain access to new knowledge outside Canada, and enables them to pursue opportunities to participate in the top conferences and meetings in research sectors. This participation and the contacts made allow them to expand their networks, advance in the system, and do research with the leaders in their specific field.

3.2 Conclusion on the Relevance

The findings from this evaluation lead the evaluators to conclude that GG is relevant since it is the only Program that meets a strong demand from the Canadian research community for flexible financial support in the early stages of exploring potential international business collaborations. It is well aligned with elements of the GCS. GG is a useful tool for the Canadian Trade Commissioner Service both in learning about activity in research sectors and in offering firms and organizations in these sectors services that can assist them in advancing through the stages of R D and commercialization.

Top of Page

4.0 Success

4.1 Assessment

Specifically:

4.1.1 - To what extent has GG met its objectives and expected results? To what extent are project results consistent with program objectives? Has there been an enhanced capacity of Canadian organizations to further their international objectives?
4.1.2 - How successful has the Program been in achieving its expected results?
4.1.3 - What are the challenges / barriers to success?
4.1.4 - Have there been significant intended / unintended impacts?
4.1.5 - What have been the key aspects to achieving success?

4.1.1 Achievement of Objectives

GG makes a significant contribution to the formation of partnerships because it is a source of support for small meetings bringing together top international researchers. These meetings are crucial for establishing new contacts and for allowing in-depth discussions but that are difficult to fund.

Twenty-nine percent of respondents said that GG had contributed to the formation of a joint partnership. Eighty-one percent said GG contributed to their intent to pursue collaborative projects and 32 percent said it contributed to a memorandum of understanding between their organization and an overseas counterpart.⁽³⁾

When asked to cite the most important result from GG funding to their organizations, some funding recipients identified elements that feed into partnership formation. GG funding allowed organizations to build understanding and capacity about working and learning in another country, or permitted them to get their name known in the right circles in industry, academia and research centres. The level of awareness among foreign researchers of Canadian research capacity is an important factor in how willing they are to form partnerships with Canadians. By assisting Canadian researchers to meet with their overseas counterparts, GG helps raise the profile of Canadian research capacity around the world. Funding recipients do report evidence to support this finding. The questionnaire responses brought to light two cases where GG initiatives raised the profile of Canadian research groups and allowed them to attract international conferences to be held in Canada.

4.1.2 Achievement of Expected Results

GG's performance measurement system is incomplete and there is no formal mechanism to systematically track post-project results. Annual reports only provide output level results. Nonetheless, headquarters staff and regional trade commissioners maintain informal contact with past funding participants and are able to cite examples of results. Some of these examples are featured as success stories on the Program's web pages. In addition, the evidence-based questionnaire, as well as interviews with staff, allowed the evaluators to collect further evidence of achievement of expected results.

Most of the evidence points to GG having achieved, or having contributed significantly to the achievement of immediate outcomes. Chief among these are enhanced networking for international R D opportunities for Canadians and the related result of new collaborative projects. Most funding recipients noted the contribution that GG had made to the formation of partnerships. This is considered an intermediate and essential step between networking and launching new collaborative projects. Thus, 36 percent of respondents said that without GG funding partnerships would not have been formed, and 58 percent said that without GG funding partnerships would have taken longer to establish. GG's financial support allowed some Canadian research groups to increase the size of their delegation for overseas missions. This increased their credibility and impressed their foreign counterparts.

At the immediate outcome level GG is raising awareness of S T services offered by the Government of Canada. According to information gathered from staff interviews, GG does help the TCS to educate the Canadian researchers about other government S T services available to them such as the International S T Partnership Program that focuses on initiatives in Brazil, China, India and Israel.

An increased inflow of knowledge not presently available in Canada is an expected result of GG at the intermediate level, and there is some evidence that GG is contributing to it. Among respondents to the evidence-based questionnaire, 65 percent said that GG contributed to knowledge exchange, including knowledge not currently available in the scientific literature. Examples of areas of new knowledge included a microbial system to clean metal wastes, and knowledge gains in ocean sonar technologies and in methods of harvesting commercially valuable seaweed. Another example is the Canada-Japan collaboration on killing chemo-resistant ovarian cancer cells. This initiative, which began with GG seed funding, is featured on GG's web pages.

Increased foreign market penetration for Canadian products and services is the expected result of GG at the final outcome level. This evaluation gathered a small amount of evidence pointing to GG's contribution at this level. About a quarter of the funding recipients queried reported that the partnerships they formed had led to new or improved goods or services being introduced to the market, or to licensing or sales. In some cases, GG support contributed to establishing important business relationships that appeared to have strong potential to lead to contracts in foreign countries. The Program also cites the success of a cyber cartographic atlas of Antarctica developed in collaboration between Canadian researchers and international partners from the USA, Australia, New Zealand, Argentina, China, Norway and the United Kingdom. The original GG investment was modest, but it leveraged more funding, including $2.6 million from the Social Sciences and Humanities Research Council under its Initiative on the New Economy.

4.1.3 Barriers to Achieving Results

Funding recipients and TCS staff identified several barriers to success that could be addressed by modifications to the GG design or by other federal government action. These include:

• The fact that GG includes as an eligible cost the travel of Canadians overseas, but not the travel costs of non-Canadians to Canada. Some TCS staff ask whether these costs should not also be eligible if a Canadian research organization considers them a legitimate need.
• There is a lack of awareness of GG and other funding programs such as the International S T Partnership Program funded through DFAIT. The Program seems to get lost among the hundreds of programs for funding research in Canada.⁽⁴⁾ TCS staff say they still meet people who have no idea that GG exists.
• A lack of federal funding mechanisms for other stages in the process of product and service commercialization. Although GG does help researchers with the early stages of exploring potential partnerships and collaborations, there is a gap in federal support for the later stage of preparing a research grant application. By comparison, TCS staff say that Europeans counterparts are often already funded, but must wait for their Canadian partners to secure their funding.

4.1.4 Unexpected Results

According to TCS staff interviewed, GG has led to the unexpected result of creating awareness within other government departments that DFAIT is contributing to the Government of Canada's innovation agenda. GG provides DFAIT with a greater sense of belonging to the government-wide network on science and technology.

Another unexpected result is the usefulness of GG as an entry tool for regional TCS staff to learn more about the active research sectors in their region. When regional staff contact local research organizations, GG provides them with greater credibility since it gives them something tangible - a funding program - to offer research groups. A related unexpected result is how TCS staff can use GG to help clients achieve further results and coordinate with other federal or provincial support for industrial R D.

4.1.5 Program Sustainability

Many of the results that Canadian researchers have achieved with the support of GG investments will continue. GG contributed to the formation of partnerships between Canadian researchers and overseas counterparts. Many of these partnerships are now established and the partners will likely advance in their R D cycle toward the achievement of the Program's final results, without the need for more GG funding.

Another sign of sustainability is that funding recipients' dependency on the Program appears to be low. The number of years that participants have received funding serves as a proxy measure for dependency. Among the questionnaire respondents by far the majority - 70 percent - have had funding in one year only. Twenty-three percent have received funding over two years, and only two respondents, or seven percent, have had funding over three years.

4.2 Conclusion on the Success

The data gathered during the course of the evaluation show results being achieved, especially at the immediate outcome level. There is also some evidence of results at the intermediate outcome level and more limited evidence of results at the final outcome level. The Program has contributed to positive unexpected results and shows signs of sustainability. Barriers to further success include a lack of federal funding mechanisms for other stages in the R D process, the restrictions of GG with respect to eligible project costs, and a lack of awareness about GG.

Top of Page

5.0 Efficiency and Cost-Effectiveness

5.1 Assessment

Specifically:

5.1.1 - Is there ongoing performance measurement? To what extent are performance measurement frameworks adequate? To what extent does the Program monitor results?
5.1.2 - Is the Program delivered effectively and efficiently, demonstrating good value for money?

Did the results achieved enhance value-for-money?

Specifically:

5.1.3 - Is the administration of the Program beneficial and efficient with regard to outputs and results? Were the Program costs reasonable in light of demonstrated results?
5.1.4 - Are the current Program governance structures adequate?

5.1.1 Performance Measurement and Reporting

GG exhibits many of the challenges experienced by programs in other government departments in performance measurement and results-based management. The Program's performance measurement system is incomplete. The 2006 Accountability, Risk and Audit Framework (ARAF) has a partial performance monitoring system but it fails to link results statements to measurable indicators. The results reporting provided in annual reports does not follow GG's logic model. What are described as outputs in the annual report would be more properly labelled as disbursements. What are described as immediate outcomes - according to the logic model - are results at the output level, which include the approved projects.

The greatest shortcoming in Program operations is the lack of a post-project mechanism for regular measurement of results. Program staff acknowledge that they have no clear picture of how well the Program is performing and what results are being achieved, especially at the intermediate and final outcome levels. The knowledge gap about intermediate and final results exists because there is no formal follow-up mechanisms to find out from funding recipients what happened months or years after the funding relationship with GG has ended. There is informal follow-up on these later results by program staff and regional trade commissioners who are keeping in touch with clients or prospective clients, but it is not systematic. Although it is challenging to determine what results the Program is achieving at the intermediate outcome level, the data gathered through this evaluation demonstrate that it is possible. The establishment of a regular, formal mechanism for inquiring with past funding recipients about intermediate-level results will allow the Program to include this information in its annual reports.

5.1.2 Program Delivery

GG is open to applications year-round. The program manager has set a service standard of replying within six weeks to applications. This service level has been maintained, even in the face of a record number of applications in 2007. This is especially true given that applications are sent to sectoral specialists and to TCS offices in Canada and abroad for comment and review.

The Program also effectively manages disbursements. During the last six fiscal years lapsed funding has declined from 70 percent in the first year to just over 10 percent in 2006-07. Corrective measures such as better risk analysis and closer monitoring of recipient expenditures were taken to control the lapsing of funds.

Exhibit 4: Annual Comparison of Funding Usage, 2001-02 to 2006-07

In its 2005 audit of GG, the audit team found that:

• The contribution Program was managed with sound comptrollership practices and complies with legislative and regulatory requirements.
• Contribution agreements were justified under the appropriate authorities and were in agreement with the requirements of the Treasury Board and departmental Transfer Payment Policies.
• Program funds were managed with due diligence and payments were made in accordance with the policies.
• The contribution files contained the relevant documentation and indicated that proper approvals had been obtained for the contribution payments.

The audit team made no recommendations.

Regional TCS staff say lack sufficient understanding of the Program to properly assist potential applicants in their region. This knowledge gap points to a lack of training on GG for regional staff. The evaluators understood that a training plan is in the works for spring/summer 2008.

Regional TCS staff still encounter people in the research community who are unaware of GG. Regional staff are willing to promote the Program on their own, but promotional materials and communication tools are inadequate. The TCS staff have to put together their own PowerPoint presentations on the Program. Since they have no GG brochure, they print the Program web pages. These are not professional materials suited to marketing the Program in academic or private research circles.

Regional staff also say that communication about the Program from headquarters is insufficient. They feel they are not kept fully in the loop. For instance, they are not informed by headquarters about successful and unsuccessful applicants from their region. Apart from meeting a basic need to be up to date on successful applicants, this information would broaden the knowledge of trade commissioners and allow them to better judge future applications from their regions. GG projects could be registered in the client management database so that regional offices are aware of approved projects and so that results can be tracked.

5.1.3 Administration of the Program

Exhibit 5: Estimated administrative costs of the Going Global Program, 2007-08⁽⁵⁾

Exhibit 5 describes the human resources of DFAIT involved in the Program and the corresponding estimated administrative costs. A 33 percent factor has been applied to direct salary costs to account for employee benefit plans (20 percent) and the PWGSC accommodations charge (13 percent). A number of other indirect program costs were not included.⁽⁶⁾

The total estimated administrative costs of the Program appear moderately high in relation to the budget of the Program, corresponding to an approximate range between 14.0 percent and 19.4 percent of the total contributions provided in 2007-08. This administrative ratio is comparable to those for PEMD-A and CISP.

However, there are two limitations to the administrative ratio calculation:

• The data is based on statements provided either by people occupying the position or on estimations; and,
• Operational staff may allocate more (or less) time to perform the same activity.

Regardless of which scenario is closer to the actual cost, both indicate that the cost of administering the GG is high. The main reason for such a high administration cost is GG's small dollar value. Small programs tend to have high administration costs. However, GG needs to be aware of these costs, monitor them and take steps to reduce them, wherever possible.

The administration cost could be lower if a junior person were attending to the more clerical aspects of program delivery. Two other factors could provide some relief. One would be an increase in the Program budget. The other factor would be the proposed amalgamation of GG with CISP and PEMD-A whereby a joint secretariat would provide administrative support for all three programs. However, neither of these factors would guarantee that administration costs for GG would fall.

The moderately high cost of program administration must be weighed against the strong relevance of the Program to GOC priorities and to the needs of the research community as well as against the available evidence of having achieved significant results.

Based upon the responses to the evidence-based questionnaire, funding recipients expressed high levels of satisfaction with almost all elements of program delivery. High satisfaction levels were expressed for the following program elements: support by local international trade officers, application process, payment process, and reporting requirements. The one element that had a somewhat lower satisfaction rating (combined satisfied/very satisfied score of 72 percent) was level of funding. This finding is supported by comments by some questionnaire respondents to the effect that funding amounts can be so small that they are almost not worth consideration, given the cost to the organization of applying for and administering its project.

Other proxy measures for client satisfaction include number of participants who will reapply for funding and number of participants who say the GG was effective in creating new R D opportunities.

• Seventy-nine percent said GG was effective in creating new R D opportunities; and,
• Fifty-five percent said they will certainly seek GG funding in the future.⁽⁷⁾

This finding is also consistent with increasing demand for the Program and the results achieved.

5.1.4 Program Governance

The Program is currently run almost entirely by a single staff person with quite minimal administrative support. This situation is unsustainable given the higher level of effort that has been required as a result of growth in demand in recent years and the anticipated future growth. Although the files are in order, the staffing structure leaves the Program vulnerable to disruption should the current administrator leave for any reason. A staffing structure that comprised a manager and a junior-level assistant would better distribute the workload and reduce vulnerability to disruption, since the Program could be kept running temporarily by either person.

There is also a cost and efficiency issue associated with the present staffing structure. It is inefficient and costly for the program manager to be handling many administrative and clerical tasks. Having a junior person on staff would allow for a more appropriate division of labour, which in turn could lower administrative costs and free up the time of the current manager for other duties. As matters stand now, the manager is too preoccupied with client relations, processing applications and administering contributions to fully attend to other aspects of program management such as performance measurement, analysis, and promotion.

Both headquarters and TCS staff acknowledge that the application and eligibility guidelines for GG are more geared to the academia community as opposed to researchers in the private sector. TCS staff report that applicants from private firms have more difficulty than university-based researchers in completing the application. One stumbling block for private companies is that the Program requires them to publicly share technology arising from the project. Companies generally try to protect keep new technologies and intellectual property to themselves and away from competitors. Private firms usually want to be the only company involved in a project.

An application process that favours university projects goes against the plans expressed by GG managers to attract more business applications. In terms of success in achieving the expected final program results, it does make sense for the Program to seek to approve more business applications. According to regional TCS staff, business projects are typically closer to the commercialization stage when compared to academic projects.

5.2 Conclusion on the Efficiency and Cost-Effectiveness

The incomplete performance reporting framework and the lack of monitoring of results are the major shortcoming of the Program in relation to its effectiveness. The staffing structure is unsustainable and leaves the Program vulnerable to disruption, although it has not hampered GG's effectiveness to date. The Program's moderately high administration cost is an issue that requires ongoing monitoring, especially with a planned amalgamation with other programs. With respect to the Program's value compared to its high administration cost, the costs must be weighed against the relevance of the Program and the success of results achieved. The GG application process needs to be examined to ensure that private companies are not discouraged from applying since their participation can increase the Program's effectiveness in achieving results. The Program is expeditious in processing applications and has high levels of client satisfaction.

Top of Page

6.0 Lessons

The evaluation brings to light the following lessons:

• Developing international business opportunities is essentially about cultivating relationships. The relationship element is an intangible but necessary step in transforming opportunities into actual research and business collaborations.
• Relatively small amounts (as little as $10,000 or less) of seed money invested in the early stages of R D collaborations can lead to larger initiatives, attract other investments, and produce downstream results.
• Cultivating relationships is a non-linear process and not without risks.

Relationships in research and business do not necessarily follow straight paths. Some leads turn out to be dead ends while others require shifts in direction. When the R D process is understood in this way, learning what the range of opportunities are and ruling out certain avenues that lack promise is an achievement in itself and a sign of adaptive capacity.

• Since GG supports non-linear processes of relationship building, it poses challenges in applying a linear performance measurement system, such as results-based management.

This lesson points to the need for periodic evaluation of a qualitative nature since the Program's internal performance measurement system cannot provide complete clarity about what is happening.

• Due diligence for Going Global Program projects is complex.

Applications must be reviewed by sectoral specialists in complex research domains and by TCS staff at regional offices in Canada and overseas.

• Program administration is too demanding for a single staff person.

Given the large number of small contributions being managed in this type of program, the time and level of effort required create too high of a burden for a single staff person.

Top of Page

7.0 Conclusions and Recommendations

The evaluation findings indicate that GG is relevant to Government of Canada priorities in S T and in international trade. Further, there is evidence that the Program also responds to the needs of the Canadian research community for financial support in the early stages of solidifying relationships with international R D collaborators.

The incomplete performance reporting framework hampers GG's effectiveness and its ability to measure results. The staffing structure is not sustainable and leaves the Program vulnerable to disruption. The Program's relatively high administration cost is an issue that requires ongoing monitoring. The high administration cost must be weighed against the Program's relevance and achievements. The GG application process needs to ensure that private companies are not discouraged from applying. Expeditious processing of applications and high levels of client satisfaction point to efficiency in the program's management.

Data gathered for this evaluation point to significant Program achievements, including positive unexpected results. There is also some evidence of the program's sustainability.

Recommendations

Top of Page

Appendix A: Management Response

Top of Page

¹ Patton, M.Q. (1997). Utilization-Focused Evaluation. 3

^rd Edition. Thousand Oaks, California: Sage.

² At a 95% confidence level, this means that 95 times out of 100 the true value will fall within a 12-point range of the response rate.

³ Respondents could choose multiple reponses so that these percentages wold exceed 100%.

⁴ The Government of Canada's Funding Guide for International S T Cooperation lists 660 sources of grants, bursaries, post-doctoral fellowships, prizes and awards, including GG.

⁵ Rates of pay are according to Treasury Board. The effective dates for each pay category are as follows: EX (April 1, 2007); CO (June 22, 2006); and AS (June 21, 2006).

⁶ Other indirect program costs that were not included include the following: legal (i.e. advice), finance (i.e. payments, planning, budgeting), human resources (i.e. staffing and training), audit, evaluation, IT systems support, communications (i.e. telephone), general reception, etc.

⁷ Whereas fifty-five percent of the survey respondents say that they would certainly seek GG funding and thirty percent reported receive funding for more than one year, fewer may actually reapply. Reasons for not reapplying may vary but are thought to include such things as: projects are no longer viable, research may become obsolete, or other researchers may develop the same technology and obtain patents. GG may also have exhausted its funding budget and not able to fund an application.

Office of the Inspector General

^* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Primary navigation

Date Modified:

2013-01-08