The Evaluation Division of the Office of the Inspector General at the Department of Foreign Affairs and International Trade conducted this summative evaluation of the Going Global Program (GG) beginning in January 2007. The evaluation was intended to inform a Treasury Board submission seeking a joint renewal of the Going Global Program, the Community Investment Support Program and the Program for Export Market Development - Associations as components of DFAIT's Global Commerce Strategy.
The Going Global Program is a federal contribution program with an annual budget of $390,000. The objective of GG is to promote and enhance Canada's international science and technology efforts by supporting Canadian researchers in accessing international R D collaborations through partnerships with counterparts in other countries. GG is managed and coordinated by a small team in Ottawa with some support from the regional offices of the Trade Commissioner Service in Canada. The Program funds up to 50 percent of total eligible project costs, with a minimum eligible project cost of $16,000. The maximum contribution payable to any organization for an approved project is $50,000.
The objective of this evaluation was to assess the relevance, success, efficiency and cost-effectiveness of GG. It covers the period from 2001 to 2007.
The evaluation methodology included collaboration with a small evaluation group, document review, and seven semi-structured interviews with DFAIT staff at headquarters and regional offices as well as the administration of an evidence-based questionnaire.
GG is relevant to the Government of Canada's priorities in S T and within the realm of international trade. It is the only Program that meets a strong demand from the Canadian research community for flexible financial support in the early stages of solidifying potential international partnerships or collaborations for future R D initiatives. It is well aligned with elements of the Global Commerce Strategy. GG is also relevant to the work of the Trade Commissioner Service both in learning about activity in research sectors and in offering other government services for R D and commercialization.
Given the shortcomings of GG's performance measurement system, complete information about the achievement of results is not available from Program documentation. Nonetheless, the data gathered during the course of the evaluation show results being achieved, especially immediate level outcomes. Chief among these are enhanced networking for international R D opportunities and new collaborative projects. Most funding recipients noted the important contribution that GG had made to the formation of partnerships. This is a prerequisite to launching new collaborative projects. Many funding recipients also noted how GG's financial support had enabled their organizations to enhance their profile among the leading researchers in their field globally.
There is also some evidence of intermediate level outcomes (i.e. inflow of new knowledge to Canada) and more limited evidence of final level outcomes (i.e. increased foreign market penetration for Canadian products and services). The Program has contributed to positive unexpected results and shows signs of sustainability. Barriers to further success include a lack of federal funding mechanisms for other stages in the R D process, the restrictions of GG with respect to eligible project costs, and a lack of awareness about GG.
The incomplete performance reporting framework and the lack of monitoring of results are the major shortcomings of the Program in relation to its effectiveness. Although the Program's Accountability, Risk and Audit Framework has a partial performance monitoring system, it fails to link results statements to measurable indicators. The lack of a post-project mechanism for regular measurement of results means that Program staff have no clear picture of how well the Program is performing and what results are being achieved, especially at the upper levels of expected results.
The staffing structure is unsustainable and leaves the Program vulnerable to disruption, although it has not hampered GG's effectiveness to date. The Program's moderately high administration cost is an issue that requires ongoing monitoring, especially with a planned amalgamation with other programs. With respect to the Program's value compared to its administration cost, the costs must be weighed against the strong relevance of the Program and the significant results achieved. The GG application process needs to be examined to ensure that private companies are not discouraged from applying since their participation can increase the Program's effectiveness in achieving results. The Program is expeditious in processing applications and has high levels of client satisfaction.
GG is a relevant Program that responds to Government of Canada priorities in S T and international trade as well as the needs of the Canadian research community. Data gathered for this evaluation point to significant achievements, including positive unexpected results. There is also some evidence of the Program's sustainability. The Program is generally well managed and delivered, with the important exception of its incomplete performance reporting system. In order to address this and other challenges, the evaluators recommend:
The Going Global Program (GG) will continue to support Canadian researchers in accessing international R D collaborative opportunities by providing funding for the development of partnerships with key players in other countries/economies.
GG fulfills a unique niche by being the only federal contribution program that provides early-stage financing to Canadian researchers for the specific purpose of assisting them in solidifying partnerships with key players in other countries. As stated in the evaluation report of July 2003, "The program's relevance and contribution to international collaboration in R D is undeniable. Interviews conducted with fund recipients, trade commissioners and representatives from other science-based departments and agencies and granting councils underscore the uniqueness of the fund as a start-up fund to facilitate access for Canadian researchers to international research network, and as a platform to promote exploration of international R D collaborative opportunities. GG allows initial contact among researchers and/or companies to gain access to cutting-edge research and technologies that are not available in Canada."
The principal objective GG is:
To promote and enhance Canada's international science and technology efforts by supporting Canadian researchers in accessing international R D collaborative opportunities through the development of partnerships with key players in other countries/economies.
In addition, GG promotes Canada's productivity and prosperity through innovative policies, programs and partnerships designed to increase the international competitiveness of Canadian researchers and of Canadian business.
The Program's expected results are expressed at three levels:
GG supports the strategies outlined in Canada's International Policy Statement (IPS) of April 2005. The IPS clearly reinforces the need for Canada to position itself advantageously within the new global economy by making Canada a magnet for talent and investment and advancing our relationships with select partners. It states that "The Government will strengthen innovation by providing access to early-stage financing for innovative firms, and by actively encouraging university research and the commercialization of new technologies by Canadian companies and research institutions; and we will support international science and technology partnerships between Canadian firms and research institutes and their counterparts in other countries..."
Overall responsibility and accountability for GG rests with the Minister of International Trade. Responsibility for program development, implementation and administration at an operational level has been delegated to the Innovation, Science and Technology Division (IIS) of DFAIT who administers the Program.
Recipients/ Beneficiaries: GG supports projects designed to bring together Canadian researchers to foster identified opportunities for collaborative partnerships with foreign researchers. Eligible beneficiaries are Canadian researchers from private companies, universities, and non-government research centres, that require support to solidify partnership opportunities for collaborative R D initiatives, through visits abroad, workshops, or other non-research activities associated with the establishment of collaborative partnerships. Researchers at government research centres (federal, provincial, municipal, Crown Corporations) are not eligible as recipients of the funding, however, they may participate in projects funded by GG provided their expenses are covered by other sources (i.e. they pay for their own expenses).
The annual budget for contributions under GG is $390,000. Actual disbursements during the year will depend on the individual contributions approved and then the further approval of the specific expenditures submitted by the recipients. Any annual budget that is not used is returned to general revenue and is not carried forward to the following year. No budget is reserved for operations or management costs which are covered by IIS who allocate staff time and other resources, as appropriate, from the Divisional budget.
Canadian researchers from private companies, universities and non-government research centres are invited to submit applications for project funding at any time during the year, but at least six weeks in advance of the beginning of the proposed project. Applicants must complete a form detailing the project they propose to undertake and submit it to the program administrator in IIS at DFAIT headquarters. The application must include:
An initial review of the project application is completed by the program administrator. If there is sufficient information for a formal review, the application is sent to the relevant officers in the division, at post, the geographic divisions, and the regional office for their recommendation on whether to support the project. If the initial review shows that additional information is required, the project applicant is asked to provide the additional information. Once the review process has been completed, the application is sent to the relevant management in IIS for their review and approval on the project. If approved, the Program Manager has the final sign off for project approval. Applicants are then notified (via email or phone) of the decision on the project and, in the case of successful applications, the amount of funding support.
A Contribution Agreement for successful projects states the general conditions of relationships as well as the obligations and responsibilities of all parties.
The Program operates on a cost-shared basis with approved contributions expected to help defray costs associated with international activity by Canadian researchers. Three funding conditions apply for recipients.
This evaluation has been conducted in parallel with a summative evaluation of the Community Investment Support Program (CISP). While GG's Terms and Conditions from Treasury Board are valid until March 2009, this evaluation is intended to feed into a Treasury Board submission that will seek joint renewal for CISP, GG and the Program for Export Market Development - Associations (PEMD-A) as components of the Global Commerce Strategy (GCS), which was approved in 2006 in support of Advantage Canada. This summative evaluation focused on providing management with information to make decisions about the amalgamation of CISP, GG and PEMD-A expected in 2008-09.
The summative evaluation focused on the three issues as outlined in the Treasury Board "Evaluation Policy" and "Guide for the Review of Evaluation Reports." These issues are relevance, success, efficiency and cost-effectiveness.
The evaluation followed a utilization-focused approach(1) that has proven successful in other evaluations undertaken by the evaluators at DFAIT and elsewhere. It is aimed at fostering ownership and effective use of the evaluation by the intended users, primarily DFAIT program staff and management. The evaluation focused on results achieved in relation to performance objectives and endeavoured to provide useful information to assist DFAIT with its planning and decision-making about the future direction of GG.
Toward these ends, the evaluation team worked collaboratively with a small working group composed of the Program Manager, Evaluation Manager and professional evaluators. The working group met twice during the evaluation. A first meeting clarified expectations of stakeholders and allowed the evaluation team to focus the study. The second meeting gave the evaluators an opportunity to present their preliminary findings, discuss the implications and identify any information gaps. The evaluators submitted the draft report with conclusion and recommendations to an Evaluation Advisory Committee for review.
The evaluation team used a combination of the following primary methods to gather data:
The evaluation team faced no major impediments and benefited from the complete cooperation of headquarters staff and regional Trade Commissioner Service (TCS) offices. Given that this was not a full-scale evaluation, fewer interviews were conducted than would have been the case in a more extensive study. However, the evaluators do not believe that this imposed any limitation in addressing the key evaluation issues.
To determine the contribution of the Going Global Program in support of Canada's Global Commerce Strategy and DFAIT's strategic priorities.
3.1.1 - Are the GG objectives consistent with the current International Trade priorities?
3.1.2 - Is there a continued need and demand for the Program?
As Exhibit 3 shows, given the outcomes that the GG seeks to achieve, the Program supports three elements of the GCS.
|GG Outcomes||GCS Elements Supported by GG|
With respect to the DFAIT Report on Plans and Priorities 2007-08, GG is aligned with the International Trade priority - Greater economic competitiveness for Canada through enhanced commercial engagement, secure market access and targeted support for Canadian business.
GG is relevant to the operations of the TCS in several ways. Trade commissioners located in Canada find the GG a useful tool for approaching the R D community in their region since it offers the possibility of funding to researchers. Whether or not local researchers actually apply to the Program, GG becomes an entry point for client acquisition by regional trade commissioners and for learning about different R D sectors. GG also contributes to the integration function of the TCS since it allows researchers to learn about the range of federal and provincial programs on offer that support R D and international business collaboration.
Strength in the demand for GG is demonstrated by the applications received. In 2007 the Program received 78 applications - the highest since the Program started. In comparison, in 2005 the Program received 59 applications. In 2004 there were 36 applications and in 2001 there were 18 applications. The number of application confirms that demand for the Program within the Canadian research community has grown steadily.
Funding recipients say that demand is strong because the Program fills a unique niche in the cycle of R D and product development. There are no other sources of funding for those very early stages of R D collaboration when Canadian researchers are exploring potential partnerships. No other program will help cover the costs of travel to meetings and conferences to scout out the terrain for business collaborations. Moreover, the Program is completely flexible with regard to sectors and countries, whereas other programs provide funding only for initiatives in specific sectors or countries. Responses to the evidence-based questionnaire indicate that GG is one of the few, if not the only, resource for business and relationship development overseas. The response also indicated the importance of GG in supporting small focused international meetings in science. The Program continues to play a critical role in research and development by bringing together potential collaborators under circumstances favouring extensive discussion.
According to interviews conducted for the evaluation, another indicator of the demand for GG is the fact that provinces such as Ontario have designed their own complementary programs. For example, Ontario's International Strategic Opportunities Program (ISOP) is specifically targeted towards new strategic international collaboration that succeeds the initial meetings of the type supported by Going Global Program. ISOP funds successive meetings as well as the cost of setting up a secretariat to manage the international collaboration.
Demand is likely to remain strong, according to respondents to the evidence-based questionnaire. Eighty-six percent said they are certain or likely to seek GG funding in the future.
Among the respondents to the evidence-based questionnaire, 79 percent said that GG is very effective or effective at creating new international R D opportunities. Program staff and regional trade commissioners agree with this assessment. They explained GG's effectiveness by saying that it allows Canadian researchers to gain access to new knowledge outside Canada, and enables them to pursue opportunities to participate in the top conferences and meetings in research sectors. This participation and the contacts made allow them to expand their networks, advance in the system, and do research with the leaders in their specific field.
The findings from this evaluation lead the evaluators to conclude that GG is relevant since it is the only Program that meets a strong demand from the Canadian research community for flexible financial support in the early stages of exploring potential international business collaborations. It is well aligned with elements of the GCS. GG is a useful tool for the Canadian Trade Commissioner Service both in learning about activity in research sectors and in offering firms and organizations in these sectors services that can assist them in advancing through the stages of R D and commercialization.
4.1.1 - To what extent has GG met its objectives and expected results? To what extent are project results consistent with program objectives? Has there been an enhanced capacity of Canadian organizations to further their international objectives?
4.1.2 - How successful has the Program been in achieving its expected results?
4.1.3 - What are the challenges / barriers to success?
4.1.4 - Have there been significant intended / unintended impacts?
4.1.5 - What have been the key aspects to achieving success?
GG makes a significant contribution to the formation of partnerships because it is a source of support for small meetings bringing together top international researchers. These meetings are crucial for establishing new contacts and for allowing in-depth discussions but that are difficult to fund.
Twenty-nine percent of respondents said that GG had contributed to the formation of a joint partnership. Eighty-one percent said GG contributed to their intent to pursue collaborative projects and 32 percent said it contributed to a memorandum of understanding between their organization and an overseas counterpart.(3)
When asked to cite the most important result from GG funding to their organizations, some funding recipients identified elements that feed into partnership formation. GG funding allowed organizations to build understanding and capacity about working and learning in another country, or permitted them to get their name known in the right circles in industry, academia and research centres. The level of awareness among foreign researchers of Canadian research capacity is an important factor in how willing they are to form partnerships with Canadians. By assisting Canadian researchers to meet with their overseas counterparts, GG helps raise the profile of Canadian research capacity around the world. Funding recipients do report evidence to support this finding. The questionnaire responses brought to light two cases where GG initiatives raised the profile of Canadian research groups and allowed them to attract international conferences to be held in Canada.
GG's performance measurement system is incomplete and there is no formal mechanism to systematically track post-project results. Annual reports only provide output level results. Nonetheless, headquarters staff and regional trade commissioners maintain informal contact with past funding participants and are able to cite examples of results. Some of these examples are featured as success stories on the Program's web pages. In addition, the evidence-based questionnaire, as well as interviews with staff, allowed the evaluators to collect further evidence of achievement of expected results.
Most of the evidence points to GG having achieved, or having contributed significantly to the achievement of immediate outcomes. Chief among these are enhanced networking for international R D opportunities for Canadians and the related result of new collaborative projects. Most funding recipients noted the contribution that GG had made to the formation of partnerships. This is considered an intermediate and essential step between networking and launching new collaborative projects. Thus, 36 percent of respondents said that without GG funding partnerships would not have been formed, and 58 percent said that without GG funding partnerships would have taken longer to establish. GG's financial support allowed some Canadian research groups to increase the size of their delegation for overseas missions. This increased their credibility and impressed their foreign counterparts.
At the immediate outcome level GG is raising awareness of S T services offered by the Government of Canada. According to information gathered from staff interviews, GG does help the TCS to educate the Canadian researchers about other government S T services available to them such as the International S T Partnership Program that focuses on initiatives in Brazil, China, India and Israel.
An increased inflow of knowledge not presently available in Canada is an expected result of GG at the intermediate level, and there is some evidence that GG is contributing to it. Among respondents to the evidence-based questionnaire, 65 percent said that GG contributed to knowledge exchange, including knowledge not currently available in the scientific literature. Examples of areas of new knowledge included a microbial system to clean metal wastes, and knowledge gains in ocean sonar technologies and in methods of harvesting commercially valuable seaweed. Another example is the Canada-Japan collaboration on killing chemo-resistant ovarian cancer cells. This initiative, which began with GG seed funding, is featured on GG's web pages.
Increased foreign market penetration for Canadian products and services is the expected result of GG at the final outcome level. This evaluation gathered a small amount of evidence pointing to GG's contribution at this level. About a quarter of the funding recipients queried reported that the partnerships they formed had led to new or improved goods or services being introduced to the market, or to licensing or sales. In some cases, GG support contributed to establishing important business relationships that appeared to have strong potential to lead to contracts in foreign countries. The Program also cites the success of a cyber cartographic atlas of Antarctica developed in collaboration between Canadian researchers and international partners from the USA, Australia, New Zealand, Argentina, China, Norway and the United Kingdom. The original GG investment was modest, but it leveraged more funding, including $2.6 million from the Social Sciences and Humanities Research Council under its Initiative on the New Economy.
Funding recipients and TCS staff identified several barriers to success that could be addressed by modifications to the GG design or by other federal government action. These include:
According to TCS staff interviewed, GG has led to the unexpected result of creating awareness within other government departments that DFAIT is contributing to the Government of Canada's innovation agenda. GG provides DFAIT with a greater sense of belonging to the government-wide network on science and technology.
Another unexpected result is the usefulness of GG as an entry tool for regional TCS staff to learn more about the active research sectors in their region. When regional staff contact local research organizations, GG provides them with greater credibility since it gives them something tangible - a funding program - to offer research groups. A related unexpected result is how TCS staff can use GG to help clients achieve further results and coordinate with other federal or provincial support for industrial R D.
Many of the results that Canadian researchers have achieved with the support of GG investments will continue. GG contributed to the formation of partnerships between Canadian researchers and overseas counterparts. Many of these partnerships are now established and the partners will likely advance in their R D cycle toward the achievement of the Program's final results, without the need for more GG funding.
Another sign of sustainability is that funding recipients' dependency on the Program appears to be low. The number of years that participants have received funding serves as a proxy measure for dependency. Among the questionnaire respondents by far the majority - 70 percent - have had funding in one year only. Twenty-three percent have received funding over two years, and only two respondents, or seven percent, have had funding over three years.
The data gathered during the course of the evaluation show results being achieved, especially at the immediate outcome level. There is also some evidence of results at the intermediate outcome level and more limited evidence of results at the final outcome level. The Program has contributed to positive unexpected results and shows signs of sustainability. Barriers to further success include a lack of federal funding mechanisms for other stages in the R D process, the restrictions of GG with respect to eligible project costs, and a lack of awareness about GG.
5.1.1 - Is there ongoing performance measurement? To what extent are performance measurement frameworks adequate? To what extent does the Program monitor results?
5.1.2 - Is the Program delivered effectively and efficiently, demonstrating good value for money?
Did the results achieved enhance value-for-money?
5.1.3 - Is the administration of the Program beneficial and efficient with regard to outputs and results? Were the Program costs reasonable in light of demonstrated results?
5.1.4 - Are the current Program governance structures adequate?
GG exhibits many of the challenges experienced by programs in other government departments in performance measurement and results-based management. The Program's performance measurement system is incomplete. The 2006 Accountability, Risk and Audit Framework (ARAF) has a partial performance monitoring system but it fails to link results statements to measurable indicators. The results reporting provided in annual reports does not follow GG's logic model. What are described as outputs in the annual report would be more properly labelled as disbursements. What are described as immediate outcomes - according to the logic model - are results at the output level, which include the approved projects.
The greatest shortcoming in Program operations is the lack of a post-project mechanism for regular measurement of results. Program staff acknowledge that they have no clear picture of how well the Program is performing and what results are being achieved, especially at the intermediate and final outcome levels. The knowledge gap about intermediate and final results exists because there is no formal follow-up mechanisms to find out from funding recipients what happened months or years after the funding relationship with GG has ended. There is informal follow-up on these later results by program staff and regional trade commissioners who are keeping in touch with clients or prospective clients, but it is not systematic. Although it is challenging to determine what results the Program is achieving at the intermediate outcome level, the data gathered through this evaluation demonstrate that it is possible. The establishment of a regular, formal mechanism for inquiring with past funding recipients about intermediate-level results will allow the Program to include this information in its annual reports.
GG is open to applications year-round. The program manager has set a service standard of replying within six weeks to applications. This service level has been maintained, even in the face of a record number of applications in 2007. This is especially true given that applications are sent to sectoral specialists and to TCS offices in Canada and abroad for comment and review.
The Program also effectively manages disbursements. During the last six fiscal years lapsed funding has declined from 70 percent in the first year to just over 10 percent in 2006-07. Corrective measures such as better risk analysis and closer monitoring of recipient expenditures were taken to control the lapsing of funds.
In its 2005 audit of GG, the audit team found that:
The audit team made no recommendations.
Regional TCS staff say lack sufficient understanding of the Program to properly assist potential applicants in their region. This knowledge gap points to a lack of training on GG for regional staff. The evaluators understood that a training plan is in the works for spring/summer 2008.
Regional TCS staff still encounter people in the research community who are unaware of GG. Regional staff are willing to promote the Program on their own, but promotional materials and communication tools are inadequate. The TCS staff have to put together their own PowerPoint presentations on the Program. Since they have no GG brochure, they print the Program web pages. These are not professional materials suited to marketing the Program in academic or private research circles.
Regional staff also say that communication about the Program from headquarters is insufficient. They feel they are not kept fully in the loop. For instance, they are not informed by headquarters about successful and unsuccessful applicants from their region. Apart from meeting a basic need to be up to date on successful applicants, this information would broaden the knowledge of trade commissioners and allow them to better judge future applications from their regions. GG projects could be registered in the client management database so that regional offices are aware of approved projects and so that results can be tracked.
|Minimum salary||Maximum salary||Person years (%)||Minimum cost||Maximum cost|
|Sub-total of salaries||$41,190||$57,021|
|Employee benefits @ 20%||$8,238||$11,404|
|PWGSC accommodations @ 13%||$5,355||$7,413|
|Total estimated admin cost||$54,783||$75,838|
Exhibit 5 describes the human resources of DFAIT involved in the Program and the corresponding estimated administrative costs. A 33 percent factor has been applied to direct salary costs to account for employee benefit plans (20 percent) and the PWGSC accommodations charge (13 percent). A number of other indirect program costs were not included.(6)
The total estimated administrative costs of the Program appear moderately high in relation to the budget of the Program, corresponding to an approximate range between 14.0 percent and 19.4 percent of the total contributions provided in 2007-08. This administrative ratio is comparable to those for PEMD-A and CISP.
However, there are two limitations to the administrative ratio calculation:
Regardless of which scenario is closer to the actual cost, both indicate that the cost of administering the GG is high. The main reason for such a high administration cost is GG's small dollar value. Small programs tend to have high administration costs. However, GG needs to be aware of these costs, monitor them and take steps to reduce them, wherever possible.
The administration cost could be lower if a junior person were attending to the more clerical aspects of program delivery. Two other factors could provide some relief. One would be an increase in the Program budget. The other factor would be the proposed amalgamation of GG with CISP and PEMD-A whereby a joint secretariat would provide administrative support for all three programs. However, neither of these factors would guarantee that administration costs for GG would fall.
The moderately high cost of program administration must be weighed against the strong relevance of the Program to GOC priorities and to the needs of the research community as well as against the available evidence of having achieved significant results.
Based upon the responses to the evidence-based questionnaire, funding recipients expressed high levels of satisfaction with almost all elements of program delivery. High satisfaction levels were expressed for the following program elements: support by local international trade officers, application process, payment process, and reporting requirements. The one element that had a somewhat lower satisfaction rating (combined satisfied/very satisfied score of 72 percent) was level of funding. This finding is supported by comments by some questionnaire respondents to the effect that funding amounts can be so small that they are almost not worth consideration, given the cost to the organization of applying for and administering its project.
Other proxy measures for client satisfaction include number of participants who will reapply for funding and number of participants who say the GG was effective in creating new R D opportunities.
This finding is also consistent with increasing demand for the Program and the results achieved.
The Program is currently run almost entirely by a single staff person with quite minimal administrative support. This situation is unsustainable given the higher level of effort that has been required as a result of growth in demand in recent years and the anticipated future growth. Although the files are in order, the staffing structure leaves the Program vulnerable to disruption should the current administrator leave for any reason. A staffing structure that comprised a manager and a junior-level assistant would better distribute the workload and reduce vulnerability to disruption, since the Program could be kept running temporarily by either person.
There is also a cost and efficiency issue associated with the present staffing structure. It is inefficient and costly for the program manager to be handling many administrative and clerical tasks. Having a junior person on staff would allow for a more appropriate division of labour, which in turn could lower administrative costs and free up the time of the current manager for other duties. As matters stand now, the manager is too preoccupied with client relations, processing applications and administering contributions to fully attend to other aspects of program management such as performance measurement, analysis, and promotion.
Both headquarters and TCS staff acknowledge that the application and eligibility guidelines for GG are more geared to the academia community as opposed to researchers in the private sector. TCS staff report that applicants from private firms have more difficulty than university-based researchers in completing the application. One stumbling block for private companies is that the Program requires them to publicly share technology arising from the project. Companies generally try to protect keep new technologies and intellectual property to themselves and away from competitors. Private firms usually want to be the only company involved in a project.
An application process that favours university projects goes against the plans expressed by GG managers to attract more business applications. In terms of success in achieving the expected final program results, it does make sense for the Program to seek to approve more business applications. According to regional TCS staff, business projects are typically closer to the commercialization stage when compared to academic projects.
The incomplete performance reporting framework and the lack of monitoring of results are the major shortcoming of the Program in relation to its effectiveness. The staffing structure is unsustainable and leaves the Program vulnerable to disruption, although it has not hampered GG's effectiveness to date. The Program's moderately high administration cost is an issue that requires ongoing monitoring, especially with a planned amalgamation with other programs. With respect to the Program's value compared to its high administration cost, the costs must be weighed against the relevance of the Program and the success of results achieved. The GG application process needs to be examined to ensure that private companies are not discouraged from applying since their participation can increase the Program's effectiveness in achieving results. The Program is expeditious in processing applications and has high levels of client satisfaction.
The evaluation brings to light the following lessons:
Relationships in research and business do not necessarily follow straight paths. Some leads turn out to be dead ends while others require shifts in direction. When the R D process is understood in this way, learning what the range of opportunities are and ruling out certain avenues that lack promise is an achievement in itself and a sign of adaptive capacity.
This lesson points to the need for periodic evaluation of a qualitative nature since the Program's internal performance measurement system cannot provide complete clarity about what is happening.
Applications must be reviewed by sectoral specialists in complex research domains and by TCS staff at regional offices in Canada and overseas.
Given the large number of small contributions being managed in this type of program, the time and level of effort required create too high of a burden for a single staff person.
The evaluation findings indicate that GG is relevant to Government of Canada priorities in S T and in international trade. Further, there is evidence that the Program also responds to the needs of the Canadian research community for financial support in the early stages of solidifying relationships with international R D collaborators.
The incomplete performance reporting framework hampers GG's effectiveness and its ability to measure results. The staffing structure is not sustainable and leaves the Program vulnerable to disruption. The Program's relatively high administration cost is an issue that requires ongoing monitoring. The high administration cost must be weighed against the Program's relevance and achievements. The GG application process needs to ensure that private companies are not discouraged from applying. Expeditious processing of applications and high levels of client satisfaction point to efficiency in the program's management.
Data gathered for this evaluation point to significant Program achievements, including positive unexpected results. There is also some evidence of the program's sustainability.
That DFAIT proceed to expand funding for the Going Global Program, given the relevance of the Program and the growing demand. An enhanced budget will allow the Program to fund larger projects and a greater number of them, provide an opportunity to address the unsustainable staffing structure, and may also lead to improvements in the administrative cost ratio.
That program management take steps to strengthen performance measurement. The necessary steps would include revising the performance monitoring system in the ARAF in order to link results statements to measurable indicators, and establishing formal mechanisms for regular follow-up with funding recipients to identify post-project results. Progress toward achieving results should be presented in annual program reports in a format that adheres to the program logic model.
That program management examine the staffing structure to reduce administrative costs and resolve disruption vulnerabilities. The demand for the Program has grown from 18 applications in 2001 to 78 applications in 2007. Despite the growth in demand, the administration of the Program is run almost entirely by one individual. Having only one person to run the Program makes it vulnerable to service disruption should that staff member take leave for any reason. Moreover, some efficiencies could be gained by distributing administrative tasks currently performed by the Program Manager to more junior-level staff.
That program management take steps to make the best use of regional TCS staff in program delivery. Action required in this area would include providing training for regional TCS staff to better enable them to serve prospective clients and screen applications, improving the flow of information and general communication between headquarters and regional offices, and providing regional TCS staff with professional communication tools for education and awareness about GG.
That program management modify the application form and criteria in order to encourage applications from research groups in the private sector. This step may require consultations with stakeholders in the private sector research community to identify impediments to greater private sector participation in the Program. Success in attracting more private sector participation could lead to stronger program performance in achieving longer-term results.
|Recommendations||IIT Management Response and Action Plan||Time Frame|
That DFAIT proceed to expand funding for the Going Global Program, given the relevance of the Program and the growing demand.
|Management response:DFAIT management concurs with the evaluation recommendation and recognizes the relevance and usefulness of the Going Global Program in facilitating S T partnerships as an important priority for the Department.|
Action plan: In 2008-09 DFAIT management will combine three funding programs - PEMD-A, CISP and Going Global - under an umbrella Treasury Board submission. This initiative will allow Going Global to expand financially in order to respond to the increasing demand. Additional funding is expected from the Global Commerce Strategy upon approval of the combined Treasury Board submission.
That program management take steps to strengthen performance measurement.
|Management response:DFAIT management concurs with the evaluation recommendation and recognizes the importance of a well designed performance measurement system that is practical, and outcomes oriented for all of its activities.|
Action plan: As part of the amalgamation exercise (as stated in recommendation 1), all documentation pertaining to the Going Global Program will be reviewed and reassessed. The ARAF and the year end reports will be revamped to reflect the new program and adhere to the program logic model. This will allow for the development of more outcomes-oriented performance measurements and greater coherence with the program logic model.
That program management examine the staffing structure to reduce administrative costs and resolve disruption vulnerabilities.
|Management response:DFAIT management concurs with the evaluation recommendation and recognizes the crucial importance of ensuring proper support for the administration of the program to reduce the risk of service disruption and work overload.|
Action plan: With the new funding resources provided under the Global Commerce Strategy, a CO-01 position has been created and will be dedicated to the analytical and administrative portion of the Going Global program. Once the amalgamated program has been approved, an administrative office will be created to off load the administrative duties of all three programs. This will alleviate a majority of the administrative burden placed upon this small program.
That program management take steps to make the best use of regional TCS staff in program delivery.
|Management response:DFAIT management concurs with the evaluation recommendation and recognizes the important value added role of the Regional Offices in the delivery of the program to researchers in business and academia.|
Action plan: Over the coming year an action plan will be developed to train the regional office staff for program delivery. Each officer would:
The action plan would also include the following elements:
That program management modify the application form and criteria in order to encourage applications from research groups in the private sector.
|Management response:DFAIT management concurs with the evaluation recommendation and recognizes the need for greater participation of private sector researchers in international S T partnerships as a priority focus for the Department as it moves forward to implement a global innovation strategy across the department.|
Action plan: As part of the amalgamation exercise, all documentation pertaining to the Going Global Program will be reviewed and reassessed with a particular focus on delivering the program to private sector firms. A review of the application process will be undertaken to ensure that it is suited for all clients applying to the program. Among the options being considered is the development of an online application process that would be more suitable and efficient for all clients and particularly small and medium sized research intensive enterprises. Consultations with SMEs have been undertaken and comments have been noted and may be taken into consideration upon the drafting and renewal of the new terms and conditions of the program. In addition to the Regional Offices, DFAIT's Sectors Divisions will be engaged in the delivery of the program.
1 Patton, M.Q. (1997). Utilization-Focused Evaluation. 3
rd Edition. Thousand Oaks, California: Sage.
2 At a 95% confidence level, this means that 95 times out of 100 the true value will fall within a 12-point range of the response rate.
3 Respondents could choose multiple reponses so that these percentages wold exceed 100%.
4 The Government of Canada's Funding Guide for International S T Cooperation lists 660 sources of grants, bursaries, post-doctoral fellowships, prizes and awards, including GG.
5 Rates of pay are according to Treasury Board. The effective dates for each pay category are as follows: EX (April 1, 2007); CO (June 22, 2006); and AS (June 21, 2006).
6 Other indirect program costs that were not included include the following: legal (i.e. advice), finance (i.e. payments, planning, budgeting), human resources (i.e. staffing and training), audit, evaluation, IT systems support, communications (i.e. telephone), general reception, etc.
7 Whereas fifty-five percent of the survey respondents say that they would certainly seek GG funding and thirty percent reported receive funding for more than one year, fewer may actually reapply. Reasons for not reapplying may vary but are thought to include such things as: projects are no longer viable, research may become obsolete, or other researchers may develop the same technology and obtain patents. GG may also have exhausted its funding budget and not able to fund an application.