The Research and Conference Fund (RCF) was launched in December 1989, renewed in November 1995, and renewed again in August 2000. The 2004-2005 budget for the program is $881,000. The Fund was established to encourage knowledge creation and relationship building in support of Canada's socio-political and trade-related foreign policy objectives in the Asia-Pacific region through research papers, study tours and conferences under the direction of, primarily, Canadian academics. Since 2000-2001, the RCF's total expenditures amount to $2,833,083.
The purpose of this summative evaluation was to "generate findings concerning relevance, success and efficiency; derive lessons learned; and, provide recommendations on the continuance of the program before it is scheduled to sunset in March 2005(1)." The study methodology included a review of program-related documentation, 16 case studies, and interviews with 36 stakeholders.
Since 2000-2001 (not including the last set of projects approved for 2004-2005), the RCF program has supported 113 projects. RCF support for these projects totaled $2.8 million. Projects fall into the categories of research papers, study tours, conferences and projects combined two or more of these elements. Topics have ranged from peacekeeping in Mongolia to Korean social policy to human rights in Afghanistan. The combined total budget of the 16 case studies was $1,576,640. The total FAC share contributing to these projects was $699,122 (44 per cent of total project costs).
The RCF program was found to be successful in achieving its objectives of enhancing critical relations and creating crucial knowledge in support of Canada's socio-political and economic priorities respecting the Asia-Pacific region. The RCF has enabled FAC personnel to build and access a community of interested partners, including Canadian and world experts, dedicated to broadening and deepening Canada's connections and understanding in relation to the Asia-Pacific region resulting in the production of better, more strategic and informed foreign policy advice. By and large, these results are incremental, i.e., they would not have occurred in the absence of the program. Nor would program outcomes likely sustain over the long term without continued RCF support. The program was found to be effectively and efficiently managed. Desk officer are involved in project management and the approval of the proposals is well documented. Although there are many example of solid and effective management, some enhancements are warranted.
The RCF program was found to be relevant inasmuch as it supports Canadian foreign policy objectives considered to be of high priority.
From the perspective of these findings, RCF renewal is warranted. If Branch directions and approaches remain relatively stable, and resources are available, it is recommended that the Research and Conference Fund program be continued at the current or a similar level of funding and that refinements and enhancements be instituted to:
The Research and Conference Fund (RCF) was launched in December 1989, renewed in November 1995, and renewed again in August 2000. The Fund, with a 2004-2005 budget of $881,000, "was established to proactively address knowledge gaps related to Canada's Asia-Pacific policy in the peace and security, public diplomacy, economic and commercial areas." As a complement to research undertaken by the Asia Pacific Foundation of Canada (APFC), the RCF provides FAC(2) with a means of obtaining the perspective of a wide range of academics and organizations. "Eligible recipients include Canadian and foreign institutions, and organizations and individuals for which clearly defined support would enhance Canadians' knowledge base of the Asia-Pacific region, strengthen Canadian linkages with the region, and improve the capability of Canadians to engage with the region(3)."
The purpose of this summative evaluation is to: "generate findings concerning relevance, success and efficiency; derive lessons learned; and, provide recommendations on the continuance of the program before it is scheduled to sunset in March 2005(4)." The evaluation addresses the following questions(5):
Relevance. To what extent have the RCF program and its funded projects been congruent with FAC's priorities and strategic objectives in the Asia-Pacific region, and met the needs of FAC's partners in Canada and the Asia-Pacific region? This includes an analysis of how RCF's activities intersect with those of FAC, CIDA, or other similar Canadian governmental or non-governmental activities, how effectively the RCF has evolved to respond to changing external conditions, and its contribution to the strategic objectives and priorities of the Asia-Pacific Branch, FAC and the Government of Canada.
Success. To what extent has the RCF achieved its short, medium and long-term outcomes? This includes an analysis of whether or not Canadian academics, businesses, NGOs and policy makers have increased their cooperation with their Asia-Pacific counterparts and benefited from additional knowledge that enhanced their ability to explore opportunities and formulate policy directed at the region.
Efficiency. How cost-effective has the RCF program management approach and program design been for achieving desired results? This includes an analysis of the extent to which results-based and performance-oriented program planning, monitoring and reporting are in place, the efficiency of current financial planning, budgeting and control systems, and program alternatives.
The RMAF and RBAF for the RCF component of the APIP have also been reviewed and updated taking into consideration changes that have occurred in the program, in the department, and in Canada's foreign policy directions generally since 2000-2001. The result of this activity is a new RMAF and a new RBAF for RCF component of the Asia-Pacific Initiatives Program. These are available under separate cover.
The methodology associated with this study involves three complementary data collection and analysis activities(6). These are:
Data from the three sources were analysed independently. Findings were then integrated to develop answers to the evaluation questions. Before proceeding to the analysis, however, the next chapter provides an up-to-date profile of the program.
The RCF was launched in 1989 as one of seven elements of the Asia-Pacific Initiatives (API) Fund(7), which provided funding for all API grants and contributions programs. Today, the RCF is one of three API programs. The August 2000 submission to Treasury Board for the APIP requested an allocation of $2 million per annum, beginning in FY 2000-2001 and terminating FY 2004-2005. In FY 2003-2004, APIP's total budget was $1,981,000 of which $831,000 was earmarked for the RCF, $1,000,000 for the APFC and $150,000 for the Canadian Education Centre Network (CECN). These figures do not include the December 2004 call for proposals.
|FY||Budget||Expenditures||Completed Projects per Division||Total|
8 Calculated by subtracting APFC expenditures from PSD and PND expenditures.
9 Calculated by subtracting APFC expenditures and paye from total expenditures and paye.
10 The Carleton University/Kovalio G&C project is not counted in FY 2002-2003 as it took place, yet despite repeated requests to provide reports, these were not submitted. Due to non-compliance, the funds lapsed.
11 This amount does not include funds allocated to projects approved in the second half of this fiscal year. The final expenditure for 2004-2005 is expected to be considerably higher than the amount shown here.
12 One file has been completed at the time of writing. Others are ongoing.
The RCF supports projects that "provide [FAC] with a means of obtaining the perspectives of academics and NGOs on the region. [The RCF] provides a flexible response mechanism to requests for the funding of specific research and conference activities that would assist in meeting Canada's objectives in Asia-Pacific(13)."
Table 1 traces the RCF's budget, including a breakdown of projects by sponsoring(14) division. Since 2000-2001, not including projects approved following the second call for proposals in 2004-2005, the RCF has supported 113 projects. Project support levels in a given fiscal year have ranged from $1,185(15) to $81,900(16) with an average support level of $25,071 per project. The total number of projects per year has remained fairly steady. In later years, as the numbers of projects sponsored by Japan Division (PJP) decreased, the distribution of projects among divisions became more even. Project breakdown by division is as follows: 23 percent of projects were sponsored by the Japan Division (PJP), 13 percent by the China and Mongolia Division (PCM), 21 percent by the Korea and Oceania Division (PKE), 14 percent by e3 Operations and Asia-Pacific Policy Division(17) (PEP), 16 percent by the Southeast Asia Division (PSE), and 12 percent by the South Asia Division(18) (PSA). Since 2000-2001, the North Asia and Pacific Bureau (PND) sponsored projects have received 71 percent of expended funds ($2,011,489). Since 2000-2001, the budget allocated to the RCF has climbed from $681,000 to $881,000. Actual expenditures have yet to reach budgeted figures due to a variety of factors discussed below.
Projects typically fall into the categories of research papers, study tours, and conferences in support of Canada's trade and/or socio-political(19) objectives in the Asia-Pacific region. The majority of projects are directed by Canadian academics, many of whom work at Canada's largest universities. Some projects, however, are directed by academics from other countries (in the Asia-Pacific region, or working in world-renowned institutions in the United States and elsewhere) or by representatives of NGOs or business associations.
PEP runs an RCF proposal call-out twice yearly. The call-out is advertised on FAC's website and is circulated among divisions, desk officers in the Branch, and to missions. It is also distributed on a limited basis through selected other channels such as CANCAPS and the monthly newsletter Canada in Asia-Pacific(20). Potential proponents are directed to the RCF website which includes a general description of the RCF program and its objectives, and contains links to a description of APIP, a statement of current Branch priorities, and the RCF application form. While considerable information is provided, explicit detail on how proposals are to be rated and ranked is absent. The complete application procedure and selection criteria are described in Appendix E.
Proposals come in either independently from proponents who have seen RCF advertising or through the initiation or intervention of a Branch desk officer. For example, a desk officer needing more in-depth knowledge or insight concerning a particular policy issue may contact a researcher with whose work he or she is familiar to suggest a related research project, resulting in the submission of a proposal.
Twice-yearly call-outs result in the submission of 20 to 35 proposals per call. Proposals are first reviewed by PEP staff. Non-compliant or ineligible proposals are removed, and remaining proposals are distributed for review among the appropriate divisions. Following division review and ranking of proposals, PEP and division personnel meet to make final decisions on the acceptance of proposals and offers of contribution. Once agreement to proceed has been achieved, administrative and financial procedures are put in place and a desk officer is assigned to monitor and, ultimately evaluate each project.
Recent recipients of RCF funding include proponents from several Canadian universities, colleges, centre for International studies and other international associations.
The evaluation question relates to the extent to which the activities and results of the RCF program are congruent with the achievement of the broader, strategic aims of the Department and of Canada. Canada's socio-political and trade interests respecting the Asia-Pacific region are determined and pursued by the Prime Minister, departmental ministers, FAC, and the Branch. These actors adopt or promote policy, agreements and relations in the region. In turn, their activities are informed by advice from Asia-Pacific personnel. The evaluation question is: to what extent do RCF-funded projects demonstrably improve Branch personnel's capacity to formulate high-quality strategic policy advice?
The evidence from all sources suggests that RCF-funded projects enhance Branch personnel capacity and that, without the RCF, a level of policy work - i.e., the broad, strategic, in-depth level - would remain undone. A chain of communication appears to link RCF-funded projects with policy formulation and advice. The advice given by desk officers to senior officials is a result of analysis, knowledge and connections, much of which benefit from RCF-funded projects. This is the case on a project-by-project basis. For example, the Study on Terrorism and Threats to Regional Security project led directly to advice respecting Canada's terrorism policy for the region. It is also the case across numerous projects. For example, a number of projects over the years related to Korea and the DPRK have contributed to the creation of a network of academics, NGOs and policy makers in Canada and in the region as well as an impressive, growing and singular body of knowledge concerning the region, all of which enables Branch personnel to inform policy development on an ongoing basis and to respond to crises in an informed and measured way as they arise.
According to departmental interviewees, in the early years of the program, some RCF-supported projects may have been selected for non-strategic reasons. The advent of PEP and the improvement of the program's management brought with them a more strategic, top-down, priority driven approach. The degree of relevance of individual projects examined as case studies varied, as did their distribution and reception. After study tours such as An Opportunity for Canadians to Better Understand the Khmer Rouge Legacy in Cambodia and Developing a Dialogue with the Islamic Communities in South and Southeast Asia, the Canadian government was presented with recommendations to improve Canadian capacity to engage with the Asian regions. Many conferences formulated policy recommendations for both governments. For example, "the most significant intermediate outcome [of the Roundtable for the Afghanistan Independent Human Rights Commission] is that of the Government of Canada's foreign policy formulation and implementation in Afghanistan." The Conference Indonesia Outlook - The New Shape of Government in Indonesia and Implications for Bilateral Relations made recommendations such as increased Canadian government support for new legislative institutions and increased civil society linkages with Muslim organizations. In the Canada Korea Social Policy Research Project narrative report, emphasis was put on tangible follow-ups; the project was thought to have had "significant impact within Korean academic circles and in the Korean ministries of Health." In other cases, such as Workshops on Canada-ASEAN Dialogue on Regionalism, increased collaboration remained something that might occur in the future. For example, directors noted that "there is some possibility that a collaborative project might be organized...to expand the results of their research paper and to publish academic articles on the topics of the project."
There appear to be no viable alternatives to the RCF to produce this analysis, knowledge and set of connections. Other government granting programs focus either on purely academic research or otherwise disallow many of the essential expenditures of RCF-supported projects such as NGO and business participation in conferences, and overseas travel and accommodation. Contracting for the same or similar kinds of activities and outputs as generated by RCF-supported projects would be prohibitively expensive. According to interviewees, most projects would not be undertaken without RCF support; despite the fact that other bodies contribute significantly to projects, it appears that in most cases the RCF program is the driving and defining force. Without RCF support as a lever, it is unlikely that relations between FAC desk officers and members of the academic community would be widely sustained. While researchers are interested in collegial relations with other concerned parties and, to an extent, appreciate the experience of being involved in policy making, the primary attraction to FAC is project funding.
Canada's priorities in the Asia-Pacific region are stated as follows(21):
and also as follows:(22)
In support of these objectives, RCF contributions are stated as supporting the following types of activities(23):
Developing sector-specific support mechanisms: Creating opportunities for the development of special initiatives that capitalize on Canada's strengths and abilities in specific sectors and which offer opportunities to Canadian organizations, business and academia to collectively capitalize on identified trends which they would be unable or unprepared to undertake on their own, thereby reducing the risks to them (e.g. a project to promote specific Canadian technologies in the region);
Enhancing engagement with the region: Initiatives that promote and/or reinforce Canadian involvement in, and interaction with the region and enable Canada's participation in other regional fora and promote awareness of opportunities in the region;
Increasing knowledge about the region: Research that will strengthen Canada's policy and economic knowledge base of the Asia-Pacific region, that will focus on emerging markets and technologies and that will foster new links in specific sectors; and,
Domestic capacity building: Meetings, workshops, conferences and publications that focus on Canada's key objectives in Asia Pacific. Such activities will enable and facilitate Canadians to establish networks and pursue initiatives in the region and exchange knowledge.
As cited in the terms of reference, RCF-supported activities are expected to result in the following long, intermediate and short term outcomes:(24)
Long Term Outcomes
Short Term Outcomes
The RCF program's objectives and their link to broader departmental policy aims(25) are contained within these declarations. However, an examination of the above text reveals considerable overlap and some ambiguity. For the purposes of the evaluation, based on the perspective of those involved in the RCF program and in RCF-supported projects, these objectives, activities and desired outcomes shall be summarized by the program's two main foci:
(1) relationship building, and
(2) knowledge creation
in support of Canada's socio-political and trade priorities respecting the Asia-Pacific region. Program success shall be evaluated against this simplified definition.
Relationship building occurs between different actors at different levels. For convenience, consider six actors: government, academia and civil society within Canada and the same three groups in the Asia-Pacific region. The evaluation question is: to what extent have RCF-supported projects led to meaningful improvements in relations among these actors?
Data from interviews with both FAC personnel and project directors and other stakeholders, as well as the analysis of the 16 case studies, support the finding that relationship building is one of the key results stemming from RCF-supported projects. The RCF is a unique program in that it provides an opportunity for government policy makers to interact with academics, NGO representatives and businesspersons having interests both in Canada and in the Asia-Pacific region. According to one senior official, partnerships between FAC personnel and other government department personnel, provincial players, NGO leaders, and academics, represent the most important benefit of the RCF program. The RCF is seen as a tool to build bridges among these stakeholder groups, bridges which support the development of quality policy advice.
To this effect, perhaps the most important relations stemming from RCF-supported projects are those between FAC and Canada's academic community. These links create a network consisting of Canadian and regional players that, according to interviewees, could not exist in the absence of the RCF program. For example, in working with a Canadian expert on Japan, a Branch desk officer gains access to that expert's network of contacts, both academic and political, in Canada and in Japan.
Departmental interviewees, particularly Branch desk officers, invariably refer to the RCF program as a "tool" providing a convenient and flexible means to make connections, and to acquire in-depth knowledge, critical to the development of sound policy advice. Interviewees noted that without the RCF, much of their work would necessarily be transactional and reactive as opposed to strategic. By way of example, Canada's China policy is a highly sensitive, high stakes policy; before the institution of any new measures, Canada must be sure of their context and potential consequences. Access to the academic community ensures that government policy-making is benefiting from multiple viewpoints, and allows HQ, mission and academic views to be taken into consideration. Hence the RCF is relied upon as one tool to build connections and knowledge in support of this policy.
Other government departments have contributed to RCF-supported projects over the years. Generally this appears to have been mutually beneficial. For example, the Study on Canadian Cooperation with India in Science in Technology was supported by Industry Canada under their Highly Qualified Professionals Agenda. Industry Canada views India as a priority market and found this project to be a good means to improve relations and exchange with the Indian telecommunications technology industry. In most study tours examined as case studies project directors felt they had contributed to Canada's reputation in regions where Canadian interests have traditionally existed. Much of the Study on Canadian Cooperation with India in Science and Technology consisted of developing lower-level arrangements to substantiate Canada's high-level political willingness to support Canadian and Indian science and technology collaboration. In the case of the Developing a Dialogue with the Islamic Communities in South and Southeast Asia study tour, all participants stated that this was the first time an independent researcher was charged by his government with the responsibility of noting the world view of Muslim intellectuals, and felt accordingly flattered(26).
In general, desk officers evaluating conferences felt that Canada had benefited diplomatically. In many cases, such as the Conference on Rural to Urban Resettlement: Three Gorges Project, the conference was felt to have created an opportunity to "brand Canada as an innovative, dynamic, open and diverse society...To this end the contribution from the Universit[ies] and the Canadian private sector imparted a strong 'Canada Brand' on the region and created tremendous good will(27)." Some conferences - Conference on Rural to Urban Resettlement: Three Gorges Project, Smart Border Symposium: Border Cooperation and Capacity Building for the Asia-Pacific Region, Roundtable for the Afghanistan Independent Human Rights Commission, and, Korea DPR Roundtable - sought to develop areas where "there has previously been little interaction between Canadians and [others] on this issue(28)." Respecting the Roundtable for the Afghanistan Independent Human Rights Commission, many conference participants commented that they had not been able to discuss sensitive issues in Islamic law in an open forum before. The director of a project which brought together ASEAN and Canadian academics and policy-makers noted, "we got ASEAN to take Canada seriously." Both project directors and Branch desk officers also cited instances of improved Canadian profile in the Asia-Pacific region through connections made among people in academic and civil society institutions, as well as through media coverage of RCF-supported events. Many project directors saw the intermediate outcome of the conferences as the development of solid, long-term working relations between Canadian academics and their counterparts in South and Southeast Asia. Directors of the eight conference projects examined as case studies generally felt that they "provided a unique opportunity for developing contacts(29)," as well as opportunities to advance Canadian ideas. The director of the Conference on Rural to Urban Resettlement: Three Gorges Project saw the conference as the first step toward more formal linkages, saying "it is critical for the development of formal agreements that on-the-ground functions such as a conference are held to 'test the waters' and determine the compatibility of the interested institutions." In the case of the Roundtable for the Afghanistan Independent Human Rights Commission, relationships were created with "Afghans with a demonstrated commitment to the protection of human rights, respect for Islamic principles and community organizations in support of rehabilitation of their society."
The SEAscape Electronic Bulletin was thought to act as a tool that "facilitated networking and coordination by providing direct links to many Canadian businesses, associations, councils and other government departments with an interest in Southeast Asia(30)." As such, project directors felt they had contributed to strengthening Canadian business linkages with the region.
Knowledge creation is the other route by which Canadians - and FAC in particular - gain an enhanced understanding of the Asia-Pacific region leading to better policy. All four project types purport to generate knowledge. The evaluation question is: to what extent have RCF-supported projects generated knowledge related to policy?
The findings suggest that meaningful knowledge creation is one of the results of RCF-supported projects. As the evaluators of the projects, desk officers receive information on the various subject areas covered by the projects: business intelligence, security related information, and political information. As their understanding of the Asia-Pacific region is enhanced, they are better able to formulate policy directed toward it. For example, the recent Policy Options for Canada conference respecting Canada's relations with Afghanistan supported panellists in writing papers, which were presented at the conference, and making policy recommendations for consideration by FAC. Another group of beneficiaries of knowledge creation is citizens of the Asia-Pacific region. Often, the Canadian experience is showcased as a potential model from which South and Southeast Asians can draw useful parallels.
According to interviewees, the RCF program provides a policy capacity that complements the department's internal capabilities. The community that is supported and nurtured by the RCF is stable and long-lasting and supports a knowledge base that continues beyond the typical tenure of desk officers. By opening its doors to proposals from any and all proponents, the RCF attracts additional members to, and continues to build, this community.
The main short-term outcome of the three research projects examined as case studies was identified as the desk officers' self-reported increased knowledge of trade, investment and terrorism in the Asia-Pacific region. Examples of the direct beneficiaries to whom the Impact of Potential Australia/U.S. and New Zealand/U.S. Free Trade Agreements on Canada analysis was distributed were "interested client groups," government officials at posts in Australia, New Zealand, and the divisions of FAC. The project director felt that the "research study ... would be of considerable value as a reference document to inform senior government officials." In the case of the SEAscape Electronic Bulletin, directors noted that feedback from the readers had been extremely positive(31).
While most conferences, such as the Southeast Asia Conference: Investment Opportunities, Emerging Trends, Key Contacts and Strategic Partners, focussed on increasing Canadians' knowledge of the region, some focussed on enabling people in the Asia-Pacific region to better understand Canada. In these instances, it was typically the Canadian model that was held up as a potential example for South and Southeast Asian countries, and some conferences were designed with a view to South and Southeast Asians being able to take advantage of Canadian expertise. The director of the Conference on Rural to Urban Resettlement: Three Gorges Project described his conference as a "showcase of Canadian talent and knowledge of an increasingly important...issue." Judging from the directors' remarks, Canada's expertise was valued by the Canada Korea Social Policy Research Project participants, who felt that "Canada's social policies are an attractive model...for many of the East Asian countries." Other conferences, such as the Southeast Asia Conference: Investment Opportunities, Emerging Trends, Key Contacts and Strategic Partners, were purposefully organized for the benefit of Canadians with interests in the region. Still others - e.g. Roundtable for the Afghanistan Independent Human Rights Commission - consisted of moderating discussions among South and Southeast Asians. For the organizers of the Roundtable, the conference was an opportunity to "bring Afghans from around the country together." And the last instances - Korea DPR Roundtable, Conference Indonesia Outlook - The New Shape of Government in Indonesia and Implications for Bilateral Relations, Workshops on Canada-ASEAN Dialogue on Regionalism - were thought of as a dialogue between interested parties of different nationalities.
While some projects focussed strictly on trade or on socio-political issues, many covered both areas. Similarly, while some emphasized knowledge creation over relationship building and vice-versa, some had both as their outcomes. Examples of these include the Study on Canadian Cooperation with India in Science and Technology project which combined a study tour to accumulate knowledge with a high-profile delegation interested in forming new relationships. Directors of the Canada Korea Social Policy Research Project and Workshops on Canada-ASEAN Dialogue on Regionalism projects found that the main short-term outcome of research followed by a conference was the increased knowledge that resulted from contrasting and comparing experiences. Emphasis was put on the fact that both Canadians and their counterparts in other countries were learning about each other, and themselves, at the same time.
The RCF is flexible, and both project topics and project types can vary in response to emergent issues. For this reason, the issues covered in RCF projects have accommodated FAC's priorities over the years. The RCF's goals and parameters are distinct from those of any other source of funding and, without it, projects (or at least portions of projects) would not get done.
Analysis of interviews with FAC personnel and RCF-supported project directors respecting the success of the RCF program in achieving its objectives leads to the finding that the program is successful. According to stakeholders, projects supported by RCF contributions have enhanced critical relations and/or created crucial knowledge in support of Canada's socio-political and trade priorities respecting the Asia-Pacific region. By and large, these results would not have occurred in the absence of the program. In-depth case study of 16 representative projects corroborated this finding. The RCF appears to be an important tool, enabling Branch personnel to build and access a community of interested partners dedicated to broadening and, more importantly, deepening Canada's connections and understanding in relation to the Asia-Pacific region resulting in the production of better, more strategic and informed foreign policy advice.
In light of the apparent success of the RCF program, and noting that any source of funding imposes an administrative burden on its recipients, the evaluation question respecting program efficiency is: how efficiently does FAC manage the RCF program - from the proposal stage through the funding, monitoring and evaluation stages - to achieve desired results, and is the RCF the most cost-effective way of achieving these results?
A 2001 audit of the RCF program noted weaknesses in program administration. Improvements were instituted as a result. With the advent of PEP and the dedication of new resources two years ago to the management of the RCF, major further improvements in the administration were achieved. Today, there is a relatively low administrative burden on fund recipients. Records management is excellent, which facilitates any prospective reforms. The twice-yearly call for proposals, the processing and selection of proposals as well as project administration procedures appear close to optimal. In general, recipients report that they enjoy dealing with FAC and state that FAC imposes acceptable transaction costs on them.
The only complaint from project fund recipients was that the timing of the RCF application deadlines does not necessarily coincide with academic life. Interviewees from academic institutions noted that, in particular, senior academics tend to favour long-term research programs involving major conferences, books and policy implications. Even in the case of small projects, academics are not always equipped to respond to a proposal call in late Fall for project completion by March 31 of the next year.
Branch desk officers were occasionally critical of how individual projects were conducted. In the case of the Impact of Potential Australia/U.S. and New Zealand/U.S. Free Trade Agreements on Canada, FAC was reportedly not consulted during the course of the study. As a result, the first version of the final report omitted certain subjects that desk officers wanted included(32). The problem with the SEAscape Electronic Bulletin was that the first drafts were "not deemed to be of sufficient quality for publication" and often had to be edited and rewritten by PSE staff who sometimes requested that the writer undertake additional research where insufficient information was provided. The main challenge in implementing the Workshops on Canada-ASEAN Dialogue on Regionalism project was working with the ASEAN Secretariat to finalize the work plan, time frame and resource allocation. As the desk officer noted: "the key risk in undertaking such an initiative, is that the two partners may not have the same expectations of the project process and outcomes...This is particularly challenging when working with an overseas institution that may not be aware of Canadian government funding procedures, and when a Canadian partner institution may not be aware of how the foreign partner operates if working with the organization for the first time." The Southeast Asia Conference: Investment Opportunities, Emerging Trends, Key Contacts and Strategic Partners was a unique case wherein problems in project direction were so serious that they noticeably affected project results. There was a large degree of variance between planned and actual activities, and certain fundamental outputs, such as coauthored papers between Canadians and Southeast Asians and the summary research paper which was to set the context for the entire program, were either late or were never completed.
These individual problems notwithstanding, RCF-funded projects were considered by all sources to be highly cost-effective. All four types of projects - study tours, papers, conferences, and mixed projects - benefit by leveraging program funds. A few thousand dollars in travel and accommodation expenditures can bring together a highly qualified group of Canadian and Asia-Pacific researchers and policy makers.
Taken together, the findings suggest value for money. Value is created through several factors. Participating partner institutions are involved in projects; in addition to the RCF contribution, most projects receive funding from other sources - often the project director's home university. The combined total budget of the 16 case studies was $1,576,640. The total FAC share contributing to these projects was $699,122 (44 per cent of total project costs)(33). Assuming this is at least somewhat representative, it is apparent that FAC dollars are substantially complemented by other contributors. Projects are conducted by university faculty, whose base salaries allow that they do not usually need to charge professional fees, only expenses such as travel, accommodation, equipment, etc.. This means that RCF dollars are highly leveraged; for a relatively small expenditure, the department gains access to the time and services of highly qualified, motivated professionals.
Funds expended on projects have never reached the available budget in a given year. Highest annual expenditures so far totalled $790,053 in FY 2003-2004, approximately $41,000 under that year's budget ceiling of $831,000. Program managers explained that only qualified project proposals whose anticipated results are in line with the needs of the divisions receive funding approval. All other proposals are rejected. Due to circumstances beyond the control of the administrators, there tend to be uncertainties regarding program reference levels. Larger government exercises, such as expenditure reviews, can have the effect of diminishing the administrators' ability to solicit proposals.
The administration and the management of the RCF has many strengths. The procedures and systems associated with the RCF program enable it to function smoothly. Desk officers are involved in project management and the approval of the proposals is well documented. The monitoring process includes a review of the original project plan and budget. Project evaluation criteria examine the extent to which a given project complies with the provisions of the contribution agreement.
Although there are many examples of solid and effective management procedures, there are aspects that could be strengthened, as follows:
The evidence supports the conclusion that the RCF is a unique, cost-effective tool generating in-depth analysis by Canadian and world experts informing the development of high-quality policy advice on priority issues respecting Canada's trade and socio-political objectives in the Asia-Pacific region. The program supports three types of projects - study tours, conferences and research papers - plus occasional projects which combine these elements. The evidence suggests that all three types can be successful in achieving meaningful objectives.
Following some years of administrative difficulties, two years ago the Branch bolstered the resources associated with the management and administration of the RCF, resulting in what can now be considered a highly efficient and effective administration model.
The flexible nature of the RCF allows the program to respond to changing priorities by choosing to support projects accordingly. As such, when used by the Branch to gain in-depth policy-related knowledge, and/or to build relations with key stakeholders in the region and in Canada, the RCF is relevant to the pursuit of Branch and Department goals.
Projects range in kind and in subject matter, as well as in scope and cost. The analysis, combined with insights and suggestions made by interviewees, suggests a set of characteristics defining effective projects. Effective projects are carried out by qualified, capable experts typically with the support of their home institutions. Effective projects tend to be narrowly focussed achieving depth (rather than breadth). They are projects not previously undertaken or undertaken elsewhere. Effective projects are well planned in terms of objectives, roles, and project management and risk mitigation strategies. Often planning is done jointly with FAC desk officers and managers to maximize mutual benefits of the project. Effective projects almost always involve significant cost sharing. While RCF-supported projects tend to begin with RCF support, complementary support comes from other government departments, universities, NGOs and associations and/or other governments. Finally, and perhaps most importantly, effective projects are strongly linked to Branch priorities adding value and support to policy development respecting issues of current importance.
The analysis and interviewee observations also suggest elements of optimal program management. As noted above, the program is currently well managed. However, the following features are seen as potentially constituting valuable additions and enhancements to the existing program management model.
Publicize RCF more widely. Presently, the Fund and calls for proposals are advertised on FAC's Asia-Pacific website, and in the Canada in Asia-Pacific monthly newsletter. Additional efforts could help reach the full community of Canadian and appropriate international scholars and other players who have interests in the Asia-Pacific region and could potentially use the RCF to undertake activities that would benefit Canada's Asia-Pacific priorities. Such efforts would attract and broaden the "virtual network" that FAC wishes to nurture and could also generate proposals for considerably more projects than the RCF can support. This would lead to more competition among proposals and, ultimately, higher quality work. Possible channels include more systematic outreach to selected Canadian universities by PEP and other Bureaux personnel, advertising through the websites and newsletters of a wider variety of associations, and direct mail-outs to the appropriate faculties in post-secondary institutions across Canada. In addition to the networking in which Branch personnel are already engaged, these kinds of measures will naturally lead to the creation of a bigger, stronger research and policy community.
Enhance the transparency and objectivity of the selection process. Presently, project proponents fill out a proposal form in which they describe their project, its potential results, and how its execution would reflect Branch priorities. However, the current application form and instructions contain ambiguities and do not contain proposal rating formulae. This leaves room for interpretation by proponents unsure of how project objectives, activities and results line up with funding criteria. Interviewees called for greater clarity and transparency in the process to ensure that qualified proponents develop their proposals in line with department needs and to ensure that the competitive process is fair and open. Refinements to the existing application form and instructions could include:
Strengthen long-term partnerships with selected researchers and institutions. FAC should work to further developing long-term partnerships with key institutions (including other government departments) and researchers. Interviewees spoke of the value of collegial relations and ongoing dialogue between officers in sponsoring divisions (as well as mission personnel and departmental executives) and members of the research community. These relations spawn projects and facilitate the proposal writing process by helping researchers to fit their proposals to Branch needs and interests and helping FAC personnel to understand the larger research programs and interests of proponents. As part of its yearly report, program management may wish to undertake an analysis of proposal acceptances/rejections including overall statistics, gross breakdowns by type, source, and/or size, and a summary of reasons for rejection. Results of this analysis could feed back into program refinements.
Strengthen project monitoring, evaluation and follow-up. The practice of assigning to each approved project a desk officer from the sponsoring division, and requiring the desk officer to evaluate the project at its conclusion, should be continued. Desk officer involvement is a critical element in successful project management. Generally, there is room to strengthen the project monitoring and evaluation function in line with results-based management principles. Specifically, monitoring at selected mid-points would help prevent later problems. Evaluation at the conclusion of the project should focus not only on compliance with the provisions of the contribution agreement but also on the extent to which anticipated results were achieved (along with explanations and lessons learned) and how results were applied to the advancement of Branch priorities, including policy development. A system is needed to ensure, following project completion, that results and policy applications are appropriately disseminated among key players within the department as well as throughout the community at large. Results could be in the form of desk officer evaluation reports, published articles, abstracts, internet links, conference proceedings, bibliographies, etc.. Some of these would necessarily lag behind project completion by six months to one year. Channels could include the RCF web page, newsletters, departmental internal memoranda, and other vehicles.
Institutionalize the RCF program management model. All procedures and systems associated with the smooth management of the RCF program should be institutionalized such that they can be executed by future, qualified personnel.
The study has found the RCF to be a cost-effective program successful in achieving results consistent with Branch objectives. From the perspective of these findings, RCF renewal is warranted. If Branch directions and approaches remain relatively stable, and resources are available, it is recommended that the Research and Conference Fund program be continued at the current or a similar level of funding and that the following refinements and enhancements be instituted to:
Since the completion of the evaluation, a one-year extension was approved by the Secretary of the Treasury Board to the Terms and Conditions of certain transfer payment programs which were scheduled to expire on March 31, 2005. This included certain discretionary (class) grant and contribution programs such as the Asia Pacific Initiatives Program, of which the RCF is a component.
While the one-year extension will allow the Division to continue the program for an additional year, it is important to note that the RCF will be transformed as a result of two factors:
1. As a result of the expenditure review process, funding for the RCF was reduced from $981,000 in FY 2004/05 to $450,000 for FY 2005/06. Due to the reorganization of the department, some uncertainty exists as to whether these funds were allocated for the RCF alone or were intended to be shared with other programs. While agreement on this issue is expected in the near future, the unclear status of funding may have an impact on the managers' ability to disburse the funds in a manner that fully complies with the evaluation's recommendations.
2. As a result of the reorganization of FAC, the Asia Pacific Branch was amalgamated with a number of other geographic branches into a Bilateral Relations Branch. There is broad recognition of the effectiveness of the RCF throughout the new Branch, and a significant amount of interest in reshaping the RCF into a fund that would support policy research not only in Asia-Pacific priority areas but in areas of relevance to Canada's bilateral and multilateral priorities in Latin America, Europe, Africa, and the Middle East as well. Significant effort will need to be expended this fiscal year to look at this and other options for the future of the RCF.
The management response and action plan laid out below must be understood in the context of the above uncertainties and potential developments. Whatever the scope of the RCF beyond this year, we would expect to be able to take the lessons learned from the evaluation and apply them to the new context.
During the early years of the program, the RCF call for proposals was internal to the Asia Pacific Branch. Via word of mouth, people were informed of the fund and requested to submit proposals. This resulted in a limited number of project applications.
In order to publicize the RCF more widely, it was decided (for the first time) in fiscal year 2003-2004 to advertise the call for proposals on the internet. This was accomplished via our e-newsletter, Canada in Asia-Pacific, at /asia/newsletter-en.asp, which currently has over 1,750 subscribers. The number of proposals received as a result of this first public call for proposals exceeded expectations. Since then, solicitation has been done primarily through our e-newsletter. In that same year, the manager of the fund visited key universities in Canada which specialize in Asia-Pacific related studies (University of Toronto, York University, University of British Columbia, University of Victoria) and met with numerous faculty. One of the key purposes of this outreach was to share information about the RCF. Outreach regarding the RCF has been continued in subsequent years.
In fiscal year 2004-2005, once again to broaden dissemination, the link to the RCF call for proposals was provided to Canada's network of Asia Pacific security experts, the "Canadian Consortium on Asia Pacific Security (CANCAPS)" through the CANCAPS office at York University. This method of dissemination has proven to be effective, and in future we intend to publicize the RCF through other university and association electronic newsletters /web sites.
The managers of the fund agree in principle with the suggestion to disseminate information about the fund even more broadly in future years. However, it must be recognized that dissemination efforts must be commensurate with the level of funding available. Assuming sufficient and stable funding, we would like to expand the network wider by having our missions abroad provide the information to their local partners in the Asia-Pacific region in a more systematic manner than is presently the case. This would be achieved by requesting missions to provide a link on their web sites to the Asia-Pacific newsletter.
Finally, other government departments having interests in the Asia-Pacific region would also be notified of the call for proposals and asked to share this information with their academic and civil society contacts.
In order to enhance the transparency of the funding process and ensure greater consistency in the management of the program, PEP made a number of changes to its administration of the RCF in 2003-04.
Following the deadline for receipt of project proposals, PEP now reviews all applications in consultation with the relevant geographic divisions. Proposals are ranked according to explicit fund criteria/parameters, GOC priorities and availability of funds. A formal record of decisions reached is written by PEP, shared with geographic divisions and Directors General and filed, at which point applicants are notified of the decision reached.
Improvements were also made to the application form to provide clearer and more concise instructions to prospective applicants. Further improvements will be made consistent with the findings of the evaluation.
Finally, in response to the evaluator's recommendation, PEP will use its e-newsletter to publicize its funding decisions after all subsequent calls for proposals.
Over the years, the RCF has been a key mechanism through which the Asia Pacific Bureaux have been able to build and strengthen long-term partnerships with key researchers and institutions. FAC, for instance, has been a long-term funder of the Canadian Consortium for Asia Pacific Security and provides ongoing support to a significant number of other academic networks such as the Japan Studies Association of Canada. Prominent Canadian scholars in Asia-Pacific related studies have frequently tapped into the RCF to fund their research, and we hope to see such support for excellence in Canadian academia continue.
At the same time, however, we are cognizant of a new generation of scholars at top Canadian universities who are slowly taking over from leading scholars moving into retirement. The focus of the RCF's managers over the past few years has been to meet these new scholars, become acquainted with their work, and encourage them to engage on policy issues of critical importance to Canada-Asia relations. As such, while long-term partnerships must be an important consideration in the management of the RCF, this must be balanced with an emphasis on building ties with new scholars.
In order to strengthen long-term partnerships we will continue providing information on Branch and GOC priorities to interested applicants. Transparency in this respect allows researchers to align their research with GOC/Branch work.
The evaluation revealed that a geographic desk officer is assigned to each approved RCF project. This practice continues and has been strengthened over the last few years by implementing a "Geographic Project Officer Assessment" process. A reporting template was created by PEP which aligns with the RMAF. After the completion of a project, the geographic officer files a report that evaluates the work done by the institution. These evaluations form a crucial part of the roll-up to the annual report. They also provide a vehicle for the geographic officer to provide honest feedback on institutions and project results. The completed geographic assessment becomes an historical record and highlights both opportunities for future engagement and challenges encountered with individual researchers and institutions.
In fiscal year 2003-2004, the annual report on the APIP program was formalized. No formal record could be found of annual reports before this time. PEP has made a practice of completing the annual report within four months of fiscal year end, and is currently in the process of preparing its third annual report.
Dissemination of research papers/reports has been left to the discretion of the geographic officer as some of the material is sensitive. However, where possible, the institution or researcher who has received funding is asked to publicize their findings on their web sites and to share their findings with other academics. PEP proposes to send an e-mail to geographic divisions at the end of each fiscal year providing the titles of projects funded and the location of the reports. In this way, geographic officers can browse the file directory if they are interested. We will also work with other fund managers in the department to identify and implement best practices for disseminating research results to the broader community.
PEP has already undertaken a significant amount of work with respect to this recommendation. We currently:
A deck on the RCF process was created by PEP and is available on the Asia Pacific website. This deck provides a step-by-step explanation of the management process underpinning the RCF and is available as a reference tool for future program managers and geographic officers undertaking projects.
Managers of the program view this fund as a work in progress. Improvements will be continually implemented in order to keep the program current, viable and flexible.
1 Summative Evaluation of the Research and Conference Fund of the Asia-Pacific Initiatives Program: Terms of Reference, October 2004.
2 In 2003 the Department of Foreign Affairs and International Trade (DFAIT) became Foreign Affairs Canada (FAC) and International Trade Canada (ITCan). For convenience, the department shall be referred to as Foreign Affairs Canada (FAC) in this report.
4 Summative Evaluation of the Research and Conference Fund of the Asia-Pacific Initiatives Program: Terms of Reference, October 2004.
5 Adapted from Summative Evaluation of the Research and Conference Fund of the Asia-Pacific Initiatives Program: Terms of Reference, October 2004.
6 Adapted from Summative Evaluation of the Research and Conference Fund of the Asia-Pacific Initiatives Program: Terms of Reference, October 2004. In consultation with SIE and RCF program managers (PEP) the focus groups and surveys suggested in the Terms of Reference were eliminated from the methodology because no populations existed that could effectively be accessed through these approaches. In their stead, the number of interviewees was increased and the case study component of the methodology - deemed to be highly useful - was enhanced. Rather than "2-3 case studies," 16 were conducted. It was further determined that little in the way of relevant externally published literature existed and that the literature that does exist is often found within RCF project files. Therefore the literature review component of the methodology was integrated with the document and file review component.
7 In addition to the RCF, there was the Asia-Pacific Fund, Canadian Education Centres Network, APEC Central Funds Contribution, Korean Peninsula Energy Development Organization, Centre for Dialogue SFU, and the Canadian Chamber of Commerce. (Source: Audit of the Research and Conference Fund of the Asia-Pacific Initiatives Fund, Audit Division, September 2001.)
13 Source: Internal e-mail, PEP0048 Research and Conference Fund - Call for Proposals, April 6, 2004.
14 Whether initiated by a geographic division or by an external researcher, on approval each project is assigned for supervision and evaluation to a "sponsoring" geographic division.
15 This was the amount expended for Khmer Rouge study tour (FY 2003-2004), examined as a case study.
16 This was the cost of the University of Ottawa e-commerce conferences in Bangkok and Penang (FY 2001-2002).
17 PEP monitors projects that do not fit clearly within the purview of a geographic division.
18 Of course, these proportions do not reflect overall division activity. For example, PSA division draws significantly on the work of the APFC.
19 In addition to trade, Canada's foreign policy objectives relate to security and counter-terrorism, cultural development, and a variety of political aims. Department officials sometimes refer to two main concerns: trade and political. In this paper, the term "socio-political" is used to describe the full range of objectives outside of trade objectives.
22 Quoted from Branch mission, Commitments and Results for 2004-2005.
24 Quoted from Summative Evaluation of the Research and Conference Fund of the Asia-Pacific Initiatives Program: Terms of Reference, October 2004, which adapted outcomes from the Results-based Accountability Framework (RMAF) finalized in January 2002 as part of the Treasury Board submission approving RCF funding.
25 The RCF currently responds to the departmental priorities of both FAC and ITCan despite the split in the department. This is consistent with the RCF's program objectives and was deemed appropriate given that the fund is sunsetting and the objectives of the program are being reviewed in the context of the process for program renewal.
26 The improvement of Canada's image, generally thought to be a predictable outcome of the study tours examined as case studies, nonetheless varied according to the region that was the object of the tour. In particular, the image of Canada in the eyes of Muslim leaders was different from their Cambodian or Indian counterparts. After speaking with Muslim citizens in the region the researcher found Canada to be set apart from a wider, more negative perception of the West and other English-speaking countries. However, while the image of Canada as "a country with a political, economic and social conscience" is unquestionably a good thing, he warned that along with opportunities for Canada in that region, this image brings responsibilities to the government and people of Canada, responsibilities of which Canadians must be made aware.
27 Desk officer final assessment report, Conference on Rural to Urban Resettlement: Three Gorges Project.
28 Desk officer final assessment report, Korea DPR Roundtable.
29 Final narrative report, Canada Japan Symposium on Peace and Security.
30 Final narrative report, SEAscape Electronic Bulletin.
31 SEAscape directors felt that the government's appreciation of their project was symbolized by the Minister of Foreign Affairs and International Trade holding up a printed copy of the publication at a trade conference relating to Southeast Asia.
32 Researchers at the Estey Centre eventually added them, but this resulted in a one-month delay in the delivery of the report. Desk officers recommended that in future projects of that type, a progress report be written into the contract to "enable early detection of any problems/omissions encountered."
33 Study resources did not permit an analysis of co-funding across all projects supported over the lifespan of the RCF.
34 A system could be developed that is not unlike the rating grids used to evaluate contractors' proposals in open bid situations. PEP may wish to experiment and pilot test a system.