The Evaluation Division of the Office of the Inspector General at the Department of Foreign Affairs and International Trade conducted this summative evaluation of the Community Investment Support Program beginning in January 2007. The evaluation was intended to inform a Treasury Board submission seeking a joint renewal of CISP, Going Global Science and Technology Program and the Program for Export Market Development - Associations.
CISP is a federal contribution program first approved by Treasury Board in 2004. CISP grew out of the Program for Export Market Development - Investment, which operated from 1998 to 2004. In 2007-08, CISP had a contribution authority of $4.5 million.
The objective of CISP is to support community efforts to attract, retain and expand foreign direct investment. The Program is managed and coordinated by a program team in Ottawa, which is part of the Investment, Innovations and Sectors Branch (IIT). CISP projects are delivered through the regional offices of the Trade Commissioner Service in Canada located in each of the 10 provinces. These offices are responsible for the promotion of trade, the commercialization of science and technology, and investment attraction. CISP funds up to 50 percent of project costs to a maximum of $300,000 annually for each municipality or municipal organization.
The objective of this evaluation is to assess the relevance, success, and efficiency and cost-effectiveness of CISP. It was undertaken approximately a year after the completion of a Results Capacity Check and examined the extent to which DFAIT managers have taken corrective measures in response to the recommendations of that review. It covers the period from 2004 to 2007.
The evaluation methodology included collaboration with a small evaluation group, a review of documents, and 12 semi-structured interviews with DFAIT staff at headquarters and regional offices. Further information was collected from an evidence-based questionnaire.
CISP is relevant to Government of Canada and DFAIT strategic priorities through its support of the directions outlined in the Global Commerce Strategy and DFAIT's FDI Attraction Strategy. Funding recipients said there is an ongoing need and increasing demand for the Program. The majority of stakeholders see CISP as an effective means of building the capacity of municipalities to attract and retain FDI.
Deficiencies in CISP's performance reporting made it difficult for the evaluators to come to firm conclusions on the extent to which CISP achieved its long-term outcomes, namely increased capacity and attracted investment proposals and interest, leading to the ultimate result of FDI. However, by gathering data from project documentation, the recipient questionnaire and interviews with DFAIT staff, the evaluators conclude that CISP likely contributed to the attraction of more than $840 million in FDI and about 4,600 jobs between 2004 and 2007. As well, there is evidence that CISP also built the capacity of communities to attract FDI, largely through the development of FDI attraction tools such as community profiles, strategies and web sites. The Program also strengthened partnerships among communities, and between different levels of government.
CISP results in attracting investment and creating jobs are sustainable beyond the life of the Program, as are some of the tools it has helped to develop. However, the sustainability of small projects, especially training through the Economic Developers Association of Canada for economic development officers, is questionable. Given the need for constant renewal of investment attraction efforts, it may be difficult for municipalities to graduate from CISP for many years without alternative funding.
CISP has made progress in dealing with some weaknesses identified in the Report of the Results Capacity Check (2007) namely in the areas of strengthening financial management and providing clearer strategic direction. However, significant challenges remain.
Weak performance reporting compromised the Program's ability to demonstrate results as reporting remained at the output level. CISP management has not used its RMAF effectively as a management tool. For example, management has updated the logic model, but not the performance indicators, as recommended in the Results Capacity Check.
CISP is a program that is decentralized to DFAIT's regional offices in Canada. It is the responsibility of the regional offices to delivery the projects. These offices fall under the auspices of the TCS Regional Offices in Canada (WTD). Because it is decentralized CISP incurs increased cost. Invest in Canada Bureau (IPP) provides funds annually to all regional offices to defer partial costs and new resources have also been allocated to regional offices for specific elements of the Global Commerce Strategy. Additional staff are expected to be hired by WTD pending funding in FY 2008-09. The evaluation could therefore not assess the impact of these additional resources on the delivery of CISP projects.
The estimated cost ratio of CISP administration was calculated between 12.8 percent and 16.1 percent for 2007-08. However, these costs must be weighed against the labour-intensive nature of a decentralized program, the high satisfaction rate of recipients and the Program's significant contributions to FDI attraction and strengthened partnerships.
In December 2007, financial management of CISP was transferred to DFAIT. This means that CISP management currently has day-to-day access to the Program's financial situation. CISP has implemented measures to control its consistent funding lapses, including increased project monitoring and the realignment of the Program's funding year to the calendar year of municipalities. However, the lack of an integrated corporate program/financial management system is an ongoing financial risk to the CISP as well as other G C programs at DFAIT.
Funding recipients are largely satisfied with the program delivery. When asked how the Program could be improved, many called for greater flexibility in the activities eligible for CISP funding and the ability to adjust their budgets to reflect changing opportunities. CISP management is making progress in rectifying the weaknesses in program delivery identified by the Results Capacity Check, such as standardizing selection criteria and centralizing program administration.
CISP has likely contributed to attracting hundreds of millions of dollars worth of investment and thousands of jobs. It has strengthened partnerships between communities and different levels of government. Although CISP has made progress in dealing with many of the weaknesses identified in the Results Capacity Check, a number of challenges remain, for which the evaluators recommend:
The Office of the Inspector General, and in particular the Evaluation Division, was requested by the Investment, Innovation and Sectors Branch (IIT) to conduct a summative evaluation of the Community Investment Support Program (CISP). The evaluation was intended to inform a Treasury Board submission seeking a joint renewal of CISP, Going Global Science and Technology Program (Going Global Program) and the Program for Export Market Development - Associations (PEMD-A). A summative evaluation of Going Global Program was conducted in parallel with this study, and a formative evaluation of PEMD-A was recently completed.
This summative evaluation follows approximately a year after the completion of a Results Capacity Check of CISP. The summative evaluation examines the extent to which DFAIT managers have taken corrective measures in response to the recommendations of the Results Capacity Check. It also aims at providing management with information to make decisions about the amalgamation of the administration of CISP, Going Global Program and PEMD-A, which is expected to take place in 2008-09.
CISP is a contribution program that falls under the Invest in Canada Bureau of IIT. The Program was rolled out in 2004 following the re-profiling of the Program for Export Market Development - Investment (PEMD-I), which provided recipients with assistance in attracting and retaining foreign direct investment (FDI) from 1998 to 2004. The objectives of CISP are threefold:
The Program funds community capacity-building projects through contribution agreements. The projects focus on the development and implementation of internationally competitive investment attraction and retention strategies.
For communities in the early stages of investment attraction, the Program aims to:
For communities at a more advanced stage, the Program aims to:
CISP activities as expressed in its current logic model (revised April 2007) are:
The short-term expected results are:
The long-term results are:
Achievement of the short- and long-term results leads to the ultimate result of attracting and retaining FDI.
In 2004 Treasury Board set CISP's annual budget at $5 million, with $4.5 million for contributions and $500,000 for administration. Actual disbursements during the year depend on the individual contributions approved and expenditures claimed by recipients.
CISP is managed and coordinated by a program team in Ottawa which is part of the Investment, Innovations and Sectors Bureau (IIT). CISP projects are delivered through the regional offices of the Trade Commissioner Service in Canada responsible for trade, the commercialization of science and technology, and investment attraction. The regional offices are located in each of the 10 provinces and function as the local arms of the IIT Branch. Through their trade functions, regional offices assist small- and medium-sized enterprises to expand and diversify their export efforts. The regional office investment role involves acting as full partners in the investment attraction process, including the delivery of CISP.
Before December 2003 the regional offices fell under the jurisdiction of Industry Canada. The transfer of the regional offices to DFAIT took some time. For example, until recently the financial management for CISP continued to be undertaken through Industry Canada's financial management system. It was not until December 2007 that DFAIT assumed full responsibility for the financial management of the regional offices including the transfer of financial authority under the Financial Administration Act.
Under CISP, DFAIT funds up to 50 percent of project costs to a maximum of $300,000 for each municipality or municipal organization per year. Two categories of activities are eligible for funding:
Capacity-building support valued at up to $50,000 is designed for communities that need to prepare for investment-related promotions. Communities at this stage need to conduct research to identify their strengths and weaknesses in key sectors. Examples of supported activities include:
For communities that have completed basic research, CISP provides support valued at between $20,000 and $300,000 for strategy planning and implementation. Examples of activities supported within this category include:
Potential CISP recipients submit proposals/applications to the nearest regional office every year. The regional officer for CISP screens the proposals, and those that meet the basic criteria are assessed by provincially/territorially-based CISP adjudication committees.
The CISP Adjudication Committee includes one representative each from:
The national office at DFAIT headquarters provides comments on applications valued at more than $100,000 and selectively reviews projects of a lesser value. These comments are then forwarded to the appropriate adjudication committee who approved proposals based on a competitive basis to ensure that the best applications are funded according to set criteria. The objective of the adjudication process is to ensure:
Until the end of 2007-08, approved proposals were effective for one fiscal year (April to March) and all approved work had to be completed by March 31. For 2008-09, CISP is realigning its funding year to the calendar year and recipients will have from April 1 to December 31 to complete their projects.
The summative evaluation focused on the three evaluation issues as outlined in the Treasury Board Evaluation Policy and Guide for the Review of Evaluation Reports. These issues are: relevance, success and efficiency and cost-effectiveness. The inter-related evaluation objectives were to:
The evaluation team worked collaboratively with a small working group composed of the Program Manager, and the Evaluation Manager. The working group met twice during the evaluation. The first meeting clarified expectations of stakeholders and allowed the evaluation team to focus the review. A second meeting allowed the evaluation team to discuss the findings of the review and their implications. The evaluators submitted the draft findings with conclusions and recommendations to an Evaluation Advisory Committee for review.
The evaluation team used a combination of the following primary methods to gather data:
The evaluation team faced no major impediments. However, given that this summative evaluation took place just over a year after the completion of the Results Capacity Check, the scope of research was limited to document review, a dozen interviews with DFAIT staff and the recipient questionnaire.
The evaluation was able to respond to all but one of the questions outlined in the Terms of Reference. The evaluation team was unable to answer the question: "Are the current program governance systems adequate?" because interviews with adjudication committee members fell outside the scope of the review. This issue was previously addressed in the Results Capacity Check.
Advantage Canada (2006) is the federal government's strategy to build a more competitive Canadian economy. Flowing from that national strategy, the GCS is DFAIT's policy framework for a suite of programs to enhance Canada's international commercial engagement and competitiveness. The GCS aims to carve out a niche for Canada in global value chains(2) and gives priority to high-technology sectors where Canada has a clear comparative advantage. Under the GCS, the role of CISP is to build the capacity of local communities to attract, retain and expand FDI in partnership with provincial and territorial governments and the private sector.
In the DFAIT Report on Plans and Priorities 2007-08, CISP aligns with the International Trade priority - Greater economic competitiveness for Canada through enhanced commercial engagement, secure market access and targeted support for Canadian business.
DFAIT's FDI Attraction Strategy (2007) is designed to increase Canada's declining share of Northern American FDI in a fiercely competitive global environment. The strategy has four main pillars: policy advocacy, proactive marketing, systematic prospecting and aftercare services and partnerships. CISP falls under the last pillar with the dual task of building the capacity of municipalities to market themselves as locations for investment and to provide aftercare to investors once they get established in a community.
The strategy outlines priority sectors where DFAIT will focus its investment attraction efforts including advanced manufacturing, information and communication technologies (ICTs), and life science. CISP management examined the Program's 2007-08 projects to see how well they aligned with these priority sectors. It found that the vast majority of CISP funding went to projects in priority sectors. CISP's 2008-09 funding application provides bonus points for project proposals in priority sectors and the Program plans to only fund projects in those sectors in the future. Since 2006-07, CISP has supported projects to retain and expand investment from foreign companies already located in Canadian communities.
All of the DFAIT staff interviewed for the evaluation agreed there is a continuing need to build the capacity of municipalities to attract and retain FDI. They saw CISP as an essential component of DFAIT's investment efforts. Funding recipients also strongly support the Program in its current format; 77 percent agreed or strongly agreed with the statement "CISP is an effective way for DFAIT to promote FDI."
Responses to the recipient questionnaire also demonstrated a strong need for the Program. Seventy-one percent of recipients said that the FDI attraction efforts in their communities would stop or be significantly reduced without CISP support. Among the barriers facing communities in attracting FDI, CISP recipients most frequently identified lack of funding as their primary challenge.
Demand for CISP funding appears to be increasing. The number of communities receiving support has grown from 126 in 2004-05 to 151 in 2007-08. Despite the fact that CISP does not possess an information management system to systematically capture the number of proposals and the amount of funding requested each year, the Results Capacity Check did find that demand had grown steadily and funding applications submitted for 2006-07 totalled $9.7 million, more than double the program funding available. Interviews with regional office staff also confirmed that the dollar value of funding applications continued to grow for 2008-09. Demand for the Program is likely to remain strong as 81 percent of recipients said they "will certainly seek CISP funding in the future."
When asked how the Program could be improved or better designed, DFAIT staff and funding recipients suggested no fundamental changes, only tinkering with details of Program delivery and areas of focus.
Overall, CISP appears relevant to Government of Canada and DFAIT strategic priorities through its support to the directions outlined in the GCS and DFAIT's FDI Attraction Strategy. CISP is highly valued by recipients and DFAIT staff. Most said there was an ongoing need and increasing demand for the Program. The majority of stakeholders see CISP as an effective means of building the capacity of municipalities to attract and retain FDI.
The evaluators set out to assess the extent to which CISP has achieved the long-term results in its most recent logic model, namely increased capacity and attracted investment proposals and interest, leading to the ultimate result of FDI. However, the evaluators were unable to come to firm conclusions on the extent to which these results have been achieved because of deficiencies in CISP's performance reporting. The evaluators were, however, able to comment on the Program's success by gathering data from project documentation, the recipient questionnaire and interviews with DFAIT staff.
CISP final project reports contained little information on the short and long-term results of the Program so the evaluators examined three sources of evidence to determine its success in attracting investment proposals and interest: letters submitted to the Program by funding recipients; recipient responses to the questionnaire; and success stories prepared by the Program. Some of the limitations to the data include the following:
Overall, 34 communities and organizations out of the more than 240 that received funding said that CISP-supported activities had contributed directly to the attraction of FDI proposals and interest, or actual FDI. Table 1 provides a summary of the 21 communities or organizations (out of the 34) that were able to estimate the value of investments or the number of jobs created through CISP-supported FDI attraction efforts. Together, these communities said that CISP had likely contributed to attracting of FDI investments worth $840 million and creating more than 4,600 jobs.(3)
|CISP Partner- Investor||$ of FDI in millions||# of jobs created|
|Trail, BC - M-Tech||0.7||––|
|Guelph Partnerships for Innovation, ON - Licensing and Business Partnerships||0.1||––|
|Crowsnest Pass Economic Development, AB 5 tourism investments||2.0||––|
|Greater Moncton - Korean Immigrant Investors||20.0||––|
|Summerside, PEI - Chinese Hotel Investment||7.5||100|
|Edmonton Economic Development Corporation, Alb - Schlumberger ($9M), Bauer Equipment (3M)||12.0||248|
|Société de développement économique de Sherbrooke, QC - TLD||39.0||282|
|Société de développement économique Bromont, QC - Norway and Latin America||31.0||––|
|Ottawa Centre for Research and Innovation, ON - Dell, Stratolight||48.0||––|
|Kingston, ON - Lyreco||20.0||80|
|Canada's Technology Triangle (Kitchener, Cambridge, Waterloo ON) - 7 investments||––||2,025|
|Greater Toronto Marketing Alliance, ON - 13 investments||––||274|
|Brantford, ON - Ferrero||150.0||600|
|Montreal International, QC - GlaxoSmithKline||50.0||––|
|Toronto Financial Services Alliance, ON - Franklin Templeton||––||50|
|Quinte Economic Development Commission, ON - Kellogg, ANR Food/ZEB Rice||132.0||––|
|St. Catharines, ON - Scandinavian Program||––||41|
|Vancouver Island Economic Developers Association - Shellfish Aquaculture, multiple investments||15.0||620|
|Bow Island, Alb. - Conestoga Supply||––||15|
|St.Thomas, ON - Tagaki Manufacturing||14.0||100|
|Guysborough County Regional Economic Developers Association, N.S. - East Coast Gateway Container Terminal||300.0||220|
As Table 2 indicates, CISP appears to have contributed to investment attraction in communities of all sizes with no clear pattern to indicate the Program has been more successful with larger or smaller communities.
|Community size (population)||$ of FDI in millions||# of jobs created|
|Small (less than 10,000)||333.7||235|
|Medium (10,000 to 100,000)||338.5||1420|
|Large (over 100,000||169.1||3000|
Geographically, one-third of the communities that said they had attracted investment proposals and interest were located in Ontario, which receives about 40 percent of CISP spending. Ontario was the province where CISP was clearly most successful, having attracted more than 40 percent of the FDI in dollar terms and nearly 70 percent of the jobs.
The questionnaire asked recipients whether they would have attracted investment without CISP funding. Of the 22 recipients who provided evidence of investment proposals and interest 45 percent (10 responses) said that those results would not have occurred without support from CISP. Another 54 percent (12 responses) said they did not know whether those investment proposals or interest would have been generated without CISP support.
Since program reporting was confined largely to the output level, it was difficult to determine the degree to which CISP had achieved its long-term result of increased capacity or even the following short-term results:
The available evidence comes largely from the recipient questionnaire and interviews with DFAIT regional office staff. Eighty percent of recipients agreed or strongly agreed that "CISP funding has had a significant impact on the ability of our community/organization to attract FDI proposals." About half of those who commented on this statement said that CISP support was important in assisting in the development of FDI attraction tools such as community profiles, market studies, strategies and web sites.
Regional office staff also believe the Program has been effective in building the capacity of communities to attract FDI. They told the evaluators that the Program has been successful in developing a community of people who are knowledgeable and attuned to FDI developments.
Both DFAIT staff and recipients said the new tools and knowledge developed through the Program had increased the confidence of communities to pursue FDI.
Two of CISP's objectives are: to encourage partnerships within and across communities in the promotion of Canada as a location for investment; and to encourage cooperation among all levels of government, and between government and the private sector. It appears that CISP has been successful in meeting these objectives though it was impossible to measure the extent of that success.
In response to the questionnaire, 24 percent (10 responses) of recipients said CISP had helped to develop stronger partnerships with other levels of government or increased regional cooperation. The Program helped develop collaborative partnerships between communities, as staff responsible for CISP encouraged community organizations working in the same sector or region to combine their efforts. CISP has enhanced DFAIT's role in regional investment partnership.
One of the most successful partnership examples is the Ontario Technology Corridor, which brings together Ontario's three largest ICT clusters - Ottawa, Toronto and the Kitchener-Waterloo region. The partnership has allowed the organizations to promote themselves as a corridor for investment and multiply their investment promotion efforts. For example, while one partner promoted the corridor in Silicon Valley another was working on potential investors in the United Kingdom.
CISP also facilitated cooperation with DFAIT missions abroad, international trade officers in Canada, the provincial government and other municipalities. DFAIT staff at headquarters and the regional offices told evaluators CISP played an important role in facilitating the Department's overall FDI attraction efforts by allowing it to deal directly with other levels of government. CISP project adjudication committees include members of regional, federal and provincial economic development agencies, and provincial governments. Several staff members said the Program allowed for the coordination of investment attraction efforts by forcing municipalities to discuss their plans with DFAIT and providing the Department with influence in directing those efforts.
These relationships had important spin-offs such as allowing municipalities to leverage funding from other levels of government. According to DFAIT staff, representatives on adjudication committees would, at times, recommend that the province or regional development agency fund portions of viable proposals that CISP lacked the resources to support.
CISP has produced a number of sustainable results in its contributions to attracting FDI, creating jobs and facilitating partnerships. As well, FDI attraction tools, such as strategies and web sites, will be used by communities over a number of years and will need to be updated on a regular basis.
Over the past few years, CISP's budget has declined whereas the number of projects has grown with a larger number of communities receiving smaller amounts of money. CISP's Terms and Conditions allow the Program to fund up to 50 percent of eligible project costs. Although the percentage of project costs covered by the Program is not systematically available, it appears CISP support rarely reaches the 50 percent level. Some regional staff suggested that the maximum allowable funding amount under CISP (i.e. $300,000) should be changed because it raises unrealistic expectations among funding recipients. In some cases, CISP support amounts to less than $1,000.
Many of the smallest projects provide support for municipal economic development officers to take courses through the Economic Developers Association of Canada (EDAC). Several DFAIT staff questioned the value of this training, since only a small portion of it focuses specifically on foreign investment. There was no evidence that this training had been evaluated. The evaluators also questioned the sustainability of small, one-off projects such as training where there is high turnover of economic development officers.
Sustainability can be difficult for a program such as CISP since the nature of FDI attraction requires ongoing efforts and constant renewal to stay apace in a highly competitive investment environment. Several DFAIT staff said they thought it would be difficult for municipalities to graduate from the Program as municipal budgets are already stretched to pay for basic services.
Because CISP has no program management system and rigorous follow-up process at regional office, reporting results continues to be deficient. This makes it difficult to assess the extent to which CISP achieved its long-term outcomes. However, it appears that CISP has likely contributed to the attraction of more than $840 million in FDI and about 4,600 jobs. As well, there is evidence that CISP also built the capacity of communities to attract FDI, largely through the development of FDI attraction tools, such as community profiles, strategies and web sites. The Program also strengthened partnerships between some communities and different levels of government.
CISP results in attracting investment and creating jobs are sustainable beyond the life of the Program, as are some of the tools it has helped to develop. However, the sustainability of smaller projects, especially training for economic development officers, is questionable. Given the need for constant renewal of investment attraction efforts, it may be difficult for municipalities to graduate from CISP.
One of the recommendations of the Results Capacity Check was "that senior management responsible for CISP provide clear, strategic direction for the Program." Such direction was needed to clarify whether the Program should focus on larger urban communities with the greatest potential for attracting FDI in the short-to-medium term, or cater to smaller urban and rural communities, which may take more time to develop effective FDI attraction strategies.
Senior management recently approved a new direction for CISP. It focuses the Program on building the capacity of communities to respond to potential investors by excluding CISP funding for overseas prospecting activities (eg. conference attendance, meeting with investment leads). DFAIT staff identified that this, at times, had led to uncoordinated efforts overseas that have overlapped with the Department's own work in identifying and engaging potential investors and the promotion of communities who are not ready to undertake prospecting activities. This resulted in promoting Canada's advantages incoherently thus losing investment opportunities for all Canadians.
This will reverse the trend in recent years whereby CISP funding and projects have tended to be concentrated in proactive outreach and promotion activities overseas. DFAIT headquarters staff said this has, at times, led to uncoordinated efforts overseas that have overlapped with the Department's own work in identifying and engaging potential investors.
This renewed focus is likely to boost the funding allocated to smaller communities at the expense of larger ones. The assumption is that smaller communities have greater need for federal support for their FDI attraction activities, and larger communities will fund much of their own overseas investment promotion. The questionnaire provides some evidence to support this theory. Seventy-six percent of smaller communities said their FDI attraction activities would stop or be significantly reduced without CISP support, while only 56 percent of large urban communities said the same thing.
CISP's new direction addresses another of the Results Capacity Check recommendations by proposing to "support only projects that are strategically planned over a multi-year platform." The Program's current design is aimed to attract investment over the long-term while providing funding one year at a time. Developing the capacities of municipalities and associations to attract FDI, especially those with little experience, can take many years. The majority of CISP stakeholders shares this view.
Multi-year funding will potentially contribute to the Program's ability to demonstrate results, as recipients will be required to report on the outcomes of several years of support. Sustainability could also be enhanced if recipients of multi-year funding are required to develop an exit strategy that examines how results achieved through the Program will be maintained when funding ends.
CISP has difficulty demonstrating results because reporting is largely confined to outputs. This is, in part, owing to the short-term nature of CISP funding. As well, the Program has provided little guidance to recipients on reporting. Until recently, neither the CISP application, nor the reporting templates asked recipients to identify and report against performance indicators to demonstrate they had achieved any of the expected program outcomes. CISP management developed a draft on-line reporting template to rectify this problem. It has yet to be used.
CISP hired a consultant to compile annual reviews of final project reports for 2005-06 and 2006-07. Those reviews found that a significant number of funding recipients (17 percent in 2006-07) had either failed to submit formal reports or submitted reports missing important financial data. While these reviews go some way toward the annual progress reporting recommended in the Results Capacity Check, they fail to provide a comprehensive picture of the Program. As well, they do not highlight lessons and good practices and do not provide recommendations for program improvement.
CISP developed a comprehensive RMAF as part of its 2004 Treasury Board submission. However, the Program did not use it effectively as a management tool. The Results Capacity Check recommended that the RMAF be updated. In 2007 the Program did revise its logic model but no work was done on updating the other parts of the RMAF such as the indicators, which are essential for results reporting. CISP management is planning to establish a new Accountability, Risk and Audit Framework (ARAF) to be developed as part of approval of the Treasury Board submission for program amalgamation.
To capture longer-term results, the RMAF called for an annual follow-up survey of funding recipients following project implementation. To-date follow-up with funding recipients is not conducted consistently across Canada and the reporting is incoherent.
CISP is a decentralized program delivered through DFAIT's regional offices with the support of a small program team located in Ottawa. These offices fall under the auspices of the TCS Regional Offices in Canada (WTD). While CISP receives $500,000 for program administration annually at headquarters, until recently resources (both human and financial) were not sufficient for the adequate delivery of CISP or its predecessor PEMD-I at the regional level. This was confirmed by the management response in the Results Capacity Check. The Invest in Canada Bureau (IPP) has provided resources to all regional offices to defer partial costs of administrative support and additional staff are expected to be hired by WTD pending funding in FY 2008-09.
IIT plans to establish a grants and contributions administrative unit and an information technology system to support CISP, PEMD-A and the Going Global Program. This centralized admin unit is expected to reduce the regional offices admin costs in the future.
The evaluation team undertook to estimate how much time staff spent on administering the Program. Table 3 shows that in 2007-08, CISP spent between $576,000 and $723,000 representing between 12.8 percent and 16.1 percent of the program budget on administration.(5) This administration ratio falls within the range for PEMD-A and the Going Global Program.
There are two limitations to the administration ratio calculation:
CISP managers reported that regional staff have strong relationships with the municipalities and associations funded through the Program and are highly client-focused. This is an important attribute, as successful capacity development requires ongoing coaching and mentoring. The questionnaire showed 92 percent of recipients were satisfied or very satisfied with the support they received from the staff delivering CISP.
Thus, CISP's administrative costs must be balanced against the labour-intensive nature of a decentralized program, the high level of satisfaction with regional staff support and the significant results achieved. However, some tasks currently performed by certain levels of operational staff are questionable.
Future budget reductions raise the question of program viability. Several regional office staff told the evaluators that if CISP funding fell below $3 million per year, the Program should be eliminated as the amounts disbursed could not justify DFAIT's administration costs and the effort required by municipal organizations to develop project proposals. This is especially true in smaller provinces, five of which had budgets worth less than $200,000 in 2007-08.
|Invest in Canada Bureau (IPP)|
|Sub-total of salaries||$143,008||$180,374|
|Employee benefits @ 20%||$28,602||$36,075|
|PWGSC accommodations charge @ 13%||$18,591||$23,449|
|Regional offices in Canada|
|Sub-total of salaries||$290,205||$362,941|
|Employee benefits @ 20%||$58,041||$72,588|
|PWGSC accommodations charge @ 13%||$37,727||$47,182|
|Total estimated annual administrative costs of CISP||$576,174||$722,609|
In December 2007, the financial management of CISP was transferred to DFAIT. This means that CISP management now has day-to-day access to financial transactions. From December 2003 to December 2007 Industry Canada continued making payments to funding recipients. The handover of financial management fulfils one of the recommendations of the Results Capacity Check.
CISP is making progress with another significant financial management issue namely, lapsed funding - see Figure 1. The Program and its predecessor, PEMD-I, have consistently lapsed funds. In 2004-05, CISP lapsed 26 percent of its budget, and in 2005-06 it lapsed 31 percent. In 2006-07, the Program began implementing a number of measures to reduce the percentage of lapsed funds. This included the approval of more money at the beginning of the year; increased monitoring by regional offices; and, an additional adjudication round for supplemental funds. These measures helped to reduce the funding lapse.
Reasons for the funding lapses are varied. They include a lack of program management capacity and the absence of a project management system. There was also a lack of access to Industry Canada's financial management system during the period prior to December 2007. Additional reasons include: the high turnover of economic development staff; high number of ineligible claims; and misalignment of CISP's federal fiscal-year funding with municipalities who use the calendar year. Annual funding also meant some municipalities simply failed to complete their projects within the one-year timeframe.
Over the past year, CISP management has moved to further improve project monitoring. Headquarters requested regional office staff contact all funding recipients at the end of the second and third quarters to update project progress and projected spending. CISP management said it received reports on about 70 percent of the projects. In March, headquarters sent a letter to all funding recipients asking them to provide a signed commitment of their project spending to the end of the fiscal year. Less than a dozen communities failed to submit a final report in FY 2007-08 - a significant improvement over previous years. Furthermore, CISP is incorporating penalty points in the scoring system to encourage reporting and communication obligations.
As mentioned previously in 2008-09 CISP management moved to realign the Program's funding year with the calendar year used by municipalities. As a result, CISP management will know the actual amount of Program spending well before the end of the fiscal year in order to make adjustments so as to further reduce lapsing.
CISP management faces a challenge common to all grants and contributions programs, namely the lack of an integrated corporate program/financial management system. Financial commitments and transactions are tracked under DFAIT's financial system - IMS. IMS is, however, limited in functionality. As a result, starting in FY 07-08, headquarters developed a spreadsheet to track all necessary information. This information is then reconciled with IMS regularly to accurately manage financial information and forecast financial results. Regional offices access financial information through this spreadsheet which is posted on the common computer drive.
In the absence of a program/financial management system, IIT plans to automate the management of program information by integrating CISP with PEMD-A's on-line application and claims systems on an interim basis until a corporate system is developed.
The results of the questionnaire demonstrated that three-quarters of funding recipients are satisfied or very satisfied with CISP program delivery. The lowest level of satisfaction at 66 percent was for the level of funding likely due to the funding reductions in 2007-08.(6) When asked how the Program could be improved, 34 percent called for greater funding flexibility in covering expenses, such as travel, marketing and printing costs now ineligible under the Program, and in allowing organizations to switch budget items to reflect changing opportunities.
CISP management is making progress in rectifying the weaknesses in program delivery identified by the Results Capacity Check. According to the RMAF, all projects are to be assessed according to criteria set out in the Treasury Board submission. The Results Capacity Check found it was unclear whether the selection criteria were being used consistently or with adequate rigour. For the 2008-09 funding year, CISP management developed a new project application and scoring grid with standardized criteria for provincial adjudication committees.
The new application form, for the first time, requires applicants to identify anticipated project outputs and outcomes along with performance indicators. However, some recipients had difficulty with the new application, with 15 percent saying it was too long, repetitive and needed to be simplified.
To improve program management, CISP has centralized much of the administration including the preparation of contribution agreements and the payment of claims. In theory, this should relieve the administrative burden of the regional offices. However, several regional office staff expressed concerns to the evaluators that the heavy workload at headquarters might lead to delays in getting out contribution agreements and in paying claims. Headquarters informed us that all efforts were being made to make payments in a timely manner and that only one payment was delayed due to an address change.
With the payment of claims, some of the problems flow from the Program's cumbersome claims process. Claims must include all receipts, as well as a financial report. The regional offices must verify each receipt and, with the centralization of administration, photocopy all documentation and forward it to headquarters for payment. In 2008-09 programming period, CISP has removed recipients' obligation to submit copy of receipts and cheques. The planned amalgamation of CISP with PEMD-A is expected to further streamline this process as PEMD-A uses an on-line claims system. Oversight and controls are ensured through regular recipient audits.
CISP has made progress in dealing with some weaknesses identified in the Report of the Results Capacity Check (2007) namely in the areas of strengthening financial management and providing clearer strategic direction. However, significant challenges remain.
DFAIT has assumed full responsibility for the financial management of the regional offices and CISP has changed the funding year from a fiscal year to a calendar year to better align with funding recipients. The Program has implemented measures to control funding lapses. However, the lack of an integrated corporate program/financial management system is an ongoing risk. CISP's administration costs are between 12.8 percent and 16.1 percent of its program budget. These costs must be balanced against the high satisfaction rate of recipients and the Program's significant contributions to FDI attraction and strengthened partnerships. The management response to the Results Capacity Check indicated that new resources for CISP would be requested as part of the GCS.
DFAIT management has also provided clear strategic direction for the Program that will refocus it on building the capacity of communities to respond to potential investors. Under this new focus, the Program plans to provide multi-year project funding enhancing the potential for the Program to produce demonstrable and sustainable results.
Weak performance reporting has compromised the Program's ability to demonstrate results as reporting has remained at the output level. The Program has largely failed to use its RMAF as a management tool. While it has updated its logic model, it has yet to update the other parts of its RMAF as recommended in the Results Capacity Check.
Successful capacity development is a long-term endeavour that cannot be accomplished with short-term funding. Multi-year funding is critical to the achievement of results. International studies show that the most successful capacity development gets beyond technical training of individuals, and includes support management systems, networks and adaptation strategies, while paying particular attention to leadership, organizational culture and the incentives that shape individual behaviour.(7) A more holistic approach to capacity development could help municipal councils make better decisions on FDI attraction and retention.
Successful delivery of regional programs and partnership building is dependent on adequate support and good communication between headquarters and the field
Communication is often a weakness of decentralized programs, where it is assumed that technical inputs alone will lead to successful program delivery. Headquarters needs to provide consistent leadership and be responsive to the needs of regional staff that interact directly with municipalities and associations. Building strong relationships and fostering good communication are particularly important in decentralized programs.
The formation of regional and sectoral alliances allows communities to pool their resources, focus their investment-attraction strategies and derive mutual benefit from investment. It also allows for the development of networks, information sharing and mentorship relationships.
International studies(8) have shown that the bulk of new investment comes through the expansion of initial investments, emphasizing the importance of servicing existing investors.
CISP is relevant to Government of Canada and DFAIT strategic priorities particularly in connection with the GCS. The Program is highly valued by recipients and DFAIT staff, who stated that there was an ongoing need and increasing demand. The majority of stakeholders saw CISP as an effective means of building the capacity of municipalities to attract FDI.
While deficiencies in reporting make it difficult to assess to the extent to which CISP has achieved its long-term outcomes, it appears that CISP has contributed to attracting FDI worth hundreds of millions of dollars and thousands of jobs. As well, evidence suggests that it has strengthened partnerships between communities and different levels of government.
CISP has made progress in dealing with many of the weaknesses identified in the 2007 Results Capacity Check including the transfer of financial management to DFAIT from Industry Canada, measures to control funding lapses and clear strategic direction for program renewal. However, challenges remain in performance reporting and in ensuring regional offices have adequate resources to promote, manage and deliver the Program.
The evaluation, therefore, recommends:
That senior management seek renewal of CISP given the Program's contribution to FDI attraction/retention/expansion and job creation as well as its key role in facilitating the Department's relationship with provincial and municipal organizations involved in investment and trade. The level of funding for the renewed program needs to ensure it has adequate resources (both human and financial) to achieve meaningful results.
That senior management responsible for CISP follow-up on the management response in the Results Capacity Check and re-examine the level of human and financial resources needed to adequately deliver the Program at the regional level. Action on this recommendation has the potential to improve program management, provide a project management training course to RO staff, and increase the coaching and mentorship needed for successful capacity development.
That senior management accelerate plans to put in place an integrated corporate program/financial management system which will also assist the Program to streamline its application, claims and reporting processes. Such a system would reduce the risks associated with manual tracking of program commitments and disbursements by program managers.
That CISP management strengthen its systems for capturing performance information and reporting on results. Action on this recommendation includes: development of a new ARAF in concert with the amalgamation initiative with PEMD-A and Going Global Program; using the ARAF as a management tool; updating it regularly; integrating the expected outcomes and performance indicators into application and reporting templates; putting in place a systematic way to capture unexpected results; and developing an annual progress report.
That CISP management establish a minimum level of project funding to increase efficiency. Minimum funding levels and the planned transition multi-year funding will both help ensure capacity-development initiatives go beyond training. Guidance to adjudication committees should ensure the Department provides sufficient funding to support the best projects, rather than dispersing small amounts of money to too many communities or associations.
|Recommendations||IIT Management Response and Action Plan||Time Frame|
|Recommendation 1: That senior management seek renewal of CISP given the Program's contribution to FDI attraction/retention/expansion and job creation as well as its key role in facilitating the Department's relationship with provincial and municipal organizations involved in investment and trade.||Management response:|
IIT has launched the program renewal process in consultation with departmental and non-departmental stakeholders and the Treasury Board Secretariat (TBS).
In order to maximize the allocated resources necessary to achieve desirable results, IIT undertook a study and consultations, in summer 2007, on the possibility of amalgamating three IIT G C programs; CISP, PEMD-A and Going Global S T Program. The analysis demonstrated that such amalgamation would result in streamlined procedures and reduced overall program administration costs. In recent years, DFAIT has developed expertise in managing G C programs. Department-wide collaboration would ensure that adequate resources and systems would be put in place to support the amalgamated program. The TBS submission therefore proposes amalgamation of these three programs into a single program with three pillars. The investment pillar would represent the continuation of CISP.
Recommendation 2:That senior management responsible for CISP follow-up on the management response in the Results Capacity Check and re-examine the level of human and financial resources needed to adequately deliver the Program at the regional level.
|By March 31 2009|
|Recommendation 3: That senior management accelerate plans to put in place an integrated corporate program/financial management system which will also assist the Program to streamline its application, claims and reporting processes.||Management response:|
This implementation of this sound recommendation is a DFAIT corporate issue. A department-wide effort is required to implement such an enterprise system, including the involvement of the corporate finance branch. The process will take considerable time.
|By March 31, 2009|
|Recommendation 4: That CISP management strengthen its systems for capturing performance information and reporting on results.|
|By March 31, 2009|
|Recommendation 5: That CISP management establish a minimum level of project funding to increase efficiency.||Management response:|
This issue has been addressed in the proposed Terms and Conditions of the new amalgamated program.
1 At a 95% confidence level, this means that 95 times out of 100 the true value will fall within an 8-point range of the response rate. Confidence levels may vary depending on the response rate of each individual question.
2 Global value chains refer to the dispersion of production where firms locate different parts of their business across the world so that each part can operate at optimum efficiency.
3 Some communities provided estimated figures on the dollar value of FDI attracted, while others provided information on the estimated numbers of jobs created. In a few cases information was provided on both. In some cases, no information was available.
4 This table uses the data from Table 1.
5 In calculating the admin cost, a 33 percent factor has been applied to direct salary costs to account for employee benefit plans (20%) and the PWGSC accommodations charge (13%). Other indirect program costs that were not included include the following: legal (i.e. advice), finance (i.e. payments, planning, budgeting), human resources (i.e. staffing and training), audit, evaluation, IT systems support, communications (i.e. telephone), general reception, etc.
6 Fourteen percent of recipients called for increased funding when asked how the program should be changed/redesigned.
7 See, for example, Morgan, P. (2006). The Concept of Capacity (draft version). Maastricht: European Centre for Development Policy Management; and Teskey, G. (October 2005). Capacity Development and State Building. Issues, Evidence and Implications for DfID. London: Governance and Social Development Group, Department for International Development. Retrieved September 5, 2007 from http://www.ecdpm.org/.
8 According to UNCTAD. Aftercare: A Core Function of Investment Promotion, Geneva, 2007 (http://www.unctad.org/en/docs/iteipc20071_en.pdf) up to 70 percent of FDI in some countries is linked to existing investments.