Foreign Affairs and International Trade Canada

international.gc.ca

Common menu bar

• Français
• Home
• Contact Us
• Help
• Search
• canada.gc.ca

Breadcrumb

1. Home
2. >
3. About the Department
4. >
5. Accountability

Formative Evaluation - Program for Export Market Development - Associations

(May 2008)

(PDF Version, 726 KB) ^*

• List of Acronyms
• Executive Summary
  • 1.0 Introduction
    • 1.1 Background
    • 1.2 Context of the evaluation
  • 2.0 Evaluation Methodology
    • 2.1 Review of PEMD-A documents and files
    • 2.2 Analysis of PEMD-A activities and results database
    • 2.3 Case studies
    • 2.4 Informant interviews
    • 2.5 Evidence-based questionnaire
    • 2.6 Evaluation limitations
  • 3.0 Description of Program and of Program Activity
    • 3.1 Program objectives
    • 3.2 Allocation process
    • 3.3 Claims process and reporting of performance
    • 3.4 Program resources and expenditures
    • 3.5 Program Clientele
  • 4.0 Relevance
    • 4.1 Consistency of PEMD-A with needs of national sectoral associations
    • 4.2 Opening PEMD-A to horizontal associations
    • 4.3 Duplications with other governmental programs
    • 4.4 Coherence with the Global Commerce Strategy
    • 4.5 Relevance with respect to DFAIT priorities
    • 4.6 Conclusion on the relevance of PEMD-A
  • 5.0 Success
    • 5.1 Measuring and reporting performance
    • 5.2 Outputs and breakdown of PEMD-A funded activities
    • 5.3 Outcomes of the program
    • 5.4 Overall impact of the program
    • 5.5 Conclusion on the success of PEMD-A
  • 6.0 Efficiency and Cost-effectiveness
    • 6.1 Program design
    • 6.2 Appropriateness of application, selection and claim process
    • 6.3 Appropriateness of resources utilization
    • 6.4 Conclusion on the efficiency and cost-effectiveness of PEMD-A
  • 7.0 Conclusions
  • 8.0 Recommendations
• Appendix A: Management Response

List of Acronyms

CEC

Canada Export Centre

DFAIT

Department of Foreign Affairs and International Trade

GCS

Global Commerce Strategy

IBD

International business development

ICT

Information and communications technologies

OEM

Original equipment manufacturers

PEMD-A

Program for Export Market Development- Association

RMAF

Result Management and Accountability Framework

SME

Small and medium enterprises

TBS

Treasury Board Secretariat

TCS

Trade Commissioner Service

Executive Summary

The Program for Export Market Development (PEMD) was introduced in 1971 with the goal of increasing Canadian prosperity and competitiveness in the international marketplace. Throughout its existence, PEMD has taken many administrative forms and has seen its budget vary considerably. In the early 90s, the Program was restructured into four sections, three of which catered to company initiatives, while one was specifically designed to address trade association initiatives.

PEMD-A provides financial support through contribution agreements to Canadian national and sectoral trade associations for export promotion of their members' products and services. Eligible activities relate to the improvement of market access or the development of market intelligence. In addition, activities must be for the benefit of the associations' entire industry - members and non-members alike. These activities are particularly important for small and medium-sized enterprises (SMEs), which are often unable to afford such undertakings on their own.

The purpose of the formative evaluation was to assess the extent to which PEMD-A is meeting its strategic and operational objectives and is achieving the outcomes as identified in the Result Management and Accountability Framework (RMAF). The formative evaluation addressed three evaluation issues as outlined in the Treasury Board's Evaluation Policy and Guide for the Review of Evaluation Reports. These issues are: relevance, success, efficiency and cost-effectiveness.

This evaluation utilized several methodologies including:

• A review of PEMD-A documents, reports and files;
• An analysis of PEMD-A activities and results database;
• Case studies of selected trade associations;
• Face-to-face and telephone interviews with key informants; and,
• An electronic evidence-based questionnaire of all PEMD-A trade association recipients.

I - Relevance

PEMD-A program objectives are relevant to Canada's international commercial policy framework known as the Global Commerce Strategy (GCS). However, the Program Terms and Conditions could be improved to better align with the GCS. With PEMD-A funding, national trade associations are able to contribute to international business development and to improve the Canadian economy. PEMD-A also supports DFAIT's 2007-08 strategic priority for international trade by supporting "greater economic competitiveness for Canada through enhanced commercial engagement, secure market access and targeted support for Canadian business."⁽¹⁾ PEMD-A is also a key element for the promotion of foreign direct investment and science and technology which contributes to DFAIT's strategic outcome of increasing awareness and knowledge of Canadian capabilities as well as increasing interest in Canada as a location for investment.

The evaluation examined the 61 trade associations who made-up the clientele of the Program in 2007-08. Thirty percent (representing 19 associations) belonged to sectors that match the priority sectors of the GCS, and were involved in activities aiming to position their sector in a global supply chain. This group also accounted for 42% of the PEMD-A funds approved for 2007-08. They consist mainly of mature industries like aerospace, automotive industries which already have strong export volumes. Other associations in this group were in developing or emerging industries like ICT, life sciences, bio-tech and environmental industries. These have a high export potential and SMEs often play a leading role in innovation and sector growth. Most associations in this group, except some from emerging industries, are of a significant size and have adequate managerial and financial capacity.

The remaining 70% of the trade associations (representing 43) belong to sectors not included in the GCS. This group of associations have little or no involvement in IBD activities for the positioning of Canadian companies in global value chains. This group also accounted for 58% of the PEMD-A funds approved for 2007-08.

The evaluation found that PEMD-A is not aligned with the GCS and to align it with GCS would mean the following:

• Exclusion of more than half of the current client associations because they are not in the right sectors; and,
• Major modifications to many elements of the program design.

The implementation of such changes may face some difficulties since DFAIT may not be able to impose markets to associations. Associations in some sectors will have to be regrouped, and several associations have very limited capacities.

Information gathered through interviews and questionnaires indicates that the PEMD-A answers the needs of most associations by allowing Canadian trade associations and their members to undertake international business development (IBD) activities. The activities are mainly in the form of participation in international trade shows and conferences and outgoing missions. The vast majority of associations reported that their IBD efforts would be either stopped or significantly reduced if it were not for PEMD-A assistance. This is of particular importance to small associations who would not participate in IBD activities otherwise. Although many associations would like to see some modifications to the Program, mainly regarding the incrementality rule and the inclusion of some administrative expenses, there is a general satisfaction with the Program as it is currently designed. Overall, it can be concluded that PEMD-A meets the needs of trade associations in marketing their products on the world stage.

At the federal level, there is a limited level of duplication with other programs. At the provincial level, some duplication exists since provinces also promote their sectors and companies internationally.

II - Success

Output-level results

An analysis of PEMD-A amounts claimed by the 14 selected associations during the 2002-03 to 2006-07 period indicates that 78% of PEMD-A funding was spent on direct contact activities, 15% on marketing tools and 8% on other marketing activities. The analysis also pointed out the relative importance of trade shows (46% of claimed amounts) and networking events (19%) in the IBD agenda of trade associations.

Key informants consider that PEMD-A is effective in obtaining output results. However the level of unused funds averaged 31% for the period and can be explained by: 1) the difficulty of associations in planning activities well in advance (up to18 months); 2) a lack of resources to deliver the activities; 3) a lack of participation from members; and, 4) unforeseen events.

Outcome- and impact-level results

Information gathered from interviews and the evidence-based questionnaire suggests that the expected outcomes of the Program are achieved to a significant degree by:

• Increased awareness of international business opportunities: The exposure and relationship building achieved through PEMD-A supported activities have increased potential business opportunities, particularly in new or emerging markets, with a lower risk of market entry/development costs.
• Increased awareness of Canadian industrial and commercial expertise in foreign markets: This has been identified as the most important result achieved in the last two to five years.
• Increased IBD by Canadian companies: Many associations, which benefited from PEMD-A support for several years, reported increased sales and market development through distribution agreements that would not have happened without PEMD-A. In some industries, association members increased their participation in the global supply chain of their sector. Most associations said however, that it takes time to take advantage of real business opportunities, particularly in new or emerging markets, and that it is difficult to link increased exports sales to PEMD-A support.

These benefits mostly accrued to SMEs, for which IBD has been accessible and affordable through the work of associations and the support of PEMD-A.

The allocation of PEMD-A funds through a competitive process has not been achieved and has not resulted in more effective and efficient IBD activities by associations because no real competition for funding has taken place.

Overall, the Program has an incremental impact, since information obtained through interviews and the questionnaire indicated that most associations would have reduced or stopped their IBD activities in the absence of PEMD-A. This suggests that PEMD-A is an effective tool for promoting IBD activities. No data is available on the export sales generated by the Program to conduct a cost-benefit analysis to determine if it is economically justified or not.

Furthermore, the impact of the Program cannot be limited to increased exports, because, as was pointed out in one study, for some sectors trade associations play a key role as enablers of innovation because "their activities are driven by the needs of their clients and because they have specialized knowledge of the context in which technology is applied and new products developed."⁽²⁾ Results are unequal between associations, and it can be concluded that a relatively small number account for significant positive results, while several others obtain only limited results.

III - Efficiency and Cost-Effectiveness

The introduction of the online application and claims system in 2002-03 is considered a great improvement in terms of cost-effectiveness and efficiency. The online system is also praised by associations and PEMD-A management. However program efficiency and cost-effectiveness have been hampered by the loose application of its eligibility criteria. The open character of the Program has attracted a diverse group of associations. Several have only a marginal involvement in marketing products internationally and a limited capacity for the provision of IBD services. This has had a direct impact on the quality of the Program's performance information and on the ability to measure results.

Eligibility Criteria

The Program is designed to favour activities that tend to benefit SMEs but this criterion is not rigorously applied. Given that the PEMD-A clientele is very heterogeneous with respect to membership and that information collected on association membership is inaccurate, it is not possible to determine the degree to which associations really serve SMEs rather than large companies or individuals.

As well, many associations are not national since they only have a provincial clientele. The Program considers these associations as national if they represent companies having specific regional products for export. However, this does not always appear to be the case.

The "managerial and financial capacity" of associations is an eligibility criterion that is also loosely defined and applied. It was found that although the majority of associations are reporting that they have the human and financial resources necessary to carry out their IBD activities, a significant proportion of associations have only a few employees and very small budgets. The evaluation also found that there is an inverse relation between the association's size and the ratio of unused funds. Program managers indicated that the limited capacity of some associations has led to inefficiency and poor results.

Relevance of Supported Trade Promotion Activities

The general perception from associations is that the activities supported by the Program are relevant for achieving IBD results. Some associations would like to get support for a wider range of marketing activities, and particularly would like to see trade events that occur within Canada as part of PEMD-A eligible activities.

Since May 2006, PEMD-A has been funding the participation of individual member companies to the Canada Export Centre (CEC) - a private permanent trade exhibition located in Vancouver. Three associations who were interviewed and who took advantage of the CEC exhibition said that it contributed only marginally to their IBD objectives and that the number of international visitors was limited.

Appropriateness of Level of Funding

The evaluation concluded that the level of funding was generally sufficient considering the capacity of associations. However, many associations believe that the federal government does not recognize the real costs that associations must incur in the delivery of trade promotion activities. They aregue that if funding could be expanded to cover a portion of administrative and operational costs, they could undertake more IBD activities and thereby use more PEMD-A funding.

Many associations also believe that the $25,000 set aside for assistance to companies is not sufficient and that by limiting the financial assistance to companies, the Program is reducing its chances of success.

Unused Funds

During the last 5 fiscal years, the amounts claimed by associations averaged 69% thus resulting in unused funds of 31%. The percentage of unused funds was clearly related to the capacity of the associations as reflected by the number of their employees and the amount of their PEMD-A funding. For example, in 2006-07, associations with less than 20 employees were on average unable to spend 48% of their funding whereas associations with 20 or more employees were unable to spend only 26% of their funds. Likewise, the rate of unused funds was higher for associations who were approved lower amounts - 32% for amounts approved ranging between $100,000 and $150,000 compared to 55% for associations with approved amounts ranging between $25,000 and $50,000. Also some sectors were able to spend their funds better than others. For example, manufacturing (4%) and mining equipment services (9%) performed better than automotive (90%) and aerospace and defence (67%). The high level of unused funds is a matter of efficiency that could be improved with more capable associations and a more focussed selection of sectors. Unused funds are also a result of Program inflexibility particularly in the use and reallocation of funds by associations.

In order to reduce the ratio of unused funds, the Program should be more flexible with respect to: 1) the reallocation of unspent amounts between packages of activities; 2) the modification of packages of activities 3) the maximum amount per package (i.e. $50,000); and 4) the claims before the end of an activity package to reduce cash-flow problems.

Appropriateness of Application, Selection and Claims Process

As already mentioned, there is a general satisfaction with the online application. The support received from sector officers to guide associations in the preparation of their application is adequate. However, some associations note the high turnover of sector officers. The practice of rotating staff into new sectors diminishes the benefit that these individuals are able to bring to their jobs.

The Review Board selection process, which requires that activity packages of each application be scrutinized and approved by a board of three senior DFAIT managers in a 3-day session, is considered to be largely ineffective and inefficient for many reasons: too much information to digest; lack of adequate information on past results; turnover of board members; pressure of time; and lack of clarity of criteria.

Performance Measurement and Reporting

Performance measurement is a weakness of the Program because it has proved difficult to monitor output- and outcome-level results.

The evaluation considers that the Program's performance measurement strategy should be substantially improved, because the existing tools to assess performance reveal some weaknesses in the logic model and the reporting of results.

Program management has accomplished considerable work to remodel the performance measurement framework that will contribute to a better assessment and monitoring of future results. This work is still in progress and, if the Program's Terms and Conditions are significantly modified, then the changes proposed will have to be reconsidered to take into account new objectives of a redesigned Program.

Appropriateness of Resources Utilization

The administrative cost of the Program was calculated to be a minimum of $440,283 and a maximum of $536,935 taking into account salaries, 20% for the employee benefit plans; and 13% for the PWGSC accommodations charge.⁽³⁾ These costs translate into an estimated admin ratio of between 14.6% and 17.8% for 2006-07.

Recommendations

• Identifying the positioning of Canadian firms in the global value chain as a PEMD-A objective;
• Reviewing the range of promotion activities that can be funded in order to enable associations to intervene effectively;
• Restricting the Program to sectors with high export potential and high or medium S T intensity, where integration in the global value chain is essential for the growth and survival of Canadian business participation; and,
• Modifying other elements of the Program to make it coherent and efficient.

• impose a minimum of $50,000 for the approval of an application;
• allow associations to hold a percentage of PEMD-A funding in order to cover their administrative costs, particularly in the case of flow-through funds for partnering organizations;
• require associations to provide more details with respect to association membership to better identify SMEs and the extent to which SMEs benefit from PEMD-A;
• require associations to provide more details with respect to financial and managerial capacity to better identify sources of funding and FTEs;
• allow more flexibility with respect to the incrementality criterion, by applying this concept to results rather than activities;
• allow more flexibility in the identification, design and costing of activity packages, Consistent with a multi-year strategy, especially for Year-2 and Year-3;
• allow more flexibility in the transfer of funds between different activity packages;
• increase funding allocated to assist companies to a minimum of $50,000; and,
• standardize activity packages and output results in order to facilitate the grouping of results and their aggregation. It is also recommended to revise the current application and reporting procedures in a way to match results achieved by associations with the logic model.

Top of Page

1.0 Introduction

1.1 Background

The Program for Export Market Development (PEMD) was introduced in 1971 with the goal of increasing Canadian prosperity and competitiveness in the international marketplace. Throughout its existence, PEMD has taken many administrative forms and has seen its budget vary considerably. In the early 90s, the Program was restructured into four sections, three of which catered to company initiatives, while one was specifically designed to address trade association initiatives.

PEMD-A provides financial support through contribution agreements to Canadian national and sectoral trade associations for export promotion of their members' products and services. Eligible activities relate to the improvement of market access or the development of market intelligence. In addition, activities must be for the benefit of the associations' entire industry - members and non-members alike. These activities are particularly important for small and medium-sized enterprises (SMEs), which are often unable to afford such undertakings on their own.

In 2002, the PEMD-A was reviewed and several recommendations were made. Among them were: 1) a reconsideration of the restraints on PEMD that have led to its decline in scope and scale; 2) a reconsideration of the dispersal of resources among various instruments of trade promotion (namely: exports, investment, and science and technology) and among departments of the federal government; and 3) a consideration of a more integrated approach to international business development. In 2002, the review concluded that

"PEMD-A is strategically important, well established and well accepted by its clientele. It reaches SMEs by working with many associations of all sizes whose membership is mostly SMEs. For a small budget the outlay program provides DFAIT Trade Commissioner Service sector staff with an entrée to these associations and with the opportunity to work with them directly to improve their capabilities in trade promotion."

The Summative evaluation in 2005 concluded that PEMD-A was cost-effective by increasing "its range and benefits while holding its administrative expenses constant." The evaluation concluded that "PEMD-A roughly breaks even, economically. The net economic benefit was between 'plus $1 million' and 'minus 0.48 million' per annum." Additionally, PEMD-A "had an important positive (but unmeasurable) strategic value."

Since 2002 DFAIT has funded activities totalling approximately $11.8 million to an average of 55 participating associations each year.

1.2 Context of the evaluation

In December 2006, the Global Commerce Strategy (GCS) in support of Advantage Canada was approved. To ensure a better alignment of the PEMD-A mandate to the new strategy, the department is currently undertaking, in parallel to this evaluation, a revision of the Program including its Terms and Conditions. A new organizational proposal of the PEMD-A has also been adopted, which will see the administration of the Program moved from the SME Support and Services Division to the Business Sectors Bureau Division.

Top of Page

2.0 Evaluation Methodology

The evaluation focused on the three issues as outlined in the TBS "Evaluation Policy" and TBS "Guide for the Review of Evaluation Reports." These issues are relevance, success and efficiency and cost-effectiveness.

The methodology used to analyze these issues included:

• a review of PEMD-A documents, reports and files;
• an analysis of a PEMD-A activities and results database;
• case studies of selected trade associations;
• face-to-face and telephone interviews with key informants, and,
• an electronic evidence-based questionnaire of all PEMD-A trade associations recipients.

2.1 Review of PEMD-A documents and files

The review of PEMD-A documents and files has been a major component of the evaluation. The objective of the file review was to provide a general description of the Program with respect to:

• background
• current design and evaluation framework;
• delivery
• clientele
• activities and results
• budgets and expenditures.

Documents reviewed included but was not limited to the following:

• Reports on previous evaluation studies (2001; 2005)
• Results Based Management and Accountability Framework (2005);
• Recommendations Report on performance measurement (2007);
• Management files (work plans, management reports);
• PEMD-A process map;
• Trade Association Handbook (2002-03 to 2007-08);
• Trade Association applications (2002-03 to 2007-08);
• Review Board documents (2006-07 and 2007-08);
• PEMD annual reports (2002-03 to 2006-07);
• Program statistics;
• Program administration expenditures (2002-03 to 2007-08); and,
• Program human resources (2002-03 to 2007-08).

2.2 Analysis of PEMD-A activities and results database

A multi-year (2002-07) database was exported from the program management system. This database, which records all the activity packages funded since 2002-03, included many variables. However, a preliminary review of the database showed that in many instances the data was incomplete, so that the database could not be used in the evaluation. Instead, evaluators had to generate a database from other public sources containing the following data:

• the size and membership of the associations (number of employees, number of members and SME members, total funding);
• the source of their funding (private and public sources);
• the number of years of PEMD-A funding;
• program funding (requested, approved and claimed); and,
• distribution of funding by type of activity.

2.3 Case studies

Case studies were developed for a group of 14 trade associations on the basis of file reviews and face-to-face or telephone interviews. To ensure that the group represents the PEMD-A clientele, selection criteria were used including: a mix of sectors, location, and size, longevity of PEMD-A participation, level of unsed funds historically, quality of reporting and level of input from project officers.

The case studies were used to analyze the evaluation issues and questions more in depth. The topics covered were:

• description of associations with respect to mission, membership (SMEs and non SMEs), size, funding, activities;
• export promotion strategy, export potential and export activity of members;
• participation to PEMD-A including: funding approved and claimed by year; activity packages; and, member participation; and,
• results including output and outcome level results.

2.4 Informant interviews

A total of 28 interviews were conducted with stakeholders of the Program to obtain in-depth qualitative information, unexpected difficulties and lessons learned. Interview protocols were developed on the basis of the evaluation framework for each category of interviewees. Stakeholders interviewed included PEMD-A managers (n=3); Board Review members (n=2); sector officers (n= 2) in addition to a focus group; people involved with the GCS (n=2) and associations' representatives (n=14).

2.5 Evidence-based questionnaire

An evidence-based questionnaire was developed by the evaluation team around the issues of relevance, performance, efficiency and cost-effectiveness of the Program. The questionnaire was sent on October 19, 2007 to 94 associations who had received PEMD-A funding since 2002-03. The electronic questionnaire was closed on November 13, 2007. The response rate of the survey was 36%. ⁽⁴⁾

2.6 Evaluation limitations

The evaluation was conducted with few limitations. Making the arrangements for interviews (in-person and by telephone) and site visits occurred from September to November 2007 with no restrictions. Additionally, the evaluation team had access to information, documents, databases and files in a timely manner.

The evaluation itself was conducted at a time of Program reorganization. An organizational proposal was adopted during the evaluation whereby the administration of the Program moved from the SME Support and Services Division to the Business Sectors Bureau Division. To ensure a better alignment of the PEMD-A mandate to the GCS, DFAIT also set out to amalgamate PEMD-A with two other DFAIT programs: the Community Investment Support Program (CISP) and the Going Global Science and Technology Program under a common set of Terms and Conditions and common administrative structure.

Top of Page

3.0 Description of Program and of Program Activity

3.1 Program objectives

The PEMD-A mandate is

"to contribute to the creation of employment and prosperity in Canada by assisting the Canadian business community in taking full advantage of the international business opportunities."

The objective of the PEMD-A is to assist national and sectoral trade associations in the generic international business development activities undertaken on behalf of their members and sector. The logic model of the Program given in the RMAF further specifies the expected results of the Program as:

Intermediate outcomes:

• increased awareness of international business opportunities
• increased awareness of Canadian industrial and commercial expertise in foreign markets

Long-term outcomes:

• increased international business development by Canadian firms
• increased Canadian employment, job creation and economic growth.

Through this, the benefits to Canada are:

• increased total employment and job creation and retention by enhancing the capability and effectives of international business development by Canadian business;
• encouraging and promoting the pursuit of excellence in the development of international markets by allocating funds on the basis of competition; and,
• helping Canadian businesses become more effective in international business development initiatives.

There are three types of international business promotion activities that are funded by PEMD-A:

• direct contacts such as: as trade shows, outgoing missions and incoming visits;
• marketing tools such as: web site development targeting foreign customers, print materials and other materials that promote the export of associations' goods; and,
• other marketing activities such as: research, awards programs, and indirect marketing that promotes relationship building, or improves access to foreign markets.

The eligibility criteria of associations to the Program help to somewhat clarify the very broad objectives of the Program. To be eligible for PEMD-A, an association must:

• be a national trade or industry association (or a regional association with a national perspective);
• be registered as a partner-client (industry association) with the Virtual Trade Commissioner
• be incorporated;
• not directly sell products that are sold by its members; and,
• be sector-specific.

Generally all expenses used to promote international business development are considered eligible with some published exceptions.

3.2 Allocation process

Throughout the year, eligible associations work alongside Trade Commissioners at Headquarters and at Regional Offices to develop packages of activities based on the three types of international business promotion activities identified above.

The concept of packages of activities was introduced as a way to group market development activities which have identical objectives or which lead to the same type of results. Applicants may submit up to ten packages of activities, with a maximum of five activities per package. Although there is no minimum amount for individual packages of activities, the maximum amount per package is $50,000.

The cost of activities is shared equally between the association and the federal government. Annual non-repayable government contributions range from a minimum of $20,000 to a maximum of $150,000, before optional supplemental funding.

The associations must provide matching funds originating from private sources. Associations funded entirely by governments are not eligible (i.e. federal, provincial or municipal). If an association receives other government funding, support will be granted only when the association's contribution for the proposed activities comes from another (private) source, like membership dues or sponsorships.

The application period stretches from mid November to late January, during which sector officers assist associations in finalizing their application based on the goals and objectives of the Program. All applications are then submitted online by associations and reviewed and rated by sector officers according to pre-set criteria. Minimum scores are required for each criterion in order for the applications to be recommended for funding to the Review Board. The Board does the final evaluation and rating and either approves or rejects the applications. Trade Commissioner Service officers are also requested to provide information and advice on the merits of applications from the market viewpoint. For the last two years, sector officers have appeared at the Review Board to answer questions with regard to the applications and their sectors.

Applications are assessed by the review board according to the following criteria:

• export strategy (30%);
• incrementality (30%), that is, the presence of new or expanded international market development initiatives;
• membership benefits (15%) and;
• the association's performance in the past two years (25%).

All applications that get a minimum score of 50% will have their packages of activities evaluated by the Review Board. Funding decisions are made on a package by package basis, ranked primarily according to the incrementality of the activities but taking into consideration several other factors such as the appropriateness of performance indicators, capacity of associations to complete the proposed activities, past performance, etc.

Adjudications are held in January - February with the announcements made at the end of February so that successful applicants can initiate activities due to start as of April 1. For each successful association, a contribution agreement is signed by both PEMD-A and an authorized officer of the association.

Limited supplemental PEMD-A funding, based on funds unused by trade associations may be made available to associations in September to provide a second opportunity to the funding recipients to add activities to their current program, and also allow PEMD management to reallocate unused funds prior to the end of fiscal year. Supplemental funding is allocated to associations that request it by taking into account the completion rate of packages initially approved for the year and the merit of the new request. Supplemental funding rounds have been held in the fall of 2003-04, 2005-06 and 2006-07.

Several of the recommendations made have been put in place in time for the 2006-07 application round. The major modifications made to the Program were to increase the ceiling allowed per applicant from $100,000 to $150,000 effective in April 2006, and to create a mechanism allowing associations to allocate up to $25,000 for partnering with other organizations.

3.3 Claims process and reporting of performance

Associations can claim a payment only after all activities in a package are completed. They must submit their claims within 30 calendar days of completion of a package with a description of the expenditures incurred and report their activities against the performance indicators they chose for each activity in the package. Associations are not requested to submit receipts with claims but they must provide a tangible "Proof of Activity" that resulted from the package. Associations are audited selectively to control financial risks.

When an association completes its final package of activities for the year it must also submit a final report comparing the planned results to the actual results achieved.

3.4 Program resources and expenditures

Table 1 presents basic indicators of PEMD-A budgetary allocations and expenditures, as well as the number of applications received and approved from 2002-03 to 2006-07.

Budget figures for PEMD-A are available since 2005-06 only. In prior years, budgetary provisions were made jointly for PEMD-Industry and PEMD-A. The budget was decreased from $4.02 million in 2005-06 to $3.62 million in 2007-08.

The number of applications received increased from 53 in 2002-03 to a maximum of 67 in 2006-07, and stood at 62 in 2007-08, while that of applications approved went from 50 in 2002-03 to 64 in 2006-07 and declined to 61 in 2007-08.

The amounts approved for contributions to associations averaged $3.1 million from 2002-03 to 2004-05, but increased sharply to an average of $4.7 million in 2006-07 and 2007-08, partly as a result of the maximum having been augmented to $150,000.

As seen in Table 2 the total claims of the Program from 2002-03 to 2006-07 was $11.80 million. Starting from a low of $1.79 million, claims averaged $2.1 million from 2002-03 to 2004-05, and increased by 60% to $3.38 million in 2006-07. Total claims for 2007-08 dropped to $3.30 million.

Table 2 also shows that from 2002-03 to 2006-07 the amounts claimed by associations averaged 69% of the total funding approved each year. In order to reduce the level of unused funds, program managers have established a supplemental round of funding so that unused funds by some associations are made available to others. The supplemental funding appears to have raised the percentage of claims approved in 2005-06 and again in 2006-07.

Also, during 2003-04 to 2007-08 PEMD-A has approved $19.05 million of funding or 86% of the amounts requested by associations, excluding supplemental funds. The approval rate was stable at 83% until 2007-08 when it increased to 92%. The total amount of funding approved over 2002-03 to 2007-08 was $21.72 million.

3.5 Program Clientele

Over the 2002-03 to 2007-08 period, the PEMD-A has approved applications from a total of 101 different associations. The number of recipient associations for each sector is presented in Table 3, as well as the number of withdrawals and new applicants in each year.

Although the numbers of applications received and approved per year were fairly stable over the period, there was a 22% turnover rate of recipient associations, with averages of 11 withdrawals and 11 new applicants per year.

Sectors having the highest participation are:

• Service Industries Capital Projects (20 associations);
• Forest Industries (12 associations);
• Arts Cultural Industries (10 associations);
• Information Communication Technologies (10 associations);
• Consumer Products (9 associations); and,
• Building Products Construction (8 associations).

Funds were allocated to 19 sectors as described in Table 4. Sectors with the highest percentage of approved and claimed funding over 2002-03 to 2006-07 were:

• Service Industries Capital Projects: 18.3% of approved and 15.9% of claimed;
• Consumer Products: 11.9% of approved and 15.0% of claimed;
• Forest Industries: 11.0% of approved and 12.4% of claimed;
• Arts Cultural Industries: 8.7% of approved and 9.4% of claimed;
• Information Communication Technologies: 8.2% of approved and 7.6% of claimed;
• Building Products Construction: 7.0% of approved and 8.4% of claimed.

Together these six sectors requested 65% and received 69% of the total PEMD-A funding during the last five years.

Top of Page

4.0 Relevance

Assessment approach

The relevance issue, as presented in the Terms of Reference of the evaluation, required examining if the Program continues to be a strategic priority for the Government of Canada and DFAIT. Specifically the evaluation team examined the following areas:

4.1 - Consistency of PEMD-A with needs of national sector associations
4.2 - Opening of PEMD-A to horizontal associations
4.3 - Duplication with other government programs
4.4 - Coherence with Global Commerce Strategy
4.5 - Relevance with respect to DFAIT priorities

4.1 Consistency of PEMD-A with needs of national sectoral associations

Perspective of Associations

Information gathered through interviews and the electronic questionnaire indicates that the Program answers the needs of most associations.

Perspective of DFAIT

DFAIT managers concurred in saying that there is a justification for a program like PEMD-A allowing associations to go to other countries showcasing Canadian products. It was also felt that several other countries provide much greater support to international marketing by their trade associations and that Canada should maintain its comparatively modest assistance.

The view among sector officers was that PEMD-A is important for most trade associations and that it makes a difference particularly in sectors that are struggling and need assistance e.g. the apparel industry. It is also useful for emerging industries like the life sciences sector and well-established sectors like wood and building products. On the other hand, it is believed that several associations do not have an international perspective to start with and that the funding offered by PEMD-A in some way creates the need for international marketing by those associations.

4.2 Opening PEMD-A to horizontal associations

The terms "horizontal associations" and "vertical associations" are used to describe trade organizations in relation to the goods produced or services offered in the marketplace. Associations are said to be "horizontal" if their products or services meet the needs of more than one industry and "vertical" if their goods and services cross various markets. Vertical associations may conduct trade promotion activities in various markets. Horizontal associations, on the other hand, would focus their product or service to meet the needs of only one industry. For example, associations in the information, communication and technologies (ICT) sector could include horizontal markets such as: computer storage, desktop graphics, computer design. It could, however, also include vertical markets outside the ICT sector. These could include: banking, healthcare, government, insurance, etc.

DFAIT recently contemplated expanding PEMD-A to horizontal associations and requested that it be part of the evaluation to assess the relevance of this proposal. The evaluation team was requested by PEMD-A management to ask trade associations about their perception vis-à-vis opening the Program to horizontal associations. findings are based on the opinions expressed during the focused interviews.

The evaluation found that trade associations were not in favour of opening the Program to horizontal association. They provided the following reasons:

• Limited funding: Horizontal association would dilute the funding available unless additional funds would accompany the Program expansion;
• Program objectives blurred: Horizontal associations would blur the objectives of the Program if associations are not sector-based;
• Eligibility criteria more difficult: Horizontal associations would make the eligibility criteria to the Program more complex and selection criteria would be more difficult to apply; and,
• Results reporting problematic: Horizontal associations would make reporting results more difficult to assess.

4.3 Duplications with other governmental programs

The evaluation did not make a full inventory of other government programs that are used by trade associations at the same time as PEMD-A. There was, however, a review of the assistance provided by other government programs to the 14 associations included in the sample and their distribution of public funding ratios and dollar amounts. This information was based on the data supplied by associations in their PEMD-A application.

In principle the PEMD-A does not allow duplication of its funding with other programs because, according to its Terms and Conditions, "PEMD-Associations support will be granted only when the required 50% contribution by the association for proposed activities comes from private sources. Private revenues are non-public funds and include membership fees, sponsorship, etc."

The findings of the evaluation show that at the federal level programs offering assistance similar to PEMD-A are limited to two sectors mainly: a) the arts and cultural industry; and b) the forestry and woods products industry.

Arts and Cultural Industries

There are at least two programs of Heritage Canada providing financial assistance to associations:

• The Trade Routes program is the only other program offering assistance for export promotion activities. It has an annual budget of $2.0M to meet the needs of individual companies and associations, with contributions averaging $20,000 and generally not exceeding $50,000. It requires that public funding be limited to 75% of project costs.
• The Canada Music Fund's Support to Sector Associations Component offers to associations some assistance to pursue objectives other than export promotion, with a maximum of $250,000 and a 50 : 50 funding ratio.

The evaluation team identified only two cases of dual use of Trade Routes and PEMD-A:

• the Association of Registered Interior Designers of Ontario received $54,000 from Trades Routes and $22,500 from PEMD-A in 2004-05; and,
• the Association for the Export of Canadian Books (AECB) received $7,500 from Trades Routes and $66,400 from PEMD-A in 2005-06.

The Canadian Independent Record Production Association (CIRPA) was the largest recipient of Trade Routes in 2005-06 and 2006-07, with contributions of $80,000 and $50,000, but it received no PEMD-A funds in those two years. CIRPA also received close to $200,000 in 2005-06 and 2006-07 from the Canada Music Fund-Support to Sector Associations Component, plus money from the Ontario Media Development Corporation, from the Foundation Assisting Canadian Talent on Recordings (FACTOR) and from the Society of Authors and Artists of Canada (SOCAN).

Heritage Canada also offers funding for export promotion activities through the International Marketing Assistance component of the Book Publishing Industry Development Program. This program is managed by the Association for the Export of Canadian Books for Heritage Canada (AECB) and is the main source of funding of that association.

Provincial governments also support arts and culture with programs aiming at several sectors, among which the new media and the film and television production segments are the largest funding recipients and the same trade associations that PEMD-A supports.

Forestry and Wood Products

The forestry and wood products is a sector⁽⁷⁾ receiving extensive assistance from federal and provincial programs for export promotion initiatives of trade associations. At the federal level, the Canada Wood Export Program (Canada Wood) is the major federal program and is directed at non-US markets, while PEMD-A is directed at the USA market.

Canada Wood is a five-year ($35M) program launched in 2002 and renewed in 2007 for an additional two years ($20M). It is a cost-shared program funded by Natural Resources Canada in partnership with forest product associations from across Canada and with provinces and industry partners aiming to diversify Canada's wood products exports to offshore markets.

The magnitude of PEMD-A assistance to the Forestry and Wood products sector, which amounts to about $0.5M per year for the USA market, is dwarfed by that of Canada Wood, given that the USA market accounts for at least 80% of Canadian exports.

There is at least one case of apparent duplication, that of BC Wood, which has received PEMD-A funds in the last two years for activity packages related to non-US markets.

There seems also to be some duplication with provincial programs in British Columbia, mainly with the Forestry Innovation Investment (FII) program which provides large amounts of funding for international marketing to associations⁽⁸⁾ that are also supported by PEMD-A. These associations and their level of funding from FII in 2006-07 are:

• Western Red Cedar Lumber Association: $1.54M;
• Canadian Wood Council: $1M;
• BC Wood: $1M;
• B.C. Shake and Shingle Association: $0.2M

The Quebec Government also provides support to "Bureaus de promotion des produits forestiers du Québec," an association also supported by PEMD-A and by Canada Wood Export.

Other Industries/Sectors

Apparel:

The Canadian Apparel Federation used to receive some support for international marketing from Industry Canada, while also receiving PEMD-A funding. The Quebec government is investing $83M over 5 years in the Pro-Mode program, which has an important international marketing component, which may lead to duplication with PEMD-A.

Aerospace:

The Quebec Government provides support to the Quebec Aerospace Association (QAA), which is also supported by Economic Development Canada. QAA is heavily involved in promoting the participation of Quebec aerospace companies in the international supply chain. QAA organizes outgoing and incoming missions with the support of the Quebec Government, and this has led in the past to duplication with the efforts of the Aerospace Industries Association of Canada and to the cancellation of activities to be funded by PEMD-A.

Education:

Three not-for-profit corporations receiving funds from PEMD-A depend heavily on public funds for their international activities. There may be duplication of PEMD-A with their regular funding sources used for international marketing in developing countries.

Information and Communication Technologies:

• PEMD-A has supported the Canadian Microelectronics Corporation, a not-for-profit corporation involved in microelectronics research projects, having university and corporate members and tied to Queen's University. It is 97% funded by the National Research Council, which means that PEMD-A has funded a public entity.
• The Canadian Photonics Consortium reports that some of its PEMD-A funding could not be spent because the activities were funded by Ontario or Quebec.
• Quebec also supports Nano-Quebec, a provincial trade association devoted to nanotechnologies, which promotes the participation of Quebec firms in the global supply chain of the sector.

Environment:

The Hydrogen and Fuel Cells Association benefits from wide government support at the federal and provincial level with 85% of its $4.3M budget coming from public sources. It has also received funding from the Industrial Research Assistance Program (IRAP-Industry Canada) for a trade show in Germany. IRAP offers financial assistance for international marketing and technology showcasing activities to other industries.

4.4 Coherence with the Global Commerce Strategy

The Global Commerce Strategy mentions that the Program for Export Market Development be:

"formally reviewed and adapted to ensure alignment of its terms and conditions with the Global Commerce Strategy."

The Global Commerce Strategy, which is an integral component of Advantage Canada and of the government's overall economic agenda, will integrate departmental program activities and policy objectives related to a) opening new markets, b) promoting foreign investment and innovation, and c) strengthening services to business, in collaboration with key partner departments and agencies. The GCS will include new approaches for DFAIT to meet the evolving and increasingly complex needs of Canadian firms. It will reorient DFAIT's services to give Canadian firms a competitive edge in global markets, whether they are attempting to export goods or services, penetrate global value chains, establish operations abroad, develop strategic partnerships or connect with global innovation networks.

The evaluation found that the PEMD-A in its present form is not aligned with the Global Commerce Strategy. First, PEMD-A is open to sectors and markets, while the GCS clearly aims at specific sectors and markets. Secondly PEMD-A supports the IBD work that associations do for the benefit of all of their members, while the GCS fundamentally aims to integrate Canadian firms in global value chains which requires working more at the level of individual companies.

The openness of the PEMD-A creates a mismatch between its current clientele and the clientele that could be targeted once it is aligned to the GCS.

To illustrate this the evaluation produced an analysis of the PEMD-A clientele⁽⁹⁾ showing that out of the 61 associations that made-up the clientele of the Program in 2007-08, only 19 (or 30%) belonged to sectors that match the priority sectors of the GCS, and were involved in activities aiming to position their members in a global supply chain. This group, however, accounted for 42% of the PEMD-A funds approved for 2007-08.

Other criteria used to analyse the compatibility of the current clientele to a Program designed along GCS objectives were:

• the science and technology intensity of the industry represented by the association, on the basis that priority is given to high-technology sectors in coherence with the Science and Technology policy of DFAIT and of other government policies aiming at innovation;
• the export potential of the industry, on the basis that only associations representing industries which have a significant export potential should be assisted by PEMD-A;
• the stage of development of the industry, on the basis that there is a stronger justification to assist emerging and developing industries, than there is to assist a mature industry.

It is also not clear that the IBD activities presently supported by the Program would be effective tools to allow trade associations to intervene in global value chain positioning of their members and contribute to the implementation of the GCS.

It appears that, if PEMD-A is to contribute to the objectives of the GCS, then:

• the eligibility criteria of the Program should be modified to reflect a strategy of picking winners in terms of sectors and markets, which would eliminate many of the current PEMD-A participants; and,
• the design of the Program should be modified to better enable trade associations to position their member firms to participate in the global value chain of their industry, as suppliers, importers or investors, and as sellers or buyers of technology.

The alignment of the Terms and Conditions with the Global Commerce Strategy would then have the following implications:

• The use of selection criteria restraining PEMD-A eligibility to GCS sectors would mean that more than half of the current clientele would be excluded.
• The range of activities to be funded by the Program would be extended to include the promotion of:
  • the participation of companies all along the supply chain;
  • inward and outward investment;
  • technology transfer and R D.
• Other elements of the Program would also have to be deeply modified:
  • greatly enhanced sector strategies to be developed jointly with associations, with assistance of DFAIT and other government departments (Industry Canada, and the participation of other stakeholders;
  • much greater collaboration between associations and sector officers and post officers to work on priority sectors and markets;
  • multi-year contribution agreements for sector strategies and activity programs to replace current 1-year contribution agreements;
  • modification of online application and claims management system to reflect those changes; and,
  • definition of new success indicators, at the output and the outcome level, would have to be adapted to the results expected in the promotion of global value chain participation. These indicators should correspond to the key priority indicators of DFAIT's Plans and Priorities.

PEMD-A management has already started to integrate Global Commerce Strategy priorities by mentioning in the PEMD-A Handbook for 2008-09 that it will give consideration to a series of priority sectors and markets.

4.5 Relevance with respect to DFAIT priorities

In the RMAF, the objectives of PEMD-A are defined in very broad terms with respect to the eligible clientele and to the results expected at the outcome and impact level. The objectives of PEMD-A being so generic, they could not but coincide with the general objectives of DFAIT in the area of international trade, as they existed before the introduction of the GCS.

PEMD-A has historically been an open program whereby DFAIT was inviting the participation of many associations provided that they met the general objective of the Program.

The openness of the Program has attracted a clientele of trade associations that are quite diverse in terms of sectors and membership role in the area of IBD. This impact on the relevance of PEMD-A are as follows:

1. The implementation of the Global Commerce Strategy. GCS is aiming at priority sectors and markets, which precludes maintaining the Program as a program open to all sectors and associations. This decision means that DFAIT will have to adopt eligibility criteria to select associations that can work in harmony with the GCS for the integration of Canadian firms in global value chains and exclude others even if they obtain good results through their PEMD-A participation.
2. The accountability obligation of DFAIT. Accountability obligations require a more rigorous results-based management approach to PEMD-A and therefore supporting associations that cannot show results or that show negligible results may need to be discontinued.
3. The support of Canadian exports in traditional markets and sectors. The program helps to showcase Canada and it is a public policy objective that the Program helps to achieve. Although this priority is not mentioned in the 2007-08 Report on Plans and Priorities, DFAIT managers indicated in interviews that the GCS aims to concentrate incremental resources on priority markets and sectors, but that the exclusion of traditional sectors and markets from the GCS does not mean these sectors will not continue to be supported by DFAIT. If this were the case, PEMD-A would continue to be relevant for those sectors.

4.6 Conclusion on the relevance of PEMD-A

Overall, it can be concluded that the PEMD-A, to a large extent meets the needs of trade associations who want to get involved in international business development for their members, but that, for DFAIT, it is no longer relevant in its present form, as a program open to all sectors and markets and pursuing broadly defined objectives of increased awareness of international business opportunities by Canadian companies and increased awareness of Canadian industrial and commercial expertise in foreign markets.

Although the Program still meets the general objectives of DFAIT and is not an outright duplication of other government programs, it is not in line with the objectives of the Global Commerce Strategy to advance the international business interests of Canadian companies and promote their integration in global value chains in priority sectors and markets and to connect them with innovation networks.

These elements put together mean that the realignment of the PEMD-A Terms and Conditions along the orientations of the GCS would require a substantial revision of its objectives and of its eligibility criteria.

Top of Page

5.0 Success

Assessment approach

In order to assess the results of the Program with respect to results, the following sources of information were used:

• PEMD-A database describing the activities funded by the Program and the results achieved;
• the claims status, package performance and final results reports submitted by the associations;
• face-to-face and telephone interviews with associations, sector officers and program managers; and,
• an electronic questionnaire of 94 associations who participated to PEMD-A during the 2002-07.

The evaluators used the Result Based Management and Accountability Framework (RMAF) as the framework to assess results.

The evaluation will assess results in the following areas:

5.1 - Measuring and reporting performance
5.2 - Outputs and breakdown of PEMD-A funded activities
5.3 - Outcomes of the program
5.4 - Overall impact of the program

5.1 Measuring and reporting performance

The Program has developed a Results Based Management and Accountability Framework (RMAF) describing the logic model of the Program as well as the performance, evaluation and reporting strategy. The reporting of results has however been a weak point of the Program. The following excerpt from the 2006-07 Annual Report summarizes the difficulties encountered.

Tracking the actual quantifiable results from the program's contributions is challenging for two key reasons. First, associations can only report to the program results which their member firms have reported to them. Second, results from export market development activities may generate leads and opportunities, but do not always have immediate quantifiable results.

The inadequate reporting of results has led DFAIT managers to ask for more concrete results than showcasing or sales leads, year after year.

The evaluation considers that the Program's performance measurement strategy should be substantially improved, because the existing tools (i.e. logic model and results reports) to assess performance show several weaknesses. This has been part of the agenda of program management staff, which has accomplished considerable work to remodel the performance measurement framework. Table 5 summarises the findings of the evaluation as well as the actions taken by the PEMD program staff to address program performance measurement issues.

It is believed that efforts allocated to improve the performance measurement framework are considerable and will certainly contribute to a better assessment and monitoring of future results. However, given that the work is still in progress, the evaluation cannot at this stage provide a definitive judgment on the appropriateness of this new performance measurement framework. Furthermore, if the Terms and Conditions are significantly modified to make them coherent with the Global Commerce Strategy, then the changes proposed to the RMAF will have to be again reconsidered to take into account new objectives of a redesigned program.

5.2 Outputs and breakdown of PEMD-A funded activities

In the current application procedure, associations are required to group their proposed activities into packages defined on the basis of three categories of trade promotion activities. They are:

• Direct contacts: participation to trade shows, outgoing missions, incoming missions and participation to information and networking events;
• Marketing tools: web site development targeting foreign customers, print material and other material that promote the export of the associations' goods and services; and,
• Other marketing activities: market research and survey and indirect marketing that promote relationship building, or improve access to foreign markets.

Given that associations are provided with only three categories of trade promotion activities and that the packages of activities that they submit may include a mix of these categories, it is not possible to quantify the output level results of the Program. Additionally the PEMD-A information system does not allow for such things as the tracking of outputs on trade shows, support for incoming or outgoing missions, the number of websites developed or the number of market studies undertaken. The information system also does not report or measure the gaps between proposed and completed activities.

The aggregation of results by type of trade promotion activities is also not possible because associations are requested to report on the performance of packages of activities and not on categories of activity. Packages of activities are not standardized, which prevents the aggregation of activities at the output level.

In the absence of aggregated data, the evaluation had to base its analysis of results on the opinions of key informants, which also cannot be aggregated. These opinions were generally positive. For a majority of the people interviewed, PEMD-A is effective in obtaining output results. A more in-depth review of final reports prepared by the selected 14 associations confirms this perception, although the quality of reporting varies from one association to another.

In order to assess the relative importance of these trade promotion activities, the evaluation team examined a compilation of amounts claimed. Table 6 gives the distribution of PEMD-A funding by category of trade promotion activities during 2002-03 to 2006-07.⁽¹⁰⁾ It shows that 66% of the amounts spent by recipient associations were allocated to direct contact activities, 24% to marketing tools and 10% to other marketing activities.

A more detailed breakdown is provided in Table 7 for the 14 associations included in the sample⁽¹¹⁾ of the PEMD-A clientele for which interviews were carried out. These associations claimed $3.3M or 28% of the total $11.8M claimed during the 2002-03 to 2006-07.

The above table indicates that the selected associations used 78% of their PEMD-A funding for direct contacts activities, 15% for the development of marketing tools and 8% for other marketing activities. This distribution is more reliable than that provided for all recipient associations. It points out to the relative importance of trade shows (46% of claimed amounts) and networking events (19%) in the IBD agenda of trade associations.

5.3 Outcomes of the program

The expected outcomes of PEMD-A are:

Intermediate outcomes:

• Increased awareness of international business opportunities
• Increased awareness of Canadian industrial and commercial expertise in foreign market

Long-term outcomes:

• Increased international business development by Canadian firms
• Increased Canadian employment job creation and economic growth

It was difficult to assess the extent to which these outcomes have been achieved because the current reporting system does not allow capturing outcome and impact level results.

In their application, associations are asked to explain how their packages of activities will contribute to a series of key outcome results, but they are not requested to identify performance indicators for these results. Furthermore, these key results are not clearly linked to the overall outcomes of the Program. The key results proposed are the following:

• Increased sales by member companies
• Improved market access
• Increased exposure to international clients
• Number of potential agents, distributors, service contractors
• Number of prospective leads and contacts
• Number of secure licensing agreements and negotiate joint ventures
• Number of exporting SMEs
• Industry satisfaction with PEMD-A supported activities

The reporting process is based on activities of one year, while in most cases, IBD results occur over the long-term, often after more than 3 years.

In spite of the fact that the measurement of PEMD-A outcome level results is a major problem, information gathered from interviews and the questionnaire suggests that the Program leads to significant outcomes that many associations would not have achieved without PEMD-A funding. The results achieved can be summarized as follows:

• Increased awareness of international business opportunities. Exposure and relationship building realized through PEMD-A supported activities have allowed the following:
  • Associations have made IBD accessible and affordable to SMEs, who would not have participated in trade shows or international events without PEMD-A assistance and the support of their association. Many companies continue to participate on their own because they have obtained concrete results in terms of contacts, leads and prospects for future sales; the impact on SMEs is further discussed below;
  • Other associations have formed alliances with foreign associations to exchange information of benefit to member companies;
  • Identifying potential business opportunities, particularly in new or emerging markets, with a lower risk of market entry/development costs, as well as informing potential customers about their products and skills, in trade shows, have been very significant results for many associations and their members.
• Increased awareness of Canadian industrial and commercial expertise in foreign markets. This was identified by questionnaire respondents as the most important result achieved in the last two to five years. The result was further exploited by enhancing their members' capacities to access new markets, leading afterwards to greater exposure in foreign markets, and resulting in new leads, contacts, agreements and sales.
• Increased international business development by Canadian firms. Many industries, which benefited from PEMD-A support for several years, reported increased sales and market development through distribution agreements that would not have happened without PEMD-A. Association members in other industries increased their participation in the global supply chain of their sector, including 2-way business for technology, venture capital, supplier-deals, joint ventures management. However, most associations mentioned that it takes time to take advantage of real business opportunities, particularly in new or emerging markets, and that it is generally difficult to link increased export sales to PEMD-A support. In other cases, associations representing emerging industries, which are not at the export sales stage yet, could not report any such results.
• Increased Canadian employment job creation and economic growth. This is the ultimate impact of the Program. For many associations there is a direct relationship between jobs creation, increased exports and the PEMD-A although it is very difficult to quantify those results and to link them to PEMD-A funded activities.

Interviews conducted with recipient associations and with trade commissioners confirm the key role that associations often play for exporting SMEs and newly exporting SMEs:

• Trade associations offer a platform to SMEs for an organized participation to trade shows and other events. The associations are an essential vehicle for many SMEs who want to showcase their products, but who cannot do it on their own;
• Although larger firms are also members of the associations, they mostly conduct their IBD activities independently;
• Trade associations often deliver substantial cost reductions to SMEs who participate in trade shows;
• The cost advantage offered by trade associations for participation to trade shows by SMEs has been greatly enhanced by the PEMD-A's partnering mechanism allowing them to cover 50% of air travel costs and per-diem; and,
• Some associations have established specific mechanisms to promote SME development and participation. For instance, BIOTECanada has created an "Emerging Companies Advisory Board," while the Canadian Apparel Federation has created the "Dynamic Exhibitor Program" by which it selects the most promising newly exporting SMEs to go to trade shows at subsidized costs, after which they go on their own.

5.4 Overall impact of the program

The concept of incrementality of IBD activities funded by the Program will be examined at two levels, that of the application process and that of the overall program level.

5.4.1 Application process

The incrementality concept was introduced to prevent associations from becoming dependent on PEMD-A for funding their core activities. Incremental activities are defined in the guidelines as those that expand existing initiatives or create new ones. The incrementality criterion has a weight of 30% in the evaluation process and associations must demonstrate the incrementality of each of their proposed packages of activities, otherwise the packages can be rejected. The Review Board has applied a 3-year limit for the same activity.

In interviews, many concerns were raised with regard to incrementality, which can be summarized as follows:

• Definition of core activities: In the Terms and Conditions, there is no clear definition of core activities. These are not eligible for funding. Furthermore, the criterion is difficult to apply consistently year to year and across a broad spectrum of associations.
• Length of process: While the Review Board considers activities that are repeated during three years as core activities, most associations believe that results are achieved only after several years of participation in the same event. For many associations, incrementality must be seen within a multi-year IBD strategy because international business development is a long term undertaking.
• Key events: In many sectors, like aerospace, biotech, building products, life sciences, plastics, to name only a few, there are key international events, either trade shows or conferences, that companies and associations must attend because they are unique occasions to showcase products and technologies and to learn about market and technology trends worldwide. Associations must participate in these events year after year. However, repeated participation makes those activities non-eligible for PEMD-A funding. Many associations complain that the incrementality rules are counter-productive because their objectives and role evolve each time they participate so that they become more efficient and focussed. In their view, emphasis should be placed on how a proposed activity is building on past success and experience rather than on its repetition.
• Incremental activities vs. incremental results: For most associations, what should be incremental are the results not the activities. Carrying out the same activities several times may lead to incremental results and be cost-effective for associations, while creating new ones every year entails additional cost and risk and may lead to non-optimal results.

5.4.2 Program level

The use of the incrementality criterion for the evaluation of applications implies that the overall results of the Program are incremental. In other words, PEMD-A brings associations to carry-out IBD activities generating results that would not have materialized without PEMD-A. Some of these results are summarized below.

The rationale of the Program is based on the idea that government support will enable trade associations to achieve better IBD results, and this is valid even if the activities supported were core activities under the Program's incrementality rules. In a results based approach, the criterion of incrementality should not be applied at the level of activities, but at that of results achieved.

Information obtained from the associations through interviews and the questionnaire indicates that most respondents would have reduced considerably their IBD activities in the absence of PEMD-A support. Many would have completely stopped. This suggests that the Program is an effective tool for inducing IBD activities.

It is not possible to quantify the level of IBD activities that would take place without PEMD-A and the increase in exports resulting from PEMD-A support. However, the results achieved by some associations could potentially be impressive and probably justify the existence of the Program. From a cost-benefit stand point the dollars in the contribution budget would be a good investment. Furthermore, the impact of the Program cannot be limited to increased exports, because as was pointed out in one study, trade associations play a key role as enablers of innovation.⁽¹²⁾

5.5 Conclusion on the success of PEMD-A

Given the shortcomings of the available information and the level of effort that would have been required, the evaluation did not attempt to measure the outcome level results of the Program, so as to make comparisons between associations and to aggregate program results over the various sectors, either quantitatively or qualitatively.

The conclusion of this analysis is that PEMD-A increases the awareness of international business opportunities and the level of international business development efforts of Canadian firms. PEMD-A substantially reduces the costs and the risks of IBD for Canadian firms. Furthermore, these results are most important for SMEs, as it was found that many trade associations effectively promote the international business interests of SMEs.

Although the Program yields results that vary widely between associations because of the diverse make-up of its clientele, it can be said that some associations obtain significant results in terms of increased export sales by firms participating in trade shows, of distribution agreements, business contacts, sales leads etc. The Program also effectively supports the work of some trade associations belonging to emerging or developing industries who are trying to participate in the global supply chain by acquiring or selling technology, by attracting venture capital or management talent and by entering into licensing or joint-venture agreements.

The best proof of the success of the Program is that trade associations, year after year submit applications for trade promotion activities for which they contribute 50% of the costs. This means that the Program has an incremental impact on IBD and IBD results. Overall the Program performs reasonably well given its relatively modest cost per year including administration. However, stricter application of eligibility criteria would eliminate some associations that perform poorly and significantly improve the cost-benefit ratio of the Program.

Top of Page

6.0 Efficiency and Cost-Effectiveness

Assessment approach

The evaluation assessed efficiency and cost-effectiveness in the following three areas:

6.1 - Program design
6.2 - Appropriateness of application, selection and claims process
6.3 - Appropriateness of resource utilization

6.1 Program design

The appropriateness of program design has been examined in the following areas:

6.1.1 - Reach of PEMD-A
6.1.2 - Eligibility criteria
6.1.3 - Trade promotion activities
6.1.4 - Level of funding

6.1.1 Reach of PEMD-A

During the period 2002-03 to 2007-08, applications from a total of 101 associations were approved and approximately 10 other applications were rejected. The number of participating associations increased from 50 in 2002-03 to 64 in 2006-07 and 61 in 2007-08. In comparison to 2001-02, when there were no more than 25 associations, the Program has significantly expanded its reach.

Since the information campaign that was conducted throughout Canada in 2002-03, DFAIT has not seen the need to conduct marketing activities, as it felt that the Program reached a sufficient number of associations. It was considered that sector officers should be the marketers given their knowledge of existing associations and the extent to which these could meet PEMD-A eligibility criteria.

The analysis of the clientele categorization conducted in the evaluation shows that PEMD-A recipient associations belong to several categories differentiated by their role, mandate, type of memberships and size. Some are trade associations of manufacturing industries devoted to the provision of technical and marketing services to their members, including IBD services. Other associations are primarily involved in lobbying or representation of their members and are only marginally involved in IBD. Other associations are retail trade associations, professional associations. Finally, some are not associations but rather corporations created by networks of higher education institutions.

The open character of the Program has attracted several associations for whom the provision of IBD services is a marginal role. This has a direct influence on the results of the Program since associations working on professional certification or education marketing in developing countries will have little impact on exports. When the Program accepts the requests of associations who have little to do with exports, it stretches its objectives considerably.

6.1.2 Eligibility criteria

The following eligibility criteria were examined:

• Association must be a national trade or industry association (or a regional association with a national perspective);
• Associations must have the managerial and financial capacity to undertake their proposed activities; and,
• Membership of SMEs.

Criterion: Associations must be a national trade or industry association

The evaluation has found that this criterion is not strictly applied and appears to be stretched to accommodate applicants. For instance:

• A significant number of associations who received funding in 2007-08 (representing 16%) are not national since they only have a provincial clientele. There are many such cases as shown in the examples of the partial list below:
  • Quebec members only:
    • Association des manufacturiers d'équipements de transport
    • Groupe CTT
    • Promotion de produits forestiers (Q-Web)
  • BC members only:
    • BC Wood;
    • Western Red Cedar Lumber Association
  • Ontario members only:
    • Canadian Tooling Machining Association;
    • Canadian Association of Moldmakers Inc.
    • Canadian Independent Record Production Association (CIRPA)
    • Computer Animation Studios of Ontario;
    • Association of Registered Interior Designers of Ontario.
• The Program considers associations as being "national" if they represent companies having specific regional products to export, but this does not appear to be always the case.
• The program managers wanted to avoid having two associations of the same sector participating in the same events and in some instances encouraged joint applications by regional associations. This was successfully done with MEDEC (from Ontario) and AITS (from Quebec) who presented a joint application in 2007-08.
• In some sectors, funding is allocated to associations representing the same sector. For instance, in 2006-07, associations from the Forest Industries and the Building Products Industries, who in many cases promote very similar products, are represented by 14 associations.

Criterion: Associations must have managerial and financial capacity

This criterion was analyzed using the number of employees and the financial resources of associations as indicators of capacity.⁽¹³⁾

Figure 1 shows a scatterplot diagram of the distribution of associations according to number of employees and total revenue. It was found that although the majority of associations have in principle the human and financial resources to carryout their IBD activities, a significant proportion of associations have very few employees and very small budgets (70%). It seems therefore that the capacity criterion is sometimes overlooked in the adjudication process.

Figure 1: Scatterplot of Associations by Number of Employees and Total Revenue, 2005-08

Out of the 79 associations who had PEMD-A funding approved during the 2005-06 to 2006-08 period:

• 12 associations (15%) had 1 employee;
• 8 associations (10%) had 2 employees,
• 4 associations (5%) had 3; and,
• 12 associations (15%) had 4-5 employees.

Looking at financial resources:

• 6 associations (8%) had less than $100,000;
• 15 associations (19%) had $100,000 to $300,000; and,
• 7 associations (9%) had $300,000 to $500,000.

The threshold level of human and financial resources required to carry out IBD activities can vary, but it appears that several associations have a limited capacity to do so in an effective and sustainable way. Furthermore, several DFAIT managers raised the issue of the limited capacity of associations, saying that low capacity was leading to program inefficiency and limited results.

Criterion: Membership of SMEs

The Program is designed, as described in the PEMD-Associations Handbook "to favour activities which tend to benefit businesses with annual sales of less than $25 million (Small - Medium size Enterprises/SMEs)." The intent of the Program was to favour manufacturing SMEs and, to a lesser extent, service sector SMEs.

In the evaluation of applications, a weight of 15% is given to membership benefits and higher scores are granted to associations that focus on SME participation. The evaluation of the Review Board is based on the declared intentions of the associations for making the SMEs benefit from their activity packages and on the relative importance of SMEs in the membership of each association as declared on their application.

The findings of the evaluation supports the fact that PEMD-A mostly benefits the SMEs.

• Data from the applications indicate that most associations have a majority of SME members, 70% of the 79 associations who presented applications from 2005-06 to 2007-08 reported having 50% or more SME members, and 47% having 75% or more SME members. However, data analysis also reveals that some associations have no or very few SME members,⁽¹⁴⁾ while others have an extremely high number of SME members.⁽¹⁵⁾
• The electronic questionnaire gave the following results with respect to SME membership benefits:
  • 41% of respondents reported that PEMD-A funded activities benefit mostly SMEs, 21% reported that they benefit equally to larger companies and SMEs, while another 32% reported that activities benefit most of their members, among which SMEs are well represented.
  • increasing the number of exporting SMEs was a high priority objective for 60% of associations; and,
  • for 60% of responding associations, the association conducts IDB activities mostly for the benefit of SMEs, while larger companies conduct their IDB activities mostly outside of the association.

6.1.3 Trade promotion activities

Regular promotion activities

PEMD-A supports three categories of trade promotion activities:

• direct contacts (e.g. trade shows, outgoing missions and incoming visits);
• marketing tools (e.g. web site development targeting foreign customers, print materials and other materials that promote the export of associations' goods); and,
• other marketing activities (research, awards programs, and indirect marketing that promotes relationship building, or improves access to foreign markets).

The general perception from associations is that these categories of activities are relevant and cost-effective for achieving IBD results, although results may not be captured in a one-year planning framework. However, considering the different stage of development of industries, associations of more mature sectors would like to see the scope of the Program broadened to support a wider range of marketing activities in order to better respond to the needs of all their members.

Many associations have also expressed the need to include international market development trade events that occur within Canada as part of PEMD-A eligible activities.

Canada Export Centre (CEC)

Since May 2006, PEMD-A has accepted to fund through their trade association the participation of individual companies to the Canada Export Centre, a private company created in 2004, which operates a permanent trade exhibition in Vancouver to showcase Canadian products, services and sectoral capabilities to international delegates and business people. Eligible costs include only the exhibition fee and the cost of showcase development, both funded at 50%. In practice, PEMD-A supports 50% of booth expenses up to $8,000, plus 50% of booth set-up.

In both 2006-07 and 2007-08, there were seven associations who channelled funding to companies to have a booth at the CEC, with one newcomer in 2007-08, so that overall, eight associations have used that funding opportunity.

Three of the eight associations were included in our selected sample for interviews. The three managers of those associations gave mixed reviews of their participation to CEC. They said that it contributed marginally to their IBD objectives and that the number of international visitors was limited.

Associations also mentioned that, from their point of view, the channelling of flow-through funds to companies for the CEC had some negative impacts because they were incurring additional administrative costs without any additional revenues.

6.1.4 Level of funding

The PEMD-A funding consists of an annual non-repayable government contribution ranging from a maximum of $150,000 to a minimum of $20,000 per applicant. This excludes the optional supplemental funding allocated in the fall of each year. Recipient associations must contribute from private sources 50% of eligible expenditures. Within the annual maximum of $150,000 associations can set aside $25,000 for assistance to partnering companies.

Table 8 presents the average amounts requested, approved and claimed during the 2002-03 to 2007-08 period. It suggests that the level of PEMD-A funding is sufficient given that, associations have on average requested much less than the maximum ceilings allowed of $100,000 for 2002-03 to 2005-06 and of $150,000 for 2006-07 to 2007-08. Furthermore, the high level of unused funds confirms that overall the level of funding is sufficient.

It can also be seen in Table 9 that only 15 associations (representing less than 25%) have requested the maximum amount or close to the maximum amount of $150,000 in 2006-07 and 2007-08. In both years the majority of applicants requested less than $100,000 and close to a third requested less than $50,000. A similar situation can be observed for the distribution of approved amounts.

When taking into account the supplemental funding that was allocated to associations in 2006-07, the evaluation team noticed that none of the seven associations that were granted funding above the $150,000 maximum succeeded in spending all of their approved amounts, that group being able to spend only 73% of the money allocated. Also, only four associations have claimed amounts over $150,000. They are:

• Canadian Apparel Federation
• Canadian Association For The Advancement Of Music The Arts
• BIOTECanada
• Canadian Association of Mining Equipment and Services for Export

The evaluation has also observed that there is a clear relationship between the capacity of the association, in terms of human and financial resources budget, and the amount of PEMD-A funding that they can use.

It can be concluded from the figures presented above, that the level of funding is generally sufficient. This is supported by information obtained through interviews with associations. Recipient associations feel that given their capacity, the level of funding is appropriate. However, many associations consider that they could carry-out much more IBD activities if they were less constrained from a staffing and financial perspective. In their view, the federal government does not recognize the real costs that associations must incur in the delivery of trade promotion activities. If funding, for instance, could be expanded to allow associations to recover a portion of the overhead expenses (i.e. administrative and operational costs) they must make to support program delivery, then they could undertake more IBD activities and use more PEMD-A funding.

Many associations also believe that the $25,000 set aside for assistance to companies is not enough. Member companies are often struggling to attend key events and international opportunities, especially smaller enterprises that cannot participate otherwise. If PEMD-A allows them to get out there, the exposure they will get is what can generate new leads and prospective business opportunities, ultimately contributing to the achievement of program results. By limiting the financial assistance to companies, the Program is reducing its chances of success.

Between 2002-03 and 2006-07, an average of 69% of approved amounts was claimed by trade associations so that the level of unused funds averages at 31%. Two supplemental founding rounds appear to have reduced the unused funds ratio (refer to Table 2 and Table 8).

In 2006-07, the unused funds ratio, taking into account supplemental funding, was above 50% for 37 associations while 12 associations had unused fund ratios of more than 80%. At the other end of the scale, 18 associations had unused fund ratios of 20% or less, out of which, 11 had ratios of 10% or less.

When examining the size characteristics of the associations having high unused funds ratios, a clear inverse relationship can be seen between the number of employees and the total funds of associations. Very small associations with respect to both the number of employees and financial resources perform very poorly, while the larger associations perform clearly better, as shown in Table 10.

The data also show that unused funds are inversely related to the level of approved amounts -32% for amounts approved ranging between $100,000 and $150,000 as compared to 55% for associations with approved amounts ranging between $25,000 and $50,000.

Finally, there is also a clear relationship between the sector and the level of unused funds, some sectors obtaining much better results than others. The best performing sectors are:

• Manufacturing Technologies: 4%
• Mining Equipment, Services and Technology: 9%
• Plastics: 20%
• Bio-industries: 20%
• Consumer products: 24%
• Information Communications Technologies: 25%

The worst performers are:

• Automotive: 90%
• Aerospace and defence: 67%
• Building products and construction: 50%
• Service industries and capital projects: 50%

Many factors have been reported by key informants to explain the problem of unused funds. They are summarized as follows.

From PEMD-A managers sector officers' perspectives, the high level of unused funds is largely caused by:

• The weak capacity of associations resulting in 75% of them submitting applications at the last minute for activities that are not well planned or financed, given the 50-50% cost sharing conditions;
• Associations present more projects than they can accomplish, so as to keep the option of funding unplanned activities through the reallocation of unused funds;
• The system prescribes that claims should be presented at the latest 30 days after the completion of an activity package, but many associations present their claims up to 6 months late which prevents the reallocation of funds.

From the associations' perspective, the problem is caused by:

• Applications are prepared so much in advance of events that the proposed level of participation of members is very approximate, matching funds are not guaranteed and unforeseen events can easily occur;
• There is a lack of flexibility to reallocate unspent amounts since it is not possible to transfer funds between different packages of activities;
• There is a conflict between good planning and cash-flow management;
• The follow-up of unclaimed funds by PEMD-A administrators is too loose.

In order to resolve the problem of unused fund or at least to reduce its acuteness, the following actions should be considered:

• Apply a tighter follow-up on unclaimed amounts, especially for associations which have recurrent unused funds.
• Introduce more flexibility in funding arrangements;
• Introduce multi-year planning so that unused funds could be allocated to the next year.

6.2 Appropriateness of application, selection and claim process

The appropriateness of application and selection process has been assessed by examining the following areas:

6.2.1 - Timeline
6.2.2 - Appropriate support of sector and post officers
6.2.3 - Effectiveness of the selection process
6.2.4 - Claim process

6.2.1 Timeline

The online application and claim process that was established in 2002-03 is praised by program stakeholders. Associations find the automated system very user-friendly in comparison to other governmental programs.

Applications are received once a year at the end of January and there is a two-month period where associations are encouraged to work on their proposal with sector officers before sending their final application including the following:

• general information on the association;
• international business experience;
• international business strategy;
• packages of activities with performance indicators;
• anticipated key results; and,
• financial information.

Announcement of application results is made beginning of March.

One problem that was raised by trade associations in interviews is that the application cycle for submission starts in November and ends in January. This timeframe coincides with year-end workload based on a calendar year. This could explain why many associations submit their application at the very last minute. The preparation of the application, including the elaboration of the international business strategy and the definition of packages of activities, is sometimes done very rapidly, in a cut-and-paste manner. Finally, for some associations, the announcement of results in March makes it difficult to plan any events during the first months of the year.

The establishment of a multi-year funding system would eliminate timeline problems.

6.2.2 Appropriate support of sector and post officers

The role of the sector officers in the implementation of PEMD-A is the following:

• Support the associations in the preparation of their application;
• Perform administrative tasks including the verification of applications to see if they comply with general Terms and Conditions of the Program; upload of electronic correspondence into PEMD website; follow-up with posts officers to seek their feedback on proposed packages of activities; etc.
• Provide recommendations to association for improvement/enhancement of their strategy or activities; and,
• Evaluate the applications according to pre-set criteria, through the online system; present final application to Review Board for approval of funds.

The role of trade commissioners is limited. They play a small role in providing comments and feedback on activities proposed by trade associations.

Information obtained from the associations, both in interviews and in the questionnaire reveals a high degree of satisfaction with the support received from sector officers at Headquarters in the preparation of the applications as well as with follow-up assistance and technical services provided during the year. The level of satisfaction with the support received from DFAIT officers posted abroad is good, but many associations believe that some officers lack the sector expertise necessary to play a more supportive and informed role on the trade development front. Post officers interviewed were not always very familiar with PEMD-A, and said that they are seldom informed about results of activities for which they were asked to provide comments.

The high turnover rate of sector officers was another concern raised by some associations. The positive working relationships that exists between many associations and their respective sector branch officials is greatly enhanced by the high level of sectoral expertise these staff are able to develop over the course of their careers. The practice of rotating staff into new sectors diminishes the benefit that these individuals are able to bring to their jobs.

It was mentioned that sector officers could have a more proactive role and that they could get more deeply involved in assisting the formulation of sector strategies and objectives by the association, as well as in the evaluation of their results. The eventual revision of PEMD-A in accordance with the Global Commerce Strategy should provide an occasion for sector officers and for post officers to work in that direction with the associations.

6.2.3 Effectiveness of the selection process

The PEMD-A Review Board is normally composed of three members selected among senior DFAIT managers. Meetings are held over three days during which all applications are scrutinized according to specific criteria and each package of activities is to be approved. Sector officers are invited to the review process to inform and comment on applications and on the strategic objectives and capacity of associations.

The conclusion is that the current process has many weaknesses which negatively affect the quality and consistency of the selection process. More particularly, the following concerns have been raised:

• Lack of stability of the review committee: The effectiveness of the process is affected by the high turnover rate of Review Board members. This prevents the build-up of knowledge and experience and leads to a lack of continuity in the decisions taken.
• Participation of sector officers: The invitation made to sector officers to be present at the review committee has improved the process because sector officers can provide valuable information on strategic objectives of associations and the sectors they represent. However, their views are not always taken into consideration by the Review Board who feel that, in some cases, they lack objectivity and become too personally involved in the support of the applications of their associations.
• Evaluation criteria: The evaluation criteria used by the Review Board are not well defined and not consistently applied, which leads to a lack of consistency between decisions.
• Amount of information and time allocated: The members of the Review Board, who are not necessarily familiar with the project files, have to absorb large amounts of information and must, in no more than three days, review all the activity packages of more than 60 applications. According to board members interviewed, the workload in the time allocated is too heavy to produce informed and consistent decisions. Furthermore, the information provided to board members in many cases does not allow them to judge the performance of associations and their ability to implement their proposed packages of activities.

One of the broad objectives of the Program was to promote excellence in the development of international markets by allocating funds on the basis of competition. The approval rate of applications has been high. In 2006-07, 85% of the 64 applications were approved, out of which, 30 approved in full. In 2007-08, the approval rate increased to 92%, with 50 applications approved in full.

It is necessary to have a Review Board, but its mandate should be changed from one of analyzing and approving packages of activities, to one of reviewing and approving multi-year strategies and multi-year applications. This assumes that PEMD-A will move to a multi-year process.

The effectiveness of the Review Board will depend on the quality and the relevance of the information submitted. Therefore associations should be requested to provide more strategic information in their application, which could include:

• An International Business Strategy describing, not only the international context of the sector, but principally what is the strategy of the association to further the international business interests of its members, possibly in relation with the Global Commerce Strategy consideration for the sector.
• A presentation of the mandate and membership of the association and a demonstration of how the International Business Strategy and the participation in PEMD-A will contribute to its fulfillment.
• The capacity of the associations to implement a multi-year IBS should be demonstrated.

In the context of multi-year applications, the role of sectors officers would be expanded to provide coaching and assistance in the preparation of multi-year strategies and applications. They would prepare progress reports, based on the reporting of associations, and their own analysis, for presentation to the annual sessions of the Review Board. The Review Board would decide, upon information provided by the sector officers, if each recipient association is in compliance with the terms of its contribution agreement and reaches the targeted results required for the continuation of PEMD-A funding. These targets should be realistic and in line with what can be accomplished by trade associations. The evaluation criteria should be revised in the context of a multi-year strategy and allow for more objectivity.

It should be noted that the PEMD management is in the process of proposing a revision to the board process, which would address many of the issues raised above. The revised selection process would consist in a 3-tier approach involving:

• Sector officers ranking applications;
• Advisory committee of program managers making recommendations on applications
• Review Board endorsement and approval.

6.2.4 Claims process

Once all activities within a package have been completed, associations may submit a claim online for payment for that package. As mentioned earlier, claims must be submitted within 30 calendar days of the completion of a package. Unassigned funds can be reassigned to other activities within a same package but a legal amendment to the contribution agreement is needed to reallocate unspent funds from a package of activities to another. Receipts are not required to be submitted with claims, but PEMD-A retains the right to audit all claims, for three years following the expiry date of the agreement.

In general, the online claim process works very well and is appreciated by the associations as well as by the PEMD staff. However, it emerged from the interviews that the Program would benefit from being more flexible, particularly with respect to the following points:

• The reallocation of unspent amounts between different packages of activities. Currently, this cannot be done without a legal amendment to the contribution agreement;
• The possibility of modifying packages of activities more easily to take advantage of emerging opportunities;
• The maximum amount per package is limited to $50,000; and,
• The possibility of claiming before the end of an activity package. For many associations disbursements related to PEMD-A activities may create cash-flow problems. This has an impact on the type of activities they will propose, preventing them for instance to undertake larger and longer duration marketing projects.

A more flexible approach for claims might contribute to reduce the unused fund ratios. For instance, in many occasions, packages of activities have cost less than was expected or activities had to be cancelled for some unexpected reason, however, because funds have to be reallocated within 30 calendar days, and because it takes an amendment to the contribution agreement to reallocate the funds to another package of activities or to propose a new one, associations are less inclined to do so and the funds remain unassigned.

Furthermore, in the context of multi-year contributions more flexibility in the funding and claim process will be required.

Possible revisions of claims process

• The minimum amount for an application could be raised to $50,000 as a way to eliminate weak associations and improve efficiency.
• More flexibility should be allowed in the identification, design and costing of activity packages, as is consistent with a multi-year strategy, especially for Year 2 and Year 3.
• More flexibility is needed, especially with the transfer of funds, which should be allowed between different activity packages.

6.3 Appropriateness of resources utilization

The appropriateness of resource utilization has been assessed by examining the following areas:

6.3.1 - Program management and administrative costs
6.3.2 - Financial reporting

6.3.1 Program management and administrative costs

The management of PEMD-A was until September 2007 a shared responsibility between the SME Support and Services Division and the Business Sectors Bureau Division. As of September 2007, it was decided to transfer the entire responsibility of the Program to the latter division. This move should be completed by March 31, 2008.

Until recently, the SME Support and Services Division was responsible for program design and delivery, while the Business Sectors Bureau Division was mainly involved in the interface between the department and the associations through sector officers. Table 12 describes the human resources of DFAIT involved in the Program and the corresponding estimated administrative costs. A 33% factor has been applied to direct salary costs to account for employee benefit plans (20%) and the PWGSC accommodations charge (13%). Other indirect program costs that were not included include the following: Legal (f. ex. advice), Finance (f. ex. payments, planning, budgeting), Human Resources (f. ex. staffing and training), audit, evaluation, IT systems support, communications (f. ex telephone), general reception, etc.

It was estimated that the contribution of sector officers to PEMD-A represents approximately 12% of their annual working hours.⁽¹⁶⁾ Their involvement in PEMD-A activities is higher during the application period from December to March, when they assist associations in developing their application and ensure they meet the deadline. When the final application period closes, sector officers then evaluate applications through the online system during the first two weeks of February. Throughout the year, sector officers approve or reject modifications to planned activities, approve claims and monitor the progress of associations. Trade commissioners also allocate some time to PEMD-A activities. However it is not a significant part of their time.

The total annual administrative costs of the Program correspond to an approximate range between 14.6% and 17.8% of the total contributions provided in 2006-07. This admin ratio falls within the range for Going Global and CISP. It amounts to approximately $7,075 and $8,500⁽¹⁷⁾ per association, based on 64 approved applications in 2006-07.

There are two limitations to the admin ratio calculation:

• The data is based on a combination of tracking of hours and statements provided either by people occupying the position or estimations; and,
• Operational staff may allocate more (or less) time to perform the same activity.

6.3.2 Financial reporting

When associations submit a claim, they have to describe the expenditures incurred and to report against the performance indicators they chose for each activity in the package. They are not requested to send any receipts but to provide a tangible Proof of Activity that resulted from the package. DFAIT retains the right to audit all claims for three years following the expiry date set out in the legal agreement. On average, the department conducts nine audits per year and the cost incurred annually is approximately $81,000. This procedure was introduced in 2002-03 and simplified the processing of claims by allowing officers to concentrate more on the substance of association and member activities and less on administrative formalities.

6.4 Conclusion on the efficiency and cost-effectiveness of PEMD-A

The relative diversity of the associations making up the PEMD-A's clientele is largely the result of eligibility criteria that are broadly defined and liberally applied and of funding rules that generally correspond to the needs of trade associations.

Despite efficiencies, the Program suffers from a high and persistent deommitment of fund ratio caused by the difficulty of associations to plan their IBD expenses in advance, by the limited capacity of several of them, as well as by the Program's strict rules for the reallocation of unused funds. The Review Board process is considered as not functioning satisfactorily because it does not lead to a competitive allocation of PEMD-A funds and renders decisions that are not always consistent because of the limited information and time available to board members and of the pressure felt to engage available budgets.

As was observed in the previous (2001) evaluation, performance reporting still remains a weak point of the Program since it has not been able to report satisfactorily on the results achieved at the output and outcome level.

Finally, in spite of its highly automated processes, the administrative costs of the Program represent an estimated ratio of between 15% and 18% of contribution expenditures.

Top of Page

7.0 Conclusion

The overall conclusion of the evaluation is that PEMD-A, as it is presently designed and delivered, generally answers the needs of trade associations. For the greater part of recipient associations, the Program also achieves its expected outcomes of increased awareness of international business opportunities and international business development by Canadian firms, and of increased awareness of Canadian industrial and commercial expertise in foreign markets.

However, PEMD-A in its present form is not relevant with respect to the international trade priorities of DFAIT which are now embodied in the Global Commerce Strategy. The GCS aims to integrate Canadian firms in the global value chains of industries and gives priority to high-technology sectors and to high potential markets, criteria which can be met by only a fraction of the current PEMD-A clientele.

The alignment of the Terms and Conditions of PEMD-A to the Global Commerce Strategy will require an overhaul of the Program in such a way as to base its logic and design on the achievement of strategic goals by trade associations over a multi-year period, rather than on the execution of trade promotion activities that are approved on an annual basis. Additional tools adapted to CGS objectives and more flexibility in the use of funds should also be considered.

The consequence of aligning the PEMD-A with the GCS would be that several sectors and associations would not be eligible for PEMD-A financial assistance. The impact of such a change on the IBD activities of trade associations who for many years have relied on PEMD-A should be taken into consideration.

Top of Page

8.0 Recommendations

• Identifying the positioning of Canadian firms in the global value chain as a PEMD-A objective;
• Reviewing the range of promotion activities that can be funded in order to enable associations to intervene effectively;
• Restricting the Program to sectors with high export potential and high or medium S T intensity, where integration in the global value chain is essential for the growth and survival of Canadian business participation; and,
• Modifying other elements of the Program to make it coherent and efficient like: 1) enhancing sector strategies to be developed jointly with associations, with assistance of DFAIT and the participation of other stakeholders; 2) improving collaboration between associations and sector officers and post officers to work on priority sectors and markets; 3) implementing multi-year contribution agreements for sector strategies and activity programs replacing current 1-year contribution agreements; and 4) modifying the on-line application and claims management system to reflect these changes.

• impose a minimum of $50,000 for the approval of an application;
• allow associations to hold a percentage of PEMD-A funding in order to cover their administrative costs, particularly in the case of flow-through funds for partnering organizations;
• require associations to provide more details with respect to association membership to better identify SMEs and the extent to which SMEs benefit PEMD-A;
• require associations to provide more details with respect to financial and managerial capacity to better identify sources of funding and FTEs;
• allow more flexibility with respect to the incrementality criterion, by applying this concept to results rather than activities;
• allow more flexibility in the identification, design and costing of activity packages, consistent with a multi-year strategy, especially for Year-2 and Year-3;
• allow more flexibility in the transfer of funds between different activity packages;
• increase funding allocated to assist companies to a minimum of $50,000; and
• standardize activity packages and output results in order to facilitate the grouping of results and their aggregation. It is also recommended to revise the current application and reporting procedures in a way to match results achieved by associations with the logic model.

1. meaningful in terms of being understandable and comparable over time;
2. reliable such that the data be verifiable and not susceptible to manipulation; and
3. practical and financial feasible and with timely data.

Top of Page

Appendix A - Management Response

Top of Page

¹ DFAIT Report on Plans and Priorities, 2007-08

² The impact of industry associations, Margaret Dalziel, School of Management, University of Ottawa, 2006.

³ Other indirect program costs that were not included include the following: Legal (f.ex. advice), Finance (f. ex. payments, planning, budgeting), Human Resources (f. ex. staffing and training), audit, evaluation, IT systems support, communications (f. ex telephone), general reception, etc.

⁴ At a 95% confidence level, this means that 95 times out of 100 the true value will fall within a 10-point range of the response rate. Confidence levels may vary depending on the response rate of each individual question.

⁵ Source: IMS and PEMD-A database. Information for FY 2007-08 is based on preliminary data.

⁶ Source: PEMD-A database

⁷ There is some overlapping in the classification of associations by PEMD-A in the Building Products and Forest Industries sectors. BC Wood and the Western Red Cedar Lumber Association are placed in the Forest Industries, while they market wood products used in construction and products that are similar to those of the Canadian Wood Council placed in the Building Products sector. Similarly, the Structural Board Association is placed in the Building Products sector, while the Composite Panel Association is placed in the Forestry sector.

⁸ Source: Forestry Innovation Investment Ltd, Statements of Financial Information, Fiscal Year March 31, 2007.

⁹ The categories of associations, according to their role or type, identified in the analysis of the clientele are the following:

• Global value chain positioning: 19 associations in 2007-08
• Promotion of exports: 7 associations;
• Technical and lobbying services, with some IBD activity: 16 associations;
• Technical and lobbying services, with marginal IBD activity: 10 associations;
• Professional associations: 6 associations;
• Higher education associations: 3 associations.

¹⁰ In the program's database, amounts spent by associations are not reported by package of activities rather than by category of activities, therefore classification work had to be done and the results presented are approximations, not exact figures.

¹¹ To generate that breakdown activities included in packages were classified according to the detailed categories of trade promotion activities.

¹² The impact of industry associations, Margaret Dalziel, School of Management, University of Ottawa, 2006.

¹³ It must be noted that in several cases, the information provided by associations appears inaccurate.

¹⁴ Associations with no or few SME members are the Composite Panel Association, the Structural Board Association, the Canadian Airports Council, national associations whose members are regional associations like the Canadian Wood Council and the Forest Products Association of Canada, organizations of higher education institution posing as associations of universities and colleges.

¹⁵ Associations with an excessively high number of SMEs are professional associations and retail trade associations.

¹⁶ Number of hours worked (related to PEMD) is equal to 2,300 hours which is equivalent to 290.5 days at 7.5hr/day. This represents approximately a 1.5 person-year.

¹⁷ Calculated as follows: $3,020,000 ÷ 64 = $47,180 × admin cost ratio %

Office of the Inspector General

^* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Primary navigation

Date Modified:

2013-01-08