Foreign Affairs, Trade and Development Canada
Symbol of the Government of Canada

Foreign Affairs, Trade and Development Canada

international.gc.ca

Archived Document

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated since it was archived. Web pages that are archived on the Web are not subject to Government of Canada Web Standards; as per the Communications Policy of the Government of Canada, you can request alternate formats by contacting us.

Formative Evaluation of the Global Partnership Program

International Security Branch

(May 2006)

(PDF Version, 639 KB) *

Table of Contents

Executive Summary

Background

The Evaluation Division (ZIE) of Foreign Affairs and International Trade Canada (DFAIT) undertook a formative evaluation of the Global Partnership Program (GPP). The evaluation was conducted by an independent consulting company (Goss Gilroy Inc.) and relied on an extensive literature and document review, interviews with key stakeholders in Canada and abroad, focus groups with program staff and case studies of significant projects in each of the four streams of the GPP: nuclear submarine dismantlement; nuclear and radiological security, chemical weapons destruction; and redirection of former weapons scientists. It focussed on key evaluation issues in the areas of:

  • Program Relevance (both as the program was conceived and as it evolves);
  • Program Results;
  • Program Design and Delivery; and,
  • Program Efficiency and Effectiveness.

Evaluation Findings

Relevance: The Global Partnership Program remains highly relevant to Canadian domestic and international policies and priorities. It was implemented as a major Canadian foreign policy initiative to address the serious threat presented by Weapons and Materials of Mass Destruction (WMMD) in Russia and the other countries of the former Soviet Union (FSU) through a G8-led Global Partnership.

The design and development of the GPP was informed by a wide-ranging analysis of the nature of the threat posed by WMMD, the experience of partners already engaged, the resources required to address WMMD threats, the likelihood of contributions by other countries, and the implementation risks involved. The program has also undertaken a recent re-assessment of the evolving security threats posed by WMMD and made adjustments in priority areas of activity to respond.

Finally, the GPP has contributed to Canada's standing among international partners in the field. The program is viewed in security policy circles in the United States as an example of Canada's willingness to make a substantive contribution to North American security. Any reduction in Canada's participation in the Global Partnership would have serious consequences for its international standing and for international efforts to deal with WMMD.

Achieving Results: The GPP has achieved significant results at the short-term and intermediate outcome levels, which is to be expected at this point in the program life-cycle. Capacity in Russia has been created and strengthened in a number of areas relating to securing and eliminating WMMD. In particular, the program has strengthened planning systems, transferred knowledge on safe practices and contributed to significant investments in critical WMMD destruction infrastructure. In more concrete terms, the GPP has contributed to the defuelling of ten nuclear reactors, dismantling of five nuclear submarines, securing of nuclear materials and radiological sources, the ongoing construction of a major chemical weapons destruction facility, and the redirection of some 1,400 former weapons scientists through 62 collaborative science projects. Canadian funding through the program has also been an important component in joint program support by other G8 and non-G8 partners.

The program has not been able to report significant increases in Canadian capacity with the exception of collaborative international science projects through the Re-Direction of Foreign Weapons Scientists Stream (RWFS), although Canadian companies have been contracted for monitoring project implementation and other professional services (e.g. Raytheon Canada, NSS).

Program Design and Delivery: It is clear from the evaluation that, in establishing the program, a large number of strategic choices were made to manage costs, risk, and complexities effectively. The choices have been largely documented. The current program design is consistent with original program documents and aligned with best practices in such areas as project selection criteria, project focus and project management structures.

The GPP has managed to put in place sufficient delivery capacity to achieve its intended outputs and outcomes to this point in time. At the same time, the program has had to deal with some weaknesses in departmental systems and processes which are not always responsive to its needs. These include information management systems, project management systems, human resource management systems, risk management systems and results-based management (RBM) practices.

Efficiency and Effectiveness: Over time, the GPP has steadily strengthened its management and delivery capacities, in part by bringing together a critical mass of complementary in-house resources. This group has the capacity to handle an increasing number of complex projects and offer economies of scale in program delivery and, with its strong web of Canadian and international partnerships, is well positioned for the future. On the other hand, stream and program level results monitoring will need to be strengthened to respond to higher activity levels and associated project and program risks.

Conclusions and Future Directions 

The evaluation found GPP to be an effective and dynamic program that has overcome many challenges. In a relatively short time the program drew on the energy and dynamism of its staff to put in place an effective project management and implementation system customised to the needs of each of the four programming streams. GPP has developed a reputation among international partners for rapid, pragmatic, and high quality program delivery.

However, as the GPP enters an intense implementation phase, it must put in place at the program level the required structures, systems and processes to support strategic planning, results-based management, human resource planning, and risk management. GPP managers recognize these challenges and are implementing actions to meet them. These key integrative features have not yet progressed to the point where they can be assessed by this evaluation.

Finally, there are important strategic and operational challenges facing the program in the near to medium-term future. These include deepening the program's response to the challenge of biological security, dealing formally with pressures to expand the geographic spread of program activities beyond Russia and the FSU, and strengthening the commercial sustainability of activities in the RFWS stream. The program is well positioned to confront these challenges, not least of all through a strengthened strategic planning process.

Recommendations

In light of the findings of this evaluation, a number of specific recommendations were made to the Global Partnership Bureau (IGX), aimed at ensuring a successful steady state of implementation of the Global Partnership Program.

Recommendation 1: Deliver as planned the following key program level documents - program-level strategic action plan, risk registry, communications strategy, and HR management and learning plan.

Implementing this recommendation will allow the program to build on positive results demonstrated at stream and project level by strengthening integrative mechanisms and processes which will improve cross-stream management.

Recommendation 2: Determine what is the leading program-level design document (or develop it if it does not exist) and align planning and reporting against results in this document.

While the program and its constituent streams are the subject of a variety of planning and results-focussed documents, there was an absence of a single defining document regularly summarizing planned and actual results at a program level. By implementing this recommendation, the program will have available an up-to-date summary of overall strategy and approach, key expected results, and alignment of the various streams, at any point in time.

Recommendation 3: Implement systematic program level reporting against outputs and outcomes with indicators.

This recommendation responds to findings in Section 5.2.2 on Program Results, which point out the absence of summative statements on short and intermediate outcomes (by result across the streams) in current reporting. Its implementation follows on from recommendation 2 and should provide program management with regular information on results achieved at a program level (rather than stream by stream), which should be useful both for internal management and external reporting.

Recommendation 4: Require annual work planning at stream-levels / sub-stream levels and analyse the costs and benefits of workplanning at (or rolling up workplans from the stream level to) the program level.

A combined workplan at the program level would enable an overview of all GPP undertakings during a specific period. It would also enable the capture of non-stream interventions or undertakings. Annual work plans would also assist in knowledge transfer as the program experiences turnover in staff.

Recommendation 5: Require stream and sub-stream level programming frameworks with logic models and indicators where they do not currently exist.

Not all streams or sub-streams have a requirement to produce a logic model in the context of an RMAF, specifically NRS. However, all streams and sub-streams should have a programming framework which articulates the overall strategy and approach, the resources, the project pipeline, results in a logic model and indicators. The requirement for this was as presented in some detail in Sections 5.4.1 and 5.4.2.

Recommendation 6: Address information management issues to eliminate fragmentation of data and information storage.

The evaluation found out that program managers tend to use their own systems due to perceived weaknesses in systems at program and corporate levels. As recommendations 1 to 5 are implemented, there will be an opportunity to gather and communicated lessons learned at the project, stream and program level.

Recommendation 7: Further strengthen efforts for the development of DFAIT-wide systems and processes designed to support high-risk and high-cost programming for IGX and related IFM programs.

Implementation of this recommendation could focus on those "toolbox" elements most likely to promote operational excellence in financial, project, risk, and results-based management, while providing the flexibility to adapt to particular contexts.

Successful implementation of these integrative management components at the departmental level assures compliance, provides synergies and economies of scale, promotes refinement in approaches to risk and performance management, encourages ongoing corporate learning, and over time will better meet departmental management accountability requirements.

Global Partnership Program (IGX) Management Response Summary

The Department agrees with the overall outcome of the evaluation that the Global Partnership Program remains highly relevant to Canadian domestic and international policies and priorities, and that GPP/IGX has managed to develop a reputation among international partners for rapid, pragmatic, and high-quality program delivery. It also accepts the view that, as the Program enters its intense implementation phase, IGX management needs to strengthen its program-level structures, systems and processes to support strategic and human resource planning, as well as its results-based management processes.

The management response to this evaluation, incorporated in the last Chapter of this Report, indicates that IGX has already implemented some of the recommendations and is well positioned to confront all outstanding challenges through a strengthened strategic planning and priority setting process.

Top of page

List of Acronyms

ADM
- Assistant Deputy Minister
BNP
- Biological Non-proliferation
BTWC
- Biological and Toxic Weapons Convention
CATS
- Corporate Application Tracking System
CBRN
- Chemical, Biological, Radiological and Nuclear
CFIA
- Canada Food Inspection Agency
CIDA
- Canadian International Development Agency
CNRS
- Centre National de la Recherche Scientifique
CSIS
- Canadian Security Intelligence Service
CW
- Chemical Weapons
CWC
- Chemical Weapons Convention
CWD
- Chemical Weapons Destruction
DFAIT
- Department of Foreign Affairs and International Trade
DND
- Department of National Defence
DoE
- Department of Energy (US)
DPR
- Departmental Performance Report
EBRD
- European Bank for Reconstruction and Development
EIA
- Environmental Impact Assessment (Russian)
EU
- European Union
FM
- Fissile Materials
FSU
- Former Soviet Union
FSUEP
- Federal State Unitary Enterprise Engineering Plant (Russia)
FTE
- Full-time Employee
FY
- Fiscal Year
GDP
- Gross Domestic Product
GGI
- Goss Gilroy Inc.
GoC
- Government of Canada
GPAG
- Global Partnership Advisory Group
GP
- Global Partnership
GPP
- Global Partnership Program
GPSPF
- Global Partnership Special Projects Fund
HEU
- Highly Enriched Uranium
HR
- Human Resources
IAE
- International Assistance Envelope
IAEA
- International Atomic Energy Agency
IGX
- Global Partnership Bureau
IMF
- International Monetary Fund
IMS
- Information Management Software
IPS
- International Policy Statement
ISTC
- International Science and Technology Center
IXS
- Program Services Division of Foreign Affairs and International Trade Canada
LPAS
- Local Public Address System
MDB
- Main Destruction Building
MEPhI
- Moscow Engineering Physics Institute
MOU
- Memorandum of Understanding
NACD
- Non-proliferation Arms Control and Disarmament
NATO
- North Atlantic Treaty Organization
NDEP
- Northern Dimension Environment of Partnership
NGO
- Non-governmental Organization
NPS
- Nuclear Powered Submarines (Russian)
NPT
- Non-Proliferation Treaty
NRCan
- Natural Resources Canada
NRS
- Nuclear and Radiological Security
NSERC
- Natural Sciences Engineering Research Council
NTI
- Nuclear Threat Initiative
NWS
- Nuclear Weapons States
NNWS
- Non-Nuclear Weapons States
OGDs
- Other Government Departments
ODA
- Official Development Assistance
PHAC
- Public Health Agency of Canada
PM
- Program Manager
PMBOK
- Project Management Book of Knowledge
PMF
- Project Management Framework
PMP
- Program Management Professional
PNPI
- Petersburg Nuclear Physics Institute
PWGSC
- Public Works and Government Services Canada
R&D
- Research and Development
RBAF
- Risk-based Audit Framework
RFP
- Request for Proposal
RFWS
- Redirection of Former Weapons Scientists
RMAF
- Results-based Management and Accountability Framework
ROSATOM
- Federal Agency for Atomic Energy
RPP
- Reports on Plans and Priorities
RTG
- Radioisotope Thermoelectric Generators
S&T
- Science and Technology
SPM
- Senior Program Manager
SPF
- Special Projects Fund
STTAG
- Science, Trade, and Technology Advisory Group
START
- Stabilisation Reconstruction Task Force
STCU
- Science and Technology Center in Ukraine
TB
- Treasury Board
UK
- United Kingdom
UN
- United Nations
US
- United States of America
USSR
- Union of Soviet Socialist Republics
WMD
- Weapons of Mass Destruction
WMMD
- Weapons and Materials of Mass Destruction
ZIE
- Evaluation Division, Foreign Affairs and International Trade Canada

Top of page

1.0 Introduction

GGI is pleased to present this draft formative evaluation report of the Global Partnership Program (GPP) to the Department of Foreign Affairs and International Trade Canada (DFAIT). This report is presented with the following sections:

The Executive Summary
Section 1.0 includes this brief introduction
Section 2.0 presents the background to the evaluation
Section 3.0 demonstrates the detailed methodology employed for this evaluation
Section 4.0 contains an updated detailed review of the characteristics of the program
Section 5.0 presents evaluation findings by evaluation area with a summary of key findings
Section 6.0 summarizes the report's conclusions and the evaluation team's recommendations

Top of page

2.0 Background

2.1 Global Partnership Program Background

The Global Partnership Program (GPP) was established within the Global Partnership Bureau at the Department of Foreign Affairs and International Trade Canada (DFAIT) in order to deliver Canada's contribution to the G8 Global Partnership Against the Spread of Weapons and Materials of Mass Destruction, launched by leaders at the 2002 G8 Kananaskis Summit. This initiative committed the G8 members to raise up to (US) $20 billion to address non-proliferation, disarmament, counter-terrorism, and nuclear safety issues, to ensure that terrorists did not acquire weapons of mass destruction or related materials. Canada committed up to $1 billion over 10 years. The primary focus of the GPP has initially been on Russia.

Corresponding to the four priority areas identified at Kananaskis, the GPP has four main project streams:

  • Stream 1: Nuclear Submarine Dismantlement, (NPS)
  • Stream 2: Disposition of Fissile Material;
  • Stream 3: Chemical Weapons Destruction (CWD) and Biological Non- Proliferation; and
  • Stream 4: Redirection of Former Weapons Scientists (RFWS).

In addition, the GPP established the Global Partnership Special Projects Fund (GPSPF) in 2004.

The Kananaskis Summit developed a set of six "principles" for preventing the proliferation of weapons of mass destruction. While applying to the four priority areas, these principles also establish the basis for an extensive list of potential projects over the life cycle of the GPP that are complementary and supplemental to those under the four priority areas. These special projects are funded from the GPSPF. In the case of projects in the four streams, there should be no duplication or overlap between projects funded through the GPSPF with those funded as a discrete GPP project. In addition, the GPSPF allows for the funding of special projects in the area of biological non-proliferation (BNP).

2.2 Formative Evaluation Background

Goss Gilroy Inc. was contracted by the DFAIT evaluation division (ZIE) to undertake this formative evaluation in the early part of the 2006 calendar year. The formative evaluation of the Global Partnership Program has been conducted according to the terms and conditions of the Treasury Board submissions that stated the first evaluation of the GPP would take place in 2005/06. The release of $51.3 million to the Global Partnership Bureau for 2006/07(1) is conditional upon Treasury Board acceptance of the evaluation report.

In addition to this formative evaluation, the Treasury Board submissions state that over the ten-year life of the GPP, a minimum of three other formal evaluations of the GPP are to be conducted:

  • during year five in order to adjust future directions well before year six;
  • at the end of 7.5 years; and
  • at the conclusion of the GPP, a summative evaluation of the entire GPP will be conducted.

Individual projects will also be subject to ongoing monitoring and evaluation against established elements of project performance and expenditure targets.

2.3 Prior Audit Activities

The Office of the Inspector General has already undertaken a review and an internal audit of the Global Partnership Bureau. In June 2004, an "Advise and Assist" review of the Global Partnership Bureau (IGX)(2) was completed by the Audit Division. The purpose of this exercise was to provide advice and assistance to the management of IGX, in its efforts to establish the appropriate results-based management control framework required to effectively deliver the GPP. This review made a number of recommendations focusing on project management, audit and evaluation, governance, document management, risk management, RMAF / RBAF reviews, human resource issues, and support services.

In the spring of 2005, an internal audit of the Global Partnership Bureau was undertaken to satisfy the requirement of the Treasury Board approved RBAF. This internal audit set out recommendations in the areas of planning and budgeting, project management, human resource planning, risk management, communications, governance, operational and work planning, and information technology.

Top of page

3.0 Methodology

The formative evaluation commenced data collection on February 14 and completed the majority of collection activities by March 14. There were four principal data collection methodologies employed in the evaluation:

  • Literature and Document Review
  • Key Stakeholder Interviews
  • Focus Groups
  • Case Studies

3.1 Literature and Document Review

GGI commenced the process of identifying and reviewing key documents, reported information, and pieces of literature, at the program and project levels and with respect to the program context, as soon as the evaluation started. This process continued throughout the data collection phase and provided the basis for many of the findings of the evaluation. GGI rigorously maintained a categorised list of documents and literature which served as an ongoing reference base. The list provides basic information for each document (type, title, author, date). The documents are categorized at the GPP level and by stream. In addition there are background documents on the context of the GPP and those related to evaluation and monitoring.

3.2 Structured Interviews with Key Stakeholders

GGI, with the assistance of ZIE and IGX, developed a list of key stakeholders for the different categories of evaluation issues. GGI then developed interview guides to ensure that the structured interviews covered all of the appropriate evaluation issues. In total, 34 interviews were undertaken (including case study interviews).

3.3 Focus Groups

Focus groups were not included in the Request For Proposals or the GGI proposal, but were added as a data collection method in the workplan. In the formative evaluation focus groups were used in a greater fashion than anticipated and proved to be a fruitful method. Three focus groups for a total duration of ten hours were held as follows:

  • A relevance focus group with program management;
  • A design and delivery focus group with program management;
  • A design and delivery focus group with PM4s.

3.4 Case Studies

Four case studies were completed as per the Terms of Reference, to provide more detailed and stream-specific insights into the relevance, design and delivery, achievements and effectiveness of the program:

  • Dismantlement of Russian Nuclear Submarines (including NPS 645)
  • Petersburg Nuclear Physics Institute (PNPI)
  • Railway Construction Project at Shchuch'ye
  • Digital Technology for the Detection and Control of Fissile Material in Devices with Pulsed Neutron Sources

The case studies were completed through key stakeholder interviews and document and literature reviews.

Top of page

4.0 Program Update

Before discussing evaluation findings, it is worthwhile examining the evolution of the GPP over time. This will provide the reader with the program context for the analysis which follows.

The evolution of the GPP from its establishment by Memorandum to Cabinet in August 2002, through to this formative evaluation in March 2006, can be characterized by three phases. Based on interviews with senior IGX management, the GPP was described as going through an Initiation Phase that lasted almost two years. During this "entrepreneurial" phase, IGX senior management were putting in place the necessary legal framework, obtaining authorities from Treasury Board, and staffing key positions, particularly those of the Senior Program Managers for each of the priority areas. The second phase began once the Canada-Russia Bilateral Agreement was signed in June 2004 and projects could be entered into on a bilateral basis, in addition to the multilateral delivery mechanism employed in the first phase. This phase also reflects the addition of support personnel, including some PM-4s required to begin project development and a more formalized approach to project management with the development of the Project Management Framework in Phase II Treasury Board submission. The third phase starting in July 2005 reflects a situation where the program has entered into full-scale implementation, where most, if not all, positions are staffed and consideration is being given to program expansion and turnover of human resources. Key management tools are being put into place and program managers are seeing projects through to the implementation stage. Exhibit 4.1 presents a summary of the evolution of the GPP.

Exhibit 4.1: Evolution of the GPP

(Click image to enlarge)

Evolution of the GPP

Exhibit 4.2 presents a summary of the projects initiated under the GPP according to the start and end dates provided by the Senior Program Managers. As can be seen from this exhibit, the first year of the project saw little project activity in terms of funds being disbursed. This is consistent with the start-up period when efforts are being placed on developing the legal agreements and reflects the lag time required to implement projects. The exhibit also clearly shows that the program is now starting to enter into its peak activity phase, with many projects in each stream entering the implementation stage.

Exhibit 4.2: Summary of GPP Projects

(Click image to enlarge)

Summary of GPP Projects

Top of page

5.0 Evaluation Findings

This section presents the findings of the evaluation of the Global Partnership Program. It is organized under the main issues areas of the Terms of Reference with each sub-section dealing explicitly with the key issues identified in the evaluation Terms of Reference and refined in the Evaluation Workplan. Each issue is analysed in terms of the data reviewed by the evaluation team and a summary finding provided for the issue concerned.

5.1 Relevance

5.1.1 Other Countries' Participation

Evaluation Issue: How does Canada's GPP compare to other countries' participation in the Global Partnership initiative? Have any G8 countries withdrawn from providing funding for any of the priority areas? If yes, what reasons are given?

Interviews with key stakeholders in Canada (both inside and outside of the program) as well as with representatives of partner bilateral agencies in the US and Europe and with some multilateral agencies provide the following insights regarding the comparative position of Canada and other nations as supporters of the Global Partnership:

  • The main differentiating features among G8 countries noted was the comparative dollar volumes committed (in relation to population) and the relative speed with which countries and programs moved to commit resources and implement projects. In this area, it was noted that several fairly large countries in Europe were slow to make solid commitments of resources and to disburse funds on projects. No G8 countries were reported by any of those interviewed to have withdrawn from the Global Partnership;
  • Key stakeholders interviewed rated Canada's participation highly in terms of both the dollar volume committed in relation to a relatively small population (in G8 terms at least) and to a practical approach to seeking mechanisms for making significant investments early in the program. For this dimension, Canada was rated in the higher performing group of countries along with Germany, Norway, the UK, and the US.
  • The primary comparative dimension of Canada's participation in the Global Partnership noted by key stakeholders outside the country was a pragmatic approach to assessing needs, choosing delivery mechanisms, making commitments, and managing accountability. Key stakeholders felt that this approach enhanced Canada's credibility among GP participating countries, especially with the US and the Russian Federation.
Summary of Findings

At least to this point in time, Canada's participation in the Global Partnership through the GPP is rated highly by partner agencies in terms of both the size of the financial commitment and the program's responsiveness to the risks and opportunities of the GP in a pragmatic and practical manner. This is seen as an effective response to the significant and visible political commitments made at Kananaskis and to the evolving threats addressed by the Global Partnership.

5.1.2 GPP and Canadian Domestic and Foreign Policy Commitments

Evaluation Issue: To what extent are the GPP mission and objectives relevant and consistent with Canada's evolving foreign and selected domestic policy objectives?

A key issue concerns the relevance of the GPP and its consistency with Canadian foreign and domestic policy priorities. The formative evaluation found high levels of relevance and consistency between the GPP and a number of Canadian foreign and domestic policy goals during the period 2002 to 2006. In terms of the key foreign policy elements the linkages are as follows:

  • The GPP responds to a commitment made to the G8 meeting in Kananaskis under the presidency of Canada and follows through on further initiatives by Canada under its presidency after the summit;
  • The GPP is consistent with Canada's long-standing commitment to and support of the Nuclear Non-Proliferation Treaty (NPT), the Chemical Weapons Convention (CWC), and the Biological and Toxin Weapons Convention (BTWC) as re-iterated in foreign policy statements and reviews;
  • It is broadly consistent with the Non-Proliferation, Arms Control & Disarmament (NACD) regime;
  • The GPP constitutes a concrete expression of priority placed by Canada on supporting North American security and strengthening Canada/US Relations as expressed regularly by the Ministers of both Foreign Affairs and Defence;
  • Relevant to emphasis on economic security and the need to protect Canada from potential economic consequences of attack using nuclear, radiological, chemical or biological materials.
The following table presents a detailed analysis of the domestic and foreign policy relevance and consistency of the GPP.
Canada's Foreign Policy FrameworkGPP Congruence
Canada's International Policy Statement

a) To prevent the spread and reduce WMD stocks b) To strengthen international export control regimes on proliferation-sensitive technologies and build the capacity of countries to enforce them c) to contribute to UN, NATO, and G8 efforts to counteract terrorist organisations and to cut off their support networks d) to establish START to plan and coordinate rapid and integrated civilian responses to international crises.
GPP is directly relevant to a) as per the GPP's guiding Principles which asks for an overall reduction in the quantities of WMD-related materials in existence

The GPP is relevant to b) as a contribution to the IAEA are used to strengthen country capacity to detect and control proliferation of WMMD.
Speech from the Throne to open the 1st session of the 38th Parliament

The speech addressed the International Policy Statement and the National Security Policy.
As referenced above, the GPP is consistent with and relevant to these two policy statements.
UN Security Council's Resolution 1540

The resolution affirms the need to contribute to non-proliferation activities and combat the threat to international peace and security posed by nuclear, chemical, and biological non-proliferation, as well as their means of delivery.
The GPP represents a direct contribution by Canada as a strong supporter of Resolution 1540. The GPP is part of Canada's efforts to meet an important foreign policy commitment.
Non-proliferation, Arms Control, and Disarmament Regime (NACD)

Non- Proliferation Treaty

Provides the only legally binding obligation to reduce and ultimately eliminate NWS' nuclear weapons.

Biological and Toxic Weapons Convention

1. not to acquire microbial, biological agents, or toxins 2. to destroy all biological non-proliferation under their jurisdiction 3. not to transfer or manufacture these objects whatsoever. 4. to prevent the development, production, stockpiling, acquisition, or retention of biological non-proliferation in its territory.

Chemical Weapons Convention

Each State Party undertakes 1. Never to develop, use, engage, or assist in the making of chemical weapons 2. To destroy chemical weapons it possesses 3. To destroy chemical weapons it abandoned in another territory 4. To destroy any production facilities located in any jurisdiction under its control 5. Not use riot control agents as warfare methods.
The GPP is part of Canada meeting important foreign policy treaty commitments as a State Party/ supporter of the NPT, BTWC, CWC and the Convention on the Physical Protection of Nuclear Material, as it enhances the security of nuclear materials at sites.
Anti-Terrorism Action Plan

a) Protect Canadians from terrorist acts

b) Keep the Canada-US border secure and open to legitimate trade.
GPP is consistent with these two policy pillars

a) It aims to reduce the likelihood of terrorist accessing WMMD or scientific knowledge in this area

b) Some ISTC projects, such as a detector for fissile materials, may contribute to border security.
National Security Policy

a) International Security
- Preventing the proliferation of weapons of mass destruction

b) Transportation Security

c) Public Health Emergencies
a) The GPP is directly relevant to the National Security Plan's international security policy.

b) Some ISTC projects, such as a detector for fissile materials, may contribute to the transportation security policy of the National Security Action Plan.

c) GPP is relevant to public health emergencies policy pillar through the emphasis on Biological Security.
The Chemical, Biological, Radiological, and Nuclear Strategy

Pillar 1 - Prevention

a) the elimination of nuclear weapons and all weapons of mass destruction will reduce the likelihood of terrorists acquiring CBRN weapons. b) Support, in collaboration with our allies, international efforts to deter and prevent states with CBRN weapons and weapons-capable materials, from using them and/or transferring them to others. c) Work with the international community to improve the security for the storage and/or movement of CBRN weapons-capable materials, as well as the safe destruction of CBRN weapons. d) Continue to strengthen our national program to identify and intercept CBRN-capable materials, weapons, and persons who may want to use them with malicious intent.
The GPP is directly consistent with the first pillar of the Chemical Biological, Radiological, and Nuclear Strategy in the areas of the elimination of nuclear weapons; collaboration with our allies and international efforts; and working with the international community (a-c).

GPP, through the GPP funded ISTC project for development of a means of detection of fissile materials, is partially consistent with item d.
Science and Technology:

The Science and Technology for the New Century: A Federal Strategy

The government set out three goals for the federal investment in S&T:

1. ensure that Canada is among the best in the world in applying &commercializing S&T for sustainable job creation &economic growth;

2. ensure that Canada applies S&T to improve the quality of life for our citizens through the creation of fulfilling jobs and through the most effective social, environmental and health care program s in the world;

3. create in Canada world centres of excellence in scientific discovery, to build a broad base of scientific enquiry, to foster Canadian participation in all major fields of science and technology, and to ensure that new knowledge can be acquired and disseminated widely from Canadian sources and from around the world.

Framework for S&T

The Government of Canada is committed to ensuring relevance in federal S&T by consulting regularly with advisory bodies, clients, partners and interested Canadians; by keeping current on advancements in domestic and international S&T; and by using a variety of tools to ensure that Canadian federal S&T is compared with S&T performed internationally, to ensure excellence.

Innovation Strategy

Renew the Government of Canada's science and technology capacity to respond to emerging public policy, stewardship, and economic challenges and opportunities.

- The Government of Canada will consider a collaborative approach to investing in research in order to focus federal capacity.
The RFWS stream of the GPP, through supporting R&D that is linked to priority Canadian S&T sectors, and through the involvement of OGDs and private sector partners and collaborators, is consistent all three goals of the Federal S&T strategy.

The work of the stream is also broadly consistent with the Framework for S&T as it provides a window on international S&T (specifically in Russia and Ukraine).

The stream is also broadly consistent with the role of the GoC envisioned in the Federal Government's Innovation Strategy.
Summary of Findings

The GPP was found to be highly relevant and consistent across a broad range of foreign and domestic policies, strategies, commitments, and action plans. In the security area these range from the commitments at Kananaskis, to specific treaties, the national security policy, and the CBRN strategy. Through the work of the ISTC and Canadian-funded projects there is also relevance and consistency with a range of GoC S&T priorities and strategies.

5.1.3 Analysis Informing GPP Design

Evaluation Issue: What policy base and analysis (of international, recipient, and Canadian priorities and capacities) informed program design?

Many components of the analysis and policy base informed the design of the GPP. Post 11 September 2001 and in the period running up to Canada's presidency of the G8, staff at what eventually became GPX, consulted OGDs (DND, CSIS, Health Canada, etc.) as well as potential partners including the US, EU, UK, to undertake three essential pieces of analysis:

  • estimate the severity and complexity of the threat posed by the situation re weapons materials and former weapons scientists in the former Soviet Union;
  • gain some idea of the scale of the investment required to make important steps to bring these threats under control; and,
  • find out how much support was available within and outside the G8 to invest in solutions to the problem.

US sources made it clear that they could commit $10 billion USD if the other G8 members could match it for a total of $20 billion. Based on this commitment and expressions of interest from other G8 members, the Canadian commitment of CAD$1 billion over ten years was made.

Both before and after Kananaskis, the Canadian team was consulting with experienced partners already working in nuclear, radiological and chemical security in Russia and other states of the FSU, to first develop the principles which would be needed to guide programming and secondly to inform the negotiations of Canada's bilateral programming agreement with Russia.

Many lessons were learned from the experience of other countries active in WMMD reduction including the US and UK. These lessons informed and were incorporated into the design of the GPP. With Canada's chairmanship of G7/G8, Canada brought forward an international analytical effort culminating in the Kananaskis principles that formed the basis of the GPP and continue to provide strategic direction. Another area of analysis was the capabilities, needs, and associated risks of Russian and other countries of the FSU.

There was evidence of analysis supporting the decision where to locate the program within the Government of Canada. CIDA and DFAIT were both considered as choices. However the program location was chosen to be DFAIT, given non-proliferation and disarmament dimensions, which were determined to be outside of the scope of CIDA's mandate.

An area of potential analysis where the formative evaluation found no documented evidence of analysis was DFAIT programming capacities and broader stream level Canadian comparative advantage. The capacity of DFAIT to implement the GPP was generally discussed but there was no documentation to suggest that this capacity had been formally assessed as part of the process of initiation of the GPP. In addition, there was no evidence that the capacity within Canadian society or industry had been assessed vis-à-vis the four streams, to seek out areas of comparative advantage.

Summary of Findings

A wide range and depth of analysis informed the design of the GPP. This included the nature of the threat, the resources required to address it, the likelihood of partner nations contributing to address the threat, the lessons of other threat reduction efforts in Russia, and the needs, capacities, and risks of Russia and the countries of the FSU. There was some evidence of analysis of program location (DFAIT versus CIDA) but no formal assessment of DFAIT programming capacities nor broader stream level Canadian comparative advantage.

5.1.4 Adaptation to Changing Context

Evaluation Issue: How have the vision, strategic focus, program design and delivery mechanisms adapted since inception to ensure the continuing high relevance of this program , taking into account changes in threats/needs, the participation of other partners, and Canadian foreign policy objectives and priorities?

The fourth relevance issue addresses the ability of the program to adapt to changes in threats / needs, the participation of other partners, and changes in domestic and foreign policy objectives.

In terms of changing threats and needs data collection from key informant interviews and the literature review showed that there were a number of important changes taking place. However, it should be stressed that all parties see the threats as continuing at least at the level of 2002. Part of the change has been the rate of progress in dealing with those threats. Specifically:

  • Progress on nuclear submarine dismantling by the Russians and with assistance from the US, Norway, UK and Canada is such that the threat in Northern Russia will soon be adequately dealt with. There remains the major problem of the Pacific Fleet;
  • Relatively derelict former Soviet Naval bases in the North Atlantic remain a major environmental threat;
  • The threat of chemical weapons stockpiles is being addressed in a systematic manner at Shchuch'ye. Once the destruction facility at Shchuch'ye becomes operational (2008), the stockpiles will be eliminated in a four year period; and
  • The feared diaspora of former weapons scientists has largely not taken place. The original generation is nearing retirement age but there is a need for the West to stay engaged in this issue.

Partner agencies in Canada and internationally emphasized some important threat areas which should/could be the focus of priority actions in the future, including the following:

  • The Soviet Pacific fleets decommissioned nuclear submarines;
  • Biological agents (not necessarily weaponized) in laboratories inside and well beyond the FSU and the scientists capable of producing weapons from those agents;
  • Cross border movements of nuclear and radiological materials that could be used by terrorists. This "2nd line of defence" threat means that there is considerable work to be done in countries of the FSU that border on Russia as well as in other continents (Africa, South Asia); and
  • Fissile materials in countries outside the FSU.

The GPP has adapted to the above changes in a number of different ways. Program reporting documents and interviews with stakeholders outside GPP indicate that the four streams remain relevant. Some experts feel that the problem of nuclear submarines (Atlantic Fleet) and chemical weapons is well on the way to being resolved and that investments will be required in new priorities. There is a general view that the ISTC is worthwhile as a means for the G8 countries to remain engaged in scientific cooperation with an important potential source of threats. Fissile material remains a critically important threat area, especially since it is relatively easily transportable with major economic and social consequences.

A majority of the external key stakeholders felt there was a strong need to engage more actively in biological safety and security as a growing threat area both inside and outside Russia and the FSU. The GPP is examining the possibility of shifting some of its work on decommissioning nuclear submarines to the Pacific from the Atlantic Fleet, taking into account an increasing ability of the Russian Federation to financially support this work. It recently undertook a review of priorities and has done some re-profiling in that area. The GPP is also continuing to advocate with its GP partners that GPP funds and actions be usable to deal with biological threats (not necessarily as Bio-weapons). Program staff and external key informants note the recent emphasis on the threat of infectious diseases by Russia as the current Chair of the G8 presidency as an example of progress on this front.

Contributions to the IAEA have been used to address NRS issues outside of the Russian Federation (though within the FSU) in terms of both cross-border movements and reduction of WMMD. In addition there is considerable discussion within the program about the expansion on geographic scope. Both the use of IAEA contributions and support to the STCU represent an adaptation to the recognition of the globalized nature of the WMMD threat and risk.

Another key aspect of the changing context of GPP has been the commitments and activities of approximately 13 non-G8 states, including especially Norway. While no country (G8 or otherwise) has left the Global Partnership or indicated they will, some have been very slow to ramp up their activities and to disburse funds. However, most of the adaptation of the program in response to the participation of partners has been to complement the programming of other partners as they ramp up. The best example of this has been in the NRS stream, specifically with respect to radioisotope thermoelectric generators (RTGs). Canada's participation in addressing the RTG issue has been to complement the work of other partners. This has involved addressing specific bottlenecks in the work of other partners to transport radioactive materials and through the development of a master plan to address the RTG issue. However, there is recognition by the GPP of the changing profile of donors in this area, and since the number of donors and resources are both increasing, the NRS stream has shifted its focus away from RTGs to physical protection.

In terms of changes in foreign and domestic policy priorities, there have been no significant changes away from the priorities that support the GPP. Rather there has been a reinforcement of those priorities and an increase in the relevance and consistency of the GPP. Therefore no adaptation has been necessary.

Summary of Findings

The GPP has adapted to a number of changes in the emphasis and understanding of the threats and needs over the three years of GPP implementation. These have included recognition that all four priority areas remain relevant, solutions are engaged in chemical weapons and submarines (Northern Fleet) and the fact that, to date there has not been a diaspora of former weapons scientists. Threat perception and GPP plans are increasingly focused on biological safety and security, cross-border transportation of dangerous material, and the Pacific submarine fleet. The GPP acts in a very complementary fashion to the work of partners and has been seen to adapt to increased partner involvement by focusing on other priority areas. There have been no significant shifts or changes in domestic or foreign policy that the GPP has had to adapt to, rather policy changes have reinforced the priority placed on the GPP.

5.1.5 Relevance of the Special Projects Fund

Evaluation Issue: What was the original rationale for the establishment of the Special Projects Fund, and to what extent does this rationale continue to be relevant?

The Special Projects Fund (SPF) was created through TB Phase II, in October 2004. The primary rationale for the establishment of the SPF was the need to support the six principles of the Global Partnership, especially with their reference to preventing terrorist interests from acquiring, in particular, biological non-proliferation or related material. The SPF was also intended to provide an operationally simple and rapid means of matching opportunities (especially those arising from the work of the ISTC) with relatively small amounts of funding. The SPF allows this to be done without seeking separate TB approvals for the amounts allocated.

In terms of the ongoing relevance of this rationale, the SPF continues to fit with recognition at Kananaskis of threats and needs outside of the four priorities. This is most relevant with respect to biological security. In practical terms the SPF has provided the flexibility to undertake consensus-building activities outside of the four priorities in an operationally simple manner.

Summary of Findings

The original rationale for the SPF was to support the six Kananaskis principles in the Russian Federation and the countries of the FSU in an operationally simple manner. This rationale is still relevant, especially in the area of biological security.

5.1.6 Stream Relevance to RMAF Results

Evaluation Issue: To what extent do the four identified project streams continue to be relevant for achieving the mandated program results?

There are two groupings of mandated program results represented in the initial program level RMAF logic model:
Result LevelGrouping 1 - WMMD ProliferationGrouping 2 - RFWS
Short-Term OutcomesCapacity is created to significantly secure and eliminate WMDPeaceful scientific research projects are undertaken
Intermediate OutcomesWMD Stockpiles are significantly reduced WMD Stockpiles are securedWeapons scientists are employed in peaceful pursuits

Given these mandated results, the streams for chemical weapons destruction, submarine dismantlement, and nuclear and radiological security, all remain fully relevant to grouping one, because they are all directed at reducing or securing stockpiles. The Chernobyl investment is relevant to the first grouping if a slight expansion in the original wording of the result from WMD to WMMD is taken into consideration.(3) The RFWS stream remains fully relevant to grouping two, as it is engaged in funding peaceful research projects.

Summary of Findings

All four streams and the Chernobyl investment were found to be relevant to the achievement of mandated results.

5.1.7 Views on Relevance by GPP Partners and Beneficiaries

Evaluation Issue: To what extent is the GPP viewed as relevant by primary beneficiaries and other program partners?

Every external partner interviewed indicated that the Global Partnership as a whole and Canada's GPP remained highly relevant. In particular, US and European key informants pointed to the fact that Canada's significant financial commitment and "pragmatic, practical and activist" approach to programming helps to ensure that the GP is not perceived as "An American show." The fact that GPP has worked to promote participation in the Global Partnership by non-G8 countries is also seen as a major indicator of the relevance of Canadian participation. Finally, the GP (and Canada's GPP) are viewed by external stakeholders as evidence of a major Canadian foreign policy success backed up first by significant financial commitments and then by programming actions.

Beneficiaries also cited the program as being highly relevant and responsive to their needs. Senior staff at PNPI stated that Canadian investments were targeting the areas of highest priority according to their own ranking. They stated that in the absence of Canadian support they would not have been able to complete the required physical protection upgrades on their own in a reasonable timeframe. They also stated that the Canadian team has been responsive to their needs.

Summary of Findings

Beneficiaries and partners see the GPP as highly relevant and as adding value to the greater GP. Canada is seen as having delivered an international relations success at Kananaskis, a significant and respected financial commitment, and followed that up with timely and pragmatic program delivery that is meeting priority needs.

5.1.8 Canada's Reputation and Credibility

Evaluation Issue: To what extent does the GPP contribute to Canada's international reputation and credibility?

Representatives of external partner agencies in the US and in Europe, including bilateral and multilateral partners are strongly of the view that participation in the program has been a major benefit to Canada's reputation and credibility. They cite the following:

  • Canada's commitment and skill in supporting negotiations before and during the Kananaskis summit;
  • The size of Canada's financial commitment to the GP given its relatively small population;
  • Complementarity and synergy with work of partners / Strategic use of multiple channels: The pragmatic approach taken to directing funding through bilateral (collaborating on existing initiatives with the US , UK, Norway) and multilateral initiatives (EBRD, IAEA) while negotiating an agreement with Russia.
  • The energy and technical competence of key GPP staff members as they interact with their partners inside and outside Canada;
  • The GPPs demonstrated ability to disburse funds on concrete project activities at a much faster rate than a number of GP partners; and,
  • The compact nature of the secretariat for the GPP which combines all four priority areas in a single organizational unit.

In particular, US-based partners pointed out that Canada's active participation in GPP is regularly reported to and remarked on positively by US Congressional and Senate committee members. There has also been credibility built within the US intelligence community. Consequently Canada is seen as a front-runner in the GP, often cited as the first or second ranked participant, in terms of action and commitment.

A number of US and European based key informants indicated that they would welcome Canada taking a more active role in the area of bio-safety and security given what they see as a strong comparative advantage in the close cooperation of Health Canada, Agriculture Canada and the Canada Food Inspection Agency (CFIA). They point, in particular, to the Winnipeg facilities of Health Canada.

Summary of Findings

Canada's participation in the Global Partnership, in terms of the commitment it has shown, the skill and energy of the staff, the complementary and synergistic manner of working with partners, the strategic use of multiple channels, and the demonstrated emphasis on implementation, have all served to significantly enhance the reputation and credibility of Canada. Canada is now seen as a front-runner in the GP in terms of action and commitment.

5.1.9 Consequences of Reduction/ Non-Renewal

Evaluation Issue: What could be the domestic and international consequences if the GPP was renewed at reduced levels?

A strong consensus exists among external stakeholders that a move by Canada to reduce or eliminate its commitments under GPP would have negative consequences, both internationally and domestically.

International consequences for the GP and wider WMMD reduction efforts could include the following:

  • That such a move by Canada would be interpreted externally as an indication that the threats addressed by GPP are no longer a priority and would weaken the commitment of other G8 and non-G8 countries;
  • A number of external partners would draw the conclusion that Canada was downgrading or reversing its commitment to the GPP because it found it too difficult, time-consuming, complex or risky to continue to program in this area. This would call into question Canada's confidence in its Russian partners and give considerable pause to other countries supporting these projects.
  • Most significantly there would be an adverse impact on WMMD proliferation reduction efforts due to decreased or interrupted GPP activity.

The possible implications for Canada could include the following:

  • Canada's ability to be taken seriously in multilateral and bilateral discussions relating to joint actions to counter nuclear, radiological, chemical and biological non-proliferation would be severely undermined; and,
  • As the GP and GPP are so closely associated with Canada, diminished or reversed commitments by Canada would undercut whatever enhanced credibility and reputation for Canada had been generated by the success of Kananaskis and the Global Partnership.

US partner agencies stressed very strongly that the GPP was viewed in very senior security policy (and political) circles in the US as a concrete expression of Canada's commitment to global and North American security. In their view it represents not only a policy commitment but a willingness to share in the heavy financial lifting of global security. In that way some liken it to Canada's commitments to NATO operations in Afghanistan. Any reversal at this point would seriously undermine the perception that Canada is willing to share the significant costs of ensuring North-American security. It could also represent a negative factor in Canada-US relations.

At a project level, beneficiaries at PNPI indicated that they might be able to undertake the upgrades on their own but it would take them 10 years or more with the small amount of financial resources that they have available for this type of infrastructure. Therefore without the GPP they would have looked for another donor but this, they indicated, would not have been easy.

Summary of Findings

Reducing or discontinuing funding for the GPP could have significant international consequences and impact negatively on Canada's reputation and relations with the US. Internationally it would likely lead to an erosion of general support for the GP, it would be taken as a vote of no confidence by Canada towards Russia, due to risks / complexity of program implementation, and it would have an adverse operational impact on WMMD proliferation reduction. For Canada, it would likely lead to an undermining of credibility in key international security areas and could seriously undermine relations with the US on North American and global security matters.

5.1.10 Geographic Scope Beyond the Former Soviet Union

Evaluation Issue: To what extent would broadening the geographic focus increase program relevance?

When considering a widening of geographic scope there are many factors to consider in assessing whether this would increase program relevance. In terms of the threats, there is apparently a clear need to deal with nuclear materials and biological agents and laboratories well outside Russia and the countries of the FSU. In terms of nuclear, radiological and fissile materials threats, key informants note there are "interesting and interested" states in North Africa, South Asia and Latin America in particular. In terms of biological agents, laboratories and scientists to be addressed, key informants referred to Singapore, the Philippines, and other countries in South and Southeast Asia and Latin America.

A number of key informants also pointed to the risk of globalization and widespread shipment of fissile materials as a key threat area and suggested Canada could play a major role in helping countries establish appropriate border controls and radiological monitoring mechanisms.

The risk of broadening the program geographically might be a perception on the part of Russia that the G8 are backing away from commitments made at Kananaskis. On the other hand, several key informants pointed out that as increasingly these threats are successfully dealt with in Russia, and as Russia's oil and gas revenues increase and its underlying economic condition continues to improve, the GP could allow Russia to move into a cooperative role with G8 countries and other partners in dealing with threats well outside its borders. This would represent a maturing of the relationship between Russia and her G8 and non-G8 partners and would serve partners' needs for improved security against critical threats.

Another potential challenge of broadening geographic focus would be an increase in program complexity. Working in more countries might require a suite of bilateral treaties, for example. There is also a risk of less focus on program, as projects might become fragmented as they are dealing with very different issues in a variety of country contexts. Finally the nature of risks faced by the program would also change.

Summary of Findings

The broadening of geographic scope would have some benefits that would increase the relevance of the GPP. These include an ability to address the geographic diversity and globalized aspect of threats. It would increase program relevance and enhance responsiveness to risks. This is only the case for biological security, NRS and RFWS, and to a lesser extend CWD. It does not apply to submarine dismantlement. The challenges of a broadening of geographic scope would be a signal to Russia that support may be diluted, less focus to the program, and an increase in program complexity.

5.2 Results

This subsection presents the current level of results achievement by the GPP as assessed by the formative evaluation based on data generated by all four evaluation methods.

5.2.1 Results Achievement

Evaluation Issue: To what extent has the GPP achieved its planned results? Have any of the individual projects resulted in unintended/unexpected positive or negative results?

As stated in section 5.1.6, there are two groupings of mandated program results represented in the initial program level RMAF logic model:
Result LevelGrouping 1 - WMMD ProliferationGrouping 2 - RFWS
Short-Term OutcomesCapacity is created to significantly secure and eliminate WMDPeaceful scientific research projects are undertaken
Intermediate OutcomesWMD Stockpiles are significantly reduced WMD Stockpiles are securedWeapons scientists are employed in peaceful pursuits

The planned results achieved by the program are presented below.

Grouping 1 - WMMD Proliferation:

Short-term outcomes:

ST Outcome 1: Capacity is created to significantly secure and eliminate WMD

Capacity to secure and eliminate WMMD has been created across the GPP. However, the indicators in the initial RMAF logic model are not specific enough to measure changes at the short-term outcome level. For the purposes of the formative evaluation, capacity is understood to comprise the following:

  • Infrastructure and equipment
  • Skills, abilities, knowledge, and understanding
  • Systems, processes, plans, and policies
  • Financial resources
Submarine Dismantlement

Since the initiation of Canada's program for the dismantlement of decommissioned nuclear submarines, there has been substantial creation and maintenance of capacity for securing weapons and materials of mass destruction.

Under Implementation Arrangements 1 and 2, a minimum of 500 Russian workers have been in full time employment with up to 2,000 others employed on related dismantling activity from within Zvezdochka's 8,000 shipyard worker complement. Whenever possible, the dismantlement of two or more submarines takes place concurrently, resulting in employment for several shift teams. Practically all of these workers have continued to develop specialized skills involved with each of the 13 stages of submarine dismantlement. These skills range from detailed knowledge for preparing the submarines's transportation through highly specialised nuclear engineering skills for defueling the submarine's twin nuclear reactors. The substantial volume of both nuclear and hazardous wastes to be handled and treated also requires appropriate specialist skills, which have been further refined through work on this project.

GPP has provided worker safety equipment, been instrumental in the shipyard adopting new engineering and safety procedures and arranged for nuclear safety training of Russian shipyard workers in Canada. All of these measures have built capacity to secure and eliminate WMMD in a manner that stresses safety.

Canada has also contributed funding for a significant expansion of the "Harris Pad"; a specialised concrete pad area for scrap metal handling. Canadian funded expansion includes a special drainage system to stop 99 percent of the pollutants from entering the sewage system and a high intensity lighting system to allow around the clock work (due to the fact the shipyard is located near the Arctic circle there is a significant loss of daylight for much of the year). The increased scrap handling area will be directly providing the increased capacity needed to support the US' funded dismantling of a Typhoon ballistic missile submarine which will be taking place 2006 through 2008 in parallel to Canada's activities.

Nuclear and Radiological Security

NRS stream project and institutional support has built capacity in a number of different areas. GPP is building Russia's capacity to solve the RTG problem. By funding the development of a Master Plan, Canada is helping Russia to create a comprehensive picture of current and required capabilities, and how to fill the gap. GPP funding of RTG transportation infrastructure, such as transportation and security shielding containers, is strengthening Russia's capacity to transport highly radioactive sources in a safe and secure manner.

GPP is also including sustainability and training plans into all of its projects. All physical protection projects, including PNPI and ITEP, have sustainability plans built into the Implementing Arrangement, such as spare parts, warrantees and a commitment to future training, as part of an effort to build long-term capacity.

Training is a key part of these efforts. For example, an important part of the PNPI project is the provision of equipment for the training room. Training on detection equipment is also an integral part of the Chernobyl Exclusion Zone project. In addition, GPP has developed specific capacity building training projects. GPP is funding the development of 3 training courses at the Russian physical protection training centre in Obninsk, as well as the development of another training equipment test bed. Obninsk is the main training facility for physical protection specialists in Russia and the rest of the CIS.

The GPP-funded workshop on the IAEA Code of Conduct provided capacity building expertise to Russian speaking countries on how to implement the Code. The GPP-funded IAEA International Protection Advisory Service mission to a Central Asia state produced vital recommendation on how this state can increase its physical protection capacity.

The provision of equipment in all physical protection and border security detection projects also contributes directly to building the capacity of countries of the former Soviet Union to significantly secure and eliminate WMD.

Chemical Weapons Destruction

Capacity improvement in terms of the provision of infrastructure is still ongoing in the CWD stream and has not reached the short-term outcome level yet.

Intermediate Outcomes:

Intermediate Outcome 1: WMD Stockpiles are significantly reduced

The submarine stream and NRS have both contributed to the achievement of the reduction of WMD stockpiles.

  • 10 nuclear reactors have been defuelled
  • 4 submarines have been dismantled
  • A quantity of WMMD has been reduced through a recovery operation in Central Asia

Intermediate Outcome 2: WMD Stockpiles are secured

The NRS stream has contributed to the securing of WMD stockpiles as follows:

  • WMMD in the Chernobyl Exclusion Zone more secure (due to the operation of handheld detectors)
Grouping 2 - RFWS:

Short-term outcomes:

ST Outcome 1: Peaceful scientific research projects are undertaken

Under the RFWS stream, there has been significant progress in terms of peaceful scientific research projects underway involving former weapons scientists from former WMMD institutes of the FSU considered priorities for proliferation risk reduction.

Currently there are 62 projects underway, redirecting a total of 1,408 former weapons Scientists of whom:

  • 160 are Missile Scientists,
  • 134 are Chemical Scientists,
  • 287 are Biology Scientists,
  • 598 are Nuclear Scientists and,
  • 72 are categorised as "other".
Significant Unintended Results

The formative evaluation was also tasked to determine if there had been any significant unintended or unexpected results, either positive or negative. In terms of the projects and interventions there were no reported unintended results. For the GPP as a whole, there have been unintended increases in Canada's credibility, through demonstrated commitment to the GPP, which has enhanced Canada's reputation in a very positive manner.

Summary of Findings

In terms of short-term and intermediate outcomes, there have been results achieved at both levels. Capacity has been created in a number of areas, most significantly in the securing and elimination of WMD. This has included skills, knowledge, infrastructure, equipment, and plans. WMMD stockpiles have been reduced and secured including ten nuclear of reactors defuelled, four submarines have been dismantled, and a quantity of WMMD through a recovery operation in Central Asia. There has also been redirection of 1,408 former weapons scientists through 62 science projects.

5.2.2 Results by Program Stream

Evaluation Issue: How are the results from the individual projects rolled up into the program 's results?

The second results issue requires an analysis of reporting by the GPP to understand how the results from individual projects in the four streams are "rolled-up", or synthesized and summarised at the program level. In order for this roll-up of results to occur there needs to be a results framework at the program level. For the purposes of the formative evaluation, the initial program level RMAF logic model was taken to be that results framework.

The following GPP narrative progress reports were analysed for this issue:

  • 6 month report to TB
  • GPP Comprehensive Report on Programs 2002-2005
  • Annual Report 2005

In those reports the formative evaluation found an absence of summative statements based on program RMAF short-term and intermediate outcomes. (One possible approach to rolling up the results would be the format used in the previous section 5.1.1). Rather, program level progress reporting was not organised around results or did not contain results information and stream and project level reporting contained a mixture of activities and results. This may be due in part to the fact that the GPP is for the most part in the early stages of results achievement. Once a larger volume of short-term and intermediate outcomes are achieved by the program, reporting may focus more on these. This is not to say that the performance measurement information on results achievement is not collected, but to note that reporting occurs more at the activity than the results level.

Summary of Findings

Results from individual projects are not rolled up into program level results and are not reported according to a program level results framework. This may be due in part to the early stage of results achievement for most of the GPP.

These findings are linked to recommendations 1, 2, and 3.

5.2.3 Impacts of Canadian Funding on Others

Evaluation Issue: To what extent have Canadian-funded GPP projects led to increased international commitments, funding or action by other countries?

The fourth results issue posed a challenge in terms of attribution for the formative evaluation. The evaluation was tasked to determine if any specific GPP projects had triggered commitment, funding, or action by other countries. For this issue it was very difficult to separate out the cause and effect relationship between GPP projects and what other countries did, given the overarching policy dialogue and complementarity at the sub-stream, stream, and program levels.

In the Chemical Weapons Stream, Canadian initiatives have led to commitments and funding by other donor countries. In October 2005, DFAIT and the British Foreign Office sent a joint letter to other potential non G-8 donor countries inviting them to make donor contributions to CW destruction at Shchuch'ye. As a result of this coordinated initiative, the Netherlands, New Zealand, Ireland, Sweden, and Belgium are making contributions. Sweden, Netherlands, and New Zealand referred specifically to the letter and indicated they would contribute 4.4 million Krona, 3 to 4 million Euro, and 1.2 million New Zealand dollars respectively.

Summary of Findings

The GPP as a whole has facilitated the commitment, funding, and action of other countries in the GP. The commitments of non-G8 countries extends beyond the dollar value of their Program contributions; rather these commitment demonstrate a broader engagement of the community to address such issues. The railway construction project clearly led to increased funding of other countries. GPP activity in submarine dismantlement and NRS has influenced and enabled the actions of other countries, but it is more difficult to attribute this to a single project.

5.2.4 Increasing Canadian Capacity

Evaluation Issue: To what extent is the GPP contributing in any way to increasing Canadian capacity?

The final results issue is focussed on the area of Canadian capacity and specifically whether the GPP was contributing to increasing Canadian capacity. This issue applies mainly to the RFWS stream, as the nature of other streams does not readily enable Canadian capacity building.

In the RFWS stream, activities through the ISTC and STCU and with Canadian collaborators and partners have significant potential to increase Canadian capacity in a number of key areas:

  • Knowledge
  • Technology
  • Products

Due to the early stages of program development, Canadian capacity increases are not reported to have taken place yet. However there are strong indications that this could be a significant unintended result of the program (unintended in that Canadian capacity is not part of the initial program level RMAF logic model as an expected result). Canadian partners and collaborators stand to benefit from successful product design and have indicated that this is expected from their side. Some key informants have questioned the commercial value of ISTC-supported research, but acknowledge that they represent a "good science."

Summary of Findings

Increased Canadian capacity is only applicable to the RFWS stream. There are other recognised potential increases to Canadian capacity. However, due to the current state of implementation, these remain at the "potential" stage. (Notable exceptions are Raytheon Canada monitoring NRS portfolio and Nuclear Safety Solutions serving as sub-contractors on submarine dismantlement.)

5.3 Design and Delivery

5.3.1 Strategic Design / Program Consistency

Evaluation Issue: To what extent were the strategic choices of the program documented and formulated into clearly defined and operationalized objectives? Do they continue to be? To what extent is the current program consistent with original documented design and delivery concepts and plans and aligned with best practices?

In the early stages of the GPP, senior management in IGX and the Department made a number of strategic choices about the design and delivery of the program. These strategic choices included identifying which country and priority areas to focus on; how to choose and implement projects; what would be the required human resources, etc. The strategic choices are reflected and demonstrated in the following areas:

i) The Kananaskis Framework

The 2002 G8 Summit was held at Kananaskis with Canada as the Chair. As Chair, Canada played a key role in initiating the Global Partnership and developing the Principles. At the Summit, the G8 leaders identified that the Global Partnership would support cooperation projects, initially in Russia, to address non-proliferation, disarmament, counter-terrorism, and nuclear safety issues. Four priority areas were specifically identified: the destruction of chemical weapons, the dismantlement of decommissioned nuclear submarines, the disposition of fissile materials and the employment of former weapons scientists. The Statement by G8 Leaders also made it clear that the G8 would be willing to enter into negotiations with other willing recipient countries, including those of the Former Soviet Union (FSU). In addition, the Statement clearly included biological security as an area of concern (Ukraine subsequently joined as a recipient).

Importantly, the G8 Statement included six guiding Principles for the Global Partnership that were drafted by Canada prior to Kananaskis and later unanimously endorsed by the UN General Assembly.(4) The Principles were aimed at preventing terrorist interests from acquiring or developing nuclear, chemical, radiological and biological non-proliferation, or related materials, equipment, technology and expertise.

The Statement also set out a series of guidelines for bilateral and multilateral cooperation projects funded through the Global Partnership. The guidelines stated that all Global Partnership projects were to observe the following elements:

  • be transparent and open to monitoring and audit;
  • adhere to accepted environmental protection and safety standards;
  • be based on clearly defined project milestones;
  • adhere to peaceful use provisions and the application of adequate physical protection measures;
  • provide full exemption from taxes, duties, levies and other charges;
  • base project implementation on international procurement standards;
  • provide adequate liability protection for project personnel and contractors;
  • provide appropriate privileges and immunities for donor representatives working on cooperation projects; and
  • ensure adequate protection of sensitive information and intellectual property.
ii) Location of the Global Partnership Program within the Government of Canada

The initial Memorandum to Cabinet of August 2002 stated that a Global Partnership Bureau would reside as a single unit within the Department of Foreign Affairs and International Trade. This decision was reinforced in subsequent Treasury Board submissions (Phase I of August 2003 and Phase II of October 2004). Creating a single unit to administer the GPP provided a single point of contact within the government that could also act as a coordinating unit; it reduced the possibility of duplication of effort or issues being overlooked; and allowed for an integrated approach across all priority areas.

iii) Organization Chart and Capacity of the Unit

According to Phases I and II TB submissions, the IGX bureau was to be lead by a Senior Coordinator (Director General level) reporting to the ADM, Global and Security Policy. Two Directors would report to the Senior Coordinator. Initially, each Director focussed on two of the four priority areas. Over time, this has evolved to one Director managing three priority areas (Nuclear Submarines, Chemical Weapons, Nuclear and Radiological Security), and the other Director responsible for Former Weapons Scientists, Biological non-proliferation, and corporate policy. The Phase I submission stated that there would be four Senior Program Managers (SPM) who would each be responsible for one of the priority areas. As progress evolved in biological non-proliferation, one SPM became responsible for that area, as well as chemical weapons, because of a specialized skill set. With the recent priority review, it has been decided to separate the biological non-proliferation program into a separate stream to be headed by a Senior Program Manager. Biological non-proliferation, as an area of interest to the IGX, was clearly identified in the TB Phase I submission.

The majority of IGX senior management have sector-relevant experience and/or abilities in the areas of non-proliferation.

The unit was designed to include program officers at the PM-4 level in each priority area. It was expected that these program officers, along with the Director, would be able to manage the priority area in the event the SPM was unavailable for an extended period of time.

IGX has dedicated legal counsel and a financial officer. The bureau also accesses technical advice from other government departments and agencies, including PHAC and the Canadian Nuclear Safety Commission.

IGX also has 4 staff based in Moscow - 2 Canada based and 2 locally engaged. The Canada based staff are an FS4 and FS3. The positions provide the program with information gathering abilities in Russia, reporting, relationship development, continuity in the field, support for missions, and high and mid-level interventions as needed. The locally engaged staff provide support in accessing various levels of counterpart organisations, meeting facilitation, translation support, and logistics.

Another source of advice is the Global Partnership Advisory Group (GPAG). This group is chaired by the IGX Senior Coordinator and consists of senior officials from a range of other government departments, including: National Defence, Health Canada, Industry Canada, Environment Canada, CIDA, NRCan, National Research Council. The original GPAG Terms of Reference were included in TB submission Phase I and were recently revised.

There are also other priority area specific expert groups that provide advice to IGX. This includes the Science, Trade, and Technology Advisory Group (STTAG), that provides guidance to the Redirection of Former Weapons Scientists Unit, as well as trilateral working groups.(5)

The organization design, including the balance between in-house resources and external resources, remains strategically aligned to deliver the program and has been adequately documented.

iv) Role of External Service Providers

Both TB Phase I and Phase II submissions acknowledge that IGX would require expert technical advice and monitoring services from external service providers. In practice, two examples include an arrangement directly with Teledyne Brown Engineering of the US, and an indirect arrangement with Bechtel. Teledyne Brown has provided technical expertise in negotiating with the Russian shipyard at Zvezdochka, and has provided technical expertise in the areas of nuclear, marine, environmental and railway engineering. Representatives from Teledyne Brown participate in the Technical Monitoring Team's monthly monitoring visits to the shipyard.

Bechtel is the principal contractor and the on-site coordinator for Canada's railway construction project at Shchuch'ye, and is contracted by and reports to the UK Ministry of Defence. Bechtel holds the contract with the Russian sub-contractor undertaking railway construction, observes on-site operations on a day-by-day basis, and routinely informs Canada of all project-related matters.

In the NRS stream the GPP hired a specialised technical team through PWGSC's open procurement process to provide Project Management and Monitoring and Advisory support for Canada's physical protection projects under the GPP. Raytheon Canada Ltd. has been selected to provide this support and final contract discussions are underway.

v) Overarching Governance / Legal Framework

As described in TB Phase II submission, all projects delivering contributions under the three delivery mechanisms described below are governed by written legal relationships contained in Implementing Arrangements. Implementing Arrangements include: Memoranda of Understanding (MOU) (for example, with an international organization); Memoranda of Agreement (i.e., for supply of in-kind contributions); and Contribution Agreements/Arrangements (with legal entities who are eligible recipients). Canada's participation in the Shchuch'ye railway project is governed by an MOU with the Ministry of Defence, U.K.

All bilateral cooperation between Canada and Russia under the GPP is governed by the Agreement between the Government of Canada and the Government of the Russian Federation Concerning the Cooperation on the Destruction of Chemical Weapons, the Dismantlement of Decommissioned Nuclear Submarines and Nuclear and Radioactive Material Protection, Control and Accountancy. Signed in June 2004, the agreement was provisionally applied until its entry into force in August 2005 upon ratification by the Russian Duma.

There are a number of significant protections for Canada included in the Treaty. These include: the establishment of schedules and milestones, contracting, access to facilities, privileges and immunities, tax exemptions, peaceful uses, intellectual property, environmental soundness, evaluation, monitoring and verification, audits and evaluation and nuclear and non-nuclear liability. The Treaty also provides Canada with the unilateral right to suspend any project without incurring liability.

It should be noted that not all G7 countries have successfully negotiated a Treaty with Russia as comprehensive coverage as obtained by Canada.

vi) Selection of Delivery Mechanisms

As documented in TB submission Phase I, when the GPP was initiated, it was decided to take a multi-phase approach to securing TB authorities to proceed with funding commitments. In order to mitigate risk while a bilateral agreement with Russia was finalized, IGX chose to make a series of contributions directly through established multilateral institutions and through a G8 partner already engaged in the project area.

After the Bilateral Agreement was signed in June 2004, TB submission Phase II documented three project delivery mechanisms available to the program: Multilateral, Bilateral Donor, and Bilateral Recipient.

The Multilateral delivery mechanism includes contributions to programs of multilateral, intergovernmental, and international institutions and organizations involved in threat reduction activities in Russia and the FSU. This includes contributions to the EBRD, the IAEA, and the ISTC.

The Bilateral Donor delivery mechanism involves contributions made by Canada via another national government(6) which has an existing bilateral agreement with Russia (or the FSU) and that allows for third party contributions. An example of this arrangement is Canada's contribution to the Shchuch'ye railway project with the U.K.

The Bilateral Recipient delivery mechanism provides for direct project support to the Russian Federation (or sanctioned entities) involved in Cooperative Threat Reduction. It also applies to FSU states, but bilateral treaties do not yet exist for FSU countries. These contributions would be governed by Implementing Arrangements that would encompass the G8 Principles and Guidelines agreed to at Kananaskis. The nuclear submarine dismantlement program and PNPI are examples.

In addition, through Contribution Agreements, NGOs provide a vehicle for the delivery of project components. For example, in the chemical weapons destruction stream, IGX is working with and supporting the activities of Green Cross, which runs a network of local public outreach offices in Russia that aims to provide independent and objective information about Russia's CWD program to the populations living in the vicinity of Russia's CW storage and destruction facilities.

vii) Commitment to a Project Focus

The IGX bureau has demonstrated a commitment to implementing the GPP on a project basis since the beginning of the program. This project focus is consistent with the approach described in the Statement by G8 Leaders at Kananaskis (..."we will support specific cooperation projects...") and particularly as described in the Kananaskis Guidelines for New or Expanded Cooperation Projects. Elements of these guidelines have been incorporated into the Bilateral Treaty (milestones, monitoring, auditing, environmental soundness, tax exemption, etc.), as well as other project Implementing Arrangements.

The commitment to a project focus has been reinforced in subsequent Treasury Board submissions (Phase I, Phase II, Nuclear Submarines, MDB-2). In TB Phase II submission, a comprehensive Project Management Framework was developed for the program. This Framework was based on the Project Management Body of Knowledge (PMBOK) and is consistent with best practices. The Framework detailed the steps, requirements, and approval processes to follow for the life cycle of a project. The project management strategies identify and mitigate risk at the project level. In the TB Phase II submission, IGX estimated that close to 70 projects covering a wide range of values and time lines would be implemented over the remainder of the program.

The project focus has become the focus for governance, management, allocation of funding, monitoring and oversight, and reporting.

viii) Qualitative Rationale for Allocation of Funding / Selection of Projects

When the GPP was launched, the TB Phase I submission detailed a number of qualitative selection criteria that would guide IGX management in deciding which projects would have the greatest impact and had the highest priority. A sample of the selection criteria included the following:

  • consistency with the priorities, principles and guidelines endorsed by the leaders at Kananaskis;
  • magnitude of the threat reduction to be achieved;
  • relevance to Canadian security interests;
  • impact of the Canadian funds;
  • consistency with Russian priorities; and
  • project risk assessment.

As described in the case study on Former Weapons Scientists, there are also a number of selection criteria that have been developed and are applied to potential projects funded through the ISTC. There were six major criteria:

  • merit of the proposal;
  • synergy and quality of the team;
  • quality of management;
  • financial structure;
  • feasibility of the proposed project; and
  • expected benefits.

According to the TB Phase II submission, four broad project selection "screens" are applied to determine the feasibility of proposed nuclear security projects. These include screening for the Enabling Environment, the Technical Capacity, and the Government of Canada Policy, as well as a screen for the Verification of Canadian Capacity. These screens are consistent with the approach outlined in the PMF.

ix) Quantitative Rationale for Allocation of Funding between Priority Areas

In the early stages of the program , senior IGX management allocated Canada's $1 billion commitment (over 10 years) between the four priority areas as follows:

  • Dismantlement of Decommissioned Nuclear Submarines: $300 million
  • Destruction of Chemical Weapons: $300 million
  • Nuclear and Radiological Security: $160 million
  • Re-direction of Former Weapons Scientists: $180 million

This allocation was determined on a notional basis and its rationale is undocumented. When the IGX was given the responsibility for the Chernobyl Shelter Fund and when the Global Special Projects Fund was created (TB Phase II submission), there was some refinement of these allocations.

The February 2006 Priority Review provided a formal opportunity to re-assess these allocations. At this time, it was decided to modify the amount allocated to nuclear submarines and redeploy the resources to nuclear security and to Biological non-proliferation.

Summary of Findings

IGX and DFAIT senior management were required to make several strategic choices at the initiation of the Global Partnership Program. The evolving nature of the many risks and threats involved in this program require that the management team continue to make strategic choices. Based on the above analysis, it is clear that the large majority of the strategic choices were documented and formulated into clearly defined and operationalized objectives, and that they continue to be relevant today. Furthermore, the current program design is consistent with the original documented design and delivery concepts and aligned with best practices. These best practices are found in the selection criteria, project focus, and project management framework.

The readjustment of funding that took place in the Priority Review of February 2006 is a favourable sign of reallocation of funding at a program-wide level based on a threat and risk assessment. In the full-scale implementation phase and with new TB authorities beginning in 2008/09, program-wide risk assessment and prioritization are essential components for determining funding levels on a program-wide basis and are increasingly important components of program design.

These findings are linked to Recommendation 1.

5.3.2 Anticipated Future Costs

Evaluation Issue: To what extent is planning and funding in place to support the anticipated future costs of all of Canada's GPP commitments to partners and beneficiaries?

At the 2002 G8 Kananaskis Summit, leaders committed to raise $20 billion towards projects addressing the spread of weapons and materials of mass destruction. This $20 billion is also known as the "10 plus 10 over 10" threat reduction initiative, in as much as the United States stated it would commit $10 billion over 10 years provided the amount was matched by G7 and other countries. As its contribution, Canada committed $1 billion. This $1 billion represents an amount that is proportional to Canada's GDP vis à vis the G7's contribution towards the $10 billion.

Originally, the intention was to program the spending at an equal amount of the $1 billion over the 10 years, or $100 million per year.

As it became evident that it would be difficult to disburse the funding on a straight-line basis, particularly in the early years of the program when the legal framework was still being put in place, it was decided to re-profile the spending pattern to more of a "bell curve." This would provide the program with the required budget flexibility to meet project needs and delivery mechanisms as indicated in the RMAF developed in the TB Phase I submission. The MDB-2 TB submission revised the planned spending to reflect an increase in spending during the peak activity years of the GPP during 2005/06 to 2008. During these years, it was anticipated that the GPP would require funds greater than the $100 million per year notionally allotted in TB Phase 1 submission (spending of new funds may not exceed $100 million annually). In order to meet this revised funding profile, authority was provided in the MDB-2 submission to defer some spending on Phase I and II projects by reducing spending on ISTC projects and on NRS projects in 2005/06 and 2006/07 to reflect a reduced program scope in those years and to defer some projects to subsequent years.

As such, IGX senior management have indicated that total commitments to disburse funds will total approximately $500 million by the end of 2007/08, leaving approximately another $500 million to be disbursed in the second half of the program, from 2008/09 to 2012/2013. There is no evidence that IGX has made any GPP-related commitments in any of the priority areas past March 31, 2008 when TB authorities expire. (Funds have been reprofiled in the RWS / NRS streams beyond 2008, subject to new authorities.)

With the TB authorities set to expire at the end of fiscal year 2007/08, there are mounting pressures to begin planning for the renewals and associated programming.

Financial management of the program takes place with the assistance of a dedicated financial officer charged with monitoring commitments and consolidating forecasting data. Each of the Senior Program Managers retain their own spreadsheets tracking project funding and disbursements as part of their project monitoring and oversight. There is also a strong program level forecasting system in place, providing in-year, three-year and ten-year forecasts.

The source of funding has the potential to become an issue for the program in the latter years. As part of Budget 2005, it was announced that an increasing portion of the Global Partnership Program funding would be sourced from the International Assistance Envelope (IAE), rather than entirely from the fiscal framework. Exhibit 5.3.2 presents the source of GPP funds(7) from 2007/08 to 2012/2013 as indicated in the MDB-2 TB submission.

Exhibit 5.3.2: GPP Source of Funds 2007/08 to 2012/13
 2007/082008/092009/102010/112011/122012/13
Fiscal Framework$67 M$33 M    
IAE$33 M$67 M$100 M$113 M$113 M$113 M
Summary of Findings

IGX has demonstrated that planning and funding is in place to meet the needs of the program in the future. This is evident by the role and activities of the financial officer, the planning and forecasting system in place by the SPMs, and by the request to re-profile the spending pattern of the program to meet the demand for the heaviest disbursements during the peak activity phase of the program.

There is some concern that there is increased risk to the stability of allocation of funds in the years 2009/10 to 2012/13 when 100 per cent of the GPP funds will be sourced from the IAE. This concern is based on the fact that the Global Partnership represents one program in the Peace and Security component of the IAE, along with Landmines, Counter-terrorism, etc. In addition, Peace and Security represents only one of five components also funded from the IAE, along with Development, International Financial Institutions, Crises, and Development Research.

The concern is that the GPP may face an unstable funding environment if there are trade-offs between funding programs within the IAE and that the IGX may not be able to fully meet future commitments.

5.3.3 Delivery Capacity

Evaluation Issue: To what extent is the delivery capacity, including human resources, in place for achieving intended outputs and outcomes?M

- To what extent are current design and delivery systems and mechanisms appropriate for achieving mandated results? (information systems supporting decision making; integration of compliance, risk and performance measurement requirements in GPP design; monitoring of projects and tracking results)?

- To what extent have the necessary supporting policies, guidelines, procedures, plans and agreements been put into place?

- To what extent are they facilitating or hindering the achievement of planned results? To what extent are the existing design and delivery approaches supportive of effective decision-making, including depth of analysis and project selection?

To address the above important questions, the consultants have defined "delivery capacity" in a specific manner. Delivery capacity refers to the key integrative functions, plans, tools, and outputs at the program level, that support effective program management in the achievement of outputs and outcomes.

The consultants recognize that delivery capacity has evolved as the program has gone through a number of phases. In the first entrepreneurial phase of GPP, delivery capacity was being built and the emphasis was on functionality and "making do" within an entrepreneurial program development framework. In the present steady state program delivery mode, delivery capacity has both internal features of self-management, capacity building, monitoring and oversight, and external features relating to support and inputs from DFAIT, OGDs, and outside contracted parties. This analysis takes into account the multiple sources of delivery capacity.

The components of delivery capacity addressed here include the following:

1) Human Resource Capacity

The program started out with a handful of dedicated, energetic, and resourceful senior staff who gradually built a cadre of highly skilled individuals now numbering approximately 27. Each of the four initial priority areas has a Senior Program Manager who has been in the position for a minimum of two years. Program Officers at the PM-4 position have largely been filled within the last year to 18 months, with minor exceptions. Staffing the positions was often an arduous process largely because of lengthy departmental procedures.

It is expected that the Priority Review undertaken in February 2006 will result in a number of expansions and reclassifications to meet emerging needs. For example, with the evolution of the program to fully developing a biological non-proliferation stream, a Senior Program Manager (PM-6) position will need to be filled and an additional PM-4 position for the expanding Nuclear and Radiological Security area. In addition, the program has now staffed a Senior Policy Advisor position with a Foreign Service Officer and has added a Communications Manager.

IGX has resources attached to the Canadian embassy in Moscow. There are four positions including two Canada-based and two are locally-engaged (LE). The senior position of the two locally engaged staff is unfilled at present leaving a gap in terms of local support. With a shift to increased project activities in the steady state model the Moscow office staff will play a role of increasing importance. In addition the expanding activities of the Former Weapons Scientists stream may impact on the two resources at the Moscow embassy as they are increasingly called upon to assist in visiting institutes, etc. The LE position is expected to be staff in the summer of 2006.

IGX also has shared access to personnel from other government departments to act as technical advisors, including Health Canada and the Canadian Nuclear Safety Commission.

The most senior positions of Senior Coordinator and the two Directors have provided continuity since the early days of the program. Currently, both the Senior Coordinator and one Director are rotational Foreign Service Officers. Turnover at the senior levels is likely in the upcoming year.

2) Human Resource Management

The IGX is headed by a Senior Coordinator who reports to the Assistant Deputy Minister, Global and Security Policy. Two Directors report to the Senior Coordinator. Three Senior Program Managers (Nuclear Submarines, Chemical Weapons, Nuclear and Radiological Security) report to one Director, while the other Director is responsible for Biological Non-proliferation and Redirection of Former Weapons Scientists, and corporate policy. This organizational design has remained essentially stable since the inception of the program.

Now that the program has reached a "steady state" it is increasingly likely that there will be turnover of existing personnel. A succession plan for senior management is being developed at the ADM-level. IGX senior management are currently finalizing a Human Resources Planning Strategy to be completed by fiscal year end 2005/06. The Strategy will address the issue of turnover and suggest strategies to mitigate any risk to the program. It is expected that the Strategy will also address training and professional development.

3) Authorities

All authorities granted under the TB submissions prepared for Phase I, II, Nuclear Submarines, and MDB-2 expire in 2007/08. This implies that no programming beyond that end date can take place. IGX is already starting to feel pressure to prepare for renewal of the authorities and is taking measures to begin the analysis and planning required.

4) Program Financial Management

IGX's financial management systems are delivered at the whole of department level (IMS). IGX has codes for projects and portfolios and has a very detailed financial management system that enables budgeting, forecasting, disbursements, and reports. All managers participate in and contribute to this system. Forecasting takes place on an in-year, three-year, and ten-year basis. It was reported that there were some shortcomings with the system, in terms of an inability to link project management tools with the IMS, however, it is felt that these are being managed.

In December of 2005 the program was required to reprofile $47 million. At the time the program was operating in the context of warrants due to the election and therefore a conservative approach was taken. Due to delays on the Russian side (which were supported by interviews with Russian counterparts) which were not even anticipated by Russian counterparts, several NRS projects were unable to start as planned. In addition the ISTC was behind schedule in its spending. Consequently the request was made for reprofiling. It should be noted that the program has changed the schedules for future NRS projects to take into account the delays and Canada has been spearheading initiatives at the ISTC to streamline the funding and approval process. Both of these initiatives should reduce the likelihood of reprofiling in the future. At the end of 2005/06 the program ended up spending $96 million versus the initial budget of $122 million prior to reprofiling.

5) Project Management

A comprehensive Project Management Framework (PMF) was developed for IGX and included in the TB Phase II submission. This framework is considered by IGX to be an integral part of IGX's authorities as mandated by Treasury Board. It is based on the Project Management Book of Knowledge (PMBOK), meeting international standards and encompassing industry best practices.

The PMF identifies well-defined and detailed processes, checkpoints, and gates throughout the project life cycle. This includes developing a Concept Paper, Work Plans for assessment, and other documents leading to a Project Approval Document which is signed off by the Senior Coordinator. The PMF describes the governance and accountabilities at each stage. The PMF emphasizes project control, risk management and adaptability.

A number of key informants reported that they found the PMF to be cumbersome, particularly if they were expected to adhere to each process / checkpoint / gate. As such, some use the PMF selectively. Senior management have stressed that the PMF provides a standardized framework to be used, but that it is adaptable to the nature and complexity of the project and also that it does require project management training, which the bureau is initiating for existing personnel.

The size and complexity of the nuclear submarine dismantlement project required a sophisticated project management system. It was decided, after consultation with PWGSC and an assessment of alternative systems, to use Primavera software which is run on a server separate from the Department's IT network. This decision was made by senior IGX management. MS Project is the primary project management system for the other streams, but it was reported that it is still in ramp-up mode and not being fully utilized. All projects are thoroughly tracked and payments made only upon completion of milestones.

There is an "I" Branch information management working group that has recently been formed to look into leveraging the department's SAP R3 v4.7 system (used for financial control) as the basis of a project management information management system. This undertaking is considered to be large and complex and may take a year or two to realize substantive results.

6) Information Management

A number of information management systems were described by IGX key informants. The core filing system is hard copy, the method established as the default system by the Senior Coordinator from the inception of the program. In addition to hard copy, there are three electronic options to storing information: the "I" drive, InfoBank, and the Corporate Application Tracking System (CATS). Both InfoBank and CATS are departmental initiatives and it is the consultant's understanding that InfoBank is meant to replace CATS. However, there are usability issues with InfoBank and therefore not all personnel use it consistently. It was reported that each manager tends to use their own system.

7) Results-based Management

The GPP RMAF and RBAF prepared for the TB Phase I submission are considered by senior management to be too general in nature to be used as guiding documents for managing the program. Performance monitoring and evaluation strategy documents were developed for several projects (i.e., Nuclear Submarines, Shchuch'ye Railway, MDB-2, Nuclear and Radiological Security Project through the IAEA, Former Weapons Scientists Initiative) were prepared by ZIE, but not finalized (by either ZIE or IGX) and approved. As such, these documents are not consulted and have not provided any guidance for managing the projects. (See Exhibit 5.4.1 for a listing of RMAFs, RBAFs, and related documents.)

That being said, it is important to note that IGX has a strong risk-based culture and each program stream retains risk registries at the stream and project level which are consulted and up-dated regularly. A risk registry at the program level is being prepared. In addition, the GPP has very measurable results (i.e., number of submarines dismantled, number of weapons scientists employed, milestones reached) and the bureau tracks, monitors, and reports on these results on a regular basis. In the case of nuclear submarines, monthly reports on milestones achieved are prepared after every site visit. The results for the program, and each stream are contained in the 6-month report to Treasury Board.

The existing program level RMAF and RBAF will have to be updated at the very latest for the renewal of TB authorities in 2007/08. IGX would be open to having an integrated, IFM-wide RMAF / RBAF.

8) Risk Management

IGX is very risk-focussed, particularly at the stream and project levels. Risk registries are maintained through the use of RIAS software which was acquired for the GPP. The registries are kept up-to-date on a monthly or even bi-weekly basis depending on the risk environment. As discussed in the nuclear submarine case study, the value of the risk registry was demonstrated in August 2005 when there was a fatal industrial accident onboard a nuclear submarine. This risk had been identified, the Canadian team was prepared to respond, and the identified response actions were followed.

Risk management at the Government of Canada level is undertaken largely through the Canada-Russian Bilateral Agreement which includes nuclear and non-nuclear liability protection, intellectual property protection, and the ability to unilaterally withdraw from any project without incurring liability. The soundness of this agreement was tested in the above mentioned submarine accident at which time the Russians stated that there would be no liability pursued against the Canadian government.

To mitigate for the potential health and safety risks faced by IGX personnel on visits to hazardous areas and sites with potential health concerns, IGX has taken a number of different general and specific a health and safety measures (e.g. first aid training, issuance of dosimeters to staff visiting nuclear-related facilities, etc.). In view of the particular risks associated with visits to biological facilities and chemical weapons site, more comprehensive Health and Safety procedures, precautions, training, and equipment have been put in place (in collaboration with the Government's Occupational Health Advisory Group (OHAG) and the Department of National Defence ).

As stated earlier, while stream and project level risks are clearly identified and monitored, there has been a gap at the program level. It is the consultant's understanding that a risk register for the GPP is being finalized by the end of fiscal year 2005/06.

9) Strategic Planning

It was reported that the strategic direction for the program is largely provided by the Kananaskis documents (Statement by G8 Leaders, Principles and Guidelines). Elements of the program's strategic direction can also be found in the Treasury Board submissions.

The GPP is not a static program, but needs to be flexible and able to respond to conditions and opportunities that emerge in a highly complex and risky environment with multiple players.

Broad strategic direction can still be planned though, and IGX senior management are committed to developing a Strategic Action Plan by the end of fiscal year 2005/06. This plan will build on the first Priority Review, held in February 2006, and will provide guidance on where the program is going from a strategic standpoint, identify financial and human resources required to get there, and discuss the range of issues that impact on the program (i.e., Canada-US relationship, benefits to Canada, disarmament and peace objectives, etc.).

10) Workplanning

Formal workplanning does not take place at the program level and is inconsistently applied at the stream level. In some respects, this reflects the nature of the streams. It is fairly straightforward to develop a workplan in the nuclear submarine stream because of the structured basis of the project and the clearly defined milestones and tasks laid out in the Implementing Arrangement. In some cases, it appears workplanning is being done on a daily / weekly basis without a long-term view.

While some IGX key informants expressed the view that annual workplans would not add value, it seems that many in the bureau, particularly those not at the senior management level, would benefit from having a broad overview of deadlines, milestones, and reporting requirements. In the event of significant turnover in the program, the availability of workplans would assist in knowledge transfer.

11) Reporting and Communications

IGX faces significant reporting requirements, including 6 Month Reports to Treasury Board, Annual Report to Parliament, briefings to GPAG, inputs to the department's Report on Plans and Priorities (RPP), the Departmental Performance Report (DPR), and reports to the G8. IGX is also required to contribute to departmental reports on Sustainable Development and on Official Development Assistance (ODA). Departmental reporting is coordinated by the Senior Policy Advisor. There are no reports that are systematically linked to RMAF results though.

Each stream provides reports to senior management on a regular basis (i.e., monthly), as well as regular verbal reports. Where partners are involved, SPMs provide reports on projects, etc., to them as well. There are also trip reports that are prepared according to a standard DFAIT format.

A Communications and Outreach Strategy has been prepared. The objectives of the strategy are to communicate and provide information to Canadian and international targeted audiences about the GPP, its accomplishments, benefits to Canada, and Canada's role as a leader in promoting peace and security. The strategy outlines the target audiences, strategic considerations, and activities and products to be used.

12) Role of Program Services Division (IXS)

With the IFM branch programming close to $250 million per year, it became clear that DFAIT was not equipped to meet the demands of the kinds of project delivery services required. In September 2005, IXS was established as a support unit for the entire IFM branch, servicing the GPP, START and Counter-terrorism programs. It has been designed to promote branch-wide approaches to program delivery that are consistent with Treasury Board and other Government of Canada requirements. Services include financial, results management, data and information management, contracting advice, project management advice, human resources, and legal services.

While IGX has had to develop many of these processes on its own, the establishment of a centralized support unit will assist in relieving some of the "leg work" involved in developing an RFP, for example, and will provide a single point of contact on the range of issues. The legal counsel will remain physically situated in IGX in order to provide ongoing advice.

13) Implementing Partners and Service Providers Functions

IGX delivery capacity is enhanced through the arrangements with Implementing Partners and the acquisition of technical expertise from service providers.

In the case of Canada's participation in the railway project at Shchuch'ye, this participation is covered by an MOU with the UK Ministry of Defence, IGX is a fully engaged partner in this project, and benefits from the significant technical expertise provided by the Ministry of Defence in such specialized areas as railway engineering and the services provided by the Royal Engineers.

In the case of the RFWS stream, in addition to project development and financing services, the ISTC undertakes site visits to the Institute through-out the project and then undertake a formal audit and evaluation with technical results, which leads to the production of a Project Closing Report.

Service providers have been engaged to provide technical expertise in areas where it is lacking in IGX, or where other government departments are unable to provide the expertise. For example, Teledyne Brown Engineering provides the nuclear submarine stream with nuclear, marine, environmental, and railway engineering expertise as needed on the monthly monitoring visits to the shipyard.

Summary of Findings

IGX has managed to put in place sufficient delivery capacity, broadly defined, to achieve intended outputs and outcomes. In many cases, the bureau has been hampered by a lack of departmental resources and policies, or by processes that were not responsive to the demands of the program. This is evident in the areas of information management systems, project management systems, human resources, and results-based management. There is no evidence whatsoever that the program has suffered in the past in any way from a lack of internal delivery capacity or from a lack of support provided by other areas of DFAIT.

The above analysis also identified a small number of gaps in strategic and work planning, and in reporting linked to the RMAF. Despite these gaps, there is no evidence the program has suffered because of the absence of some of these elements up to now.

Now that the program is moving from its entrepreneurial beginnings to a "steady state," with high levels of ongoing programming and potential significant turnover, particularly at the senior management level, it has become important for all of the above delivery capacity elements to be fully in place.

These findings are linked to Recommendations 2, 3, 4, 6, and 7.

5.4 Efficiency and Effectiveness

The final area of findings is efficiency and effectiveness. In this area of the formative evaluation the consultants considered five main issues:

  • Role of the RMAF, RBAF, and PMF
  • Performance orientation of program mechanisms
  • Evolution of GPP
  • Possibility of More Cost-Effective Approaches to Program Design
  • GPP as a Learning Program
5.4.1 RMAF, RBAF, and PMF

Evaluation Issue: To what extent have the initial RMAF, RBAF, and Project Management Framework been implemented? How are they used as management tools?

The management of IGX developed the initial RMAF and RBAF for the program as part of the program's first Treasury Board submission. Consistent with the scope of the TB submission itself, the RMAF and RBAF covered the entire program. The Treasury Board approved this submission in early September 2003, and the RMAF and RBAF, as appendices to the submission, were approved along with it.

It was stated in the Phase II Treasury Board submission that it was expected that the program-wide RMAF and RBAF would be reviewed and updated throughout the GPP's life. These documents have not been revised since the initial TB submission up to this writing (March 2006), an elapsed period of about two-and-a-half years.

Since that time, IGX has prepared RMAFs for the nuclear submarine stream, for a sub-stream of the nuclear and radiological stream, and for the two key projects of the chemical weapons stream. Some of these RMAFs were not formally finalized and approved within DFAIT. As for the stream relating to the redirection of former weapons scientists, the RMAF for the first TB submission addressed the logic model and the indicators for this stream in a satisfactory manner. Thus the consultants conclude that the overarching program-based RMAF was to some extent adapted and updated through these more targeted RMAFs.

Exhibit 5.4.1 Table of RMAFs, RBAFs, and Related Documents
DateNameStatusComments
Sept 2003GPP Program RMAFApproved in TB Submission GPP Phase INot a primary document used by the program, needs to be updated, especially logic model and indicators
Sept 2003GPP Program RBAFApproved in TB Submission GPP Phase INot a primary document used by the program, needs to be revisited with reference to new program level risk registry
June 2004RMAF for the Dismantlement of 12 Nuclear SubmarinesApproved in Nuclear Submarine Dismantlement TB SubmissionIn use and updated by submarine stream personnel
June 2004RBAF for the Dismantlement of 12 Nuclear SubmarinesApproved in Nuclear Submarine Dismantlement TB SubmissionIn use and updated by submarine stream personnel
July 2005RMAF For Project: Main Destruction Building 2 (MDB-2)Approved in MDB 2 TB SubmissionIn use by CWD stream personnel
July 2005RBAF For Project: Main Destruction Building 2 (MDB-2)Approved in MDB 2 TB SubmissionIn use by CWD stream personnel
March 2004Performance Monitoring Strategy - Former Weapons Scientists Initiative ProjectNot approvedNot used
April 2004RMAF - Nuclear Submarine Dismantling ProjectNot approvedSuperseded by Submarine TB Submission RMAF
UndatedPerformance Monitoring, Evaluation And Audit Strategies - Railway Construction Project at the Shchuch'ye Chemical Weapons Destruction FacilityNot approvedNot used
Undated (Jan 05?)Performance Monitoring and Evaluation Strategy -Nuclear And Radiological Security Project IAEA, NRSNot approvedNot used
March 2006NRS Physical Protection Logic Model and IndicatorsFinalised DraftNot in use yet. Indicators will need to be reviewed.

The consultants note that the current authorities and funding for IGX expire on March 31, 2008. Therefore, over the next two years (2006/07 and 2007/08), the management of IGX will be seeking renewal of authorities and funding for the remaining five years of the program (2008/09 to 2012/13). Renewal of GPP authorities will require updating of the program-wide RMAF and RBAF in the 2006/07 fiscal year.

The consultants found that the program today is by-and-large consistent with the program design as described in the original RMAF and RBAF. Key elements of the initial program RMAF and RBAF that have been implemented include the approach to governance and project-based monitoring, and the team structure, involving the Senior Project Manager and project teams. Despite the comments made below, the indicators on the whole remain relevant. Aside from evolving cash flow profiles, the consultants were unable to find an aspect of the program that deviated in any material way from the statements in these guidance documents.

RMAF

Given the speed with which the program has been moving, the content of the RMAF and more specifically the design of the logic model is of declining relevance and appropriateness. Some of the issues with the program logic model include:

  • Some concepts and wording are outdated (WMD versus WMMD, weapons stockpiles versus materials, site focussed versus transportation-related elements of the program);
  • Stream-level results chains connect inconsistently to the program-level logic model;
  • Capacity indicators are too general; and
  • Some indicators are not responsive to entire program scope.

Some of the above difficulties with the logic model arise by treating nuclear submarines, nuclear non-proliferation and chemical weapons in one results chain, whereas the streams are different in the costs, risks, and complexities involved, and face different risks and timelines for the realisation of benefits. In some cases, Canada's contribution has a direct effect on fissile materials (as in the case of nuclear submarines), whereas in another case (chemical weapons), Canada's contributions are part of a vast coordinated effort by several parties, and the realisation of the ultimate objective - the elimination WMD stockpiles - may only be achieved sometime after Canada's projects have been completed. The desired outcomes in the case of chemical weapons are less controllable by Canada. It will be important for the revised RMAF and RBAF to address appropriately these divergent aspects of IGX programming and results chains.

The consultants observed a tendency of IGX management to view the RMAF as an external reporting requirement rather than as a key self-management tool. Perhaps due to this perception, IGX management and staff are not frequently consulting the initial program RMAF. In addition, the consultants note that generally speaking there is insufficient program reporting against the program-wide logic model and indicators.

RBAF

The consultants note that IGX has implemented key elements of the initial program RBAF.

The program level risks identified in the RBAF have not yet been translated into a program-level risk management framework. IGX senior management has confirmed that they will complete this management tool, in the form of a program level risk registry, by the end of the FY 2005/06.

Project Management Framework

The GPP Phase II Treasury Board submission went forward to the Treasury Board Secretariat in September 2004, shortly after Russia and Canada signed their bilateral agreement. The Project Management Framework (PMF) was included as Appendix B to this submission and approved along with the submission itself. As such, it is an integral part of IGX's authorities as mandated by TBS. At the time of approval, it as recognized that the PMF is a "living document" and may be amended through the life of the GPP. In the elapsed time of 18 months since this document was approved, it has not been amended.

The Project Management Framework is a remarkable document. It is compliant with the Project Management Body of Knowledge (PMBOK), and is based on CIDA project management experience. Emphasising project control, risk management, and adaptability, the PMF provides comprehensive information on project management best practices and standards throughout the project life cycle, from project initiation through to project closure. Moreover, the document takes into account in its content the lived reality of the IGX, above all showing awareness of the risks, costs and complexities of the projects being undertaken. The consultants conclude that this is an authoritative and robust document.

It is noted that Senior Program Managers are free to select what elements of the PMF apply to individual projects, and officially a Director signs off on any departures from authoritative practice.

In the case studies the PMF was used to varying degrees. The PMF was used extensively at all stages of PNPI design and implementation and most of the components of the PMF were put into place in the project. In the case study in the RFWS stream the project was initiated and implemented according to the some of the key PMF section. The case studies in the Chemical Weapons and Submarine Dismantlement Streams were implemented before the PMF was written but they were also found to follow the key sections.

Interviews reveal that IGX staff give strong recognition to the intense project management focus of IGX and therefore use the PMF as a general reference document. The consultants observe that key requirements of the PMF are being met, especially the concept paper, the Program Approval Document, work plans, and work breakdown structures. Some interviewees report the PMF to be overly complicated; IXS is in the process of revising the document to make it more user-friendly for broader IFM project management requirements, including IGX.

This being said, the consultants note that the PMF cannot be a substitute for project management training, and that this training has in fact been taking place. Interviews confirm that the PMF and project management training are complementary; in effect such training enables IGX staff to select the most appropriate best practices and standards from the PMF, taking into account the risks, challenges and complexities of their projects.

The consultants also note that some IGX staff interact on a routine basis with contracted on-site project management and monitoring service providers such as Bechtel and Teledyne Brown, who offer a wide suite of project management capacity.

Summary of Findings

Both the strong project management approach and the recent emergence of some sub-stream and project-level RMAFs and RBAFs lessened the requirements for reviewing and updating the program-level RMAF and RBAF. However, the emerging pattern of these documents has some degree of inconsistency. Not all streams have RMAFs and RBAFs, not all sub-streams have RMAFs and RBAFs, and not all projects have RMAFs and RBAFs. As an example, given the pattern of TB approval requirements, the nuclear and radiological stream is limited to an RMAF for one sub-stream.

The program-level RMAF and RBAF, created in September 2003, have not played the role of leading program documents, regularly consulted in shaping the other integration deliverables of the program (for example, program-level strategic planning, performance measurement and risk management). There is evidence that the four TB submissions - some of which had an RMAF and RBAF in appendix- replaced the original program-level RMAF and RBAF as the leading documents guiding the direction of IGX.

There is no evidence that the program has suffered in its developmental phases due this approach to RMAFs and RBAFs. However, this needs to be seen in the context of a three year ramping up to the current phase of full-scale implementation and intense activity. It is critical that a consistent pattern of RMAFs and RBAFs emerge from now on.

Excellence in project management is a critical success factor for this program. The PMF is being generally used as a management tool and has contributed, in parallel with project management training, to the successes of the program.

These findings are linked to Recommendations 2, 3, 4, and 5.

5.4.2 Program Mechanisms and Performance Orientation

Evaluation Issue: Are program mechanisms performance oriented and to what extent are they institutionalized?

The IGX programming streams are diverse in terms of program design and management requirements, and it follows that their performance orientation and degree of institutionalization also vary somewhat (Exhibit 5.4.2).

Exhibit 5.4.2: Program Mechanisms and Performance Orientation
StreamPlanningImplementationMonitoringReporting
SubsRMAF with logic model and indicatorsMilestone-based deliveryIGX and contracted resources collect performance data and informationReports against key outcome indicators (i.e. April to Sept. 2005 Report)
NRSLogic model and indicators are completeAll projects have milestone-based delivery (due to nature of projects)IGX and partners collect performance data and information (additional contracted resources to be engaged)Most reporting activity-based, possibly due to early status of implementation
CWRMAF for Railway (status unclear) RMAF for MDB2 with logic model and indicatorsMilestone-based deliveryIGX, UK partner, and contracted resources collect performance data and informationActivity-based and RMAF-based reporting
RFWSResults included in TB I RMAF logic model with stream specific indicatorsCanadian- supported projects have milestonesIGX and ISTC collect performance data and informationMost reporting activity-based, possibly due to early status of implementation

For example, stream-level frameworks and indicators are in place for submarines and RFWS. For CW and NRS there are frameworks with indicators at the project level or sub-stream level. The project delivery "pipeline" varies between RFWS, nuclear submarines, and chemical weapons, with the first reduction in chemical weapons stockpiles in Shchuch'ye potentially taking place in 2008. The exhibit reveals that monitoring and oversight strategies also vary by stream. Reporting has been long-term outcome-oriented for the nuclear submarines stream, while reporting tends to be more activity or project focussed for the other streams. This profile of program mechanisms, as presented in the exhibit, is in part a function of the status of the main projects and achievements of each of the streams.

Summary of Findings

Tests of the extent of performance orientation lie in planning and reporting.

With respect to planning, the stream-level and project-level logic models and indicators that have been developed support the performance orientation of the program. However, the inconsistency in the availability and use of logic models and indicators at the stream and project levels will detract over the longer-term from the performance orientation of the program.

In the area of reporting the issue is the degree of alignment between reported information and program-wide RMAF results and indicators. Outside of the submarine stream, reporting tends to be activity-based, perhaps reflecting the early status of implementation and results delivery. Regardless of the nature of the activity/output chain, the consultants observe it is essential to focus on reporting against an agreed-to set of output and outcome indicators, in addition to activity-based reporting, where appropriate.

These findings are linked to recommendations 2, 3, 4, and 5.

5.4.3 Evolution of GPP

Evaluation Issue: How has the GPP's focus evolved over time and with what benefits, disadvantages, costs, and risks?

In March 2006, IGX is emerging from its entrepreneurial start-up phases and is taking on the dimensions of a full-bodied, multi-project, intense-activity, high-cost, high-risk program that was envisaged to emerge from the very beginning. In reaching this point, IGX management suggests the program has experienced two complete phases to date, one of program initiation and one of project development.

The consultants observe a program that in a continuous and dynamic manner is fulfilling the original intentions of its creators and foundation documents. Over time, the program has steadily strengthened its management and delivery capacities, in part by bringing together a critical mass of complementary in-house human resources, about 27 FTEs to date. This capacity has the ability to handle an increasing number of complex projects and over time offers economies of scale in program delivery. IGX has moved toward effective direct implementation of projects within the Russian Federation by using to advantage the Bilateral Agreement that Canada entered into in 2004. The program has also tried out and confirmed the validity and usefulness of a variety of channels for effective program delivery. The program has developed a robust set of relationships with recipients, partners, and stakeholders, relationships that will serve the program well in the future. The program is therefore very well positioned for the future.

In its current phase, the program is also facing some challenges and risks. About $500 million in programming needs to be put in place for the next five years. With a large staff, effective internal communications become more difficult to achieve, and it may become more difficult to implement internal measures, such as records management and new reporting templates. With size also comes the risk of reduced management and operating flexibilities. There is little doubt the program's increasing disbursement levels will now require more monitoring and performance reporting. And in this "large-program" context, it becomes more important for IGX management to further systematize planning, risk management, disbursement tracking and forecasting, project controls, performance measurement and reporting in general.

5.4.4 Other Cost-Effective Options

Evaluation Issue: Are there more cost-efficient approaches and program designs to achieve intended results?

This is the "greenfield" question: If DFAIT had to start the program today, what would it do differently in terms of program design and delivery? In fulfilling the requirements of this question, the consultants have reviewed a number of alternative approaches and program designs for achieving the objectives of the program.

The options below are presented for illustrative purposes only, as each option creates risks and outcomes that are unacceptable to Canada, recipients, and stakeholders under the Kananaskis Global Partnership framework:

  • In the Nuclear Submarine stream, Canada could only remove the fissile reactor core and not support the dismantlement of the submarines themselves; Canada could reduce the monitoring role of Teledyne Brown;
  • In the NRS stream, Canada could focus on a narrower set of projects;
  • In the Chemical Weapons stream, Canada could reduce the monitoring and implementing roles of the United Kingdom and Bechtel, for which Canada is paying substantially; Canada could support limiting the destruction of these weapons to the neutralization phase (dropping the expensive bituminzation phase);
  • In the steam directed at Former Weapons Scientists, Canada could enter into its own bilateral agreements and projects with Russian scientific institutes without coordinating its efforts through the ISTC;
  • More broadly, Canada could refocus its contributions away from infrastructure creation and toward subsidizing operational dimensions of WMMD elimination; and,
  • Alternatively, Canada could focus all its efforts on one or two streams, rather than share the burden with G8 Partners across a variety of WMMD streams.
Summary of Findings

On the first level, the response that the Government of Canada made to each of the strategic design choices cannot be "second-guessed in hindsight," as the original choices continue to make sense today. Furthermore, each of the possibilities described above entails higher or disproportionate or unacceptable risks for Canada or erosion of partner relationships. Even under a "greenfield" scenario, it is likely that the Government of Canada would proceed today in terms of program design and delivery in much the same way as it has done. In a start-up setting, a multi-"theme and stream" approach makes sense, the possibilities of achieving outcomes in any one area are to a degree uncertain, and progress and results in one stream build the credibility and momentum to achieve results in other sites, projects and streams.

On the second level, the consultants came to the conclusions that:

  • The availability of multiple project delivery channels is a program strength;
  • That the current channels appear to be sufficient and there is no need to search for additional channels;
  • That each stream has its preferred channels from the viewpoint of cost-effective program delivery; and
  • That emphasis on any one delivery channel across streams would diminish program design and generate adverse results.

This being said, the focus of the program is emerging as a design issue. Other scoping possibilities (geographic, stream, and technologies) are being raised. It is tempting to "be all things to all G8 partners."Greater risk sharing with G8 partners across individual streams, delivery mechanisms and projects, is also leading to some dilution of the focus of the program. DFAIT and IGX management are being challenged to find the appropriate balance between focus and risk-sharing.

These findings are linked to Recommendation 1.

5.4.5 GPP as a Learning Program

Evaluation Issue: To what extent is the GPP a learning program? What lessons have been learned and best practices acquired?

The first three-and-one-half years of this program have entailed learning on a massive scale. To be successful, the program has required mastery of several distinct layers of complexity. Of these layers of complexity, learning has occurred at three key levels in particular.

At the macro-level, learning occurred at the levels of international security, non-proliferation, and disarmament policy, and at the levels of program design (learning from other donors already active in the field in Russia), developing effective working relationships with relevant officials of the Russian federation, enabling effective project delivery on a large scale in Russia, and at the technological level of understanding the technological and environmental aspects of elimination of weapons and materials of mass destruction.

At the project level, significant learning occurred at the levels of project management standards and best practices, the selection of optimal delivery channels, coordination of donor engagement and operational involvement, the selection of appropriate construction and operations technologies appropriate to the risks and uncertainties of the context, the management of project level risks, and the management of contingencies in high-risk and high-cost projects.

Specific lessons learned from the case study project included:

  • RFWS: The value of the SPMs knowledge of key priorities outside of the GPP and lateral connections thereof. Had the counter-terrorism relationship not been seen the proposal might not have been moved forward. In this regard GPP continues to have strong formal linkages between this stream (and others) and the work of OGDs; and, The need for flexibility in decision-making. Based on new information the program was aptly able to reconsider and support a proposal that had already been rejected. The need to develop a list of Canadian priority institutes and not rely so extensively on information from the US. The program is now developing this list;
  • NRS: The lack of Russian capacity in reporting according to GPP requirements. To address this issue the GPP has developed a series of simple reporting templates that have started to be used by PNPI with significant improvements in the quality of reports.
  • Submarine Dismantlement: The accident onboard NPS 645 provided an opportunity for important lessons learned for the nuclear submarine program, and for IGX as a whole. Significantly, the accident and the mitigation measures had been identified in the risk registry (risk #038), thereby validating the registry process and reinforcing the importance of updating the registry on a regular basis. Another lesson was the importance of focussing on the entire range of work being done, not just on the high risk / high threat activities. This was an industrial accident at the low end of the danger / complexity range and it could have had far reaching implications for the program as a whole.
  • CWD: The project confirmed the approach needed in being successful in working with the Russian Federation: the need for firmness, clarity, and flexibility wherever possible, but without compromising principles.

Significant learning occurred at the level of program-building: the building of a complementary mass of human resource expertise, and the development of capacities in project management, risk management, performance measurement, human resource and financial management, and communications and program level reporting.

Summary of Findings

In this entrepreneurial phase that is now coming to an end, IGX personnel have shown a remarkable ability to adopt standards and best practices, to learn from others and from their own experiences, and to apply the lessons they have learned in the evolving context. As no single person or small group master all aspects of learning that are critical to the success of the program, the interdependence of learning curves and the sharing of learning has been an essential feature of the success of the program.

As the program grows and spends, strategic plans, RMAFs, risk registers, performance measurement tools and common reporting frames are important aspects for systematizing learning within IGX, IFM, and DFAIT.

At present IGX is fortunate with respect to learning, insofar as the senior management that created the program is continuing to serve in the program over three years later. A significant test of whether IGX represents an effective learning program will take place should any member of this senior management cadre leave the program for new responsibilities. While data and information are relatively easy to systematize and transmit, it is ephemeral knowledge and wisdom that run complex programs successfully.

There is a need to continue to strengthen program-level integrative tools that support learning processes about the optimal ways to undertake the program's activities and to best achieve its outputs, and outcomes.

These findings are linked to Recommendation 1, 2, 3, 6 and 7.

Top of page

6.0 Conclusions and Recommendations

6.1 Conclusions

The overall conclusion of the formative evaluation is that the GPP is an outstanding program that has overcome many challenges. IGX started the program from "scratch" just over three years ago, in a departmental context where program and project management was not a strong point. The achievements to date are significant. Relying on the commitment and energy of its human resources, and without notable departmental program support, IGX has put in place a project implementation and management structure that has common elements across streams, such as risk registries, and that has also adapted to the particular circumstances of each stream. In terms of its evolution, the GPP has passed through logical phases with continuous growth and strengthening and with no apparent discontinuities. IGX has made significant progress in project implementation, faster than most of its partners, without sacrificing project quality. Given its starting point and the achievements to date, the progress of the GPP is ahead of expectations.

While the progress of the GPP is laudable, the program faces some key challenges that must be addressed if the current strengths are to be built upon. Elements the program could afford not to have in place during the initial stages of growth and development, are now critical for success in the full-scale implementation mode. IGX recognizes these gaps and has committed to fill them. Most of these are at the program level and are in the areas of strategic planning, results based management and reporting, human resource management planning, and risk management. However, some of these also have to be addressed at the stream and sub-stream level.

In addition to these recognised areas of attention, IGX faces a number of major strategic and operational issues. There is a need to structure the approach that the GPP will take to the expanding area of biological security. While the program has been able to work outside of the Russian Federation (via contributions to the IAEA and with the STCU), the issue of broadening the geographic scope of the program needs to be more formally addressed. In the RFWS stream there is a need to strengthen the sustainability of redirection efforts with a focus on commercialisation. Finally there is a need to continue to address the high risk environment the program faces in the FSU. In terms of operational challenges, the GPP has to renew its authorities with TB, determine how to program the unallocated balance of its $1 billion budget, align its reporting with a program level results based framework, and maintain its current level of operations with the inevitable staff turnover.

6.2 Recommendations

In order to ensure a successful steady state of implementation, the formative evaluation recommends that the GPP undertakes the following:

Recommendation 1: Deliver as planned the following key program level documents - program-level strategic action plan, risk registry, communications strategy, and HR management and learning plan.

  • Implementing this recommendation will allow the program to build on positive results demonstrated at stream and project level by strengthening integrative mechanisms and processes which will improve cross-stream management.
  • The findings reported in Sections 5.3 and 5.4 (Design and Delivery and Efficiency and Effectiveness) suggest that any gaps identified in the evaluation occurred at the program level and were in the process of being addressed in the preparation of the above documents.

Management Response: Completed. These documents had been under development during the formative evaluation and draft copies were shared with the evaluators. A Strategic Action Plan has since been completed, reviewed and endorsed by Senior Management (IFM). It draws on best practices used elsewhere in DFAIT (RGM, BCD, IDD) and compliments the Department's Strategic Planning and Priorities Framework. Essential elements of the SAP include elaboration of 6 strategic priorities and proposed outcomes (1-3 years) together with performance indicators for the program.

The GPP SAP is supplemented with Results Based Logic Models for each of the five priority program areas, results of the Priority Review (approved by the Minister), Financial Projections, an HR Action Plan, HR Forward Plan and Learning Plan Templates (reviewed and approved by HR Senior Mgt).

A comprehensive Communications and Outreach Strategy (as acknowledged on page 51 of the report) has also been developed and endorsed by Senior Management. This provides clear objectives aligned with the current program environment and future strategic considerations, both international and domestic. The Strategy places a premium on communicating the value added that the GPP brings to Canadians and on advancing Canada's role as a world leader in promoting and contributing to international peace and security.

The Strategic Action Plan template developed by IGX has been endorsed at the ADM level and recommended for use by all Program Delivery Bureaux (copy forwarded under separate cover to ZIV and GGI).

Recommendation 2: Determine what is the leading program-level design document (or develop it if it does not exist) and align planning and reporting against results in this document.

  • While the program and its constituent streams are the subject of a variety of planning and results-focussed documents, there was an absence of a single defining document regularly summarizing planned and actual results at a program level.
  • By implementing this recommendation, the program will have available an up-to-date summary of overall strategy and approach, key expected results, and alignment of the various streams, at any point in time.

Management Response: Agreed. Program-level design documents already exist, but have not been integrated in a single document. G8 Leaders' Statements, the Memorandum to Cabinet, Omnibus Treasury Board Submissions, the Strategic Action Plan, "Priorities" Memoranda and semi-annual reports to the Treasury Board Secretariat all contain information relevant to program design and are the outcome of existing whole of government processes covering program approval and funding authorities. In the short term, IGX will consolidate the required documents into one area for ease of reference by program managers. In the longer term, IGX will integrate existing documentation into one program-level design document and modify it on an ongoing basis to reflect refinements in the direction of the program. It is hoped that this document will serve to refine our focus, simplify our planning exercises and facilitate reporting against results. IGX will also work with TBS to introduce as many common elements as possible into future TB submissions.

Recommendation 3: Implement systematic program level reporting against outputs and outcomes with indicators.

  • This recommendation responds to findings in Section 5.2.2 on Program Results, which point out the absence of summative statements on short and intermediate outcomes (by result across the streams) in current reporting.
  • Its implementation follows on from recommendation 2 and should provide program management with regular information on results achieved at a program level (rather than stream by stream), which should be useful both for internal management and external reporting.

Management Response: Agreed. Systemized reporting at the program level exists, particularly through the semi-annual reports to Treasury Board Secretariat. A revised format was used in the latest report, covering the last 6 months of FY 2005/06 for this purpose. In addition, each of the program streams regularly reports against an exhaustive set of outputs and outcomes, consonant with the RMAFs, using performance indicators. Nonetheless, we will implement more structured reporting this year at the program level against outputs and outcomes (with indicators) based on undertakings outlined in the RMAF. This will go beyond a simple a roll-up of the ongoing reporting that already exists at the stream level and introduce a more strategic approach. This procedure will also make a positive contribution to the development of a revised, program-level, integrated RMAF/RBAF.

Recommendation 4: Require annual work planning at stream-levels / sub-stream levels and analyse the costs and benefits of workplanning at (or rolling up workplans from the stream level to) the program level.

  • As noted in Section 5.3.3 on Delivery Capacity, annual workplanning would provide program staff with a broad overview of deadlines, milestones and reporting requirements. Annual work plans would also assist in knowledge transfer as the program experiences turnover in staff. Such workplanning would provide program management with a medium term outlook on levels of effort and staff requirements.
  • A combined workplan at the program level would enable an overview of all GPP undertakings during a specific period. It would also enable the capture of non-stream interventions or undertakings.

Management Response: Agreed. Workplanning does take place at all levels of the organization and is particularly pronounced at the stream level. As indicated in the report, the nature of the workplanning varies among streams given the number, breadth and scope of the projects undertaken. For this reason, IGX will institute this year a formal, standardized workplanning procedure at the stream level to take place annually. We will also evaluate the utility of rolling up the stream workplans to the program level and, if appropriate, formal, program-level workplanning will be instituted, consistent with and reinforced by the commitments in Recommendations 2 and 3.

Recommendation 5: Require stream and sub-stream level programming frameworks with logic models and indicators where they do not currently exist.

  • Not all streams or sub-streams have a requirement to produce a logic model in the context of an RMAF, specifically NRS. However, all streams and sub-streams should have a programming framework which articulates the overall strategy and approach, the resources, the project pipeline, results in a logic modeal and indicators. The requirement for this was as presented in some detail in Sections 5.4.1 and 5.4.2.

Management Response: Agreed. Programming frameworks are used extensively in all streams as indicated in the report but there is scope for increased usage at the sub-stream level. We will review IGX practices this year and ensure that frameworks rely on the logic models and that the indicators are appropriate and updated as required. This will serve to ensure consistency of availability and use of these tools as called for in the findings of the report.

In line with its mandate to promote best practices across the IFM Branch, the Program Services Division (IXS) will be engaging a consultant this summer to examine the feasibility of a common program management framework for the Branch. The aim would be to ensure PMBOK standards embodied in the GPP PMF and best practises/lessons learned from the IGX example are applied consistently throughout the Branch while respecting the authorities negotiated with Treasury Board by each program delivery group.

Recommendation 6: Address information management issues to eliminate fragmentation of data and information storage.

  • As noted in Section 5.3.3, sub-section 6, program managers tend to use their own systems due to perceived weaknesses in systems at program and corporate levels.
  • As recommendations 1 to 5 are implemented there will be an opportunity to gather and communicated lessons learned at the project, stream and program level.

Management Response: Agreed. Infobank is FAC's approved information management system but it does not meet the requirements of program delivery. IGX has consequently adopted a paper-based file system as its master information retention system. This is used by all staff, albeit with different levels of consistency and IGX is already addressing this shortcoming. As a longer- term solution, the Program Services Division (IXS) is engaged with the Office of the Chief Information Officer (SXD) to examine how Infobank could be customized to integrate the documentation and storage requirements of all program delivery groups.

There is a Branch information management working group that has recently been formed to look into leveraging the department's SAP R3 v4.7 system (used for financial control) as the basis of a project management information management system. This undertaking is considered to be large and complex and may take a year or two to realize substantive results.

The Program Services Division (IXS) is leading the development of a branch-wide information management system which integrates financial management and project management. The study using consultants is proceeding in close cooperation with FAC's Office of the Chief Information Officer (SXD) and CIDA, given its existing capabilities and experience in program delivery. Options, including sharing services with CIDA, will be examined this fiscal year with implementation scheduled to begin in FY 2007/08.

Recommendation 7: Further strengthen efforts for the development of DFAIT-wide systems and processes designed to support high-risk and high-cost programming for IGX and related IFM programs.

  • Implementation of this recommendation could focus on those "toolbox" elements most likely to promote operational excellence in financial, project, risk, and results-based management, while providing the flexibility to adapt to particular contexts.
  • Successful implementation of these integrative management components at the departmental level assures compliance, provides synergies and economies of scale, promotes refinement in approaches to risk and performance management, encourages ongoing corporate learning, and over time will better meet departmental management accountability requirements.

Management Response: Agreed. The International Security Branch has made a commitment to Treasury Board to strengthen its application of results-based management and to develop a Branch-wide results framework that incorporates the work of the three program delivery streams including the GPP. The Program Services Division (IXS) has launched a formal review to assess the extent to which results information is used in decision making and business planning. This analysis will look at the use of RMAFs, RBAFs and associated documents as the basis for managing and reporting on program performance; current practices with regard to measurement and reporting of results; identification of strengths and weaknesses of the present approach. This analysis will lead to recommendations on how to design and implement a Branch RBM strategy that will be endorsed by Senior Management, including the ADM, and will serve as the basis for development of a multi-year action plan for the Branch.

IGX has a strong risk-based culture and each program stream retains risk registries at the stream and project level which are consulted and up-dated regularly. A risk registry at the program level has also been completed. In addition, the GPP can demonstrate measurable results (i.e., number of submarines dismantled, number of weapons scientists employed, milestones reached) and the Bureau tracks, monitors, and reports on these results on a regular basis. In the case of nuclear submarines, monthly reports on milestones achieved are prepared after every site visit. The results for the program, and each stream are contained in the 6-months report to Treasury Board.

The existing program-level RMAF and RBAF will have to be updated at the very latest for the renewal of TB authorities in 2007/08. IGX would be open to having an integrated, IFM-wide RMAF / RBAF.

With specific reference to risk management, the Program Services Division (IXS) is leading the development of a Branch-level risk management framework. Following a detailed needs assessment, a Branch-level Risk Management Working Group comprised of members of all key program streams will lead the implementation and management of this activity on an ongoing basis to ensure consistency between Branch and Program-level risk management activities.

IGX is already very risk-focussed, particularly at the stream and project levels. Risk registries are maintained at both the project and program levels through the use of RIAS software which was acquired for the GPP. Risk management at the Government of Canada level is undertaken largely through the Canada-Russian Bilateral Agreement which includes nuclear and non-nuclear liability protection, intellectual property protection, and the ability to unilaterally withdraw from any project without incurring liability. The soundness of this agreement was tested following a fatal accident at the shipyard undertaking submarine dismantlement under funding from Canada. Following the accident, the Russians immediately stated that there would be no liability pursued against the Canadian government.

To mitigate potential health and safety risks faced by IGX personnel on visits to hazardous areas and sites with potential health concerns, IGX has taken a number of different general and specific a health and safety measures (e.g. first aid training, issuance of dosimeters to staff visiting nuclear-related facilities, etc.). In view of the particular risks associated with visits to biological facilities and chemical weapons sites, more comprehensive Health and Safety procedures, precautions, training, and equipment have been put in place (in collaboration with the Government's Occupational Health Advisory Group (OHAG) and the Department of National Defence).


1 Source: Treasury Board submission for MDB-2.

2 At the time of creation, the Bureau was called GPX. For the sake of consistency, the report will use the most recent initials for the Bureau.

3 Chernobyl has nuclear material which might be sought for nuclear weapons (i.e WMD).

4 Global Partnership Program Annual Report, 2005. p. 15.

5 NRS has used a number of technical experts to support the program (predating even RCL).

6 G7 country or non-G7 country member of the Global Partnership.

7 Excludes re-profiled amounts.

Office of the Inspector General

* If you require a plug-in or a third-party software to view this file, please visit the alternative formats section of our help page.

Footer

Date Modified:
2012-09-22