Common menu bar

Breadcrumb

Archived Document

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated since it was archived. Web pages that are archived on the Web are not subject to Government of Canada Web Standards; as per the Communications Policy of the Government of Canada, you can request alternate formats by contacting us.

Formative Evaluation of the Main Destruction Building 2 (MDB-2) Project

(February 22, 2007)

(PDF Version, 367 KB) ^*

Executive Summary
List of Acronyms
1.0 Introduction and Context
- 1.1 Introduction
- 1.2 Shchuch'ye Chemical Weapons Destruction Facility (CWDF)
2.0 Objectives and Context of the Evaluation
3.0 Methodology
4.0 Key Findings
5.0 Results / Success
6.0 Conclusions
7.0 Recommendations
Annex A: Management Response

Executive Summary

Introduction

In October 2005, Canada announced the decision to finance the provision of chemical weapons destruction (CWD) equipment for installation within the second Main Destruction Building (MDB-2) at the Shchuch'ye chemical weapons destruction facility (CWDF). Canada's total planned contribution is $55 million. As with other Canadian-funded projects ongoing at Shchuch'ye, the MDB-2 equipment purchase has been undertaken through a Memorandum of Understanding (MOU) with the United Kingdom (UK) Ministry of Defence (MoD).

The Treasury Board Decision letter of August 2005 approved the establishment of a frozen allotment of $2 million of the project's funding, pending an evaluation report in 2006-2007 confirming acceptable progress towards established goals and milestones. One important purpose of this evaluation is to respond to this TB requirement. In addition to assessing the results and success of the project, the evaluation also considered the project relevance, its governance and delivery infrastructure, and the efficiency and effectiveness of the project management system.

Methodology

The evaluation was based on multiple lines of evidence and conducted in three phases: Project Initiation; Data Collection; and Analysis and Report Writing.

During the Data Collection Phase, the evaluation team visited the UK partners, conducted a site visit to Shchuch'ye, and travelled to Moscow to interview Russian stakeholders and Bechtel representatives. A total of 28 key informants were interviewed from a wide range of stakeholders, including representatives of the United States, Germany, Green Cross and another federal government department. In addition, the evaluation team completed a comprehensive document and file review. The results of the key informant interviews, observations from the site visit, and data gathered from documents were synthesized by the evaluation team and form the evidence base for the findings and conclusions that follow.

Summary of the Issues, Findings, and Conclusions

The evaluation confirmed that this project is being undertaken in a complex environment, with very tight deadlines and that considerable efforts are made by IGX and the UK MoD to manage the risks and complications. Despite these challenges, significant results have been demonstrated.

Relevance

The evaluation assessed the relevance of the MDB-2 project based on the actual need of the Russian government and the project's consistency with the priorities of the federal government and DFAIT. The evaluation concluded that:

The project to fund equipment for the second Main Destruction Building remains relevant because it continues to address the needs of the Russian partner. The Russian Federation has confirmed its intentions to meet the Chemical Weapons Convention (CWC) deadline and destroy its chemical weapons by 29 April 2012. Meeting this deadline would require that the MDB-2 and the entire Shchuch'ye facility are operational in 2008. The CWD equipment funded by Canada is crucial to meeting this deadline.
The MDB-2 project continues to be consistent with the Government of Canada priorities as demonstrated at the 2006 G8 Summit in St. Petersburg when Canada announced its commitment to contribute an additional $100 million towards the chemical weapons destruction process in Russia. The MDB-2 project also contributes to three of the five strategic priorities outlined in DFAIT's Report on Plans and Priorities 2006-2007.

Governance and Delivery Mechanisms

A number of issues related to project governance and delivery mechanisms were addressed, including, the extent to which the project design and governance structure ensure achievement of results. The roles and responsibilities of the various partners for the successful completion of the Shchuch'ye CWDF were considered. In addition, the evaluation team assessed the extent to which cooperation at Shchuch'ye has led to cooperation in other areas of interest to Canada; the capacity building value of this project for Canada and the Global Partnership Team at DFAIT (IGX); and the monitoring process once CWD begins. Based on the evidence provided by key informants, document reviews and site visits, the evaluation concluded that:

The decision to implement the MDB-2 project through a second MOU with the UK has helped to leverage an existing successful relationship with the UK MoD and has provided considerable benefits to Canada. This approach has facilitated a rapid response to Russia's request, built on the UK capacity, experience and expertise in managing projects in Russia related to high financial and political risks, allowed Canada to influence the process, and provided transparency and accountability for Canadian taxpayers' dollars.
Drawbacks to the project structure include: privity of contract rests with the UK MoD, resulting in less direct control by Canada over the project, in a legal sense; perceived lack of visibility with the Russian partner; and less development of large scale project management and project delivery capacity in IGX and DFAIT as a result of efforts to minimize administrative overhead.
The strengths of the project structure far outweigh the drawbacks and many of these are being mitigated by IGX and the UK MoD and do not jeopardize the ability to achieve results.
The project provides limited or no opportunity for domestic capacity building in the area of chemical weapons destruction and efforts by IGX to engage DRDC were unsuccessful. However, this is understandable given that the GPP is focused on threat reduction and not domestic capacity building as was made known at the inception of the program.

Efficiency and Effectiveness

In order to assess the efficiency and effectiveness of the MDB-2 project, the following issues were considered: the mechanisms in place to promote efficiency and effectiveness; the procurement process and Canada's involvement; the control mechanisms that are in place to ensure sub-contractors deliver services on-time and on-budget; Canada's responsibility for monitoring the delivery, storage and installation of the equipment, as well as its reliability once operational. After an analysis of the evidence, it was concluded that:

Delivering the MDB-2 project through a second MOU with UK MoD was, and continues to be, an efficient and effective readily available option even though it incurs some project management fees.
Additional efficiencies are gained through the arrangement with the UK MoD. It provides Canada with access to management capacity and expertise not available in IGX. This approach (in effect "outsourcing") allows IGX to minimize the short-term human resource requirements that would be difficult to fill in a timely manner under DFAIT's recruiting system and that would not be cost-effective in the long-run.
Project management fees paid to the UK MoD and Bechtel reduce the amount of funding available for additional equipment. The only way to avoid this situation would be to transfer the money directly to Rosprom - an option that was not considered viable due to the lower transparency and accountability in Russia with respect to the use of foreign funds.
Piggybacking on US purchases of similar equipment for Building 101 increases the efficiency and effectiveness of Canada's project by leveraging the preparation and groundwork already done by Parsons and reducing the financial and technical risks to the project.

Results and Success

To determine the progress of the project towards achieving established goals and milestones, the evaluation considered the following issues: the status of procurement and delivery of the equipment; complications and respective mitigation measures.

The evaluation confirmed that acceptable progress has been achieved towards established goals, particularly with the delivery of the first batch of Catalytic Reactors in August, 2006 and the planned delivery of the second batch in December, 2006. There have been complications, but these are not necessarily unexpected given the complexity of the CWDF project and the challenges of working in the Russian context. Despite these complications, there is assurance that the original objectives and scope of the project will be met. Project risks are identified in risk registers prepared by IGX, the UK MoD, and Bechtel to reflect each stakeholder's respective risks. In some cases, the IGX risk register has not reflected ongoing developments.

There is sufficient evidence that acceptable project progress has been achieved and it is suggested that the MDB-2 project is worthy of continued funding.

Recommendations

In light of the above findings and conclusions, the following recommendations have been made:

Recommendation 1: Continue to work with the UK MoD to complete Shchuch'ye

It is recommended that IGX complete the MDB-2 project under the arrangement with the UK MoD with the continued oversight by IGX of both UK MoD and Bechtel. Every effort should be made to complete the project in time for the CWDF to be operational in 2008. In addition, IGX should continue to minimize project management fees while maintaining full involvement in all aspects of the project.

Recommendation 2: Update the risk registers

It is recommended that the project risk registers be updated on a regular basis to reflect any changes reported in the Quarterly Reports from UK MoD. In particular, the situation with Building 101 should be monitored closely for impacts on the second Main Destruction Building and the equipment purchases.

Recommendation 3: Revisit the relationship with DRDC

It is recommended that IGX continues to engage DRDC with respect to Canadian CWD-related activities in Russia. It is not suggested that DRDC duplicate the services of the Royal Engineers, rather the recommendation is meant to open the lines of communication between programs for mutual benefit. To facilitate a mutually beneficial relationship, it is suggested that IGX and DRDC develop a detailed agreement on services that could be provided by DRDC to the IGX program, as well as associated costs, terms and conditions. In addition, to better inform DRDC of IGX activities, it is suggested that DRDC join the inter-departmental Global Partnership Advisory Group.

Recommendation 4: Identify and utilize lessons learned from Shchuch'ye when considering options for Kizner

It is recommended that IGX identify lessons learned from the experience at Shchuch'ye. Lessons learned should focus on such areas as efficient project management structure; project governance; due diligence; and cost-effectiveness. These lessons learned should be applied when identifying a range of costed options for project delivery at Kizner. It is also suggested that the chosen option demonstrate that it:

Can be implemented in a timely manner;
Is cost-effective;
Minimizes financial and political risk for Canada;
Satisfies the Russian partner's needs;
Provides opportunities for Canadian capacity development as appropriate; and
Meets IGX commitments to provide adequate human resources in terms of numbers and skills.

Timeliness is a particularly important criteria as there is a small window of opportunity to complete the Kizner project and in effect, the IGX team will have to be ready to start implementation of this project quickly.

Top of page

List of Acronyms

CWC: - Chemical Weapons Convention
CWD: - Chemical Weapons Destruction
CWDF: - Chemical Weapons Destruction Facility
DFAIT: - Department of Foreign Affairs and International Trade
DPL: - Destruction Process Line
DRDC: - Defence Research and Development Canada (DND)
Dstl: - Defence Science and Technology Laboratory
DTRA: - Defense Threat Reduction Agency
FAI: - Federal Agency for Industry
GP: - Global Partnership
GPAG: - Global Partnership Advisory Group
GPP: - Global Partnership Program
GPWG: - Global Partnership Working Group
IGX: - Global Partnership Bureau at DFAIT
ISC: - Inter-Site Communications
LPAS: - Local Public Address System
LTA: - Leakers Treatment Area
MDB-2: - Second Main Destruction Building
MoD: - UK Ministry of Defence
MOU: - Memorandum of Understanding
MTTDPL: - Munitions Thermal Treatment Destruction Process Line
NATO: - North Atlantic Treaty Organisation
OPCW: - Organization for the Prohibition of Chemical Weapons
PM: - Performance Measurement
PMP: - Project Management Plan
RMAF: - Results-based Management and Accountability Framework
RPP: - Report on Plans and Priorities
SCWG: - Shchuch'ye Coordinating Working Group
SPM: - Senior Program Manager
T&S: - Travel and Sustenance
TB: - Treasury Board
UK: - United Kingdom
US: - United States
ZIE: - Evaluation Division at DFAIT

Top of page

1.0 Introduction And Context

1.1 Introduction

In October 2005, Canada announced the decision to finance the provision of chemical weapons destruction (CWD) equipment for installation in the second Main Destruction Building (MDB-2) at the Shchuch'ye chemical weapons destruction facility (CWDF). Canada's planned total contribution is $55 million. As with other Canadian-funded projects ongoing at Shchuch'ye, the MDB-2 equipment purchase has been undertaken through a Memorandum of Understanding (MOU) with the United Kingdom (UK). This contribution is part of Canada's commitments to the G8 Global Partnership (GP) and, in particular to the Chemical Weapons Destruction program. The Global Partnership was announced at the 2002 G8 Kananaskis Summit.⁽¹⁾

1.2 Shchuch'ye Chemical Weapons Destruction Facility (CWDF)

In 1992, the United States (US) agreed to assist the Russian Federation in eliminating its chemical weapons (CW) stockpile under the Cooperative Threat Reduction (CTR) program. The Shchuch'ye CWDF is one component of this assistance. The US committed to fund the design, construction, equipment acquisition and installation, systems integration, training and start-up of the Shchuch'ye facility.⁽²⁾

When completed, the Shchuch'ye facility will consist of 111 buildings, including a boiler house, administration building and various other support buildings.⁽³⁾ The chemical weapons will be destroyed in two attached buildings: Building 101 and Building 101A (also known as MDB-2). Building 101A is the only structure "inside the fence" that is not being funded by the US. Its construction is being funded by Russia. Russia is also funding the construction of utilities (gas, electricity, water) needed to operate the facility.

When it became apparent that Russia was unable to fund the procurement of the equipment for the building, both Russia and the US appealed to international donors to undertake this task. As described in Section 1.1, Canada agreed to commit $55 million towards the acquisition of CWD equipment for Building 101A.

1.2.1 Canada at Shchuch'ye

Canada has been funding activities at Shchuch'ye since 2000 - prior to the Global Partnership Program's (GPP) inception. Between 2000-2002, Canada contributed a total of $5.35 million towards the power supply design, access road design (and partial construction) and the construction of 105 km of natural gas pipeline (with Italy). Under the GPP, and not including MDB-2, Canada has provided funding for the railroad from Planovy to the Shchuch'ye facility ($33 million), and the local public address system (LPAS) and inter-site communication (ISC) projects ($10 million).

1.2.2 MDB-2 Equipment Project

The Global Partnership Bureau at DFAIT (IGX) consulted with Rosprom (the Russian Federal Agency for Industry responsible for chemical weapons destruction) and the UK Ministry of Defence (MoD) and developed a list of priority equipment that included:

2 destruction process lines (DPLs);
62 catalytic reactors;
Munitions thermal treatment destruction process lines (MTTDPLs);
Treatment area for leaking munitions (LTA); and
4 packages of equipment, including gas analyzers, Venturi Scrubber, and a sampling and sample delivery system.

According to the timetable presented in the Treasury Board (TB) submission, most of the equipment was anticipated to be purchased by March 2007.

1.2.3 Major Stakeholders and Beneficiaries

Exhibit A presents the major stakeholders and beneficiaries involved in the MDB-2 project. Further description of their roles and responsibilities, as well as the project governance structure, can be found in Section 4.2.2.

**Exhibit A: Major Stakeholders and Beneficiaries**
Name	Role and Responsibility	Contractual Arrangement
IGX	Provide funding ($55M) to procure the equipment through the UK MoD; and Responsible for managing the MDB-2 project with the UK MoD.	MOU with the UK MoD, 2005.
UK MoD	Administer Canadian funds as well as funds from other donors; and Responsible for managing the Principal Contractor (Bechtel).	UK-Russia Bilateral Treaty, 2001; Implementation Arrangement, 2002⁽⁴⁾ Enabling Contract with Bechtel, 2003; Annex J to the Implementation Arrangement with Rosprom, 2006; MOU with Canada 2005; and Diplomatic Notes with Government of Russia, 2006
Bechtel	Principal Contractors to the UK MoD; and, Responsible for implementing projects on behalf of UK MoD and Canada at Shchuch'ye and for contracting with and managing the Russian sub-contractors.	Enabling Contract with UK MoD, 2003; and, Non-disclosure agreement with Parsons, 2006.
US Defense Threat Reduction Agency (DTRA)	Primary funder ($1.039B) of all projects "inside the fence"; and, Responsible for managing the implementation of the CTR program and for managing the Principal Contractor (Parsons).	Treaty on Strategic Offensive Arms (Moscow Treaty), 2002; Chemical Weapons Destruction Implementing Agreement, 1992 (Amendments 1994, 1996).
Parsons	Principal Contractors to US DTRA; and, Responsible for implementing US projects at Shchuch'ye and for managing Russian sub-contractors.	Contract with DTRA, 1996.
Rosprom	Russian government ministry also known as the Federal Agency for Industry (FAI); Responsible for all CWD in Russia; Beneficiary of Canadian funding; and, Funding construction of MDB-2 (approx $55M)	Bilateral Treaty with UK, 2001; Implementation Arrangement, 2002; Annex J with UK MoD, 2006; and, Diplomatic Notes with UK 2006.
Giprosyntez and GOSNIIOKhT	Russian Design Institutes; and Responsible for reviewing, revising, and, approving the equipment shop drawings.	Sub-contractors to Bechtel.

1.2.4 Challenges of Working on Chemical Weapons Destruction at Shchuch'ye

Through interviews with key project managers, review of project correspondence and a site visit to Shchuch'ye, the evaluation team identified a number of challenges that IGX faces when working in Russia on chemical weapons destruction projects.⁽⁵⁾ Exhibit B describes the main challenges, as well as the implications for the MDB-2 project and the mitigation measures implemented to ensure project progress and success. Most of these challenges are operational in nature, while the communication challenge is considered a governance issue. On an individual basis, each challenge may seem minor. However, several stakeholders stated that these challenges are inter-related and that the compound effect of dealing with them simultaneously and continuously, makes for a difficult environment.

**Exhibit B: Challenges of Working on CWD at Shchuch'ye**
Challenge	Description	Implication	Mitigation
Shchuch'ye is a remote location	The CWDF itself is located close to the town of Shchuch'ye, in the Kurgan Oblast. This is a very remote, isolated community that is sparsely populated.⁽⁶⁾	Site visits by Ottawa-based IGX personnel require several days of travel to reach Shchuch'ye.	As required, personnel from the Canadian Embassy in Moscow or from UK MoD can go to Shchuch'ye in less time; and Bechtel maintains a representative on-site, partially at the expense of the project.
Shchuch'ye is a large industrial complex	The CWDF is comprised of more than 100 buildings, some of which are interdependent; and The CWD process consists of several steps beginning with the transportation of the chemical weapons from Planovy to Shchuch'ye by railroad, followed by inspection, unloading, draining, neutralizing, etc., through to bitumenization and storage.	Constructing this highly complex facility in such a remote location requires substantial logistical support and communication between the various stakeholders; and Decisions and changes must be coordinated between the US and Russia which can be very time-consuming, creating delays.	Project is co-managed by IGX and the UK MoD, which has experience working at Shchuch'ye with Bechtel; and, Canada and the UK participate in the Shchuch'ye Coordinating Working Group (SCWG) with the US and Russia; observe US-Parsons progress meetings; and attend meetings in the Hague to network with other donors.
Communication with Rosprom is formal and non-electronic	Communication largely takes place by written correspondence (not electronic) that must be translated and delivered through the UK and Canadian Embassies in Moscow.	Most communication is facilitated by the UK MoD; It is a formal and hierarchical relationship with most decision-making taking place at the senior level; and It is a time-consuming process.	As required, senior officials meet face-to-face in Moscow; IGX and UK MoD maintain daily contact; Efforts are underway to establish regular contact at working level; and, The SCWG meets 3-4 times a year to keep each stakeholder informed.
Relationship with Rosprom	Rosprom can be cumbersome. For example, upon Canada agreeing to fund equipment, Rosprom applied intense pressure to procure the equipment by the end of 2006; Canada and the UK undertook efforts to meet this very tight deadline; Rosprom intervened in the sub-contracting process and "blamed" the UK partner and Principal Contractor for delays; and Rosprom not always forthcoming with respect to the status of the overall facility requirements.	Despite best efforts, problems continue to arise unexpectedly; Lack of transparency leads to uncertainty, making planning difficult.	As required, senior officials meet face-to-face in Moscow; IGX and UK MoD maintain daily contact with each other; and, Canada and the UK participate in the SCWG; observe US-Parsons progress meetings; and attend meetings in the Hague to network with other donors.
Project is weapons-related	Unlike most of the other streams in the GPP, this project is providing support that will lead directly to the destruction of chemical weapons	Additional sensitivity required when dealing with the Government Agencies because these weapons are lethal; and, Additional security and safety required.	Commitment by all involved to achieving progress; and, IGX personnel have undergone security and safety awareness and training.

Top of page

2.0 Objectives And Context Of The Evaluation

The Treasury Board Decision Letter of August 3, 2005 approved "the establishment of a frozen allotment in 2006-07 of $2 million in DFAIT's Vote 10." This frozen allotment would be released pending an evaluation report confirming acceptable project progress towards established goals and milestones. The main purpose of the evaluation is to respond to this TB requirement. However, the evaluation also considered the relevance of the project, the project governance and delivery infrastructure, and the efficiency and effectiveness of its management.

This formative evaluation of the MDB-2 project follows a formative evaluation of the broader Global Partnership Program conducted in early 2006. The GPP evaluation looked at each of the five streams comprising the program: Nuclear Submarine Dismantlement; Nuclear and Radiological Security; Chemical Weapons Destruction; Biological Non-Proliferation; and Redirection of Former Weapons Scientists. The MDB-2 project is part of the Chemical Weapons Destruction stream, and was considered as part of the wider picture, but was not formally evaluated at that time.

3.0 Methodology

The evaluation was based on multiple lines of evidence and was conducted in three phases: Project Initiation; Data Collection; and Analysis and Report Writing.

3.1 Phase One: Project Initiation

This phase set the stage for the remainder of the evaluation through the development of the work plan; identification of key documents and files to be reviewed, as well as the key informants to be interviewed; planning and preparation for the visit to the UK partners, a site visit to Shchuch'ye, and a trip to Moscow to meet the Russian stakeholders.

3.2 Phase Two: Data Collection

In this phase, a large number of documents and files provided by IGX were reviewed. Additional reference material was also researched on the Internet by the evaluation team.

3.2.1 Key Informant Interviews and Site Visits

Semi-structured key informant interviews were held with a range of stakeholders, by telephone and in-person, as indicated in Exhibit C. IGX identified the key individuals to be interviewed and the evaluation team suggested additional key informants.

**Exhibit C: Key Informant Interviews**
Stakeholder	Number of Interviews	Method / Location
IGX	4	In-person (Ottawa)
Other Gov't Dept (DRDC)	4	Phone / In-person (Ottawa)
UK Partners	5	In-person (Porton Down, London)
Other GPP Partners: United States Germany	2 1	Phone Phone
Russian Partners	6	In-person (Shchuch'ye, Moscow)
Bechtel	3	In-person (Shchuch'ye, Moscow)
Green Cross	3	In-person (Shchuch'ye, Moscow)
Total:	28

Site visits were undertaken to Porton Down and London in September to meet directly with the UK project managers, at both working and senior levels. The meeting in Porton Down also allowed the evaluation team to participate in the kick-off meeting for the joint Canada-UK audit of the Shchuch'ye projects.⁽⁷⁾

In late October 2006, a site visit was conducted to the CWDF at Shchuch'ye, which included an inside tour of the MDB-2 and confirmation of the delivery of the first batch of equipment. Interviews were conducted onsite with the military commanders in charge of the building and its equipment, as well as with the onsite Bechtel representative. In addition, the evaluation team met with the Green Cross⁽⁸⁾ representative in the Shchuch'ye office.

The trip to Moscow provided the opportunity to meet with key senior Rosprom officials, as well as Bechtel representatives in charge of the equipment procurement. The evaluation team also met with two key Green Cross representatives in the head office. While Green Cross is not directly involved in the MDB-2 equipment project, the purpose of meeting its staff was to obtain an understanding of the impact of the Shchuch'ye project on the local community.

3.3 Phase Three: Analysis and Report Writing

In this phase, the evaluation team reviewed the existing RMAF in order to determine lessons learned for the development of RMAFs for future CWD projects in Russia. In addition, the team analyzed the project data and interview-based information in order to assess the degree to which the TB Submission commitments are being achieved and evaluation objectives met. A PowerPoint presentation to DFAIT staff provided a summary of preliminary high level findings. It was followed by development of the draft and final reports.

3.4 Limitations

The large number of evaluation issues meant that not all issues were focused on in-depth. Rather, the evaluation team focused on issues of highest priority, particularly issues related to efficiency and effectiveness, governance, and results. There were no sampling bias issues because there was only one site in question. None of the limitations invalidate the evaluation findings.

Top of page

4.0 Key Findings

4.1 Relevance

This formative evaluation considered the extent to which the MDB-2 project remains relevant, and in particular, the degree to which it responds to an actual and continuing need in Russia, as well as its consistency with federal government and DFAIT priorities.

The continuing need for the MDB-2 project

Key informants reported that there is a continuing need for the MDB-2 project because it supports the Russian partners in meeting the Chemical Weapons Convention (CWC) deadline of April 29, 2012. In order to meet this deadline, Shchuch'ye must be operational, MDB-2 must be complete and the destruction equipment must be available. Evidence for the continuing need was also provided in the documentation received from the stakeholders.

The Russian commitment to meet the CWC deadline is evident from a number of sources. It is reflected in the Russian Presidential Decree #510, issued in July 2001. More recently, in a statement to the Global Security Newswire, a senior Russian Foreign Ministry official reiterated that his nation would meet the 2012 deadline, saying "we should and will fulfill our obligation at any cost."⁽⁹⁾

Russia has also provided the Organization for the Prohibition of Chemical Weapons (OPCW) with a comprehensive destruction program outlining the actions it needs to take to meet the deadline.⁽¹⁰⁾ This program states that Shchuch'ye is to be operational in 2008. The 2008 deadline was reinforced by several key informants during the onsite interviews at Shchuch'ye, including respondents from Rosprom, the UK and Bechtel.

As further evidence of its commitment to destroy the CW stocks on time, the Russian Federation has increased its annual funding dramatically: from approximately $200 million in 2000 to $600 million in 2006.⁽¹¹⁾ However, the Russian Federation continues to state the need for ongoing international assistance to meet its goals. The Report on the G8 Global Partnership, released at the G8 St. Petersburg Summit in 2006, noted the importance of foreign assistance to accelerate the implementation of the chemical weapons destruction program.

Once operational, the entire Shchuch'ye facility (both destruction buildings) will have the capacity to destroy 1,700 metric tonnes of CW per year. At that rate, it is reported that all 5,440 metric tonnes of CWs stored at Planovy could be destroyed by 2011, as long as the facility operates at maximum capacity. Therefore, the equipment⁽¹²⁾ being funded by Canada is instrumental to the completion and operationalization of MDB-2 and is crucial to the destruction process. According to project documentation, the DPLs, for example, include the systems that unload the munitions, as well as demilitarization machines which drill and drain the nerve agent from the munitions. The MDB-2 equipment will be able to destroy 850 metric tonnes of chemical agents per year.

If Canada withdrew from the MDB-2 project, or reduced funding for the MDB-2 equipment, the operationalization of the CWDF would fall behind schedule while new donors and / or funding was found. As a result, the destruction of the chemical weapons would not start on time and it is unlikely that Russia would be able to meet its CWC deadline of April 2012. It would also mean that the proliferation risk of these chemical weapons would be extended.

Consistency with federal government and DFAIT priorities

The MDB-2 project is consistent with federal government and DFAIT priorities as demonstrated by three main factors: the federal government's continued participation in international organizations and treaties related to non-proliferation and counter-terrorism issues; overall government support for CWD programs; and the strategic priorities of DFAIT.

Key informants in IGX pointed out that the MDB-2 project is consistent with Canada's ongoing participation in a number of organizations and treaties, such as the G8, the Chemical Weapons Convention and NATO. This ongoing support can also be confirmed by the policies described on the DFAIT website Canadian Policy. In addition, the Government of Canada's commitment to a continued involvement in CWD in Russia was recently reinforced by the announcement made in July 2006 at the G8 Summit that Canada would commit a further $100 million toward CWD facilities in Russia.⁽¹³⁾

According to the strategic priorities outlined in the Foreign Affairs component of DFAIT's Report on Plans and Priorities 2006-2007,⁽¹⁴⁾ three of the five priorities relate to the Global Partnership Program activities, and by association, also to the MDB-2 project. These include: Strategic Priority 1 "Greater collaboration with the United States and increased cooperation with all hemispheric partners;" Strategic Priority 2 "A more secure world for Canada and Canadians, safer from the threats of failed and fragile states, terrorism, transnational crime and weapons of mass destruction;" and Strategic Priority 4 "Greater engagement with like-minded partners in the G8, as well as emerging economies such as Brazil, Russia, India and China."

In terms of greater collaboration with the United States, as well as greater engagement with like-minded G8 partners and Russia, key informants noted that Canada has become a major donor at Shchuch'ye. This observation is confirmed in the 2006 Global Partnership Working Group (GPWG) Annual Report. According to this document, Canada has contributed $33 million towards the railroad, $10 million for infrastructure projects, and $55 million towards the equipment, for a total of $98 million. In addition, prior to the GPP, Canada had contributed $5.35 million for the gas pipeline and other projects. Thus, since 2000, Canada has contributed a total of $103.35 million. The largest donor at Shchuch'ye is the US at $1.039 billion. The evaluation team was told by key informants that the Chemical Weapons Destruction Program's funding profile has increased Canada's visibility with the United States and Russia at meetings at the Hague and the G8 Working Group, as well as at progress report meetings held between the US and Russia. According to the Russian partner, "Canada is making a key contribution."

Correspondence from the UK MoD also provides further evidence of the project's consistency with DFAIT's priority of achieving greater engagement with like-minded G8 partners. A joint Canada-UK MoD letter was sent to other G8 and non-G8 countries seeking contributions to Shchuch'ye. This effort resulted in financial donations from Belgium (£60K); Czech Republic (£135K); Ireland (£56K); New Zealand (£690K); France (€6M); and others. There is no evidence that other donors have withdrawn from initial commitments.

Finally, the MDB-2 project is expected to contribute to a more secure world for Canada and Canadians by funding equipment that will lead to the neutralization and irreversible destruction of more than 5,440 metric tonnes of chemical weapons.

4.2 Governance and Delivery Mechanisms

The evaluation addressed the following issues related to project governance and delivery mechanisms:

The extent to which lessons learned from a review of the RMAF for the MDB-2 project can be utilized for the development of RMAFs for future CWD projects;
The degree to which the project design and governance structure ensure the achievement of results;
Stakeholders' roles and responsibilities for the completion of the CWDF at Shchuch'ye;
Canada's monitoring obligations once the CWD process begins;
The benefits of the cooperation at Shchuch'ye and its impact on other areas of interest to Canada; and
The capacity building value of this project for Canada and IGX.

4.2.1 Review of the RMAF

In accordance with the Terms of Reference, the evaluation team reviewed the RMAF for the MDB-2 project to assess the lessons learned for developing RMAFs for future CWD projects. The review focused on the Performance Measurement (PM) Plan (RMAF Table 1) and the Evaluation Strategy (RMAF Table 2). As a result of this review, the following suggestions are made for future RMAFs:

Focus the Performance Measurement Plan on the ongoing monitoring of outputs and results only. The PM Plan in the MDB-2 RMAF also included indicators for IGX activities which is inconsistent with the TB RMAF guidelines.
Accurately identify the PM Plan performance indicators. Data sources, methods, schedule and responsibilities should refer to ongoing monitoring activities rather than the Evaluation Strategy. Exhibit D below provides an example for "UK Project Management," as an output, while Exhibit E provides an example for the "Doubling of CW destruction capacity at Shchuch'ye" as an immediate outcome. It is particularly important to identify the valid indicators for the outcomes.

**Exhibit D: One Example of An Output in the Performance Plan**
FAC/IGX Outputs	Performance Indicators	Data Sources	Methods	Schedule	Responsible
UK Project Mgt (Task Orders, risk registers, Quarterly Reports, etc.)	UK MoD issues Task Orders to Bechtel UK MoD issues Form A to Bechtel, etc.	Copy of Task Orders, Copy of Form A, UK MoD Quarterly Reports, Bechtel Monthly reports, Bechtel Project Mgmt Plan (PMP), Project Milestone Schedule	Monitoring documents, reports, PMPs	Monthly, Quarterly	IGX Senior Program Manager

**Exhibit E: One Example of An Immediate Outcome in the Performance Plan**
Immediate Outcome	Performance Indicators	Data Sources	Methods	Schedule	Responsible
Doubling of the CW destruction capacity at Shchuch'ye	Procurement and delivery of the following equipment: 2 DPLs; Catalytic Reactors; MTTDPLs; LTA; 4 Packages of equipment (detail), etc.	Purchase orders; PMP; Project Milestone Schedule; Monthly Reports; Quarterly Reports; Pictures; Handover documents	Review documents, reports; site visits	Monthly, Quarterly	IGX Senior Program Manager

Clearly differentiate between the ongoing monitoring plan and the Evaluation Strategy. References to Evaluation Strategy data sources should not be included in the PM Plan. The PM Plan is a document for the ongoing monitoring of the project whereas the Evaluation Strategy is for periodic performance measurement.
Complete the Evaluation Strategy to include data sources, methodologies, and responsibilities for each evaluation issue. These are currently absent from the MDB-2 Evaluation Strategy. In addition, develop additional evaluation questions, including those related to cost-effectiveness.

4.2.2 Project Design and Governance Structure

This section discusses the governance structure of the overall MDB-2 project and IGX Chemical Weapons Destruction program; outlines the roles and responsibilities of the stakeholders and beneficiaries; identifies the strengths and the drawbacks of the project structure; and briefly summarizes the concerns raised about Bechtel's performance.

Governance structure of the overall MDB-2 project

The MDB-2 project is undertaken through a Memorandum of Understanding (MOU) with the UK MoD negotiated in 2005. Exhibit F illustrates the key components of this structure, as well as the main features of the various instruments.

Exhibit F: Project Structure

Project Structure

The project design and governance structure build on an existing structure established for the delivery of the railroad, LPAS and inter-site communication projects. When Canada first became involved at Shchuch'ye under the GPP, the Canada-Russia Bilateral Treaty was still under negotiation. As time was of the essence, and the Canadian government had been approached to build the railroad from Planovy to the Shchuch'ye facility, DFAIT's Senior Management decided to undertake the project through a Memorandum of Understanding with the UK. The first MOU was signed in 2003. The UK already had a Bilateral Treaty in place with Russia which allowed third parties to contribute, once Diplomatic Notes had been exchanged with Russia.

As stated in the MOU, the UK agreed to implement Canada's project, thus providing contracting services, technical advice and project management services. Since the UK MoD had an Enabling Contract with Bechtel, it was also agreed that Bechtel would implement the railroad project funded by Canada. This same arrangement was subsequently applied to the LPAS and inter-site communication projects. Because of the size of the railroad project, Canada agreed to provide financial support in the form of project management fees. These fees cover the manpower costs of the UK Defence Science and Technology Laboratory (Dstl)⁽¹⁵⁾ personnel who work on Canadian projects. These costs are recorded and charged on an hourly basis. In addition, Travel and Sustenance (T&S) costs are also paid for by Canada, when these costs are incurred with respect to a Canadian-funded project. All project management fees are reported in the Quarterly Reports to IGX. These same arrangements are in place for the MDB-2 project. The UK MoD also manages contributions from other G8 and non-G8 countries for CWD projects at Shchuch'ye. Most of these contributions are smaller than the Canadian contribution and require less project management effort. Therefore, to maximize the effect of the donation, the donors are exempt from paying any project management fees to the UK MoD.

Structure of the IGX Chemical Weapons Destruction Program

Exhibit G illustrates the IGX reporting structure for the Chemical Weapons stream of the GPP. As the exhibit demonstrates, there are minimal human resources within this stream to provide specialized project management and delivery services for a complex chemical weapons destruction project. Key informants in IGX stated that this was an explicit decision taken when the GPP was established in 2002, in an effort to minimize the administrative overhead at DFAIT. It was further noted that, where necessary, specialized skills and expertise would be sourced from external providers. As a consequence of this lean structure, when an individual leaves the Chemical Weapons Destruction program, it may give the appearance that there is insufficient human resource capacity to manage the program. However, IGX respondents pointed out that the structure is flexible enough that a very senior IGX manager can manage the program with support from more junior personnel. This is largely because the arrangement with the UK MoD allows for continuity of project management with minimal disruption.

Exhibit G: IGX Chemical Weapons Destruction Stream Organization Chart

Key informants reported that the project design and management structure provide several benefits for Canada and, despite some concerns about Bechtel, they contribute to the achievement of results. Many key informants pointed out that there are a number of considerations to bear in mind when examining the project structure and its potential to achieve results. These considerations include the need to achieve results on a timely basis, to minimize the financial and political risk and to allow Canada to influence the process. In addition, the project structure must reflect the human resource capacity (both numbers of people and skills) in IGX. Key informants also reported that, after an analysis of the options available to Canada, and given the above considerations, the arrangement with the UK MoD remains viable for continuation of the MDB-2 project.

Roles and responsibilities

The roles and responsibilities of the major stakeholders and beneficiaries were also assessed through interviews and a review of documents and files. Exhibit H summarizes the roles and responsibilities. Key informants stressed that Canada plays an influential role in the MDB-2 project and is not just "the bank." Respondents reported that Canada is a full participant at meetings with various stakeholders, such as Rosprom, the US, and Bechtel, and has chaired the SCWG. This participation was also confirmed in the minutes of various meetings. UK MoD representatives also noted that IGX personnel have attended three meetings of the Global Partnership Oversight Board. This is an inter-departmental committee of the British government comprised of Dstl, MoD, Department of Trade and Industry (nuclear program), the Foreign Office and Office of Government Commerce (with respect to contracting and procurement). During the meetings, Canada was given full speaking privileges on this Board. Key informants further observed that Canada's participation as a foreign government representative on a UK inter-departmental committee was a solid indication that Canada is considered a full partner in this project.

As the main funding partner for the equipment for MDB-2, IGX is responsible for ensuring that the necessary funds are available at the appropriate time. As manager of the project, along with the UK MoD, IGX is responsible for ensuring that the requested equipment is procured in time to meet the Russian partner's deadlines for completion and operationalization of the CWDF. According to Annex J of the UK-Russia Implementation Arrangement, Canadian responsibility ends once the equipment has been delivered and officially handed-over to the Russian partner.

Exhibit H: Roles and Responsibilities

Roles and Responsibilities

Options for the MDB-2 project

When Canada decided to fund the provision of the MDB-2 equipment, the Canada-Russia Bilateral Treaty was in place and key informants noted that the Russian partner expressed a strong interest in Canada utilizing this agreement. However, IGX officials explained that in order to implement the MDB-2 project under the Bilateral Treaty, several steps would have to be followed, including: negotiating an Implementing Arrangement; tendering for, and negotiating with, a Principal Contractor; and managing the Principal Contractor. IGX officials further explained that this option was excluded because of the time and expense it would take to negotiate the Implementing Arrangement. As discussed in Section 4.1, there are very tight deadlines to meet in order for Russia to adhere to the April 2012 timetable and a timely response from Canada was required. This option would also increase both the financial and political risks for Canada, given the difficulties of working with Russian sub-contractors, as already experienced during the railroad project. It was also pointed out that this option would require IGX to add personnel with the necessary project management, contracting, and technical expertise (including experience working with Russia). As illustrated in Exhibit G, IGX and the Chemical Weapons Stream was not set up for this kind of project management and delivery structure.

Another option available at the time was to transfer the $55 million directly to the Russian Federation. Key informants, including those from Rosprom, also noted that this option was highly favoured by the Russian partner, who asserted that this would allow for maximum usage of the financial contribution. This preference for a direct bilateral donation has also been expressed by senior Russian officials in published interviews.⁽¹⁶⁾ IGX key informants explained that this option was excluded at the time because of an assessment conducted by IGX officials that this money might not be used for the intended purposes. The mechanisms in place in Russia are insufficient to allow for the necessary transparency and accountability required by the Government of Canada. The financial and political risks were considered too high for Canada, in the event that Russia did not use the Canadian funds to procure the equipment.

Strengths of the Canada-UK Structure

A number of strengths of the Canada-UK structure were identified based on: key informant interviews, the analysis of the project structure for the railroad project under the first MOU and the analysis of the roles and responsibilities of each stakeholder. These benefits are summarized in Exhibit I. Many key informants observed that there are significant benefits for Canada from the arrangement. Further, they indicated that the arrangement is contributing to the achievement of results. Some key informants suggested that further analysis should be conducted to determine if the level of effort of the UK project managers is consistent with the project management costs paid by Canada, especially given IGX's management involvement and profile.

**Exhibit I: Strengths of the MDB-2 Project Structure**
Strength:	Comments:
1. Allowed for rapid response to Russia's request	Canada was already at Shchuch'ye working with the UK on the railroad project; MOU with UK less time-intensive to negotiate; Reinforced Canada's commitment to the Global Partnership Initiative; and, Satisfied Russian partner.
2. Builds on UK capacity and experience	Familiarity based on experience with railroad, LPAS, and inter-site communication projects; UK MoD access to Royal Engineers, contracting experts; UK MoD experience managing projects in Russia, including for other donors; and, Consistent, long-term senior management.
3. Risk Management	Canada's indirect project management role ensures the country is exempt from any direct claim of liability; and, Financial and political risks of funding not used for intended purpose are minimized.
4. Canada treated as a full partner in practical, operational sense*	IGX personnel involved on daily basis; IGX personnel review, comment on, and in some cases prepare documents; Attend meetings with key stakeholders; and, Participate on UK Oversight Board. * This arrangement does entail some supervisory/monitoring expenses for IGX.
5. UK MoD audit	Canadian funds subject to audit by UK Defence Internal Audit office; and, Provides transparency and accountability.
6. UK "Duty of Care"	Consideration given to the use of Canadian taxpayers' money for intended purposes only and to achieving maximum effect; and, MoD prepared a lessons learned report for continuous improvement.
7. Multi-donor environment	Helps to spread the financial risk within a larger group; Centralized management reduces risk of overlap and duplication; and, Ensures all needed equipment will be bought.

Drawbacks to the Canada-UK Structure

A number of drawbacks to the Canada-UK structure were identified based on interviews with a range of key informants, the analysis of the railroad project structure and the analysis of the roles and responsibilities of each stakeholder. The drawbacks are presented in Exhibit J.

**Exhibit J: Drawbacks to the MDB-2 Project Structure**
Weakness:	Comments:
1. Less direct control by Canada	UK has privity of contract with Bechtel, therefore, from a legal standpoint, Canada cannot directly contact Bechtel, nor can the Canadian legal advisor review Bechtel's contracts with the UK MoD.
2. Perceived lack of visibility with the Russian partner	Russian partner has made a presentation to the OPCW showing Canadian contribution as belonging to the UK because of the current project structure.
3. Project management fees may mean less money going to Russia	Russians have a preference for direct, bilateral contributions to maximize the funds available; Canada pays the main part of manpower, travel and sustenance costs to the UK MoD to manage Canadian-funded projects in addition to project management fees paid to Bechtel for Planning and Implementation Phases.
4. IGX / DFAIT not developing in-house project management capacity	Limited in-house expert capacity is a result of the IGX structure when the program was established and the decision made to outsource for specialized expertise.
5. Working in a multi-donor environment	Situation arose whereby France insisted on funding equipment already designated to Canada; Created complications and unnecessary confusion; Resolved by Russia requesting that France choose other equipment to fund.

While none of the key informants suggested that the project structure should be altered because of these drawbacks, key informants noted that efforts are being made to mitigate the drawbacks. For example, with respect to the limited direct control, over time Canada and the UK have implemented measures to ensure that Canada is not just seen as the funding partner. These include increased communication with the stakeholders and ensuring Canadian participation at meetings, for example, as a full participant at the Shchuch'ye Coordinating Working Group. These efforts also help mitigate the perceived lack of visibility with the Russian partner by providing IGX project personnel with opportunities to assert Canada's role as a full partner.

Key informants in IGX and UK MoD also pointed out that efforts are made to minimize project management fees. Documentation provided by the UK MoD indicates that the total project management fee⁽¹⁷⁾ for the MDB-2 project is minimal, considering the level of service provided. Representatives of the UK MoD explained that Canada only pays for the manpower costs of one and a half project officers, in addition to Travel and Sustenance charges. Canada is not paying for manpower costs of senior Counter Proliferation and Arms Control (CPAC) officials, nor for manpower costs of the technical support provided by the Royal Engineers. The services of the UK Embassy and MoD contracting are also not paid for by Canada. An assessment of whether or not the fees paid by Canada for the above services are reasonable was not part of this evaluation. The team also did not have access to data estimating what would be the cost of these services if provided by IGX personnel, thereby limiting the ability to do a direct comparison.

Key informants agreed that the project management fees of a western Principal Contractor are significant. In this case, the fees paid to Bechtel can range from 11-18 per cent for the Planning Phase and 30-35 per cent for the Implementation Phase. However, it should be noted that this arrangement was already in place between the UK and Bechtel when Canada came onboard. Documentation reports that project management fees for a western Principal Contractor in Russia can reach as high as 55 per cent.

The problems of working in a multi-donor environment are seen by staff at IGX and UK MoD as the necessary cost of access to multiple donors and additional funding.

Concerns about Bechtel

As stated earlier, concerns about Bechtel were identified by key informants, as well as in Quarterly Reports by the UK MoD. These concerns are mainly of a managerial nature, although Canadian auditors expressed some concerns about the value for money (these issues are beyond the scope of this evaluation). Examples of managerial concerns include not meeting time lines for deliverables and reporting, the need for improved quality of reporting, and unresponsiveness to UK MoD and Canadian requests for senior level meetings. In addition, there is a concern that Bechtel is so focused on reducing and managing its own risk that it creates undue delays. Despite these concerns, key informants in IGX and UK MoD expressed their overall satisfaction with Bechtel and observed that these concerns can be, and are being, dealt with to the satisfaction of all stakeholders.

4.2.3 Constraints and Mitigation Strategies

Key informants, documents and the site visit confirmed that IGX faces a number of constraints to delivering the MDB-2 project in Russia. Many of these challenges are identified in Section 1.2.4 and include the remoteness of the location; the size and complexity of the CWDF; communication difficulties with the Russian partner; and the sensitivity of working on a weapons-related project. Despite these constraints, there is evidence that IGX and the UK MoD continue to achieve progress towards providing the equipment by mitigating the challenges and continuing to collaborate with the Russian partner.

4.2.4 Other Governance and Delivery Mechanism Issues

Ultimate responsibility for completion of the Shchuch'ye CWDF

Key informants from Rosprom observed that as the "official state customer" of all chemical weapons destruction facilities in Russia, Rosprom is responsible for the successful completion of Shchuch'ye. Several key informants also confirmed this observation.

Canadian obligations for monitoring the CWD process

The evaluation assessed whether Canada had any responsibilities for monitoring the CWD process once the equipment was delivered. Key informants stated that Canada had no official responsibilities for any monitoring-related activities. It was stressed that the obligations of Canada and the UK MoD ended once the equipment was officially handed over to the Russian partner. The responsibilities of each stakeholder are described in Annex J to the UK-Russia Implementation Arrangement.

According to additional documentation, monitoring of the chemical weapons destruction process is to be carried out by the Organization for the Prohibition of Chemical Weapons. The OPCW is the implementing agency of the Chemical Weapons Convention. The OPCW Technical Secretariat, together with the Russian Federation, develops the verification regime that includes onsite inspection and monitoring with onsite instruments.

Ability to leverage cooperation at Shchuch'ye for cooperation in other areas

Respondents from IGX noted that Canadian cooperation at Shchuch'ye has enhanced Canada's credibility with other Russian partners; has benefitted the Canada-Russia bilateral relationship; demonstrated Canada's ability to deliver on commitments; facilitated the initial preparation work with Rosprom for future projects at Kizner; and has also helped Canada leverage other donors to provide funding to the Russian CWD program.

Capacity building value of this project for Canada and IGX

Canadian key informants stated that this project provides limited or no opportunities for domestic capacity building in the area of chemical weapons destruction. While it was agreed that Defence Research and Development Canada (DRDC) is the Canadian organization with knowledge of chemical weapons, the evaluation team heard conflicting reports on the level of DRDC interest in contributing to this project. IGX sought DRDC expertise at Shchuch'ye but DRDC declined after two DRDC representatives were given the opportunity to visit the Shchuch'ye CWDF at the expense of IGX.

As discussed earlier, the Chemical Weapons Destruction stream within IGX is structured to minimize the resources required for administration (project management, project delivery and contracting services) of the program in DFAIT. Instead, where required, specialized services are being outsourced. IGX representatives explained that this approach reflects decisions made at the inception of the program, partially because the program has a limited life span of 10 years, but also because of the difficulties in recruiting at DFAIT. IGX key informants noted that it takes several months to fill a position.

4.3 Efficiency and Effectiveness

In order to assess the efficiency and effectiveness of the MDB-2 project, the following issues were considered: the mechanisms in place to promote the efficiency and effectiveness of the procurement process and Canada's involvement; the control mechanisms in place to ensure sub-contractors deliver services on time and on budget; as well as Canada's responsibility for monitoring the delivery, storage and installation of the equipment and its reliability once operational.

4.3.1 Mechanisms that Promote Efficiency and Effectiveness

Key informants described several mechanisms that are in place to promote efficiency and effectiveness. These processes are also documented in the Treasury Board Submission and validated through reporting from both the UK MoD and Bechtel.

At the outset of the project it was decided to "piggyback" on US purchases of equipment for Building 101, managed by Parsons. This piggybacking was expected to increase efficiency by leveraging the preparation and groundwork already completed by Parsons. Through a non-disclosure agreement signed with Parsons, Bechtel is able to obtain the purchase orders (with costing already done) and designs for the equipment.

Due to slight differences between the two buildings, the shop drawings must still be modified and verified by the Russian Design Institutes of Giprosyntez and GOSNIIOKhT. Many key informants expressed the view that piggybacking reduced the cost and risks of obtaining the equipment, compared to a situation where Bechtel had to start from scratch. However, respondents and documents confirmed that the step involving the review and approval of the designs has, at times, been more complicated and time-consuming than initially expected by the procurement team. Key informants, and Bechtel representatives in particular, stressed that the review and approval of the shop drawings is a crucial step for the overall efficiency and effectiveness of the project. In their view, it would create significant problems if there were subsequent change orders to the designs once the equipment had been ordered. The review and approval of shop drawings are also required to minimize the risk that the equipment would not function as required.

Another mechanism to promote efficiency and effectiveness is the use of the same Russian equipment manufacturers as used by the US. This is specialized equipment that must be consistent with that in Building 101, therefore there was little to be gained by spending the time and money to find an alternative manufacturer. There is just one piece of equipment (Venturi Scrubber) being procured by Canada that is not being manufactured in Russia. It is being manufactured in the Netherlands.

Key informants also stressed that Canada approves each step in the procurement process. Both Canada and the UK are members of the Shchuch'ye Coordinating Working Group. This group also includes the United States and Russia. The SCWG meet quarterly to discuss the status of activities at the site and to ensure each stakeholder is aware of what the others are doing, to avoid overlap or gaps that would impact on the entire project.

4.3.2 Control Mechanisms to Ensure Services Delivered On-time and On-budget

A number of control mechanisms to ensure that high-quality services were delivered on-time and on-budget were identified through key informant interviews and the document review. As the Contracting Authority and project managers, the UK MoD oversees Bechtel. As part of the procurement process, as detailed in the Enabling Contract, Bechtel is required to submit key deliverables upfront. These deliverables include: a Project Management Plan (PMP) that must contain health and safety plans and demonstrate environmental compliance, risk registers, a project schedule, and other deliverables. Key informants from UK MoD stated that they scrutinize purchase proposals to ensure rates are fair and reflect the agreed rates, before issuing the Task Order to proceed with procurement. Bechtel is required to submit monthly and quarterly progress reports and pictures. Bechtel also maintains onsite representation.

According to the terms of the UK-Bechtel Enabling Contract, payments to Bechtel are made on a fixed-price basis and only upon achievement of milestones. This applies to Russian sub-contractors as well and is consistent with the Kananaskis Principles. Also, under the terms of the Enabling Contract with Bechtel, placing individual Task Orders does not carry any obligation to place any subsequent Task Orders, providing the UK MoD, and Canada by extension, with an exit strategy, if required.

The UK MoD has also initiated an audit of its projects in Shchuch'ye, including the MDB&-2 project. These audits are conducted by the Defence Internal Audit unit which invited Canada to participate.

In turn, IGX oversees UK MoD activities. There is a Department of Justice legal advisor in IGX who provides advice to the Senior Program Manager, Director and Director General on agreements such as the MOU, to ensure that the program is in compliance with the Government of Canada policies.

IGX and UK MoD representatives stated that Canada is kept fully informed by UK MoD project managers and that the two teams are in consultation on a daily basis. In effect, IGX is treated as a full partner and is provided with opportunities to review, comment on and approve all agreements and Task Orders prior to finalization. As outlined in the Canada-UK MOU, UK MoD submits written Quarterly Reports that detail all activities related to the Shchuch'ye project, including any financial transactions and project management costs. Project management costs are broken down by manpower charges (the project managers keep track of time spent on the MDB-2 project), as well as charges related to Travel and Sustenance whenever the project manager travels on behalf of Canada. In addition, UK MoD is required to keep a separate bank account for the funds that are transferred by Canada.

Within IGX, internal project management costs (largely travel related) are tracked by a financial officer. Documentation provided to the evaluation team indicates that the reporting of these costs could be clarified to make it easier to determine the project costs.

4.3.3 Canada's Formal Responsibilities for Monitoring the Delivery and Installation of Equipment

According to Annex J⁽¹⁸⁾ of the UK-Russia Implementation Arrangement, the Principal Contractor Bechtel is responsible for the delivery of the equipment and monitoring the work. The installation of the equipment is the responsibility of the Russian partner. Key informants stated that once the equipment is handed over to the Russian partner, Canada has no further obligations and is not subject to any liability. For example, if there are any mechanical problems with the equipment, the Russian partner will be responsible for pursuing action under the warranties with the manufacturers.

However, IGX has the option to monitor the delivery of the equipment and has exercised it by monitoring the delivery of the first batch of Catalytic Reactors in August 2006. IGX key informants explained that the purpose for this monitoring visit was to witness first-hand, the first delivery of a major piece of equipment. As it happened, there were slight discrepancies with the cataloguing and tagging of the equipment and IGX representatives were able to contribute to resolving the issue.

IGX representatives stated that they do not anticipate monitoring each delivery and have decided to send a representative from the Canadian Embassy in Moscow to monitor delivery of the second batch in mid December 2006. This will be a more cost-effective solution, while still allowing IGX to perform due diligence in ensuring that the solution to the cataloguing and tagging issue has been enacted. Future monitoring will depend on the nature of the equipment being delivered, as well as the circumstances. It is expected that some monitoring visits will be handled only by the UK MoD.

4.3.4 Storage of the Equipment

Annex J of the UK-Russia Implementation Arrangement clearly describes the responsibilities of the Russian partner to provide the following: storage free of charge at the site; security and maintenance of the condition of the equipment and materials; and access to the storage facility to enable the UK, the Principal Contractor and Canada to check on the equipment. All key informants agreed that these are the responsibilities of Rosprom.

Top of page

5.0 Results / Success

Determining the progress of the project towards achieving established goals and milestones was another objective of this evaluation. In order to make this determination, the evaluation team considered the following issues: the status of procurement and delivery of the equipment; complications and their causes, as well as related risks and mitigation measures; and assurance that original objectives and scope of the project will be achieved.

Key informants noted that there have been complications. However, many noted that these are not necessarily unexpected given the complexity of the CWDF project and the challenges of working in the Russian context. Despite these complications and the increased risk they pose to the project, in some cases, key informants generally agreed that the original objectives and scope of the project will be met, although not according to the time line originally put forward in the TB Submission, nor according to the original request of the Russian partner.

5.1 Status of Equipment Procurement and Delivery

Exhibit K provides a summary of the status of the equipment procurement and delivery. As indicated in the exhibit, at least one batch of Catalytic Reactors has been delivered to the site and this was confirmed during the site visit by the evaluation team. A second batch is scheduled to be delivered in mid-December 2006. These deliveries provide evidence of acceptable progress towards established goals and milestones. Many other pieces of equipment are still in the Planning Phase.

Key informants and project documentation reported that the Planning Phase can be time consuming and difficult for a number of reasons, including UK MoD requirements for Bechtel to provide detailed Project Management Plans up-front. The PMP includes a firm fixed price for the Task Order/project with agreed payment schedule based on milestones, a specific and detailed scope of work including Bechtel's activities and deliverables to execute the project scope of work (i.e. a full list of required equipment and installation works, visits to manufacturers, quality assurance), a negotiated and ready-to-award sub-contract with the Russian sub-contractor with all required Terms and Conditions as per the UK-Bechtel Enabling Contract, a cost-validation report and/or independent cost estimate to justify the negotiated price with the sub-contractor, a thorough health, safety and environmental plan for execution of the project, review of permitting requirements, shipping, warranties or any other related requirements to ensure a smooth execution and turnover to the Russian authorities on site.

Upon receipt of the draft PMP there are negotiations between Bechtel and the UK MoD to finalize the PMP, including the costs and payments schedule and , if required, obtain a bank guarantee (from a AAA-rated bank) from the sub-contractor should justifiable advance payments be required (e.g. for materials).

Once the above deliverables have been negotiated and finalized, the FAI also reviews the ready-to-award sub-contract and often requests changes to the terms and conditions, requiring further consideration and negotiation by Bechtel and the UK MoD. This last step has created several complications and can be time consuming to resolve. However, IGX officials noted that it is preferable to be thorough during the Planning Phase in order to eliminate complications during implementation. Once the Planning Phase has been completed, the procurement of the equipment can take place in a timely manner.

**Exhibit K: Status of Equipment Procurement and Delivery**
Equipment:	Status:
Catalytic Reactors	1^st batch delivered to site at end of August 2006 2^nd batch to be delivered by mid December 2006 3^rd and 4^th batches to be delivered by March 2007
Destruction Process Lines (DPLs)	In Planning Phase
Munitions Thermal Treatment Destruction Process Lines (MTTDPLs)	In Planning Phase
Leakers Treatment Area (LTA)	In Planning Phase
Packages 1,2,3	In Planning Phase
Package 4 (Venturi Scrubber)	In Procurement Phase

As indicated in Exhibit K, IGX and UK MoD have overcome a number of challenges to achieve the delivery of the first batch of Catalytic Reactors. IGX officials state the project is on track to procuring the remaining equipment in time for the completion of MDB-2.

5.2 Complications

When the project was initiated, the Russian partner indicated that the equipment was required by the end of 2006.⁽¹⁹⁾ To meet this very tight deadline, Canada, the UK MoD and Bechtel developed an aggressive project plan. However, there have been a number of complications, including the following:

UK MoD requirement that the Russian manufacturer of the Catalytic Reactors obtain a bank guarantee to protect Canadian funds from risk of default;
Inadequate cataloguing and tagging of the first batch of Catalytic Reactors upon delivery to the site;
Intervention by the Russian partner in the sub-contracting process by insisting on reviewing the sub-contracts with the Russian manufacturers;
Disagreement between the Federal Agency for Industry and MoD over the wording (and translation) with respect to Intellectual Property Rights on the DPL designs; and
Bechtel not providing information to the stakeholders on a timely basis.

The UK MoD and IGX have made considerable efforts to manage these complications, while striving to remain on track for the ultimate deadline of operationalization of the CWDF in 2008. The complications have not jeopardized the project and are further evidence of the complex and difficult environment the project personnel face. In each case, the stakeholders have been able to resolve the complication to the satisfaction of all involved through patient negotiation and cooperation.

The site visit clearly demonstrated that the Shchuch'ye CWDF is a large and complex undertaking, requiring many elements to fall into place simultaneously. A delay in one area will easily have a domino effect on other projects at the site. Key informants described how delays in the construction of some buildings have impacted on the MDB-2 project. In particular, the construction of the US-funded Building 101 has been stalled for an extended period of time, pending the resolution of a dispute over the choice of a Russian sub-contractor.

As a result of this dispute, the UK MoD reported that it has become apparent that US funds will not be sufficient to complete the building as planned. As indicated in the IGX risk registers of August and October 2006, this development has the potential to significantly impact on the project (level 4 of 5). Key informants explained that further delays with Building 101 may mean Canada would be procuring some equipment ahead of the US or in parallel, thereby losing the efficiencies of the piggybacking process.

IGX view of the situation involving the risk of insufficient US funds:
The US and Russia will soon sign an agreement to complete the Shchuch'ye CWDF;
Under the agreement, the Federal Agency for Industry (FAI) will contract directly with the Russian sub-contractors;
The US will hand-over the remaining unawarded contract packages to the FAI;
All remaining US funds (up to ceiling of $1.093B) will be made available;
Parsons will continue to manage all equipment procurement for Building 101;
FAI expect to be able to complete the facility within the US funding envelope; and
Outstanding equipment to be purchased by Canada will remain on track with the US.

Despite recognition in Quarterly Reports that US funding will be insufficient to complete Building 101 as planned, the evaluation team found that the IGX risk registers of August and October 2006 had not reflected this as an increased likelihood. Rather, the likelihood is still indicated as a "1" out of 5, which was the same rating provided in the Treasury Board Submission.

According to IGX, the likelihood rating had not changed because the situation was still fluid and in the process of being resolved (see box), in which case the delay would not impact the MDB-2 project negatively.

5.3 Achievement of Original Objectives and Scope

It is clear from the UK MoD and Bechtel progress reports, as well as the project Gantt Chart that deadlines for manufacture, delivery and hand over of the equipment continue to be pushed back. Nonetheless, the evaluation team was assured by all stakeholder groups that the original objectives and scope of the project will be met. This means that the equipment to be procured by Canada will be purchased and delivered to the Shchuch'ye site most likely by the end of 2007 - in time for MDB-2 to be operational in 2008.

Top of page