Evaluating Intercultural Competency Training

Over the past six years the Centre has been engaged in an ongoing, systematic evaluation of its Pre-departure training programs. The evaluation strategy is based on Donald Kirkpatrick's four levels of learning evaluation. The first phase of the evaluation research study focused on assessing learning resulting from participation in the Pre-departure training. This work was completed in 2005 and found measurable improvements in participants' intercultural knowledge following their training.

This second phase of the evaluation strategy assessed the retention and transfer of learning gained during pre-departure training to an international field setting. It also examined the relationship of learning to intercultural effectiveness (IE) and performance in the field – that is, how intercultural knowledge and skills learned in the Pre-departure training are applied in an international work environment. The behaviour-based competency model of the Interculturally Effective Person (IEP) provided the theoretical foundation for the evaluation research design, the measurement framework, data collection methods and instruments used. This model identifies intercultural competencies considered essential to successful intercultural performance in the field.

Why evaluate retention and transfer?

The findings from the transfer level evaluation are used to inform the ongoing development and design of the Centre's Pre-departure training programs and the further refinement of the IEP model. The results will also contribute to the broader development of knowledge and theory in the area of intercultural effectiveness, where transfer evaluations such as this are rare. This work helps to build knowledge of specific factors that may potentially facilitate or hinder intercultural effectiveness and performance of assignees while on international assignment.

Research Methods and Objectives

The specific objectives of this evaluation were to assess the retention and transfer of learning gained during training to an international field setting and secondly, to examine the relationship of learning gained during CIL's pre-departure intercultural effectiveness training to later performance in the field. Fifty-five assignees who had completed the CIL Pre-departure courses participated in the transfer evaluation. Ages ranged from 22 to 56 and the professional backgrounds of participants varied considerably. The length of assignments ranged from six months to two years. Transfer of learning was assessed using qualitative and quantitative data collected pre-training, post-training and some months later in the field. Assignees were assessed over this period using three parallel forms of a self-administered questionnaires designed to measure assignees' level of intercultural effectiveness across five intercultural competencies.

To ensure the reliability of the intercultural performance measures being used, four types of data measures were used:

1. Coded qualitative interview data was tagged numerically to indicate whether each reported intercultural behaviour or experience was a positive or negative one;
2. Interviewer ratings, using a 12-item checklist, based on observation of assignee's behaviour during the extensive interview process;
3. Independent data coding based on the reading and analysis of the transcribed interview data for each assignee using a six-item measure of in-field intercultural performance;
4. A global rating of each assignee's performance provided by a lead researcher, based on independent review and analysis of assignees' interview data.

Conclusions

The transfer level evaluation demonstrated that Intercultural Effectiveness learning gained during the Centre's Pre-departure training was retained and transferred to the field in this group of assignees. Moreover, IE learning was positively correlated with an independent measure of intercultural performance in the field, providing support for both the IEP model and set of competencies defined within it. As such, the evaluation demonstrated the value of the pre-departure training provided by the Centre for Intercultural Learning and the conceptual IE framework in improving assignees' IE knowledge, skills and in-field performance.

The knowledge and insights gained from the analysis of the in-depth interviews with assignees in the field also provides useful direction for the ongoing development of CIL's intercultural training programs, tools, online resources and screening tools that can be used in the selection of applicants for international assignments. Some directions for consideration in the further development of the Pre-departure training include addressing issues of gender and intercultural gender relations in greater depth, and incorporating training to support the development of competencies such as creative problem solving and personal resilience.

For a full copy of the report, Evaluating Transfer of Learning from the Centre for Intercultural Learning's Pre-departure Training to the Field, contact the Centre at: cultures@international.gc.ca