RESEARCH Evaluation of the National Train-the-Trainer Program for Hospice and Palliative Care in Korea

project requirement, Abstract Background: To evaluate the effectiveness of the National Train-the-Trainers Program for Hospice and Palliative Care Experts (TTHPC) sponsored by the National Cancer Center of Korea between 2009 and 2012. This program was developed to improve the teaching skills of those in the field of hospice and palliative care (HPC). Materials and Methods: Training was offered in eight 1-day sessions between 2009 and 2012. The effect of the program was measured using Kirkpatrick’s model of educational outcomes. First, levels 1 and 2 were evaluated immediately after the 1-day program (n=120). In 2012, the level-3 evaluation test was administered to trainers who offered at least one HPC training (n=78) as well as to their trainees (n=537). Results: The level-1 evaluation addressed participant reactions to and satisfaction with the program. Participants (n=120) were generally satisfied with the content, the method, and the overall course (mean range: 3.94-4.46 on a five-point Likert scale). The level-2 evaluation (learning) showed that participants gained knowledge and confidence related to teaching HPC (4.24 vs. 4.00). The level-3 evaluation (behavioral), which assessed trainers’ application of teaching skills to HPC, showed that trainees rated the teaching methods of trainers (mean range: 4.03-4.08) more positively than did trainers (p<0.05). Female trainers were more likely than were male trainers to plan sessions in consideration of their trainees’ characteristics (4.11 vs. 3.58; p<0.05), and nurse trainers were more likely than physician trainers to use a variety of instructional methods (4.05 vs. 3.36; p<0.05) Conclusions: We conducted systematic evaluations based on Kirkpatrick’s model to assess the effectiveness of our train-the-trainers program. Our educational program was practical, effective, and followed by our HPC experts, who needed guidance to learn and improve their clinical teaching skills.

services in Korea (Kang et al., 2013). The program has been presented 31 times by 13 HPC institutions from 2009 to 2013, and a total of 1101 practitioners have participated in it. Most importantly, learner-centered instructional methods (e.g., role-playing scenarios, small-group casebased discussions, and interactive lectures based on adult learning principles) were developed to enhance the educational effectiveness and quality of this program. We adopted a train-the-trainers approach to disseminate this program across Korea. This approach has been used widely in medical education to improve faculty teaching and disseminate educational programs (Sullivan et al., 2005;Stratos et al., 2006;Diesel et al., 2011;Pien et al., 2011;Lee et al., 2012). In the present study, we assessed the effect of the national train-the-trainers program for HPC experts (TTHPC) in Korea using Kirkpatrick's evaluation model.

Program outline and evaluation framework
The TTHPC has been offered once or twice annually since 2009. It was held eight times between 2009 and 2012, and 128 professionals attended its 8-h program. Participants were recruited from and restricted to experts in HPC (e.g., physicians, nurses, and social workers), and attendance was limited to 15-20 trainers per program. The program was designed to improve teaching skills and to enhance the knowledge needed to educate HPC practitioners. The program content was based on the principles of adult learning and included an overview of HPCSE and an examination of the theory and practices underpinning the development of specific teaching skills (e.g., interactive lectures, role playing, and small-group case-based discussion).
We designed a framework based on the first three levels of Kirkpatrick's model, provides a systematic framework for assessing educational outcomes in terms of four levels [9], to assess the program. The first level focuses on learners' satisfaction. The second level provides information about changes in learners' knowledge, skills, and attitudes. The third level addresses improvements or behavioral changes related to the training. The fourth level evaluates the impact on institutions. This study focused on Kirkpatrick's first three levels ( Figure 1).

Evaluation process
The level-1 investigation focused on the trainees' satisfaction with the educational content, lecturers, teaching methods, educational environment, and overall course. Items were rated on a five-point Likert scale (1=strongly disagree, 5=strongly agree). Level 1 also addressed satisfaction with the TTHPC program ("Met my expectations," "Exceeded my expectations," "Did not meet my expectations").
The level-2 evaluation focused on the degree to which the learning objectives were achieved, knowledge about HPC education was increased, awareness of the importance of HPC education was enhanced, and each of four domains was covered (adult learning principles, interactive lecture skills, role play, small-group case-based discussion). These items were also rated on a five-point Likert scale (1=definitely not, 5=definitely). The level-1 and -2 evaluations were completed immediately after the TTHPC.
Level 3, a behavioral evaluation, involved surveys of the participants in the TTHPC who also delivered lectures in regional HPCSE programs as well as their trainees.
Questionnaires were mailed to 78 lecturers who had delivered at least one lecture in regional HPCSE programs between 2009 and 2012. A total of 54 (69.2%) questionnaires were returned. This questionnaire requested sociodemographic information such as profession, age, educational background, and HPC experience. The behavior of lecturers was assessed in terms of the planning and execution of instructional methods.
The level-3 evaluation also focused on changes in the lecturers' HPC pedagogical practices after the program. We assessed the teaching skills of the lecturers and the satisfaction of their trainees with these skills. Data were gathered from trainees and those lecturers who delivered at least one lecture in a regional HPCSE program. Lecturers were asked 1) whether they planned the instructional approaches used in the HPCSE program in consideration  DOI:http://dx.doi.org/10.7314/APJCP.2015.16.2.501 Evaluation of the National Train-the-Trainer Program for Hospice and Palliative Care in Korea of the trainees' characteristics, 2) whether they tried to facilitate the trainees' participation, and 3) whether they used a variety of instructional methods. Items were rated on a five-point Likert scale (1=definitely not, 5=definitely).
Questionnaires were also mailed to 537 trainees who completed the regional HPCSE program between 2009 and 2012, and 100 (18.6%) surveys were returned. We assessed trainees' reactions to the teaching methods of the lecturers using two questions rated on a five-point Likert scale (1=definitely not, 5=definitely): 1) whether lecturers used a variety of instructional methods for teaching, and 2) whether lecturers facilitated trainees' participation. This survey was conducted between September 2012 and October 2012.

Participants
The TTHPC is a 1-day program for training experts in providing HPCSE training. Between 2009 and 2012, a total of 128 experts were trained to present regional HPCSE programs. After the program we distributed a selfassessment questionnaire to the participants, garnering a response rate of 93.7% (n=120). Most participants were female (73.4%), and a majority of participants were clinicians, including nurses (48.4%) and physicians (34.4%). Most participants (70.3%) were working in HPC units, with 20.3% working in a general clinic or hospital. The majority of participants had a doctoral-level (56.3%) or masters-level education (38.2%). Most importantly, more than 80% of participants had worked in HPC for at least 5 years.

Level 1: Reactions
Level-1 evaluations focused on learners' reactions to and satisfaction with the program (Table 1). They evaluated the extent to which topics were unfamiliar to them (mean rating range: 3.13-3.36) and easy to understand (mean rating range, 4.09-4.46) and whether the program relied on effective teaching methods (mean rating range: 4.10-4.44).
Learners were highly satisfied with the appropriateness of the educational environment (mean rating range: 3.94-4.46). Nearly two-thirds (70%) indicated the need for in-service training regarding teaching skills. Importantly, most learners said they highly would recommend the program to colleagues (90.8%). The overall satisfaction with program was high as measured by those who said it "met my expectations" or "exceeded my expectations" (59.2% and 37.5%, respectively).
Participants also provided the following positive comments in response to an open-ended question: "I felt nervous when I tried to teach. However, now I have become confident and realize what was wrong." "I had been ashamed to reveal my weakness, but I felt proud to participate in HPC education. I want to thank everyone who taught us." "This program maximized my potential."The contents of this program were necessary for *Scale: 1 = strongly disagree, 5 = strongly agree; **percentage of respondents who responded "definitely yes" or "yes"; TTHPC: National Train-the-trainers Program for Hospice and Palliative Care Experts; TTHPC: the National Train-the-Trainers Program for Hospice and Palliative Care Experts faculty members and essential for improving the quality of HPC." "The program provided a theoretical organization for my expertise, and I felt more confident." "I felt very satisfied, but thought it needed more time. I should try harder on my own." "I was motivated by learning new concepts." In contrast, some participants suggested the following improvements: "It needed more information about creating a syllabus and teaching materials." "I need a session about communication-skills training." "There was limited personal interaction with other participants." "This program started too early to travel from afar."

Level 2: Learning evaluation
Most participants reported that their learning objectives were achieved (3.96±0.069) and reported having increased their knowledge about teaching HPC (4.24±0.071). Participants also rated their confidence in teaching HPC and their awareness of importance of HPC education as high (4.00±0.062, 4.28±0.072, respectively). With regard to the usefulness of the four sessions devoted to the four domains of HPC education, the overall mean rating was 4.44, and scores ranged from 4.26 to 4.56 for the different sessions. The session on interactive lecture skills attracted the most positive ratings (4.56±0.066), followed by that on adult learning principles (4.53±0.071) ( Table 2).

Level 3: Behavioral evaluation
A total of 78 trained lecturers delivered at least one lecture in HPCSE programs, and 54 (69%) of these individuals completed surveys. Most respondents were female (64.8%) and worked in clinical positions such as nurses (40.7%) and physicians (40.7%). In terms of age, 41 (75.9%) respondents were in their 40s or 50s, and most respondents were from HPC units (64.8%) and had a doctoral degree (66.7%). Most had more than 10 years of clinical experience (88.9%) and more than 5 years of HPC experience (77.8%). Additionally, most had more than 5 years of HPC teaching experience (61.1%).
Nationally, a total of 537 trainees participated in regional HPCSE programs between 2009 and 2012. Of these 537 trainees, 100 (18.6%) responded to the survey. Most respondents were female (76%) and nurses (52%). More than 50% of respondents were from general hospitals   (25%) or other institutions (35%). More than half of the trainees (56%) had more than 10 years of clinical experience, and almost all (90%) had less than 5 years of HPC experience.
All lecturers and trainees were asked whether the lecturers used a variety of instructional methods and whether they facilitated trainees' participation. Lecturers were asked whether they planned instructional methods in consideration of trainees' characteristics and whether they used instructional methods to facilitate trainees' participation. Table 3 presents the result of the behavioral evaluations of lecturers and trainees. The results show that lecturers planned their instructional methods in consideration of trainees' characteristics (3.93±1.03) and to facilitate trainees' participation (3.61±0.088). The overall mean ratings for lesson planning were higher than 3.5. With regard to the execution of instructional methods, the mean self-rated scores were 3.74±0.085 and 3.61±1.16 for consideration of trainee characteristics and for facilitation of participation, respectively. However, trainees rated the instructional methods used by lecturers significantly more positively (mean rating range, 4.03-4.08) than did the lecturers themselves (p <0.05).
In terms of demographic variables, we found significant differences between male and female instructors regarding lesson planning. Female lecturers were significantly more likely to plan their lessons in consideration of trainees' characteristics than were male lecturers (4.11 vs. 3.58; p<0.05). Additionally, nurse lecturers were more likely to use a variety of instructional methods than were physician lecturers (nurse: 4.05; physician: 3.36; p<0.05) ( Table 4).

Discussion
We developed and administrated a train-the-trainers program for HPC experts to increase the availability of high-quality regional HPCSE education and to enhance the teaching skills of trainers in this field. This program is the first to address the teaching skills of HPC trainers in Korea. Indeed, it was necessary to focus on the teaching skills of trainers to enhance the communications skills and psychosocial sensitivity of the physicians, nurses, and other professionals who receive such training and provide end-of-life care to patients and their families. In response to the need to develop teaching skills, several train-the-trainers programs, such as the Education for Physicians on End-of-Life Care Project (EPEC) (Robinson et al., 2004), the Palliative Care Education and Practice at Harvard Medical School (PCEP) (Sullivan et al., 2005), the Stanford Faculty Development Center's (SFDC) End-of-Life Care Program (Stratos et al., 2006), and the End-of-Life Nursing Education Consortium (ELNEC), have been successfully implemented (Kelly et al., 2011). These programs have similar goals: to teach the core content of end-of-life care as well as to help the medical faculty develop effective teaching skills using a train-thetrainer strategy.
In contrast, our program focused on improving the teaching skills of highly experienced clinical experts and not on enhancing the clinical content of end-of-life care.
Thus, we developed a 1-day program to teach teaching skills based on adult learning principles.
It is necessary to rapidly disseminate the 60-h basic HPCSE program throughout Korea to enhance HPC services. With support from the MoHW, the National Train-the-Trainers Program for HPC Experts was successfully launched, training 128 trainers from around the country to present regional HPCSE programs.
Kirkpatrick's evaluation framework provided a useful perspective for this study. Research has shown that the majority of studies have employed a modified version of Kirkpatrick model to evaluate faculty development programs in medical education (Curran and Fleet, 2005;Lacasse and Ratnapalan, 2009;Leslie et al., 2013).
In terms of Kirkpatrick's framework, the majority of studies of train-the-trainers programs have measured outcomes at levels 1, 2, and 3 (Fitzgerald et al., 2009;Stebbins et al., 2009;Pien et al., 2011;Leslie et al., 2013). According to several review articles on train-thetrainers programs, only a small percentage of studies have conducted level-4 evaluation for trainers (Steinert et al., 2006;Lacasse and Ratnapalan, 2009;Leslie et al., 2013). Evaluations of learners' clinical performance (level 4) may be burdensome in terms of time and money. As the main purpose of this study was to develop the teaching skills of trainers, Kirkpatrick's first three levels of evaluation, which focus on teaching performance, were the most appropriate. Similarly to previous studies (Robinson et al., 2004;Sullivan et al., 2005;Stratos et al., 2006), our overall results for levels 1 and 2 were very positive in terms of course evaluations and learning achievements (Tables 2, 3).
To assess behavioral changes, the level-3 evaluation, we conducted a follow-up survey of trained lecturers and their trainees to assess how the lecturers had applied their training during teaching. Whereas most studies of faculty development programs have measured self-reported behavioral outcomes (Lacasse and Ratnapalan, 2009;Leslie et al., 2013;Karadag et al., 2014), our study relied on both self-reported evaluations and on assessments by trainees. Our results reflect positive changes in lecturers' teaching skills, and the behavioral changes noted by both lecturers and trainees were all in the positive direction. Indeed, trainees had more positive reactions to the lecturers' teaching skills did the lecturers themselves (p<0.05). Trainees reported that lecturers used a variety of instructional methods and facilitated trainees' participation (4.03±0.82 and 4.08±0.80, respectively). According to the trainees, lecturers applied their new teaching practices in regional HPCSE programs. According to our follow-up survey with 128 trained trainers, only 60.9% went on to present HPCSE programs (n=78). Because the train-thetrainers program assessed in this study was designed only for lecturers in HPCSE programs, data from only those lecturers who had conducted HPCSE programs were analyzed. Data from trainers who had not participated in this program were excluded although they reported that they had presented many HPC educational programs at an institutional level (e.g., short courses on HPC, other informal training courses in HPC units).
Finally, our findings showed that female and nurse lecturers were more likely to plan and apply learnercentered instructional methods than were male or physician lecturers, respectively (p<0.05). This finding reflects differences in the educational backgrounds of HPC practitioners. No formal training course in HPC was available to physicians at the time we created this trainthe-trainer program. However, formal training in HPC directed at nurses began in 1985, and various community and institutional-level educational courses in this area have been developed and directed at this target group (Ro et al., 1996;Choi et al., 1998). Furthermore, the ELNEC courses to improve palliative care education for nurses have been made available in Korea (Kim et al., 2011). These differences in the educational backgrounds among HPC professionals may influence their pedagogical foci and practices when providing HPC education. This study had several imitations. First, although most studies on train-the-trainers programs have employed self-report survey methods (Leslie et al., 2013), our reliance on such data may have limited our results. The use of objective tests and of assessments of perceived knowledge may provide more robust evaluations with regard to learning. Second, our use of only Kirkpatrick's first three evaluation levels excluded an assessment of results (level 4). We need to assess the effect of trainers' teaching behavior on learners' HPC practice with end-oflife patients and their families.
Despite these limitations, we performed systematic evaluations based on Kirkpatrick's model to assess the effectiveness of a train-the-trainers program. Our educational program was practical and effective in enhancing the standards of HPC. It also reached HPC experts, who required guidance to learn and improve their clinical teaching skills.