Effects of the use of a conversational artificial intelligence chatbot on medical students’ patient-centered communication skill development in a metaverse environment
Article information
Abstract
This study investigated how the use of a conversational artificial intelligence (AI) chatbot improved medical students' patient-centered communication (PCC) skills and how it affected their motivation to learn using innovative interactive tools such as AI chatbots throughout their careers. This study adopted a onegroup post-test-only design to investigate the impact of AI chatbot-based learning on medical students' PCC skills, their learning motivation with AI chatbots, and their perception towards the use of AI chatbots in their learning. After a series of classroom activities, including metaverse exploration, AI chatbot-based learning activities, and classroom discussions, 43 medical students completed three surveys that measured their motivation to learn using AI tools for medical education, their perception towards the use of AI chatbots in their learning, and their self-assessment of their PCC skills. Our findings revealed significant correlations among learning motivation, PCC scores, and perception variables. Notably, the perception towards AI chatbot-based learning and AI chatbot learning motivation showed a very strong positive correlation (r=0.72), indicating that motivated students were more likely to perceive chatbots as beneficial educational tools. Additionally, a moderate correlation between motivation and self-assessed PCC skills (r=0.54) indicated that students motivated to use AI chatbots tended to rate their PCC skills more favorably. Similarly, a positive relationship (r=0.68) between students' perceptions of chatbot usage and their self-assessed PCC skills indicated that enhancing students' perceptions of AI tools could lead to better educational outcomes.
INTRODUCTION
Artificial Intelligence (AI) is reshaping various aspects of society, including medical education. Advancements in AI, particularly machine learning, have opened the door to innovative teaching and learning approaches in the medical field. AI in medical education includes various applications, such as virtual patient simulations, intelligent tutoring systems, and adaptive learning platforms [1]. These AI-driven tools have shown promise in enhancing student learning outcomes and providing personalized learning experiences [2]. Studies have also shown that AI chatbots can enhance learning experiences in public health education [3], and recent comparisons between AI chatbots and medical students in clinical reasoning exams have highlighted their potential in medical education [4].
One area where AI chatbots hold significant potential in medical education is in foster patient-centered communication (PCC) skills; however, developing these skills in a traditional educational setting can be challenging because of limited opportunities for real patient interactions and variability in patient responses [5]. AI chatbots can bridge this gap by providing students with a safe and controlled environment to practice their crucial communication skills [6]. Specifically, AI chatbots can allow students to engage in realistic scenarios that require active listening, empathy, and clear communication of complex medical information, all of which are essential for patient-centered care [6]. These chatbots can simulate a wide range of patient behaviors and responses, exposing students to diverse communication challenges and helping them develop adaptability and problem-solving skills [7].
Furthermore, AI chatbots offer immediate feedback and guided reflection. As students interact with the chatbot, they receive real-time feedback on their communication techniques, which helps them identify areas for improvement [5]. This interactive process of practice and feedback can accelerate the development of PCC skills and build students' confidence in their abilities [4-6]. Moreover, the interactive nature of chatbot conversations can enhance students' motivation to learn by making the learning process more engaging and stimulating [8].
Nevertheless, the recent emergence of this technology has necessitated a rigorous evaluation of its impact on medical training. Furthermore, research specifically validating the effectiveness of AI in the development of PCC skills is currently scarce. The lack of specific research on AI chatbots for PCC skills in medical education underscores the importance of our study. This study aims to address this gap by evaluating the effectiveness of AI chatbot technology in developing these crucial skills among medical students.
This study specifically aimed to examine the use of AI chatbots in medical students' classrooms and investigate their impact on students' PCC skills and motivation to learn. By exploring the effectiveness of AI chatbots in this context, this study seeks to contribute to the growing body of knowledge on the application of AI in medical education and inform the development of future educational strategies and technologies aimed at preparing medical students for a rapidly evolving healthcare landscape.
On the basis of these considerations, the present study aimed to determine medical students’ experiences of chatbot use in the following dimensions: 1) How does the use of conversational AI chatbots affect the motivation of medical students to learn? 2) How does the use of conversational AI chatbots enhance medical students’ learning of PCC skills? 3) What are the relationships between students’ learning motivation and their perceptions towards AI chatbot-based PCC?
METHODS
1. Design and settings
This study adopted a one-group post-test-only design to investigate the impact of AI chatbot-based education on medical students’ PCC skills and learning motivation. The participants were 43 second-year medical students (age, 21-26 years; 25 males, 18 females) from Jeju National University College of Medicine, all of whom were enrolled in the “Patient, Doctor, and Society II” course. The study was conducted within the context of this course, which primarily aimed to enhance medical students’ PCC skills.
2. Procedures
The study was conducted over two class sessions on April 26 and June 21, 2023. The class began with detailed instructions on using the Gather Town (Gather Presence, Inc., San Francisco, CA, USA) metaverse platform and step-by-step guidelines for learning activities (Fig. 1). This process plays a crucial role in helping learners adapt to a novel virtual environment and gain a clear understanding of the upcoming learning activities. After this orientation, students accessed the Gather Town platform, selected personalized avatars, and moved into the virtual classroom space to participate in full-fledged learning activities (Fig. 2). The first section began with clinical ethics case studies. The cases were excerpted from “Clinical ethics cases for medical students and general physicians,” published by Seoul National University College of Medicine and translated with permission from the American Medical Association (AMA) Journal of Ethics. Students read the cases, identified ethical issues, shared diverse perspectives in team spaces within Gather Town, and documented their discussions on team space boards. Subsequently, the students were introduced to the AI chatbot, and they practiced doctor-patient role-plays to develop their PCC skills. As a core component of the learning process, individual students activated an AI chatbot to begin their doctor-patient role-play (Fig. 3). They subsequently engaged in doctor-patient role-play scenarios and practiced their PCC skills with the chatbot (Fig. 4).
The second session commenced with re-entry into Gather Town, followed by a review and feedback-sharing of their first session experiences, and then practicing doctor-patient role-play again in Gather Town. The session concluded with surveys.
3. Metaverse environment
To make the students’ experiences more realistic and engaging, we used Gather Town, which provides a virtual replica of a medical classroom and clinical setting. Students navigated this space using customizable avatars, enabling real-time interactions and simulations. This immersive environment facilitated group discussions and individual AI chatbot interactions, closely mimicking real-world medical scenarios. The doctor-patient role-play exercises utilized an AI chatbot based on NAVER Line (NAVER, Seongnam, Korea) platform, which was chosen for its Korean language support, free usage, and potential for personal chatbot development. Students practiced the role of doctors individually, whereas the AI chatbot played the role of patients or patient guardians.
4. Questionnaires
Data were collected at the end of the second session in Gather Town. Students completed three surveys via a link: the science learning motivation questionnaire [9], the patient-centered communication skills survey [10], and the patient-centered communication skills self-assessment survey [10]. The science learning motivation questionnaire measured five factors (a total of 24 items): intrinsic motivation, career motivation, self-determination, self-efficacy, and grade motivation. The questionnaire was modified into sets of questions that measured the students’ motivation to learn with the AI chatbot. While the science learning motivation questionnaire and patient-centered communication skills self-assessment survey were used in their entirety, we selectively used items from the perception questionnaires, excluding those that pertained to content not covered by the students' curriculum. This selection process ensured that the survey items were relevant to the students’ current levels of knowledge and experience, thus providing more accurate and meaningful data for our study.
The patient-centered communication skills survey self-evaluated six domains: relationship building, information gathering and information provision, decision-making, enabling disease- and treatment-related behavior, and responding to emotions. Among the six skills, the research team selected five question categories to measure students’ perceptions towards the use of AI chatbots in PCC skill development (24 items in total).
5. Statistical analysis
The collected data were analyzed using descriptive statistics and correlation analyses. Descriptive statistics were used to calculate the mean and standard deviation of each survey factor. Correlation analyses were used to examine the relationships among learning motivation factors, PCC skills, and students’ perceptions towards the use of AI chatbots in medical education.
RESULTS
1. Descriptive statistics
This study examined three key aspects related to medical AI chatbots: motivation for the use of AI chatbots, self-assessment of PCC skills, and perception towards AI chatbots at different interaction stages.
In the assessment of motivation, career-related goals emerged as the primary driver of medical AI chatbot use, with a mean score of 16.89 (standard deviation [SD], 2.79). Conversely, grade-improvement motivation had the least influence, scoring 14.74 (SD, 3.38). Thus, students were more inclined to use AI chatbots for long-term professional development than that for immediate academic performance (Fig. 5).
In self-assessments of PCC skills, participants rated themselves highest in providing information and fostering the relationship (mean score, 3.48 for both; SD, 0.73 and 0.70, respectively). The gathering information skill received the lowest self-assessment scores (mean score, 3.26; SD, 0.90), indicating a potential area for improvement in chatbot-based PCC learning (Fig. 6).
In the perception survey, the gathering information stage had the highest mean score (28.22), indicating that students generally considered the process of collecting data an efficient part of their AI chatbot learning. The second highest score was obtained for the fostering the relationship stage (21.59), implying that the students found fostering the relationship was viable with AI chatbot. In contrast, the disease and treatment behavior confirmation stage had the lowest mean score (7.19), with a smaller standard deviation (1.36), indicating a more uniform perception of this stage but with less overall impact than the other stages (Fig. 7).
2. Correlation of AI chatbot learning motivation, AI chatbot-based PCC skills, and perceptions towards medical AI chatbots
The table above presents the correlation coefficients among the three key variables related to AI chatbot usage: self-assessment of AI chatbot-based PCC skills, AI chatbot-based PCC learning perception, and AI chatbot-based PCC learning motivation. Correlation coefficients were calculated to assess the strength and direction of the relationships between these variables (Table 1)
AI chatbot-based PCC learning motivation and perception had a strong positive correlation, with a correlation coefficient of 0.72. This high correlation indicated that the perception of learning facilitated by an AI chatbot is closely associated with users' motivation to learn with the AI chatbot.
AI chatbot-based PCC learning motivation and self-assessment of AI chatbot-based PCC skills showed a correlation coefficient of 0.54, indicating a moderately positive relationship. This suggests that effective practice of PCC skills through AI chatbots can be linked to increased motivation and positive learning experiences.
AI chatbot-based PCC learning perception and self-assessment of AI chatbot-based PCC skill development exhibited a positive correlation, with a coefficient of 0.68. This indicated a moderately positive relationship, suggesting that enhancements in PCC skills via AI chatbots may be associated with a greater perception of learning facilitated by the chatbot.
These correlations provided valuable insights into the interconnectedness of these variables and highlighted the potential impact of AI chatbot technologies on PCC skill acquisition and educational outcomes.
3. Correlation of AI chatbot learning motivation with the perception towards the use of AI chatbots in PCC skill development
Analysis of the correlations between motivation factors and perception variables provided significant insights into how motivation influenced students' perceptions (Table 2).
Intrinsic motivation was strongly correlated with all perception variables, particularly with the third stage of providing information (r=0.580, P<0.01). This may reflect the importance of intrinsic motivation in shaping student engagement and perceptions across different stages of the learning process. Career motivation also showed strong correlations, particularly with the second stage of gathering information r=0.595, P<0.01) and the third stage of providing information (r=0.612, P<0.01), indicating that career-oriented goals may be closely linked with how students perceive and engage with different learning stages. The relatively lower correlation of self-efficacy with the stage of responding to emotion (r=0.197) suggests that self-efficacy may have less influence on this particular perception, potentially indicating the need to supplement this stage by providing more specific practice in responding to patients’ emotions.
4. Correlation of AI chatbot learning motivation with self-assessment of AI chatbot-based PCC skills
To explore the relationship between medical students' motivation to use AI chatbots for learning and their self-assessment of PCC skills, a bivariate correlation analysis was conducted to determine the strength and significance of these relationships (Table 3).
Analysis of the correlation between learning motivation and PCC skills among medical students revealed a moderately positive correlation (r=0.54). The strongest correlation was observed between “My career will involve AI chatbots”, and “I gave the attention my patient needed to the patient’s feelings and emotions” (r=0.55), indicating that a specific career motivation factor is closely related to responding to emotion skills such as active listening. Conversely, the weakest correlation was found between “Learning AI chatbot is interesting”, and “I spent enough time with my patient” (r=0.32), implying that initial learning interest may have less of an impact on specific PCC skills such as nonverbal communication.
5. Correlations of perception towards the use of AI chatbots in PCC skill development and self-assessment of AI chatbot-based PCC skills
This study found strong positive correlations between medical students’ perceptions of AI chatbot effectiveness and their self-assessed PCC skills. Key relationships included chatbot perceptions in fostering the relationship with providing information (r=0.796) and responding to emotions (r=0.729) skills. Gathering information perceptions correlated highly with the providing information skill (r=0.795) (Table 4). These findings imply that students who perceived AI chatbots as effective in patient communication also tended to rate their own PCC skills highly.
DISCUSSION
This study examined three key aspects related to medical AI chatbots: motivation for the use of AI chatbots, self-assessment of PCC skills, and perception towards AI chatbots at different interaction stages. We found that students are primarily motivated by long-term career goals while learning with AI chatbots, as indicated by a higher mean score for career-related motivation (mean score, 16.89; SD, 2.79) than that for grade-improvement motivation (mean score, 14.74; SD, 3.38). Thus, integrating AI chatbots into medical education can align with students' professional aspirations, thereby enhancing their intrinsic motivation. To enhance intrinsic motivation among medical students, the integration of AI chatbots into medical education should focus on aligning learning experiences with students’ long-term career goals. For example, chatbots can be programmed to present various patient profiles, including different medical conditions, communication challenges, and cultural backgrounds. These simulations would encourage students to apply their communication skills while receiving instant feedback from the chatbot, which could improve their competence and confidence. By continuously supporting students through more in-depth interactions, AI chatbots can foster a sense of purpose, making the learning experience more meaningful and intrinsically motivating.
Regarding the development of PCC skills, students rated their abilities for providing information and fostering the relationship with patients the highest, both scoring an average of 3.48 on a five-point scale. This highlights the effectiveness of AI chatbots in helping students practice PCC skills in a safe and controlled setting that mirrors real-world clinical interactions. However, the relatively low score for the gathering information skills (mean score, 3.26; SD, 0.90) indicates the need to enhance scenarios that challenge students to collect patient data more effectively.
The perception of AI chatbots varied across different stages of the interactions. The highest scores were observed in the information collection stage, with a mean score of 28.22 out of 30 points, indicating that students found AI chatbots particularly useful for gathering and organizing information to support their clinical decision-making. Conversely, the lowest scores were observed in the enabling disease and treatment stage (mean score, 7.19; SD, 1.36), suggesting that these areas may require more sophisticated interaction designs to enhance students’ clinical reasoning skills. These designs may include complex decision-making scenarios with ambiguous symptoms or conflicting patient information, prompting critical thinking and differential diagnosis prioritization. The branching dialogue design plays a crucial role in this context. A branching dialogue design refers to a structure that allows multiple decision-making paths, enabling learners to explore various outcomes based on their choices [11]. This approach helps students synthesize clinical data in diverse scenarios. Real-time feedback after each scenario can improve reasoning processes with AI-driven analytics, tracking performance, and personalized learning paths. Multi-stage case studies simulating patient condition progression and collaborative problem-solving scenarios can further enhance long-term team-based clinical reasoning skills [12]. These strategies can create a more realistic learning environment and better prepare students for real-world clinical decision-making.
The correlation analysis further emphasized the interconnectedness of motivation, perception, and skill development. A strong positive correlation (r=0.72) was observed between the motivation to use AI chatbots and the perception of their effectiveness in learning, indicating that motivated students are more likely to perceive chatbots as beneficial educational tools. Additionally, the moderate correlation between motivation and self-assessed PCC skills (r=0.54) suggests that students motivated to use AI chatbots tended to rate their PCC skills more favorably. Similarly, a positive relationship (r=0.68) between students' perceptions of the chatbot and their self-assessed PCC skills indicates that enhancing students' perceptions of AI tools could lead to better educational outcomes.
These findings have important implications for medical education. First, the integration of AI chatbots as training tools can bridge the gap between theoretical learning and practical applications by providing students with realistic and repeatable scenarios to develop their PCC skills. The strong association between career motivation and the use of AI chatbots suggests that these tools can also align with students' professional development goals and foster a more engaged and motivated learning environment. Furthermore, the positive correlation between student perceptions of AI chatbot effectiveness and their PCC skills underscores the importance of designing relevant and engaging chatbot interactions.
Furthermore, the integration of AI chatbots into clinical performance examination (CPX) preparation can provide students with a powerful and effective tool for individualized CPX preparation. The findings of this study showed that students rated their ability to provide information and foster relationships highly, suggesting that AI chatbots can effectively simulate real CPX situations. In addition, the high motivation related to students' long-term career goals indicates that AI chatbots are likely to motivate students to engage in CPX preparation tailored to their individual learning needs by providing customized practice opportunities based on their strengths and weaknesses. Moreover, the real-time feedback feature can significantly enhance the efficiency of CPX preparation. The high perception of AI chatbots in the information collection stage indicates that students can effectively hone their clinical decision-making skills. This can directly contribute to improving the accuracy of history-taking and decision-making abilities, which are crucial elements of the CPX. Lastly, the relatively lower scores in the enabling disease and treatment related behavior stage demonstrate the potential for improvement in AI chatbots. By strengthening this area through complex decision-making scenarios or branching dialogue designs, students can prepare more comprehensively for all stages of the CPX. Thus, AI chatbots are innovative and effective tools for CPX preparation. They can simulate real clinical situations, provide personalized learning experiences, and enable continuous improvement through immediate feedback. This will allow students to prepare for the CPX with more confidence and ultimately contribute to producing medical professionals with better communication skills and clinical abilities.
Although this study presented promising results, it had several limitations. The one-group post-test-only design limited the ability to establish causality between AI chatbot use and improvements in PCC skills. Future research should incorporate pre-post-test designs with control groups to provide more robust evidence of the impact of AI chatbot training. Additionally, the reliance on self-reported data may have introduced bias, since students may have overestimated or underestimated their abilities. Objective measures such as evaluations by instructors or the use of standardized patients would provide a more accurate assessment of PCC skills. The limited sample size and duration of the study may have also affected the generalizability of the findings. Future research should involve larger and more diverse populations and extend over longer periods to assess the long-term impact of AI chatbot training on medical education. In conclusion, this study demonstrated the potential of conversational AI chatbots to enhance PCC skills and learning motivation among medical students. AI chatbots represent a valuable addition to the traditional medical education curriculum by providing interactive, scalable, and personalized training.
Acknowledgements
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2020S1A5B5A16084033).