Start Submission Become a Reviewer

Reading: Development of a Three-Dimensional (3D) Virtual Reality Apprenticeship Program (VRx) for Tra...

Download

A- A+
Alt. Display

Original Research

Development of a Three-Dimensional (3D) Virtual Reality Apprenticeship Program (VRx) for Training of Medication Safety Practices

Authors:

Wing Yee Tsang,

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG
X close

Petrina Fan,

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG
X close

Shakunthala D/O Hans Raj,

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG
X close

Zee Jian Tan,

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG
X close

Irvin Yong Yang Lee,

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG
X close

Ivan Boo,

Serious Games Asia, 8 Burn Road, #15-13, Trivex, Singapore 369977, SG
X close

Kevin Yi-Lwern Yap

Department of Pharmacy, Singapore General Hospital, SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9, Singapore 168582, SG; Department of Public Health, School of Psychology and Public Health, La Trobe University, Melbourne, Victoria 3086, AU
About Kevin Yi-Lwern

k.yap@latrobe.edu.au

X close

Abstract

Background: Medication errors is a top concern for patient safety globally. Healthcare staff need to be competent in skillsets in medication safety practices. COVID-19 has negatively impacted programs with on-the-job training. Virtual reality (VR) provides an attractive way to conduct such training. We developed a virtual apprenticeship program (called VRx) for the training of medication safety practices.

Methods: VRx was developed in a three-dimensional virtual environment whereby trainees had to complete a series of key competency tasks in prescription processing, pharmaceutical calculations, and preparing, picking, packing and dispensing medications. Pilot testing involved two phases – an alpha-test (Ph1) with 30 pharmacy staff/trainees and a roll-out (Ph2) to 43 new hires at our institution. Participants rated their awareness of VR and perceptions of their training experience through an online self-administered questionnaire. Descriptive statistics and Chi-squared/Fisher’s Exact tests were used for analysis.

Results: Over half of participants were slightly aware of VR (Ph1 = 63.3%, Ph2 = 67.4%). One-quarter was aware of and had used VR (Ph1 = 26.7%, Ph2 = 25.6%). Majority agreed that VRx was easy to use (Ph1 = 83.3%, Ph2 = 72.1%), interactive and fun (Ph1 = 90.0%, Ph2 = 79.1%), and helped them develop prescription processing skills (Ph1 = 90.0%, Ph2 = 86.0%). More participants who were slightly aware of VR indicated that VRx was easy to use (94.7% versus 33.3% not aware of VR, p = 0.038, Ph1). Over half felt that the time spent on VRx training was acceptable, and that VRx should be part of their training (Ph1 = 80.0%, Ph2 = 67.4% each). More participants who were aware of and had used VR indicated that VRx should be part of their training (100% versus 55.2% slightly aware, p = 0.007, Ph2).

Conclusion: Overall, pharmacy staff/trainees found VRx to provide a realistic, engaging and interactive experience to acquire proficiencies in medication safety practices. VRx can complement the training and assessment needs of pharmacy staff/trainees at our institution.

How to Cite: Tsang WY, Fan P, Raj SDH, Tan ZJ, Lee IYY, Boo I, et al.. Development of a Three-Dimensional (3D) Virtual Reality Apprenticeship Program (VRx) for Training of Medication Safety Practices. International Journal of Digital Health. 2022;2(1):4. DOI: http://doi.org/10.29337/ijdh.45
148
Views
40
Downloads
  Published on 02 Jun 2022
 Accepted on 20 May 2022            Submitted on 21 Feb 2022

Introduction

Medication errors may result in patient harm and is a top concern for patient safety, globally [1, 2]. These errors are generally due to failures in the medication use process, which include prescribing, clinical review, dispensing and medication administration errors. Becker’s Hospital Review had previously reported that medication errors during COVID-19 were due to hard-to-read medication labelling, lack of training in medicine barcodes, inaccurate weight-based dosing, wrong drug from automated dispensing, and missed doses due to personal protective equipment (PPE) rationing [3]. In fact, guidance for medication optimization and simplification due to COVID-19 had been issued from US, Australia, and international sources [4].

Healthcare practitioners and staff need to be competent in skillsets in medication safety practices. The World Health Organization’s Patient Safety Curriculum Guide [5] has suggested several approaches to teach and assess medication safety – including problem-based learning, case scenarios, role-plays and Objective Structured Clinical Examinations (OSCEs). Topics include prescribing, drug administration and calculations, identifying medication errors, contributing factors and prevention strategies, and performing medication/allergy history-taking, among others. In hospital institutions, medication safety competency training and assessments are generally conducted via e-learning modules, clinical case reviews, demonstration and practice using dummy mannequins, simulations with standardized patient actors or actual patients, and narratives/walkthroughs by the learner [6, 7, 8, 9, 10, 11]. COVID-19 has negatively impacted programs with on-the-job training, and hands-on components cannot be easily delivered through online programs due to social distancing, infection control and prevention measures. Clinical clerkships and attachments were either suspended, or face-to-face training of essential skills kept to small group sizes and repeated frequently, increasing time, effort and manpower [12]. Several challenges were amplified during COVID-19, which included high labor-intensiveness (additional workload and manpower shortage to deal with COVID-19, and a high burden on trainers due to the need for more frequent training sessions to smaller group sizes), and resource-intensiveness (depleting medical supplies/personal protective equipment through frequent use), thus leading to disruptions in workflow processes, difficulties in providing timely and personalized feedback to learners, and the potential for direct patient harm. These labor and resource challenges faced during COVID-19 are not sustainable long-term. Nevertheless, skills-based competency trainings and assessments, especially in regard to medication safety are critical, and need to be assessed to an acceptable practice standard in order not to compromise patient safety.

Virtual reality (VR) is an attractive way to enable such training and assessment. The use of VR in healthcare education is not a new idea. VR technology stimulates multiple senses through the use of interactive simulations to engage and immerse users in a computer-generated virtual environment that generates feelings similar to real world events and objects [13, 14]. Immersive VR, which is what is being considered in this context, usually involves the use of three-dimensional (3D) head-mounted displays and can elicit high levels of immersion and presence (i.e. the psychological illusion of being in the virtual environment) [14]. There are several advantages to the use of VR for healthcare education [15, 16], such as developing communication and higher-order skills (e.g. analyzing and problem-solving skills), enabling students to learn at their own time and pace, allowing students to put themselves in the shoes of the healthcare professional to develop the skills and comfort towards patient interactions, and it is ethically justifiable as it can minimize the risks of medical/medication errors and complications in real-life patients.

VR technology has been used in various healthcare settings for patient care, such as for physical and stroke rehabilitation [17, 18], cardiopulmonary resuscitation [19], cerebral palsy [20], pain and anxiety [21, 22, 23], pediatric oncology [14]; as well as in medical education to teach anatomy [24, 25], communication skills and empathy [26, 27], dental and surgical education [28, 29, 30], and nursing education [31, 32], among others. Applications in pharmacy include the teaching of content-based knowledge in the medical sciences (e.g. anatomy, physiology, pharmacology, pharmacotherapy) [33, 34, 35], health communication and compounding [36, 37], and as a method of lesson engagement in long distance problem-based learning and team-based learning [38]. In a meta-analysis comparing the pass rates of medical students undergoing VR training versus traditional modes of learning, the researchers reported that the VR group performed significantly better than the traditional education group (odds ratio, OR = 1.85, 95% confidence interval, 95% CI: 1.32–2.58) [39]. In their subgroup analysis, they also found that postgraduate (OR = 1.64, 95% CI: 1.13–2.39) and hospital residents (OR = 15.73, 95% CI: 2.35–105.04) in the VR group performed better than their respective counterparts in the traditional education group. The researchers suggested that VR could be used to train complex skills and specialized knowledge in the medical field. However, to our knowledge, VR technology has not been explored as part of on-the-job competency training and assessment for pharmacy staff and trainees, which plays a role towards practice licence registrations. The closest studies that we could find in the literature were the use of virtual patient avatars for pharmacist pre-registration training, but their content was mainly focused on improving clinical/pharmacotherapy knowledge [40, 41, 42], and not specifically catered towards medication safety. The use of virtual patient avatars has been suggested to be able to provide multi-sensory feedback for learners to learn the skills and applications needed in different healthcare contexts [43]. When virtual avatars are combined with social VR platforms (i.e. platforms that use VR for interaction, collaboration and interpersonal communication), it allows the conveyance of non-verbal cues (e.g. facial expressions, posture) that can lead to better coordination and understanding of the communication to users [44]. However, the users’ perceptions of the social interactions in these social VR platforms are limited by the fidelity of the avatar’s appearance and behavior.

In light of the disruptions in clinical clerkships and attachments and a rapid uptake of digital technologies in education during COVID-19, a VR application for training of medication safety practices was deemed to be attractive at our institution. Hence, as part of our institution’s quality improvement initiative, we decided to embark on a virtual apprenticeship program (called VRx) to train our pharmacy staff and trainees (i.e., pharmacists, pre-registration pharmacists, pharmacy technicians, pharmacy assistants, pharmaceutical science students on hospital attachments) on medication safety practices during pharmacy operations. Our primary objective is to create and develop a VR platform incorporating a virtual patient encounter (VPE) to train key tasks in 2 main operational areas:

  1. Prescription (Rx) validation and processing – Checking for legality, authenticity and information completeness and accuracy of prescription/medication orders; identifying medication-related problems (e.g., inappropriate prescribing); performing pharmaceutical calculations and ensuring that drug label information are correct; and
  2. Medication processing – Error-free picking, packing and labelling of medications; ensuring appropriate packaging according to safety, stability, patient and legal requirements (e.g., light-sensitive, temperature-sensitive, expiry, etc).

A secondary objective is to evaluate the usefulness, user-friendliness, and engagement of the VRx program through pilot testing of the VPE among pharmacy staff and trainees.

Methods

Development of the VRx program

The VRx program was designed to provide a form of virtual apprenticeship training to pharmacy staff and trainees at our hospital institution, in order to continue training and assessment of medication safety practices without compromising the COVID-19 regulations and restrictions for healthcare professionals and patients. This program was developed in a 3D virtual environment using the Unity 3D engine (Unity Technologies, CA, USA). A series of key competency tasks based on the healthcare professional competency standards set by the Ministry of Health, Singapore, was built around a virtual patient encounter (VPE) in an outpatient pharmacy. These competency tasks included (Figure 1): (i) validating and processing prescriptions, (ii) checking the accuracy of medication orders, (iii) performing pharmaceutical calculations, (iv) processing medications and labels, (v) preparation of medications, and (vi) picking, packing and dispensing medications. The storyboards of the scenarios were developed using Microsoft Excel and a VR vendor was engaged to develop the virtual environment and VPE through the institution’s procurement process. The main technologies used were a Razer Blade 15 laptop and an Oculus Rift S (Facebook Technologies LLC, USA), which comprised of a 3D head-mounted display/headset worn by users to move around in the immersive virtual pharmacy environment and hand-held controllers to perform the key tasks. The graphics were bought from the Unity engine’s asset store or designed using 3DS Maya (Autodesk Inc., CA, USA) and Blender (Blender Foundation, Amsterdam, Netherlands). The backend database was developed using an open-source software (Learning Locker, Learning Pool, USA). The player’s responses and performance in the tasks were recorded real-time via a training dashboard, which could then be used for review and feedback. Programming scripts used to develop the virtual environment, VPE and training dashboard included a combination of python, C++ and javascripts.

Figure 1 Key competency tasks in VRx.
Figure 1 

The key competency tasks incorporated in the virtual patient encounter as part of the VRx virtual apprenticeship program.

Pilot testing of VRx

Pilot testing of VRx involved two phases. Phase 1 was an alpha-test conducted among thirty pharmacy staff and trainees (n = 30) at our institution from October to mid-December 2020. Participants who were pharmacists or pre-registration pharmacists would play scenarios involving competency tasks (i) and (ii) on validating and processing prescriptions and checking the accuracy of medication orders. On the other hand, pharmacy intern and pharmacy technician participants would play scenarios involving tasks (iii), (iv) and (v) on performing pharmaceutical calculations, processing medications and labels, as well as preparation of medications.

Phase 2 involved a roll-out of VRx to new hires at our institution as part of the department’s medication safety training program. All new hires from mid-December 2020 to August 2021 were recruited. All the participants in this phase completed the whole VPE, which involved all six competency tasks.

The VRx training session was conducted at the workplace during work hours in an office meeting room. Due to COVID-19 precautionary measures and isolation requirements taken by the hospital institution, training was conducted by one of the authors (SHR, i.e. trainer) with only one participant at any one time and following national guidelines on social distancing requirements. Each training session lasted approximately one hour (Phase 1) or two hours (Phase 2). During the session, the participant would be introduced to VRx and the learning objectives of the session. A verbal explanation on how to use the head-mounted display and controllers would also be provided. Participants would then have an opportunity to familiarize themselves with the equipment by navigating the virtual environment through a short tutorial, before embarking on the VPE to complete the competency tasks. Feedback was provided real-time within the VPE scenarios. Additionally, during debriefing, the trainer would provide the participant with feedback on how well they did and reiterate the learning points from their mistakes.

For both phases, a self-administered online questionnaire was created with the government’s survey platform and provided to the participants at the end of the sessions. The questionnaire was adapted from a local study by Chua and colleagues who assessed the perceptions and attitudes of our local population towards VR [45], and comprised of 12 questions, which obtained information about the participants’ demographics (job position, familiarity with VR) and their perceptions regarding their VRx experience. Participants rated their perceptions on a 5-point agreement scale (1-Strongly disagree, 2-Disagree, 3-Neutral, 4-Agree, 5-Strongly agree) on whether the: (a) VRx program was user friendly, (b) time spent was acceptable, (c) training helped to develop prescription processing skills, (d) learning was more interactive and fun than traditional on-the-job training, (e) use of VRx should be included as part of training, and (f) participants enjoyed the VRx training experience. In addition, free-form text boxes were provided for participants to provide feedback about the VRx program.

Data analysis

Descriptive statistics were used to report on the participant demographics and the proportions of participants regarding their perceptions on their VRx experience. The data from Phase 1 and Phase 2 studies were analyzed separately. The agreement scales were collapsed into three categories for analysis: disagree (1-Strongly disagree, 2-Disagree), neutral and agree (4-Agree, 5-Strongly agree). Pearson Chi-squared tests were used to determine the associations between the demographics and the perceptions of the participants. Where the expected frequencies were less than 5 for more than 20% of the cells, the Fisher’s Exact test was used instead. All analyses were done using IBM SPSS Statistics software (version 27) (IBM, NY, USA). Statistical significance was defined as p < 0.05. Exploratory analysis of the qualitative feedback provided by the participants was done by generating word clouds in the NVivo 12 Plus software (QSR International, MA, USA) using the top 1,000 most frequent words with a minimum word length of 3 letters, grouped by synonyms. Thematic analysis was conducted to identify the strengths of the VRx program, difficulties faced by the participants, and suggestions for improvement in relation to the participants’ learning experiences, technological aspects (e.g. headset and controllers) and the VPE scenarios.

Results

Pilot testing: Phase 1 (n = 30)

Thirty staff and trainees (40.0% pharmacists, 16.7% pre-registration pharmacists, 13.3% pharmacy interns, 30.0% pharmacy technicians) participated in the VRx program (Table 1). All of them (100%) completed the online survey. Majority were slightly aware of VR technologies (63.3%), although a quarter (26.7%) were aware of and had used VR before. Over two-thirds (83.3%) indicated that the VRx training program was easy to use. There were more participants who were aware of VR who indicated that VRx was easy to use (Table 2). In fact, the proportion of participants who agreed that VRx was easy to use and were slightly aware of VR (18/19, 94.7%) was significantly higher than those who were not aware of VR (1/3, 33.3%, p = 0.038). Large proportions also indicated that the VRx training helped them develop the skills needed to process prescriptions (90.0%) and that this program should be part of their training curriculum (80.0%). Despite doing their training sessions during work hours, majority indicated that the time spent on the virtual training (~1 hour) was acceptable (80.0%) and that the session made their learning experience more interactive and fun compared to traditional on-the-job training (90.0%). Almost all (96.7%) enjoyed the VRx training experience, and nearly three-quarters (73.3%) would like to see the use of other technology tools in their future training. There were significantly more pharmacists and pre-registration pharmacists (15/17 who completed key tasks (i) and (ii), 88.2%) than pharmacy interns and pharmacy technicians (7/13 who completed key tasks (iii), (iv) and (v), 53.8%) who were more open to the use of other technology tools in their training (p = 0.049).

Table 1

Demographics and perceptions of participants on the VRx program.


PARTICIPANT PARAMETERS NUMBER OF PARTICIPANTS (%)

DEMOGRAPHICS PHASE 1 (n = 30) PHASE 2 (n = 43)

Job Position:

    Pharmacy intern 4 (13.3) 5 (11.6)

    Pharmacy technician 9 (30.0) 22 (51.2)

    Pre-registration pharmacist 5 (16.7) 12 (27.9)

    Pharmacist 12 (40.0) 4 (9.3)

Familiarity with virtual reality (VR):

    Not aware of VR 3 (10.0) 3 (7.0)

    Slightly aware of VR 19 (63.3) 29 (67.4)

    Aware of and have used VR 8 (26.7) 11 (25.6)

Participant perceptions on the VRx program

The VRx training I have experienced is easy to use

    Disagree 0 (0) 2 (4.7)

    Neutral 5 (16.7) 10 (23.3)

    Agree 25 (83.3) 31 (72.1)

The amount of time I spent on the VRx training is acceptable to me

    Disagree 2 (6.7) 2 (4.7)

    Neutral 4 (13.3) 12 (27.9)

    Agree 24 (80.0) 29 (67.4)

The VRx training helped me develop skills to process prescriptions

    Disagree 0 (0) 0 (0)

    Neutral 3 (10.0) 6 (14.0)

    Agree 27 (90.0) 37 (86.0)

The VRx training makes the learning experience more interactive and fun compared to on-the-job training

    Disagree 0 (0) 2 (4.7)

    Neutral 3 (10.0) 7 (16.3)

    Agree 27 (90.0) 34 (79.1)

The use of VRx to learn prescription processing skills should be part of my training

    Disagree 0 (0) 0 (0)

    Neutral 6 (20.0) 14 (32.6)

    Agree 24 (80.0) 29 (67.4)

Overall, I enjoyed the VRx training experience

    Disagree 0 (0) 0 (0)

    Neutral 1 (3.3) 8 (18.6)

    Agree 29 (96.7) 35 (81.4)

I would like to see the use of other technology tools in my future training

    No 8 (26.7) 16 (37.2)

    Yes 22 (73.3) 27 (62.8)

Pilot testing: Phase 2 (n = 43)

Forty-three pharmacy staff and trainees (9.3% pharmacists, 27.9% pre-registration pharmacists, 11.6% pharmacy interns, 51.2% pharmacy technicians) from our institution participated in the study, with a 100% response rate for the survey (Table 1). Similar to Phase 1, majority of participants were only slightly aware of VR (67.4%), followed by those who were aware of and had used VR (25.6%) and those who were not aware (7.0%). Nearly three-quarters (72.1%) felt that the VRx training was easy to use and a similar proportion indicated that the training made their learning experience more interactive and fun (79.1%). On the other hand, over three-quarters of participants indicated that they enjoyed their VRx experience (81.4%) and felt that the training helped them develop prescription processing skills (86.0%). Equal proportions (67.4% each) felt that the amount of time spent on the VRx training (~2 hours) was acceptable and that VRx should be part of their training on prescription processing skills. In the latter group who indicated that VRx should be part of prescription processing skills training, the proportion of participants who were aware of and had used VR (11/11, 100.0%) was significantly higher than those who were only slightly aware of VR (16/29, 55.2%, p = 0.007) (Table 2). Over half (62.8%) would like the use of other technology tools in their future training.

Table 2

Proportion of participants who agreed with the perception statements about the VRx program, categorized based on their awareness of VR.


STATEMENTS ON VRX PROGRAM NUMBER OF PARTICIPANTS WITHIN THEIR VR FAMILIARITY CATEGORY WHO AGREED WITH STATEMENTS ON VRX TRAINING (%) P-VALUEa

NOT AWARE OF VR SLIGHTLY AWARE OF VR AWARE OF AND HAVE USED VR

Phase 1 (n = 30)

The VRx training I have experienced is easy to use 1/3 (33.3)** 18/19 (94.7)** 6/8 (75.0) 0.023*

The amount of time spent on the VRx training is acceptable to me 1/3 (33.3) 17/19 (89.5) 6/8 (75.0) 0.112

The VRx training helped me develop skills to process prescriptions 3/3 (100.0) 16/19 (84.2) 8/8 (100.0) 0.381

The VRx training makes the learning experience more interactive and fun compared to on-the-job training 3/3 (100.0) 16/19 (84.2) 8/8 (100.0) 0.381

The use of VRx to learn prescription processing skills should be part of my training 3/3 (100.0) 14/19 (73.7) 7/8 (87.5) 0.471

Overall, I enjoyed the VRx training experience 3/3 (100.0) 18/19 (94.7) 8/8 (100.0) 0.741

Phase 2 (n = 43)

The VRx training I have experienced is easy to use 2/3 (66.7) 19/29 (65.5) 10/11 (90.9) 0.557

The amount of time spent on the VRx training is acceptable to me 2/3 (66.7) 18/29 (62.1) 9/11 (81.8) 0.748

The VRx training helped me develop skills to process prescriptions 2/3 (66.7) 25/29 (86.2) 10/11 (90.9) 0.561

The VRx training makes the learning experience more interactive and fun compared to on-the-job training 2/3 (66.7) 21/29 (72.4) 11/11 (100.0) 0.038*

The use of VRx to learn prescription processing skills should be part of my training 2/3 (66.7) 16/29 (55.2)** 11/11 (100.0)** 0.026*

Overall, I enjoyed the VRx training experience 2/3 (66.7) 23/29 (79.3) 10/11 (90.9) 0.557

a Significance testing was conducted using Pearson Chi-square test. Fisher’s Exact test was conducted when more than 20% of cells had expected frequencies of less than 5.

* Statistical significance was observed among the 3 groups of participants.

** Statistical significance was observed between the 2 groups of participants.

Qualitative feedback

Qualitative feedback indicated that the strengths of the VRx program were in relation to providing an authentic learning experience for trainees (Figure 2a). Besides providing a realistic simulation of real-life scenarios and work processes that was interactive, fun and engaging to the participants, VRx allowed them to have a hands-on experience in a safe environment without the stress of catering towards the pressures in a physical pharmacy. The program was not only able to help the participants reinforce their knowledge and concepts, and familiarize with the picking and packing process, but it was also able to help minimize disruptions to the workplace and workflow processes. Through immediate feedback and a more structured way of training, the participants reflected that they were more aware of, and able to learn from their mistakes on-the-spot, as well as found it easier to understand, remember and apply the concepts to their work.

Figure 2 Word clouds of strengths, difficulties and improvement
                        suggestions for VRx.
Figure 2 

Word clouds from participant feedback reflecting the strengths, difficulties faced and suggestions for improvement of the VRx training program.

In terms of difficulties faced during the sessions, many of them found difficulty in navigating the virtual environment and scenarios as they were not familiar with the functionalities of the hand-held controllers and buttons (Figure 2b). They felt that they needed time and practice to adjust and adapt to the controls. In addition, some of them felt that the VR headset was heavy and that their line of vision was not optimal due to their height, which caused strain on the eyes and neck. Wearing the headset for long periods of time also caused dizziness and nausea in some participants. Furthermore, some participants felt that some parts of the instructions were unclear or vague, which resulted in them not knowing what to do in certain scenarios.

Suggestions for improvements were mainly in relation to the technological aspects, as well as on the training (Figure 2c). Technological enhancements included refining the sensitivity, use and functionalities of the controllers, providing a help screen for users to look up the controller functions, use of the gaze function to transition between the scenarios; as well as improving the resolution of the images and fonts. In regard to the training sessions, it was suggested that these sessions could be broken down into smaller segments of shorter durations, as well as adding more training scenarios to include a wider variety of drugs with different methods of packing. The participants also suggested to reduce some of the repetitive steps in picking and packing, skip some of the intuitive steps (e.g. opening and closing of bottles) and shorten the time needed to transition between steps in the tasks. Other suggestions included having a hybrid mode of both VR and real-life hands-on activities, simplifying the instructions, providing a briefing to the VR technology and controllers, tailoring some tasks to specific departments and the need to consolidate the learning for trainees at the end of the program.

Discussion

An in-house virtual apprenticeship program called VRx has been developed as part of our institution’s medication safety training. Based on the healthcare professional competency standards set by the Ministry of Health, Singapore, the VRx program focuses on competency skills training to complement on-the-job training for pharmacy staff and trainees during the COVID-19 pandemic. The activities in VRx targeted 6 medication safety domains according to the Pharmaceutical Care Network Europe guidelines (Table 3) [46]. These activities included preventing potential problems with the lack of pharmacotherapy effects on treatment effectiveness; preventing treatment safety issues resulting from missing/incomplete drug information; identification of drug-related problems related to the drug (e.g. selection of the drug, drug form, dose schedule and treatment duration); and identification of drug-related problems related to the logistics of the prescribing and dispensing process (e.g. identifying errors and legal issues with the prescription and perform an intervention where appropriate; and affixing appropriate auxiliary medication labels so that drug effectiveness is not compromised), among others. The advantage of this program is that the VR component is able to provide an authentic and safe environment for staff and trainees to learn about medication safety practices through responsible decision-making, but yet can still learn from their mistakes without experiencing the severe negative consequences that may occur in real-life patient care. Specifically, staff and trainees would be provided with immediate feedback on their mistakes, who would then need to acknowledge in order to proceed with the next step or stage.

Table 3

Summary of how VRx trains learners on medication safety aspects on prescription and medication processing.


MEDICATION SAFETY DOMAINS* ACTIVITIES IN VRX THAT TARGET THE SAFETY DOMAIN

Treatment effectiveness:
  • – There is a (potential) problem with the (lack of) effect of the pharmacotherapy.
  • Learner has to perform pharmaceutical calculations on the correct number of insulin pens to dispense to the patient based on the prescribed doses, frequencies and durations.
  • Learner has to factor in the shelf lives / expiry dates of the drugs to be dispensed, as this can impact on drug effectiveness and safety.

Treatment safety:
  • – Patient suffers, or could suffer, froman adverse drug event.
  • Learner has to check if thereis any incomplete drug information on the prescription (e.g. missing drug route).
  • Learner must not assume the drug route, as some drugs, if administered by the wrong route, willaffect patient safety (e.g. intravenous versus intramuscular injections; eye-drops versus ear-drops).

Drug selection:
  • – The cause of the drug-related problem can be related to the selection of the drug.
  • Learner has to pick all the drugs in the prescription correctly (up to 8 drugs in one prescription) without any errors (e.g. picking the wrong drug).
  • Learner has to identify specific drug-related problems, if any, during picking-and-packing (e.g. identifying if there is a drug that has no indication).

Drug form:
  • – The causeof the drug-related problem is related to the selectionof the drug form.
  • Learner has to select the correctdrug form to pick and pack (e.g. mometasone comesin different drug forms, such as nasal spray, cream, ointmentand tablet).

Dose selection:
  • – The cause of the drug-related problem can be related to the selection of the dose schedule.
  • Learner has to correlate the drug doses on the prescription to the appropriate drug strengths of tablets to dispense to patient with appropriate instructions on medication label.
  • E.g. Warfarin is a drug that may comprise of complex dosage schedules. The tablets come in 1 mg, 3 mg and 5 mg strengths. Learner has to correlate and pick the appropriate tablet strengths for dispensing, for example:
    • – Dose schedule: 5 mg dose for 4 times a week (on Mon, Wed, Fri, Sun) and 4 mg dose for 3 times a week (on Tues, Thurs, Sat)
    • – Number of warfarin tablets to be dispensed: four 5 mg tablets, three 3 mg tablets and three 1 mg tablets per week

Treatment duration:
  • – The cause of the drug-related problem is related to the duration of treatment.
  • For partial prescriptions that have been previously dispensed, learner has to interpret the prescription and identify the balance supply of drugs for the patient.
  • For non-oral drugs (e.g. nasal spray), the learner has to correlate the balance doses to the number of units (i.e. spray canisters) to be dispensed.

Dispensing:
  • – Thecause of the drug-related problem can be related tothe logistics of the prescribing and dispensing process.
  • Learner needsto identify any errors and legal issues with the prescriptionand do an intervention, where appropriate, such as clarifying withthe doctor (e.g. missing patient identifiers, missing date ofprescription, missing prescriber signature, possible forgery of prescription).
  • Learner needsto ensure that relevant auxiliary medication labels are affixed ontothe appropriate drug packaging so that drug effectiveness is notcompromised.
  • E.g. Ensure that a “keep refrigerated” or “storeat 2–8°C” label identifier is affixed to appropriate drugitems, so that cold chain supply can be maintained withoutaffecting drug effectiveness.

* Medication safety domains are classified according to the Pharmaceutical Care Network Europe (PCNE v9.1) classification of drug-related problems. Only relevant domains that are incorporated into VRx are included.

The design and development of VRx is based on several pedagogies in simulation and healthcare education. Recent systematic reviews have reported that ~70% of studies have no learning theory as a foundation of the VR interventions [47, 48]. Our project adapts Zhou’s VR Interaction Learning Model [49] (Figure 3) by merging the features of VR-supported technologies with pedagogical frameworks in VR simulations and healthcare education to design and create our VPE scenarios. There are 2 consideration factors when designing the VPE scenarios in order to bring about the intended learning benefits – VR immersion and VR roles. In the VRx program, a semi-immersive desktop VR is used by the trainer to observe how the trainee completes the scenarios, while the trainee undergoes a fully immersive VR experience in the virtual environment, using the VR headset and controllers. On the other hand, the virtual environment is designed to mimic the workplace so that trainees are able to apply and practise their medication safety knowledge. However, unlike actual on-the-job placements which are more unstructured, the competency skills that are needed by the trainee are made explicit as demonstrable task elements in order to scaffold the trainee’s learning in a more structured and obvious manner. The high proportion of participant responses agreeing that VRx helped in prescription processing skills and the qualitative feedback indicate that we are on the right track.

Figure 3 Pedagogical model of VRx following Zhou's Interaction Learning
                    Model.
Figure 3 

Pedagogical model of VRx showing the roles of the VR supported scenarios and the pedagogical frameworks used. Adapted from Zhou’s Interaction Learning Model.

The simulation-based pedagogy that we used to develop VRx was according to de Freitas’ Four-Dimensional Framework (Figure 4) [50], which provided consideration factors in terms of the context of the application, learner specification, mode of representation and pedagogic considerations. Our target population was more clearly defined through the learner specification framework (i.e. students, graduates and pre-registration pharmacists attached to our hospital, polytechnic graduates from pharmacy/ pharmaceutical science specializations, and new hires in our department). Although the levels of technological and manpower resources needed were low to medium (i.e. laptops with VR headset and controllers, and only one trainer needed per session), the levels of fidelity, realism and immersion needed to be medium to high, so that the trainees would be able to apply their knowledge and skills to realistic situations in the VPE and their skillsets could be transferrable to the actual workplace. In this respect, the virtual environment was designed after our hospital’s outpatient pharmacy (prescription and dispensing counters), medication storage and preparation areas (e.g. medication shelves and bins, refrigerators) and compounding areas (for compounding, preparation and labelling of non-sterile products). However, we were conscious that the virtual environment also needed to be generic, so that it could be transferrable to the pharmacy facilities in other hospitals. As such, our virtual pharmacy also adhered to the minimum standard guidelines for pharmacy facilities advocated by the American Society of Health-System Pharmacists [51].

Figure 4 Simulation-based pedagogy used by VRx following de Freita's 4D
                    Framework.
Figure 4 

Simulation-based pedagogy used by VRx following de Freita’s Four-Dimensional (4D) Framework.

The learning activities that were designed in the VPE were aligned to the competency learning outcomes of processing prescription and medication orders, and medication safety practices, which required the trainees to hone their skills in clinical reasoning and decision-making, effective communication, clinical documentation and organization, patient history-taking and patient education, among others. Certain cognitive elements, such as prescription processing and validation, and dose calculations, can be supported by good decision support systems for prescriptions and medication orders so as to enhance medication safety. However, there are limitations in practice settings. For example, the risks of medication errors can still occur due to issues such as missing or incorrect information on the prescription [52, 53] (e.g. missing drug administration routes or incorrect drug forms), wrong calculations, transcriptions and/or correlations of medication doses to their corresponding drug strengths and drug forms [52, 53] (e.g. converting insulin dose units to the number of insulin pens based on type of insulin prescribed), and with extemporaneously compounded medication products [54, 55, 56] (e.g. medications for patients with difficulty swallowing, reconstitution/dilution of injectable medications, pediatric doses of formulations that are not available commercially). Furthermore, clinicians tend to turn off the alert prompts in their clinical decision support systems due to alert fatigue [57], which in turn also impacts medication safety. The whole medication dispensing process depends greatly on behavioral input, whether it be assessing the appropriateness of prescriptions, interpreting prescriptions accurately, finding the appropriate medications to pick, pack and label, or making sure that the medication order is correctly fulfilled. Safe dispensing practices can be supported through highly motivated individuals with a good understanding of medications and dispensing equipment, working in a well-coordinated team [58]. On the other hand, being unaware of or under-estimating particular risks, such as not factoring in shelf lives/expiry dates of drugs; assuming certain drug administration routes/drug forms (e.g. intravenous versus intramuscular injections, or eye-drops versus ear-drops); negligence or lack of attentiveness; fatigue and stress; misinterpreting drug information (e.g. look-alike-sound-alike drugs); inappropriate/unclear communication or documentation; or being governed by inadequate guidelines and/or unclear organizational routines can also predispose to unsafe dispensing practices [58, 59]. As pharmacists are the last line of defence before the patients receive their medications, these skills still need to be instilled in pharmacy staff to ensure that medication errors are minimized throughout the whole medication dispensing process. As such, the team deemed VR to be a suitable technological method of training instruction. Furthermore, it was deemed that face-to-face briefing and debriefing by both the trainer and the trainee’s preceptor would also be needed to reinforce the content learnt through VRx. From the participants’ feedback, we intend to incorporate technical briefings on how to use the VR technologies before the trainees undergo the VPE in the future, so that they can have a more seamless and enjoyable experience.

The Miller’s Pyramid of Clinical Competency framework [60, 61] was used to provide guidance on designing our scenarios to train and assess the trainees’ skillsets (Figure 5). Embedded within the VPE scenarios were different variations of multiple-choice questions (MCQs) and short answer questions for trainees to answer in order to reinforce their knowledge during the training sessions, but could also double-up as quick assessments of the trainee’s knowledge. The VPE targets the skills competency layer of the Miller’s Pyramid by providing a virtual simulation of a standardized patient encounter, so that trainees can apply their knowledge and skills into specific demonstrable tasks that indicate their competencies and practice readiness, in the form of entrustable professional activities (EPAs) [62]. For the EPA of dispensing of medications (Figure 5), the domains of competence (knowledge, skills, attitudes and experiences) that the trainees were required to exhibit were translated into ten competencies (C1-C10 of EPA framework) which they had to be proficient in, and 11 milestones (M1-M11 of EPA framework) which they had to demonstrate. The trainees’ competency in medications dispensing would then be assessed and rated using five progressive levels of learner behavior from pre-entrustable to entrustable. As suggested in the participant feedback, we intend to incorporate a wider variety of VPEs which contain a variation of drugs and different methods of packing, so as to train a broader set of competencies through EPAs.

Figure 5 Education-based pedagogies used by VRx following Miller's Pyramid
                    and EPA frameworks.
Figure 5 

Education-based pedagogies used by VRx following the Miller’s Pyramid of Clinical Competence and Entrustable Professional Activity frameworks.

Our project was aimed at improving the knowledge and competency skillsets in medication safety practices among healthcare professionals, with a beginning focus on pharmacy staff and trainees. By providing a strong base and focus on medication safety through VRx, we aimed to put “Patients at the Heart of All We Do” (part of our organization’s mission), so that our staff would be acknowledged as trusted healthcare professionals who would be able to provide quality services for our patients. In the development of VRx, we wanted to be able to harness appropriate, evidence-based technologies, and not just use the latest technological trends, to train the newer generations of pharmacy professionals. In addition to enhancing engagement and motivation of our staff and trainees through the technology, we wanted VRx to play a dual role of enhancing their digital literacy through the simulation experience. Some of the participants were surprised that our institution’s training program had incorporated such “sophisticated technologies”. Amidst facing some technical difficulties and kinks, the participants were still excited to undergo the VRx experience, as many of them did not have the opportunity to to experience their learning in a virtual simulated environment in pharmacy school. Moreover, when this program was presented to pharmacy managers at the other hospital institutions in SingHealth (one of three integrated healthcare clusters in Singapore’s public healthcare system) [63], they were also inspired to consider adoption of such innovative simulation solutions to train and assess their staff and trainees. Besides the use of a Windows i7 laptop that is compatible with the Oculus VR equipment, VRx can be adopted and incorporated as part of their training programs.

In terms of sustainability, VRx was postulated to be able to decrease the manpower needed for training in the medication safety curriculum. According to anecdotal feedback from some senior pharmacists who tried playing VRx, they shared that VRx could potentially reduce the number of man-hours in their training programs. As the VPE scenarios were repeatable and could provide a standardized way of training and assessing the individual, the pharmacy managers also felt that the VRx training could potentially improve the quality of their medication safety training and assessment programs as well. Additionally, they suggested that the VPEs could potentially be useful to teach about rare diseases and/or rare medical or medication events that the trainee might encounter in real-life practices. As such, our team has been engaged as consultants by pharmacy educators from other healthcare institutions to help them develop similar training initiatives through VR.

Limitations and Future Work

Besides the difficulties faced by participants during the training sessions and their suggestions for improvement, we faced challenges in terms of booking of room facilities for the training sessions within our hospital institution, due to repurposing of the rooms to cater for COVID-19 isolation and quarantine requirements. In addition, as VRx was a prototype, we only had one set of equipment (VR headset, controllers and laptop) to conduct the training sessions, which limited the number of participants who could be trained within a certain time period. The equipment had to be sanitized and wiped down after each use to reduce the probability of COVID-19 infections, thus each training session could only be conducted at least 30 minutes apart.

Participants had to take short breaks between the stages to prevent eye strain. Those who suffered from severe motion sickness were not able to attend the VRx program. For those who had just graduated and did not have any working experience, they had difficulty applying what they had learnt in school to the VPE scenarios, which then required the trainer to explain and jog their memory to recall the concepts needed to complete the tasks, thus resulting in more time spent in the sessions than what was originally allocated. Due to the mask-wearing requirement at the workplace, participants had to spend additional time readjusting their masks, spectacles and/or VR headsets in order to avoid fogging of the lenses. Moreover, some of the participants required frequent reminders on the functionalities of the controllers in order to proceed with the tasks. As such, we are now in the process of seeking external funding to develop more scenarios, modify the prototype and obtain more equipment. In addition, we are developing some pre-training material on our internal e-learning platform for trainees to prepare/revise their content knowledge before participating in the VRx program.

Lastly, we did not manage to evaluate the competencies of the trainees due to the additional workload of staff and re-deployment of manpower to deal with COVID-19 at our hospital. However, we acknowledge that the evaluation of trainees’ competencies is important to identify whether the learning outcomes were achieved. Therefore, we are currently developing a post-quiz to be completed by future trainees at the end of the virtual apprenticeship program, so as to consolidate the knowledge for the trainees, identify their areas of weakness and provide a more structured and personalized feedback to them. In addition, a comparison between two cohorts – one undergoing VRx and another undergoing the traditional on-the-job training without VRx – is intended to assess whether VRx has improved the competencies of the trainees at our institution.

Conclusion

A 3D virtual reality program (called VRx) has been developed to train pharmacy staff and trainees on medication safety practices. During the training sessions, staff/trainees are immersed in a virtual pharmacy environment, and they have to complete several key tasks to hone their competency skills in medication safety via visual/motion capture technologies. Overall, pharmacy staff and trainees agreed that VRx was able to provide a realistic, engaging and interactive experience to acquire the required proficiencies in medication safety practices, hence it is intended that VRx be used as a form of virtual apprenticeship in our institution’s training program. The competencies in medication safety are generic and thus can be applied, adapted and scaled-up for training of other healthcare professions. With the global emphasis on interprofessional education, we envision that VR training will attract the medication safety curricula of many universities and training/vocational institutes in the near future.

Funding Information

This research was funded by the SingHealth Learning Technology and Adoption Grant (LTAG) awarded by SkillsFuture Singapore and SingHealth Academy.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

WYT, PF, ZJT and IL conceived and designed the study. IB developed the virtual reality simulator. SHR conducted the training and study. WYT, PF, SHR and KY analyzed the results. KY wrote and revised the manuscript. All authors agreed to the publication of the manuscript.

References

  1. Tsegaye D, Alem G, Tessema Z, Alebachew W. Medication administration errors and associated factors among nurses. Int J Gen Med. 2020; 13: 1621–1632. DOI: https://doi.org/10.2147/IJGM.S289452 

  2. Law B. WHO calls medication errors a global problem. Available at: https://www.bottarleone.com/blog/calls-medication-errors-global-problem/. Updated 12 Apr 2017. Accessed 21 Mar 2022. 

  3. Anderson M. 5 COVID-19 related medication errors reported to ISMP. Available at: https://www.beckershospitalreview.com/pharmacy/5-covid-19-related-medication-errors-reported-to-ismp.html. Updated 14 May 2020. Accessed 1 Dec 2021. 

  4. Scott IA, Rigby D, Hilmer SN. Optimising medication management during the COVID-19 pandemic. J Pharm Pract Res. 2020; 50(3): 186–189. DOI: https://doi.org/10.1002/jppr.1668 

  5. World Health Organization. WHO Safety Curriculum – Topic 11: Improving medication safety. Available at: https://cdn.who.int/media/docs/default-source/integrated-health-services-(ihs)/psf/curriculum-guide/resources/ps-curr-teach-guides/topic-11_improving-medication-safety_teaching-slides.pdf?sfvrsn=f35ab325_9. Accessed 3 Jan 2022. 

  6. Karpa KD, Hom LL, Huffman P, et al. Medication safety curriculum: Enhancing skills and changing behaviors. BMC Med Educ. 2015; 15: 234. DOI: https://doi.org/10.1186/s12909-015-0521-0 

  7. Bajis D, Chaar B, Basheti IA, Moles R. Pharmacy students’ medication history taking competency: Simulation and feedback learning intervention. Curr Pharm Teach Learn. 2019; 11(10): 1002–1015. DOI: https://doi.org/10.1016/j.cptl.2019.06.007 

  8. Lee SE, Quinn BL. Incorporating medication administration safety in undergraduate nursing education: A literature review. Nurse Educ Today. 2019; 72: 77–83. DOI: https://doi.org/10.1016/j.nedt.2018.11.004 

  9. Mariani B, Ross JG, Paparella S, Allen LR. Medication safety simulation to assess student knowledge and competence. Clin Simul Nurs. 2017; 13(5): 210–216. DOI: https://doi.org/10.1016/j.ecns.2017.01.003 

  10. Khalil H, Schliephake K. Design of an online medication safety module for clinicians. Int J Evid Based Healthc. 2017; 15(2): 63–68. DOI: https://doi.org/10.1097/XEB.0000000000000103 

  11. Waring JJ. Constructing and re-constructing narratives of patient safety. Soc Sci Med. 2009; 69(12): 1722–31. DOI: https://doi.org/10.1016/j.socscimed.2009.09.052 

  12. Akers A, Blough C, Iyer MS. COVID-19 implications on clinical clerkships and the residency application process for medical students. Cureus. 2020; 12(4): e7800. DOI: https://doi.org/10.7759/cureus.7800 

  13. Le D-N, Le CV, Tromp JG, Nguyen GN. Emerging Technologies for Health and Medicine: Virtual Reality, Augmented Reality, Artificial Intelligence, Internet of Things, Robotics, Industry 4.0. Beverly, MA: John Wiley & Sons, Inc.; 2018. DOI: https://doi.org/10.1002/9781119509875 

  14. Yap KY, Koh DWH, Lee VSJ, Wong LL. Use of virtual reality in the supportive care management of paediatric patients with cancer. Lancet Child Adolesc Health. 2020; 4(12): 899–908. DOI: https://doi.org/10.1016/S2352-4642(20)30240-6 

  15. Falloon G. Using avatars and virtual environments in learning: What do they have to offer? Br J Educ Technol. 2010; 41(1): 108–122. DOI: https://doi.org/10.1111/j.1467-8535.2009.00991.x 

  16. Kron F, Gjerde C, Sen A, Fetters M. Medical student attitudes toward video games and related new media technologies in medical education. BMC Med Educ. 2010; 10(1): 50. DOI: https://doi.org/10.1186/1472-6920-10-50 

  17. Hassett L, van den Berg M, Lindley RI, et al. Digitally enabled aged care and neurological rehabilitation to enhance outcomes with Activity and MObility UsiNg Technology (AMOUNT) in Australia: A randomised controlled trial. PLoS Med. 2020; 17(2): e1003029. DOI: https://doi.org/10.1371/journal.pmed.1003029 

  18. Laver KE, Lange B, George S, Deutsch JE, Saposnik G, Crotty M. Virtual reality for stroke rehabilitation. Cochrane Database Syst Rev. 2017; 11: CD008349. DOI: https://doi.org/10.1002/14651858.CD008349.pub4 

  19. Garcia Fierros FJ, Moreno Escobar JJ, Sepulveda Cervantes G, Morales Matamoros O, Tejeida Padilla R. VirtualCPR: Virtual reality mobile application for training in cardiopulmonary resuscitation techniques. Sensors (Basel). 2021; 21(7): 2504. DOI: https://doi.org/10.3390/s21072504 

  20. Metin Ökmen B, Doğan Aslan M, Nakipoğlu Yüzer GF, Özgirgin N. Effect of virtual reality therapy on functional development in children with cerebral palsy: A single-blind, prospective, randomized-controlled study. Turk J Phys Med Rehabil. 2019; 65(4): 371–378. DOI: https://doi.org/10.5606/tftrd.2019.2388 

  21. Wong MS, Spiegel BMR, Gregory KD. Virtual reality reduces pain in laboring women: A randomized controlled trial. Am J Perinatol. 2021; 38(S 01): e167–172. DOI: https://doi.org/10.1055/s-0040-1708851 

  22. Tarrant J, Viczko J, Cope H. Virtual reality for anxiety reduction demonstrated by quantitative EEG: A pilot study. Front Psychol. 2018; 9: 1280. DOI: https://doi.org/10.3389/fpsyg.2018.01280 

  23. Smith V, Warty RR, Sursas JA, et al. The effectiveness of virtual reality in managing acute pain and anxiety for medical inpatients: Systematic review. J Med Internet Res. 2020; 22(11): e17980. DOI: https://doi.org/10.2196/17980 

  24. Stepan K, Zeiger J, Hanchuk S, et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol. 2017; 7(10): 1006–1013. DOI: https://doi.org/10.1002/alr.21986 

  25. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: A meta-analysis of randomized controlled studies. BMC Med Educ. 2020; 20(1): 127. DOI: https://doi.org/10.1186/s12909-020-1994-z 

  26. Kron FW, Fetters MD, Scerbo MW, et al. Using a computer simulation for teaching communication skills: A blinded multisite mixed methods randomized controlled trial. Patient Educ Couns. 2017; 100(4): 748–759. DOI: https://doi.org/10.1016/j.pec.2016.10.024 

  27. Dyer E, Swartzlander BJ, Gugliucci MR. Using virtual reality in medical education to teach empathy. J Med Libr Assoc. 2018; 106(4): 498–500. DOI: https://doi.org/10.5195/jmla.2018.518 

  28. Li Y, Ye H, Ye F, et al. The current situation and future prospects of simulators in dental education. J Med Internet Res. 2021; 23(4): e23635. DOI: https://doi.org/10.2196/23635 

  29. Schmidt MW, Koppinger KF, Fan C, et al. Virtual reality simulation in robot-assisted surgery: Meta-analysis of skill transfer and predictability of skill. BJS Open. 2021; 5(2): zraa066. DOI: https://doi.org/10.1093/bjsopen/zraa066 

  30. Khan R, Plahouras J, Johnston BC, Scaffidi MA, Grover SC, Walsh CM. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev. 2018; 8: CD008237. DOI: https://doi.org/10.1002/14651858.CD008237.pub3 

  31. Chen FQ, Leng YF, Ge JF, et al. Effectiveness of virtual reality in nursing education: Meta-analysis. J Med Internet Res. 2020; 22(9): e18290. DOI: https://doi.org/10.2196/18290 

  32. Shorey S, Ng ED. The use of virtual reality simulation among nursing students and registered nurses: A systematic review. Nurse Educ Today. 2021; 98: 104662. DOI: https://doi.org/10.1016/j.nedt.2020.104662 

  33. Coyne L, Merritt TA, Parmentier BL, Sharpton RA, Takemoto JK. The past, present, and future of virtual reality in pharmacy education. Am J Pharm Educ. 2019; 83(3): 7456. DOI: https://doi.org/10.5688/ajpe7456 

  34. Ventola CL. Virtual reality in pharmacy: Opportunities for clinical, research, and educational applications. P T. 2019; 44(5): 267–276. 

  35. Roa SC. Pharmacy students use AR/VR to study impact of prescription drugs on our main organs. Available at: https://hscweb3.hsc.usf.edu/blog/2018/10/10/pharmacy-students-use-ar-vr-to-study-impact-of-prescription-drugs-on-our-main-organs/. Updated 10 Oct 2018. Accessed 21 Aug 2021. 

  36. Gustafsson M, Englund C, Gallego G. The description and evaluation of virtual worlds in clinical pharmacy education in Northern Sweden. Curr Pharm Teach Learn. 2017; 9(5): 887–892. DOI: https://doi.org/10.1016/j.cptl.2017.06.002 

  37. Yap KY-L, Tan SIBH, Yap KZ, Yap JYG. Students’ perceptions of an in-house developed pharmacy serious game for professional skills training. BMJ Simulation and Technology Enhanced Learning. 2020; 6: 293–296. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8936978/ 

  38. Coyne L, Takemoto JK, Parmentier BL, Merritt T, Sharpton RA. Exploring virtual reality as a platform for distance team-based learning. Curr Pharm Teach Learn. 2018; 10(10): 1384–1390. DOI: https://doi.org/10.1016/j.cptl.2018.07.005 

  39. Zhao G, Fan M, Yuan Y, Zhao F, Huang H. The comparison of teaching efficiency between virtual reality and traditional education in medical education: A systematic review and meta-analysis. Ann Transl Med. 2021; 9(3): 252. DOI: https://doi.org/10.21037/atm-20-2785 

  40. Thompson J, White S, Chapman S. Interactive clinical avatar use in pharmacist preregistration training: Design and review. J Med Internet Res. 2020; 22(11): e17146. DOI: https://doi.org/10.2196/17146 

  41. Thompson J, White S, Chapman S. Virtual patients as a tool for training pre-registration pharmacists and increasing their preparedness to practice: A qualitative study. PLoS One. 2020; 15(8): e0238226. DOI: https://doi.org/10.1371/journal.pone.0238226 

  42. Zlotos L, Power A, Hill D, Chapman P. A scenario-based virtual patient program to support substance misuse education. Am J Pharm Educ. 2016; 80(3): 48. DOI: https://doi.org/10.5688/ajpe80348 

  43. O’Connor S. Virtual reality and avatars in health care. Clin Nurs Res. 2019; 28(5): 523–528. DOI: https://doi.org/10.1177/1054773819845824 

  44. Hepperle D, Purps CF, Deuchler J, Wölfel M. Aspects of visual avatar appearance: Self-representation, display type, and uncanny valley. Vis Comput. 2022; 38: 1227–1244. DOI: https://doi.org/10.1007/s00371-021-02151-0 

  45. Chua SIL, Tan NC, Wong WT, et al. Virtual reality for screening of cognitive function in older persons: Comparative study. J Med Internet Res. 2019; 21(8): e14821. DOI: https://doi.org/10.2196/14821 

  46. Pharmaceutical Care Network Europe Association. PCNE Classification for drug-related problems V9.1. Available at: https://www.pcne.org/upload/files/417_PCNE_classification_V9-1_final.pdf. Updated 10 May 2020. Accessed 10 May 2022. 

  47. Radianti J, Majchrzak TA, Fromm J, Wohlgenannt I. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comp Educ. 2020; 147: 103778. DOI: https://doi.org/10.1016/j.compedu.2019.103778 

  48. Hamilton D, McKechnie J, Edgerton E, Wilson C. Immersive virtual reality as a pedagogical tool in education: A systematic literature review of quantitative learning outcomes and experimental design. J Comput Educ. 2021; 8: 1–32. DOI: https://doi.org/10.1007/s40692-020-00169-2 

  49. Zhou Y, Ji S, Xu T, Wang Z. Promoting knowledge construction: A model for using virtual reality interaction to enhance learning. Procedia Comput Sci. 2018; 130: 239–246. DOI: https://doi.org/10.1016/j.procs.2018.04.035 

  50. de Freitas S, Rebolledo-Mendez G, Liarokapis F, Magoulas G, Poulovassilis A. Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. Br J Educ Technol. 2010; 4(1): 69–85. DOI: https://doi.org/10.1111/j.1467-8535.2009.01024.x 

  51. American Society of Hospital Pharmacists. ASHP guidelines: Minimum standard for pharmacies in hospitals. Am J Health Syst Pharm. 2013; 70(18): 1619–30. DOI: https://doi.org/10.2146/sp130001 

  52. Parthasarathi A, Puvvada R, Patel H, Bhandari P, Nagpal S. Evaluation of medication errors in a tertiary care hospital of a low- to middle-income country. Cureus. 2021; 13(7): e16769. DOI: https://doi.org/10.7759/cureus.16769 

  53. Krzyzaniak N, Bajorek B. Medication safety in neonatal care: A review of medication errors among neonates. Ther Adv Drug Saf. 2016; 7(3): 102–19. DOI: https://doi.org/10.1177/2042098616642231 

  54. Sellers S, Utian WH. Pharmacy compounding primer for physicians: Prescriber beware. Drugs. 2012; 72(16): 2043–50. DOI: https://doi.org/10.2165/11640850-000000000-00000 

  55. Grissinger M. Reducing errors with injectable medications: Unlabeled syringes are surprisingly common. P T. 2010; 35(8): 428–51. 

  56. Keuler N, Bouwer A, Coetzee R. Pharmacists’ approach to optimise safe medication use in paediatric patients. Pharmacy (Basel). 2021; 9(4): 180. DOI: https://doi.org/10.3390/pharmacy9040180 

  57. Ford E, Edelman N, Somers L, et al. Barriers and facilitators to the adoption of electronic clinical decision support systems: A qualitative interview study with UK general practitioners. BMC Med Inform Decis Mak. 2021; 21(1): 193. DOI: https://doi.org/10.1186/s12911-021-01557-z 

  58. Phipps D, Ashour A, Riste L, Lewis P, Ashcroft D. Understanding dispensing errors and risk. Pharm J. 2020; 305(7943). 

  59. Björkstén KS, Bergqvist M, Andersen-Karlsson E, Benson L, Ulfvarson J. Medication errors as malpractice-A qualitative content analysis of 585 medication errors by nurses in Sweden. BMC Health Serv Res. 2016; 16(1): 431. DOI: https://doi.org/10.1186/s12913-016-1695-9 

  60. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65(9 Suppl): S63–7. DOI: https://doi.org/10.1097/00001888-199009000-00045 

  61. Royal College of Physicians and Surgeons of Canada. Understanding Work Based Assessment and CBD. Available at: https://cdn.dal.ca/content/dam/dalhousie/pdf/faculty/medicine/departments/department-sites/pediatrics/work-based-assessment-practical-implications-clinical-teachers.pdf; 2017. Accessed 30 Sep 2021. 

  62. Jarrett JB, Berenbrok LA, Goliak KL, Meyer SM, Shaughnessy AF. Entrustable professional activities as a novel framework for pharmacy education. Am J Pharm Educ. 2018; 82(5): 6256. DOI: https://doi.org/10.5688/ajpe6256 

  63. Poon CH. Public healthcare sector to be reorganised into 3 integrated clusters, new polyclinic group to be formed. The Straits Times. 2017, 19 Jan. Available at https://www.straitstimes.com/singapore/health/public-healthcare-sector-to-be-reorganised-into-3-integrated-clusters-new. Accessed 4 Jan 2022.