Mixed Reality System for Virtual Chemistry Lab

Abstract

This paper presents a new method for improving teaching aids with the design and development of a chemistry education platform - the Mixed Reality (MR) Chemistry Lab. Our system provides a new educational experience in which students can simulate a chemistry experiment in a virtual lab and interact with objects using Oculus Helmet-Mounted Displays (HMD) and hand controller devices. The proposed system aims to familiarize students with experimental procedures and safety knowledge before conducting actual experiments. Moreover, students will be able to observe microscopic atomic structure models in three dimensions. Our research also includes a five-part, quantitative evaluation system user tests perform using a quantitative questionnaire consisting of five items, including hardware equipment, immersion, education, interaction sense of control, and degree of difficulty. The evaluation results confirm that this system will be helpful to the educational experience of conducting chemistry experiments with scientific evaluation methods, and the proposed system is also expected to have a broad range of applications in many other subjects.

1. Introduction

Safety in chemical experiments must be an alarm bell that rings constantly. During an experiment in a Chinese high school, when benzene was added to a mixture of concentrated sulfuric acid and concentrated nitric acid, the mixture suddenly flew out of the test tube and splashed into the eyes of the student involved [1]. This accident happened because concentrated nitric acid, concentrated sulfuric acid and nitrobenzene are all highly corrosive substances. Therefore, this experiment is one of the most dangerous activities in high school chemistry experiments because if concentrated nitric acid is added to concentrated sulfuric acid, it can release a large amount of heat, and if concentrated sulfuric acid is mixed with the water in concentrated nitric acid, the upper layer of concentrated nitric acid could splash and harm the person handling the chemicals. Experimental precautions require that concentrated sulfuric acid be slowly added to concentrated nitric acid, and the mixture must be shaken evenly; only then can the next step be taken. Teachers and textbooks have already explained this to students, but due to their lack of familiarity with safe experimental procedures, students may forget about safety hazards when conducting an experiment. These accidents also occur in institutions of higher education, such as universities. A recent accident happened in Beijing Jiaotong University [2] that caused the death of three students who were participating in the experiment.

Safety is not a trivial matter in chemistry experiments or other subjects, and this is one of the main topics of this research. In this paper, we propose a Mixed Reality (MR) application that we designed to support real chemical experiments. Our application, the “MR Chemistry Lab” has the potential to avoid the dangers of chemistry experiments in education by combining the technologies of Augmented Reality (AR) and Virtual Reality (VR). To create this system, we used normal commercial hardware such as an Oculus HMD, hand controllers and Google Cardboard. We sought the best interaction mode for this application by analyzing the features of various student behaviors in laboratories. Students majoring in chemistry and teachers participated in experiments to evaluate the proposed system. The experimental results show that students can achieve their educational objectives more effectively by using our system. Students can observe microscopic molecular combinations and arrangement structures in tabulation from an intuitive, three-dimensional space. Our research demonstrates that it is possible to partially replace very dangerous experiments such as chemistry experiments and that the proposed application can be applied to various virtual experiments.

2. Methods

2.1 Aim

We propose a new virtual education environment by designing and developing a chemistry education system – the Mixed Reality (MR) Chemistry Lab. To develop this system, we investigated the current methods of conducting chemistry experiments and devised a safe MR educational environment composed of various virtual reality devices and contactless interaction equipment. In addition, we provided an evaluation system to verify our results. The verified results show that the proposed system improves the experience of chemistry experiments. This system will help students gain knowledge in a safe environment before performing actual experiments. This system provides a virtual education environment where students can easily acquire chemistry knowledge by easily editing the microscopic atomic structure models in a three-dimensional virtual space.

2.2 Related Research

The term "Mixed Reality” was first coined by Paul Milgram and Fumio Kishino in 1994 [3]. They introduced the concept of a virtual continuum and focused on how the taxonomy of taxonomies applied to presentation. Since then, the use of a hybrid reality has not only transcended displays but also included environmental inputs, spatial sound, and location. In MR systems, the user can perceive both the surrounding physical environment and the digital elements presented by using translucent displays, for example. In essence, MR is a highly interdisciplinary field.

A key issue in MR systems is the efficient integration of software and hardware platforms with the real world. The latest augmented reality (AR) hardware platform includes devices such as computers with cameras, smartphones, head-mounted displays (HMDs), glasses, and haptic devices.

A PC with a webcam is one of the platforms used for AR. A marker is placed within the view of the webcam to display a real-time, live feed. The Shiseido Makeup Mirror [4] is a typical example of a PC-based AR platform. Customers can take photos and then virtually try out various cosmetic styles and colors, which simplifies the trial and error process of cosmetics selection, while increasing sales and customer engagement. Kiosks are another PC-based AR platform that enables customers to use AR to learn more about the items they carry with them. One example of the kiosk platform is the Lego Store kiosk [5], which displays a complete Lego set based on the customer’s Lego box. The booth type of kiosk is also used in trade shows and conferences to provide a richer experience for attendees. The digital signs and window display that are also used are basically large, static markers that users interact with via the PC platform. The AR SandBox [6] project involves a real-time, integrated AR system that allows users to create terrain models using sand physics, and then scan them into a computer in real time, using projectors on top of the sandbox to create various graphical effects for the model. The simulated background is projected onto the sand model.

Accessing AR content using a smartphone or tablet is undoubtedly the most common method in use today. There are two categories of usage for smartphone AR: ubiquitous and constantly held. A typical case of the ubiquitous category is the Wikitude World Browser [7], which is easy to use and is widely regarded as the best AR browser. In certain respects, it is used as a “third eye” by users. When a smartphone camera is pointed somewhere, this virtual browser can provide some geographically relevant information that may be of great help during travel. The Microsoft Omnitouch [8] is a typical example of the constantly held category and is a portable device that combines a miniature laser projector with a depth-sensing camera. Users can place the device on a table or even on their shoulders. The research team recognizes that the prototype is still too big for them, but there are no major obstacles to miniaturizing it. Future versions of Omnitouch may be the size of a matchbox that can be worn like a pendant or watch.

The goal of an augmented reality system is to overlay computer-generated virtual objects onto the view of a real scene to enhance (rather than replace) the user’s perception of the real environment. One way to accomplish this superposition is through the use of an optical, see-through, head-mounted display (HMD). The HMD is a tool used to implement VR, AR or MR, and includes two display screens paired with a headset. The information/image is placed on top of the users’ view with six degrees of freedom (6 DOF), and the view is enhanced as the user’s head moves in any direction/angle. The Skully Helmet [9] is a helmet in which the display camera has a viewing angle of 180 degrees, so that a user can see visual blind spots on the left and right. The Skully Helmet also has a flat-panel display. Extensive road testing has been carried out to ensure the highest quality, functionality and safety. The Microsoft HoloLens [10] is Microsoft's first holographic, wearable, augmented reality computing device. The Microsoft HoloLens is completely self-contained, as it does not require cables, mobile phones, or a computer connection. The Microsoft HoloLens allows users to incorporate holographic images into their physical environments. The holographic lens consists of multiple sensors, advanced optics, and a custom holographic processing unit. The Daqri Smart Helmet [11] senses the surrounding area and draws a 3D environment map for real-time capture and recording of the working environment. Users only need to obtain relevant information through the display on the glasses. The Daqri Smart Helmet enhances the capabilities of people across industries by seamlessly connecting users to their work environments.

Another representative AR platform is AR glasses, but the related technologies are still not commonplace. However, the potential applications of AR glasses are increasing. It’s quite possible that AR-enabled glasses will become as common as iPads and smartphones, allowing wearers to choose continuous AR feeds based on individual needs and preferences. The Google Glass [12] display is the most representative of AR eyewear products, and products such as Vuzix [13] also exist and are available for purchase. Over time, there better technology will be available at lower prices. Meta [14] has a focus that Google Glass does not have. Meta is developing the software application Meta Viewer, a visual application designed for Meta 2 HMD, which includes many features that allow users to interact with 3D CAD models in new ways, including 3D CAD Workflow collaboration and voice commands. Since it can open and visualize AR-ready, 3D CAD models, Meta Viewer is a leader in changing the product design and development workflow. ICIS [15] has a shape similar to ordinary glasses. The lens can display the information pushed to a phone. The device also features a camera, headphones, and speakers. Users can control the device using a touch panel because it does not have a large component, such as a camera. This smart eyewear promotes natural interaction because the user can control it with gestures. The Atheer One [16] is a smart device that delivers an immersive, 3D, virtual experience. It projects 3D images and interacts with natural gestures for 3D modeling, interactive games, and more. In addition, the Atheer One projects a 26-inch virtual screen at a distance of half an arm from the wearer. The device is compatible with an existing, 2D Android app that projects information on the front screen. Contact lenses that deliver images directly on the human eye [17] are also being developed for medical and military purposes. Google and Alcon are now developing a contact lens that can measure blood sugar from the glucose in tears so that diabetics do not have to endure the pain of needles. Finally, the virtual retina displays [18] under development project images directly onto a pilot’s retina using lasers, monolithic optics, and minimal scanning equipment. The device allows the pilot to see both the surrounding airspace, and digital markers and images that appear to float in front of the pilot.

Haptic AR devices enable users to experience a more realistic environment that is enhanced by synthetic haptic interactions. The new AR feature of the Nintendo Wii U/PS4/Xbox [19] game tablet provides users with real-time, tactile feedback. When an HMD is combined with a PHANTOM stylus [20], the user can issue commands through touch and hand-space actions, and the PHANTOM stylus interacts with the augmented reality scene. When a Virtual Haptic Back (VHB) [21] system is combined with a PHANTOM stylus, a surgeon can clearly see human organs and body parts during a procedure, Moreover, this system can provide tactile feedback to the surgeon’s fingers during cutting, removal, and device movements. Finally, Maestro AR [22] is also a medical application that uses robotic surgery simulation techniques to provide 3D virtual instruments for anatomical interactions in 3D video environments. In the introduction of the many virtual reality and augmented reality applications described above, we found that virtual reality using head-mounted devices can provide a very immersive environment, with accurate and convenient interaction for users. Augmented reality provides users with stereoscopic and intuitive viewing effects, as well as interactive experiences that are closer to natural interactions, such as direct manual operation. The mixed reality chemistry lab we hope to develop combines the advantages of both.

2.3 MR device and application

VR/AR devices are characterized by their ability to insert digital content into the real world as if it were really there. In this study, we have used three VR/AR devices that are able to create a sense of “presence” – hiding the physical world and replacing it with a digital experience, as shown in Fig. 1.

Fig. 1. Mixed reality device types [3]

When users are at the physical reality extreme of the reality-virtual reality spectrum, they remain present in their physical environment and are never made to believe they have left the real world. In the middle of the spectrum (fully mixed reality), users experience a perfect blend of the real world and the digital (virtual) world. When users are at the digital reality extreme of the spectrum, they experience a completely digital virtual environment and are unaware of what occurs in their surrounding physical environment. In this application, the environment is fully virtual, so we will introduce virtual reality devices next.

2.3.1 VR devices

Since 2014, the booming development of VR has produced Oculus, HTC Vive [23], Gear VR, and other systems. VR has evolved in three directions: VR connected to a PC, VR with mobile phones, and all-in-one VR devices. VR applications have become common in our lives in areas such as games, education, design, travel, and many other fields. In our proposed application, we use the Oculus VR suite as shown in Fig. 2, which has one HMD, two hand-controllers called “touch controllers” and two sensors, all of which are connected to a PC. The touch controllers are held in the hand to control and interact with the VR Chemistry Lab application. After this set of equipment is connected to a PC, a space of approximately 3x3 square meters is needed for users to stand and move. Each PC has a space in front of it. VR has three main characteristics: immersion, real-time interactivity, and imagination [24]. First, immersion (or the so-called fusion) serves as the core feature of virtual reality that enables virtual reality systems and applications to stimulate users’ sensory receptors in a way that is extensive, matching, peripheral, active, interactive, and informative [25]. The user can get a real sense of seeing, hearing, touching, and smelling as in the real world. Second, real-time interactivity is a basic requirement, in the same way that people interact with their real-world environments. Thus, a user’s immersion will be disrupted if the interaction is unnatural or has a time delay. Third, imagination or creativity is the ideal purpose of VR. At the very beginning of a VR world, VR can be made to resemble real or non-existent things. With experience, users are be able to acquire new knowledge and ideas to help them establish new cognition.

Fig. 2. Oculus Rift HMD, hand-controllers, and two sensors [26]

2.3.2 Leap Motion

Leap Motion is a motion controller for PCs and Macs, which was released by Leap in 2013 [27]. Instead of replacing a keyboard, mouse, stylus or trackpad, the Leap Motion controller works in conjunction with these devices. With the Leap Motion software running, users simply plug the device into a Mac or PC and they’re ready to go. With one finger, one can browse the web, read articles, flip through photos, and play music.Without using any brushes, users can draw, doodle, and design with their fingertips. Leap Motion is essentially a sensor that points upward, allowing users to control programs and play games by moving their hands in a created virtual space, as shown in Fig. 3(a). In addition, the Leap Motion can visualize a user’s hands as bones and display them in fluorescent color on the screen. The Leap Motion controller tracks all 10 fingers with an accuracy of up to 1/100 mm. The Leap Motion controller is far more precise than existing motion control techniques, and the Leap Motion controller tracks hand movements at more than 200 frames per second, which is why on-screen movements are perfectly synchronized with every move. Combined with VR split screen technology, the Leap Motion controller can be used in VR scenes, as shown in Fig. 3(b).

Fig. 3. Using Leap Motion to change VR scenes [27]

2.3.3 Google Cardboard

Google has launched a cheap virtual reality device called Google Cardboard [28], which can turn Android phones into virtual reality devices.

The part that using cardboard, mobile phone and LeapMotion has three scenes, Fig. 4(a) is the main menu scene, that allow users choose the mode, Fig. 4(b)(c)(d) is the AR mode scene, here we use simple geometries as the Atomic structure model, users can see the structural model of the atom on the top of the paper in the physical world.

Fig. 4. The AR mode using Cardboard and LeapMotion

Although this method implements a mobile phone and a leapmotion connection, and the mobile phone has a certain mobility than the PC, it still cannot get rid of the PC. In my project, the application is directly run in Unity and connect to mobile phone with Unity Remote which is a downloadable app designed to help with Android, iOS and tvOS development. The app connects with Unity while you are running your project in Play Mode from the Unity Editor. When use Leap Motion as the interaction tool, users’hand is recognized as the Fig. 5. As the interaction of Leap Motion is very natural and attractive, we combined Leap Motion and Cardboard to provide an immersion experience, as shown in Fig. 5.

Fig. 5. VR Scenes using Leap Motion (a)Connect Leap Motion and mobile phone to PC(b) Using Leap Motion to obtain VR scenes in this application

2.4 MR chemistry lab using MR devices

High risk and reagent consumption conditions caused by student unfamiliarity with a reagent, an apparatus, or experimental procedures are problems in traditional chemistry experiments conducted in educational environments. Potential safety problems have led to high risks in the traditional teaching of chemistry experiments. By using this application, students will be able to repeatedly simulate experimental procedures and guarantee the authenticity of experimental results, thus reducing the probability of risk and effectively improving their learning efficiency. VR can be used to make the simulation scenes more immersive, and thus it is expected to be used in the education and professional training of many subjects.

The MR Chemistry Lab is an educational experience in which students can simulate laboratory procedures and safety measures. Users are immediately immersed inside a VR laboratory and can use the Oculus HMD or Leap Motion to interact with the environment. In real lab procedures, lab procedure and safety guides are spread out across tables, and a great deal of lab equipment can be picked up, placed, thrown, or actually used. In a virtual experiment, students can perform simulated experiments in an environment that is the closest to a real environment, familiarize themselves with the experimental process, observe and record experimental phenomena, save reagents, minimize dangers, and finally, achieve the goal of anytime/anywhere learning.

The “magnesium ribbon combustion” is a simple experiment that provides an introduction to students regarding chemistry experiments.

2.4.1 Background Knowledge for Chemistry Lab

The experimental principles and reaction equations involved “magnesium ribbon combustion” are as follows:

Equations:

2Mg+O₂ === ignite ===2MgO       (1)

2Mg+CO₂ === C + 2MgO       (2)

3Mg+N₂ === Mg₃N₂       (3)

Mg+2CH₃COOH===(CH₃COO)₂Mg + H₂(↑)      (4)

MgO+2CH₃COOH===(CH₃COO)₂Mg+H₂O       (5)

As shown in Fig. 6 and Eq. (1), (2) and (3), when a magnesium ribbon is ignited under oxygen, carbon dioxide, nitrogen, magnesium oxide and magnesium nitride are generated. This chemical reaction occurs when a user puts the magnesium ribbon into the fire. The magnesium ribbon, which is ignited under oxygen, generates magnesium oxide and gives out a dazzling white light. The chemical reaction in Eq.(4) occurs when the unburned magnesium ribbon is put into acetic acid. The magnesium ribbon produces magnesium acetate and hydrogen in an acetic acid solution, which generates bubbles. Eq.(5) shows that magnesium oxide produces magnesium acetate and H2O in an acetic acid solution. This chemical reaction occurs when the burned magnesium ribbon is put into the acetic acid. The magnesium oxide produces magnesium acetate and water in an acetic acid solution. The proposed application is divided into the Main Scene and the ChemVR Scene. In the Main Scene, the main function is to select experiments and see the atomic structure models in a 3D mode, which is necessary in the selected experiment according to the periodic table, as shown in Fig. 7.

Fig. 6. Chemistry experiments

Fig. 7. Drag the element’s atomic structure model to the front (a) Main UI scene (b) Pointing atom behavior(c) Dragging atom behaviors

2.4.2 VR chemistry lab scene

When users choose an experiment, the VR chemistry lab looks like a simple, real laboratory (as shown in Fig. 8) with laboratory furniture and experimental instruments on the table. In the magnesium ribbon combustion experiment, there is an alcohol lamp, beakers, magnesium strips, and so forth on the table.

Fig. 8. Laboratory table

3. Results and Discussion

3.1 Experimental procedures

The experimental environment for this study is an Intel(R) Xeon(R) CPU E3-1240 v3 @3.40GHz 3.40 GHz, 8GB RAM with Windows 10 (version 2017.1.1) personal (64 bit) of the Unity3D development engine. The experimental procedures that users can refer to in the experiment are as follows.

1. Choose the “Magnesium Ribbon Combustion” experiment in the Main Scene and click the “Go” button.

2. Read the guidelines or videos on the desk.

3. Take the match on the desk and put it on the alcohol lamp to light the alcohol lamp.

4. Take one of the two magnesium strips on the table and burn it on the alcohol lamp.

5. Watch and record the phenomena.

6. Put the burned magnesium strips in a beaker containing vinegar.

7. Put the other (unburned) magnesium strip in a beaker containing vinegar.

8. Watch and record the phenomena.

9. Extinguish the alcohol lamp by putting the cover on the flame.

3.2 Experiment and evaluation

This section presents the evaluation of the MR chemistry laboratory. This game demonstration of MR virtual chemistry laboratory education was developed for students engaged in chemistry. In an educational environment, students are expected to achieve the following goals:

1. To understand chemistry experiments, to learn to observe and record experimental phenomena, and to master basic chemistry research methods.

2. To observe and understand the atomic structures of different elements from different perspectives using the interaction between the handle and the model.

3. To become familiar with the experimental process. Before the actual experiment, the experimental process is rehearsed repeatedly in the virtual laboratory, especially for dangerous items and valuable reagents.

4. To stimulate student interest in chemistry and deepen their impression of reactions through repeated observation.

5. To cultivate students’ scientific attitudes toward the objective facts of experiments and the need to be cautious and strict in discipline.

3.3 Experimental subjects

We chose 45 students from the Chemistry Department of PaiChai University to be experimental subjects. They were divided into three groups: two experimental groups and one reference group. The experimental group, the Oculus HMD experimental group, and the reference group all had 15 members each. Each group was mixed in terms of gender. In addition, two chemistry teachers also participated in the evaluation. Before the evaluation, under the guidance of the teachers, all the subjects needed to become familiar with the equipment and its operation.

Four PCs were set up for the concurrent evaluations, and students queued to experience the system, as shown in Fig. 9.

Fig. 9. Students’ experience and evaluate the process using Oculus

3.4 Experimental Results and Discussion

Costello et al. [29] discussed the method to evaluate the MR application during use. The evaluation had four aspects: (a) task performance; (b) perception and cognition in the orientation perception of users, depth perception and occlusion perception of objects in a scene of virtual and real fusion; (c) collaboration and interaction among multiple users; (d) the subjective feelings of users [30]. We evaluated the results from five perspectives corresponding to five gauge dimensions: hardware equipment, immersion, education, interaction and sense of control, and equipment learning time. The evaluation methods consisted of questionnaire surveys and interviews, and the result is shown in Table 1.

Table 1.MR Chemistry Lab Evaluation Results

After the evaluation, scientific questionnaires and interviews were conducted with the students, and the respondents were invited to talk about their feelings after participating in the MR Chemistry Lab experiment. In the Leap Motion experimental group, 12 of 15 participants said that at first they were very interested in this device because they could see their hands on the screen and use them as in real life. However, they found that they often had to try several times to complete some very fine movements. In the Oculus experimental group, all participants said that the operation is very accurate. However, some of them felt dizzy after using it for a while. Six of the 15 participants said that wearing HMD devices makes them feel immersive, but the effects of the demonstration experiment are not realistic. Additionally, 12 said that using MR equipment helped them better observe and memorize the microscopic atomic structure because in a traditional class, it is difficult for students to develop a direct impression of the microscopic atomic structure from the teacher's simple display and because their knowledge, understanding, and memory might be at a low level. Teaching with MR-based software can also stimulate student enthusiasm, while the immersive environment allows students to be more focused. Compared with traditional teaching software, such as MS Power Point or Adobe Flash, the MR instrument helps to improving the operability of the experiment. The MR experiment cannot completely replace the real experiment. However, students can become familiar with the experimental process beforehand and avoid or reduce the probability of errors caused by an unfamiliar process. Most importantly, students do not need to worry about the safety of reagents and laboratory equipment, and the overuse of these. Therefore, repeated practice with the experimental process before the actual experiment will be of great benefit to actual experiments in the future.

Every coin has two sides. Our study also revealed some of the limitations of mixed reality implementations. In summary, the following are some important human factors in MR: 1) Delay. MR requires users to be able to interact with the system in real time, but system delay will directly affect interaction efficiency and task performance. 2) Location perception. Errors in registration technologies such as depth errors, orientation errors and masking errors will affect user judgment of the location of virtual objects. 3) Sense of reality. The sense of reality of the light, texture, materials and other factors of virtual objects will greatly affect a user’s cognitive ability towards the virtual reality integration environment. If the display of the virtual object is very unrealistic, it will also affect the user experience and interaction efficiency. 4) Fatigue. Many head-mounted displays cause eye fatigue and make them unsuitable for long-term use. There are also problems in the long-term use of handheld displays.

4. Conclusions

In spite of the limitations found in this MR application, this study clearly confirmed that a combination of virtual reality and augmented reality in education can improve teaching efficiency, at least when experiments are conducted. As far as we know, the aim of the traditional teaching mode is for teachers to impart knowledge to passive students. By contrast, the use of MR technology is a personalized, independent, and experiential teaching mode. With personalized teaching, each student can independently choose what to learn in the virtual environment at any time. In addition, the use of AR technology can turn static characters and pictures into three-dimensional images, thus improving the interactive experience. Microscope content is visualized in three dimensions, and abstract content is materialized. Moreover, simplifying complex learning content can better help students understand and memorize information [31]. The mixed reality technology also provides a virtual, immersive environment where students can satisfy their sense of experience and curiosity and improve their concentration [33]. In summary, an innovative method of imparting knowledge can greatly improve teaching efficiency, stimulate students’ interest in learning, and enhance their learning efficiency. In the future, we plan to extend the relevance of our study to other chemistry subjects or virtual experiment fields to further advance education