3D cursor controlled by Wii Remote in Virtual Reality

3D cursor controlled by Wii Remote in Virtual Reality environment for Physics teaching

Roberto Scalco

Department of Computer Engineering and Industrial Automation (DCA)

School of Electrical and Computer Engineering (FEEC)

State University of Campinas (UNICAMP)

Campinas, SP, Brazil

January 2015

Master's thesis advisor: Prof. Dr-Ing. Wu, Shin-Ting

1 Motivation and Objective

In Physics laboratory classes from undergraduate Engineering course, we observed that most freshmen performs experimental procedures mechanically, following blindly the script, without being attentive to correlate what they are doing with theoretical concepts. So, teachers can to develop playful strategies that encourage students, to motivate and please them in undertaking the activity.

The objective of this work is to develop a 3D controller for selecting and tracking object's motion in a 3D environment. Low-cost controller Wii Remote was chosen to the UI due to its 6DoF movement and a rumble motor for a haptic feedback. This 3D cursor can be used in educational virtual environments.

2 Contributions

This work proposes the design and development of an educational environment for the teacher to present interactive simulations about Physics subjects, using the Wii Remote as 6DoF input data interface. Wii Remote includes a rumble motor that vibrates when the user holds an object in the 3D environment, for a haptic feedback to the user.

3 Overview

Fig. 1 shows the data transmition via Bluetooth protocol and are processed to get 3D cursor rotation and position in space. If the 3D cursor is over an object, the user can pick and track its movement.

Physical parameters (mass, center of gravity position etc.) are assigned to the objects and Open Dynamics Engine calculates collisions between bodies. OpenGL was used to implement the graphic interface.

Figure 1: framework.

3.1 Cursor Movement

3D cursor moves only when the user presses A button. If 3D cursor is over a scene object, the user can press the A and B buttons to pick and move the object.

Figure 2: Button A and button B.

Adapted from Nintendo Wii Operation Manual.

An orthogonal basis for reference is created using a vector $\vec{L}_{w}$ parallel to WiiBar, gravity acceleration $\vec{g}_{w}$ and depth camera direction $\vec{p}_{w}$ .

Figure 3: Wii Remote and WiiBar basis.

Euler angles $\alpha$ , $\beta$ , and $\gamma$ describes the cursor orientation relative to the fixed basis in WiiBar.

$\alpha = arccos\left(\frac{\vec{L}_{w}\cdot\vec{a}_{proj_{L_{w},p_{w}}}}{\left\|\vec{L}_{w}\right\|\cdot\left\|\vec{a}_{proj_{L_{w},p_{w}}}\right\|}\right)$

$\beta = arccos\left(\frac{\vec{p}_{w}\cdot\vec{a}_{proj_{p_{w},g_{w}}}}{\left\|\vec{p}_{w}\right\|\cdot\left\|\vec{a}_{proj_{p_{w},g_{w}}}\right\|}\right)$

$\gamma = arccos\left(\frac{\vec{g}_{w}\cdot\vec{a}_{proj_{g_{w},L_{w}}}}{\left\|\vec{g}_{w}\right\|\cdot\left\|\vec{a}_{proj_{g_{w},L_{w}}}\right\|}\right)$

Using acceleration components from vector $\vec{a}$ , 3D cursor velocity and position are calculated for each frame i. The movement sensibility is adjusted by the factor.

$v_{\tiny{i}}=v_{\tiny{i-1}}+factor\cdot \left(a_{\tiny{i-1}}-a_{\tiny{i}}\right)\cdot \left(t_{\tiny{i-1}}-t_{\tiny{i}}\right)$

$P_{\tiny{i}}=P_{\tiny{i-1}}+v_{\tiny{i}}\cdot \left(t_{\tiny{i-1}}-t_{\tiny{i}}\right)+factor\cdot \left(a_{\tiny{i-1}}-a_{\tiny{i}}\right)\cdot\frac{\left(t_{\tiny{i-1}}-t_{\tiny{i}}\right)^{\tiny{2}}}{2}$

3.2 Survey and Results

An ambient for tests was implemented for analysis of the time to perform the tasks and the difference between the object position and the position of the target were used to determine the characteristics of a learning curve of the pointing and motion tracking objects with the Wiimote.

After the tests, users answered a survey about acceptability of the 3D cursor controlled by Wii Remote. The survey results confirmed that familiarity with Wii Remote, experience with 3D modeling software or 3D games improved the test results performance. Fig. 4 shows that 63% of users approved the haptic feedback feature.

Figure 4: Haptic feedback.

4 Physics Simulation Environment

4.1 Teacher’s software

Teachers can create, edit and save scenes in a XML file. This scenes can contain spheres, cubes, capsules, planes and triangular meshes (to import Wavefront OBJ files created in another software). Each object has geometric (position, rotation etc.) and physic (mass, type of collision etc.) properties, defined by the teacher.

Figure 5: Teacher's interface.

The teachers also sets if auxiliary visuals elements are drawn in the student's software. These elements are velocity vector, acceleration vector, rotation axis and a tracking path from object's movement.

4.2 Student’s software

Students can open a XML file created by the teacher and use Wii Remote to control the 3D cursor to move selected objects in the scenes. Users can press and hold A and B buttons to pick and move objects.

Figure 6: Student's interface.

Student Interface

We created five scenes to students to plays simulations: selecting, picking, holding and throwing objects. Physics engine ODE calculates collisions between objects and graphics are rendered by OpenGL.

4.3 Tests and Results

Students answered three tests during our experiment. These tests used some questions from Force Concept Inventory test:

Test 1: four questions. Applied at beginning of the experiment;
Test 2: four questions of the Test 1 + five questions about same subjects. This test was applied after students use Wii Remote to interact with objects in five scenes, describes below;
Test 3: same nine questions of Test 2. Applied one week after experiment.

Scene 1

Scene 2

Scene 3

Scene 4

Scene 5

Figure 7: Tests results
Common questions of the Test 1 and Test 2.	All questions in Test 1 and Test 2.	All questions of the Test 2 and Test 3

Comparing the results of the tests, the better result occurs when students answer the same questions of the Test 1 and Test 2 (first histogram). All students receive same or better score. This result shows that our proposal can be used in cases of repeatability of questions for fix incorrect concepts.

5 Download

5.1 Softwares

Teacher Editor (431 kB) – release on: 07.04.2015.
Student Simulator (834 kB) – release on: 07.04.2015.

5.2 Publications

The final version of my master thesis in Portuguese is available in pdf.

Estudo da aceitabilidade de usuários sobre o uso do Wiimote como interface de um cursor 3D [article] [pdf]
Authors: Roberto Scalco and Wu, Shin-Ting
Published in Proceedings of INTERTECH’ 2014 – XIII International Conference on Engineering and Technology Education, Guimarães, Portugal, mar. 2014
Abstract
Aiming at the application of Wiimote as a 3D cursor controller for accurate spatial placements, this article presents an analysis of test results of motion tracking objects and a survey with users who used the control Wiimote of the Nintendo Wii to move a 3D cursor. The time to perform the tasks and the difference between the object position and the position of the target were used to determine the characteristics of a learning curve of the pointing and motion tracking objects with the Wiimote. The survey results allow to confirm, from the previous experience of the users, such as familiarity with the use of Wiimote, experience with 3D modeling softwares or familiarity with 3D games, the points that should be improved in the proposal of this framework.

Desenvolvimento de um framework para uso do Wiimote como dispositivo de interações em ambientes tridimensionais aplicado a um laboratório virtual de Física [pdf]
Authors: Roberto Scalco and Wu, Shin-Ting
Published in Proceedings of SIBGRAPI’2013 – Conference on Graphics, Patterns and Images, Arequipa, Peru, aug. 2013
Abstract
This paper presents the partial development of a low-cost interactive educational environment for performing undergraduate physics laboratory experiments. Once most of students are already fond of the Wii game system, we explore the pointing and motion tracking capabilities of the Nintendo Wii Remote to make learning more playful.

Localização espacial da mão do usuário utilizando Wii Remote [pdf]
Authors: Roberto Scalco and Ricardo Silva Tavares
Published in Proceedings of 4º Semiário Mauá de Iniciação Científica, São Caetano do Sul, São Paulo, Brasil, dez. 2012
Abstract
This paper presents a proposal for developing a human-machine interface that allows the user to interact with objects modeled in a virtual space using the Wii Remote (or Wiimote), Nintendo Wii videogame control. This device allows to determine the user's hand movement in space, in contrast of the two-dimensional motion obtained by ordinary mouse, because the three acceleration orthogonal components of the movement are transmitted to the computer using Bluetooth wireless protocol. Using the acceleration components of the movement it's possible to draw a 3D cursor in the virtual space, as the user's hand movement.

Elementos de um laboratório virtual de Física controlados por Wiimote [pdf]
Authors: Roberto Scalco and Wu, Shin-Ting
Published in Proceedings of SIBGRAPI’2012 – Conference on Graphics, Patterns and Images, Ouro Preto, Minas Gerais, Brasil, aug. 2012
Abstract
This paper presents the development of a Physics learning environment in which the Nintendo Wii Remote controller is used not only to select an object via pointing and to track an object’s motion, but also to move experiment objects and measuring instruments. Its rumble feature is explored in our project to provide appropriate haptic feedback.

Uso do controle Wii Remote como cursor para manipulação de objetos tridimensionais em ambientes de realidade virtual [article] [pdf]
Authors: Roberto Scalco and Alessandro Guilherme de Freitas
Published in Proceedings of IGIP’2011 – 40th IGIP International Symposium on Engineering Education, Santos, São Paulo, Brasil, mar. 2011
Abstract
This paper presents a proposal for developing a human-machine interface that allows the user to interact with objects modeled in a virtual space using the Wii Remote control of the video game Nintendo Wii. Thus, the input device to capture data pass three information instead of two, in contrast the utilization of the mouse. In possession of the accelerations the control sends to computer, it’s possible to create a cursor to manipulate three-dimensional objects in virtual environment. As a case of study, the image of the virtual world can be projected on a screen and, for example, a Physics teacher can drag blocks, simulating a force on the body, or even move a rope designed to harmonic motions demonstrations. As a second phase of the project, there’s going to be a replacement of the public demonstration mediated by a projection screen for a immersive three-dimensional environment using a HMD device.