VR with Oculus Rift, Kinect and LabVIEW

Virtual Reality Application with LabVIEW using the Oculus Rift and the Microsoft Kinect v2

To pursue the development of the Haro3D™ library, a driver was developed to interface LabVIEW with the virtual reality headset Rift Development Kit 2 from Oculus. To demonstrate the potential of the new driver, an application was developed that also uses the Kinect V2 from Microsoft.

The current project is provided to obtain a feedback from the community about LabVIEW applications related to virtual reality, the Oculus Rift headset, the Kinect, and the 3D picture tools. Comments related to any other subject are also welcome.

The spaceship application is kind of a simple game that consists to pilot a small spaceship around a planet very similar to the earth. There are two satellites orbiting the earth, one very similar to the moon, and the other one very similar to Jupiter. Realism of gravitational effects was not a concern here. The spaceship orbits around the planet at a constant angular speed but its roll and its altitude can be controlled. The roll and altitude are controlled through the Kinect using gestures. The view of the pilot inside the spaceship is rendered in stereo in the Oculus Rift headset. A DLL was developed to obtain the position and orientation information from the Rift headset from within LabVIEW. The Oculus Rift positional tracker was positioned on top of the Kinect V2 (shown in the picture below) in a manner to make its orientation approximately match the orientation of the coordinate system of the Kinect. The position information from the Oculus Rift tracker is not used in the application. The position information from the Kinect is used instead. The head position as provided by the Kinect is used as the position of the eyes (after including the offsets associated with the inter-pupillary distance). The orientation of the view is determined by the orientation provided by the Oculus Rift tracker. In addition, the Kinect provides information for the positions of 25 body joints, making it possible for the user to see the hands and feet of his avatar, corresponding to his own hands and feet.

The two stereoscopic images in this application are rendered for an inter-pupillary distance of 64 mm. Pincushion and chromatic distortions are not compensated for. It does not appear possible to make those compensations using the current LabVIEW tools. However, it is not a major concern for the current application because the virtual world produced here is not an attempt to reproduce a normal human environment. A sitting position for the user is strongly recommended because of possible disorientation that the user might experiment with the use of the Oculus Rift headset. In some cases, the Kinect has difficulties to recognize gestures probably because of the sitting position but also because of the presence of the headset. The user should be sitting at a distance of at least 2 meters (6 feet) from the Kinect (see figure below).

When the hands and arms are low, the Kinect tends to not recognize the presence of the user. This is an issue when the user is making moves that are recorded over some time (like grabbing up or down to change altitude) because the loss of the user tracking might result in the reset of the hand tracking over time. One solution is to keep the hands extended and at or above shoulder level for the altitude change gestures. The user can turn his head and look around. He can even look at the back of the spaceship, the Rift tracker can follow the orientation up to 180° head turns and beyond. The Kinect tracks the position of the body the spaceship is positioned relative to the head position. It is therefore not possible for the user to “exit” the spaceship. If the user moves around in the real world, the spaceship will follow in the virtual world, adding to the possible disorientation sensation of the user.

With a Kinect present, the user can make the spaceship roll to its left or right by using the lasso hands (closed hand with the middle and index fingers extended) on each side of his body. Circular movements of the two hands in the clockwise direction makes the spaceship roll to its right, and counterclockwise movement make it rolls to the left. The movement changes the rolling rate, not the position of the hands. The user can open both hands to stop the rolling at the current angle at any moment. The rolling rate is capped when using the Kinect but not with the keyboard. The user can also change the altitude of the spaceship. The altitude is displayed in real time on the control panel of the spaceship. The altitude is increased when using the Kinect by closing the right hand and moving the hand down (pulling the sky down). Closing the right hand and moving the hand up decreases the altitude. Each gesture need to be clearly separated in time (at least 2 seconds) because the Kinect will interpret movement on the way up as well as those on the way down. Also, it is better to keep the left hand closed during altitude changes because both open hands are interpreted as a different gesture (stop rolling).

This application is only an illustration of the potential of the combination of the Oculus Rift, the Kinect, and LabVIEW. The use of the Kinect to control an application with a virtual reality headset is very beneficial because the user does not have to attempt to interact with elements outside the virtual world, like a keyboard or a mouse for example. Obviously, LabVIEW is not a platform for the development of games or virtual reality applications for large audiences. For rendering of situations more similar to normal human environments, the lack of lens compensation could become a major issue when using the Rift for some period of time. In any case, the capabilities of LabVIEW to quickly generate and render 3D objects are worth further exploration for virtual reality, especially for industrial and scientific applications.


Meet The Maker

Marc Dubois

More Projects