Large-scale VR Application Case: the Holodeck Control Center

The AUDI Holodeck

Our large-scale VR solution allows any SteamVR-based (e.g. Unity, Unreal, VRED) Virtual Reality software to seamlessly use the HTC VIVE headset together with most large-room tracking systems available on the market (OptiTrack, Vicon, ART). It enables easy configuration and fits into the SteamVR framework, minimizing the effort needed to port applications to large rooms.

One of our first users, Lightshape, have recently released a video showing what they built with our technology.  They call it the Holodeck Control Center, an application which creates multi-user collaborative VR spaces. In it users can communicate and see the same scene whether they are the same real room or in different locations. The installation showcased in the video is used by German car maker Audi to study cars that haven’t been built yet.

Our technology is essential in order to get the best VR experience possible on the 15m × 15m of the main VR surface, combining optical tracking data and IMU measurements to provide precise and responsive positioning of the headsets.  Please have a look at Lightshape’s video below.

Ready for the HTC Vive Pro

In the near future, this installation will be updated to the HTC Vive Pro which our software already supports. The increased pixel density of this successor of the HTC Vive will make the scenes look even more realistic. The resolution is high enough to actually read the various panels once you are in the drivers seat! Besides that, we are also busy studying applications of the front-facing cameras of the Vive Pro in order to improve multi-user interaction.

Optical-Inertial Sensor Fusion

Optical position tracking and inertial orientation tracking are well established measurement methods. Each of these methods has its specific advantages and disadvantages. In this post we show an opto-inertial sensor fusion algorithm that joins the capabilities of both to create a capable system for position and orientation tracking.

How It Works

The reliability of position and orientation data provided by an optical tracking system (outside-in or inside-out) can for some applications be compromised by occlusions and slow system reaction times. In such cases it makes sense to combine optical tracking data with information from an inertial measurement unit located on the device. Our optical-intertial sensor fusion algorithm implements this functionality for integration with an existing tracking system or for the development of a novel system for a specific application case.

The graphs below show two examples of how the signal from an optical positioning system can be improved using inertial measurements. Slow camera framerates or occasional drop-outs are compensated by information from the integrated inertial measurement unit, improving the overall tracking performance.

Combination of Several Optical Trackers

For a demonstration, we combined three NEXONAR IR trackers and an LPMS-B2 IMU, mounted together as a hand controller. The system allows position and orientation tracking of the controller with high reliability and accuracy. It combines the strong aspects of outside-in IR tracking with inertial tracking, improving the system’s reaction time and robustness against occlusions.

Optical-Inertial Tracking in VR

The tracking of virtual reality (VR) headsets is one important area of application for this method. To keep the user immersed in a virtual environment, high quality head tracking is essential. Using opto-inertial tracking technology, outside-in tracking as well as inside-out camera-only tracking can be significantly improved.

Machine Learning for Context Analysis

Deterministic Analysis vs. Machine Learning

Machine learning and artificial intelligence (AI) are important methods that allow machines to classify information about their environment. Today’s smart devices integrate an array of sensors that constantly measure and save data. On the first thought one would image that the more data is available, the easier it is to draw conlusions from this information. But, in fact larger amounts of data become harder to analyze using deterministic methods (e.g. thresholding). Whereas such methods by themselves can work efficiently, it is difficult to decide which analysis parameters to apply to which parts of the data.

Using machine learning techniques on the other hand this procedure of finding the right parameters can be greatly simplified. By teaching an algorithm which data corresponds to a certain outcome using training and verification data, analysis parameters can be determined automatically or at least semi-automatically. There exists a wide range of machine learning algorithms including the currently very popular convolutional neural networks.

Context Analysis

Many health care applications rely on the correct classification of a user’s daily activities, as these reflect strongly his lifestyle and possibly involved health risks. One way of detecting human activity is monitoring their body motion using motion sensors such as gyroscopes, accelerometers etc. In the application described here we monitor a person’s mode of transportation, specifically

  1. Rest
  2. Walking
  3. Running
  4. In car
  5. On train

To illustrate the results for deterministic analysis vs. machine learning approach we first implemented a state machine based on deterministic analysis parameters.

The result is a relatively complicated state machine that needs to be very carefully tuned. This might have been because of our lack of patience, but in spite of our best efforts we were not able to reach detection accuracies of more than around 60%. Before spending a lot more time on manual tuning of this algorithm we switched to a machine learning approach.

The eventual system structure looks noticeably simpler than the deterministic state machine. Besides standard feature extraction, a central part of the algorithm is the data logging and training module. We sampled over 1 milion of training samples to generate the parameters for our detection network. As a a result, even though we used a relatively simple machine learning algorithm, we were able to reach a detection accuracy of more than 90%. A comparison between ground truth data and classification results from raw data is displayed below.

Conclusion

We strongly belief in the use of machine learning / AI techniques for sensor data classification. In combination with LP-RESEARCH sensor fusion algorithms, these methods add a further layer of insight for our data anlysis customers.

If this topic sounds familiar to you and you are looking for a solution to a related problem, contact us for further discussion.

Sensor Fusion for Virtual Reality Headset Tracking

In order to test the functionality of our sensor fusion algorithm for head-mounted-display pose estimation, we connected one of our IMUs (LPMS-CURS2), a Nexonar infrared (IR) beacon and a LCD display to a Baofeng headset. The high stability of the IR tracking and the orientation information from the IMU as input to the sensor fusion algorithm result in accurate, robust and reactive headtracking. See the figure below for details of the test setup. The video shows the resulting performance of the system.

Field Distortion Compensation Algorithm

If the LPMS is operated in an environment with a partially distorted (non-homogeneous) earth magnetic field, there is the possibility of the orientation readings becoming inaccurate due to invalid data from the magnetometer unit. To prevent this we have extended our sensor fusion algorithm to detect such field distortion and automatically switch to operation without magnetometer. The switching between the two states happens seamlessly (without orientation jump) and, if the exposure to the distorted magnetic field happens for a limited amount of time, without any major orientation drift.

Please see the video below for a demonstration of the improved filter. An iron plate is used to distort the magnetic field. As soon as the sensor gets close to the metal surface the magnetic field vector starts changing direction deliberately. The color of the cube on the monitor turns red in case of the detection of a distorted magnetic field.