Accurate Mixed Reality with LPVR-CAD and Varjo XR-3

Anchoring Virtual Objects with Varjo XR-3

A key aspect of making a mixed reality experience compelling and useful is to correctly anchor virtual content to the real world. An object that’s fixed in its position and orientation (pose) relative to the real world should not change its pose as seen by the user when they’re moving around in the scene wearing the headset. Imagine a simple virtual cube positioned to be sitting on a real table. As the user walks around, this cube should remain in place, regardless from which perspective the user looks at it.

In more detail, correct anchoring of virtual objects to reality depends on the following points:

  1. In order to create correctly aligned content in a mixed reality experience, accurate knowledge of the user’s head pose is essential. We calculate the head pose using LPVR-CAD.
  2. The field of view provided by the cameras and the natural field of view of the human eyes are very different. Appropriate calibration is needed to compensate for this effect. The illustration below shows this inherent problem of video pass-through very clearly. Varjo’s HMDs are factory calibrated to minimize the impact of this effect on the user experience.

To get a better idea how a correct optical see-through (OST) calibration influences mixed reality performance we recommend playing with the camera configuration options in Varjo Lab Tools.

– Image credit: Varjo mixed reality documentation

Functional Testing of MR Performance

Our LPVR solution must at the very least achieve a precision that is satisfactory for our users’ typical applications. Therefore we decided to do a series of experiments to evaluate the precision of our system for mixed reality experiences and how it compares with SteamVR Lighthouse tracking.

There are 4 basic variables in the testing of this issue. Testing each one methodically should uncover where the problem resides so that we can take steps to correct it. The four variables involved in the setup are:

  1. The VR software and version (Unity, UE)
  2. The tracking system and driver (ART Smartrack 3 with LPVR, Valve Lighthouse with Stream VR)
  3. The type and use of markers (Varjo markers)
  4. The HMD (Varjo XR-3, XR-4, Meta Quest, XReal..)

To reduce variables and simplify testing, we looked at the following configurations:

# HMD Engine Tracking system Varjo markers
1 Varjo XR-3 Unreal 5.2 LPVR-CAD No
2 Varjo XR-3 Unreal 5.2 Lighthouse No
3 Varjo XR-3 Unreal 5.2 LPVR-CAD Yes
4 Varjo XR-3 Unreal 5.2 Lighthouse Yes

1 – LPVR Tracking without Varjo Markers

In this scenario we fixed a simple virtual cube on a table top in a defined position and orientation. The cube is the only virtual object in this scenario, everything else is the passed-through live video feed from the HMD cameras. The user puts on the HMD and moves around the cube: head rotations, moving closer / further away from the cube, walking around it.

While the user is moving around, we watch the video feed recorded through the HMD and observe by how much the cube is shifting from its original location while the user is moving.

We are tracking the HMD using LPVR-CAD in connection with a Smarttrack 3. No markers are used to stabilize the pose of the cube.

Important note: The tracking performance for the mixed reality use case can significantly change if the marker target that is attached to the HMD is not correctly adjusted. Please refer to the LPVR-CAD documentation or contact us for further support.

2 – Lighthouse Tracking without Varjo Markers

This scenario is in its basic setup identical to scenario #1, except that we’re using Lighthouse tracking to find the pose of the HMD. In terms of tracking performance Lighthouse tracking behaves similarly to LPVR, except that the maximum tracking volume size is limited for Lighthouse tracking and can be almost infinitely extended with LPVR.

3 – LPVR Tracking with Varjo Markers

Regardless how well the tracking of the HMD itself is working, as long as distortions of the environment as seen via the video pass-through feed aren’t perfectly compensated, there will be a discrepancy of where objects are displayed in reality and in virtual view space. As there are limits to the precision of such an optical see-through calibration (OST), another way to compensate for its effect is to get additional information about the environment directly from the video feed and align objects to it.

Such a tool are Varjo markers, ie. QR codes that are placed in the image. Using image analysis, virtual objects can be fixed to such QR codes and therefore automatically realigned to teh video feed as the user moves around. The video below shows the result of this scenario.

4 – Lighthouse Tracking with Varjo Markers

In our final test scenario we did the same experiment as in scenario 3, just with Lighthouse tracking instead of LPVR tracking.

Conclusion

See a table with our preliminary findings below:

Test Scenario LPVR Tracking Lighthouse Tracking
Without Varjo Markers 2 2.5
With Varjo Markers 1.5 1.5

– Approximate displacement error on horizontal plane (in cm)

Using the same room setup and test scene, the mixed reality accuracy of LPVR-CAD and Lighthouse tracking is similar. With both tracking systems slight shifts of 1-2cm depending on the head movement can be observed. A way to further reduce this residual drift is to use Varjo markers that further align virtual objects with the video feed from the pass-through cameras. Good results with LPVR tracking require precise adjustment of the optical target attached to the headset.

Note that our method of estimating the displacement error is rather qualitative than quantitative. With this post we made a general comparison of LPVR and Lighthouse tracking, with and without Varjo markers. A more quantitatively accurate evaluation will follow.

For customers wanting to reduce as much drift as possible, we recommenced the use of markers and optical tracking. There may be different results using Varjo XR-4 and other variations on the tracking environment or displayed content, which could warrant further testing in the future.

LPVR-AIR for Immersive Collaborative Industrial Design

Wireless Content Streaming

LPVR-AIR is LP-Research’s wireless VR streaming solution. Content is generated on a rendering computer and wirelessly streamed to a VR headset to be displayed. At the same time the pose, orientation and position, of the headset is calculated from tracking data from a camera system and inertial measurements on the headset itself.

The core tracking algorithm of LPVR-AIR is similar to our LPVR-CAD solution. We are combining this established tracking method with wireless data streaming.

This has a few significant advantages:

  • Rendering detailed VR content is computationally too heavy to do all calculations on embedded hardware on the headset itself. Therefore, content needs to be rendered on an external computer and the result is streamed to the headset. LPVR-AIR allows doing this.
  • Designers in eg. the automotive space have their own preference of applications to create content, such as Autodesk VRED. These applications usually don’t run on a headset’s embedded hardware. With LPVR-AIR dsigners can use any application that normally works with a Windows based PC.

Technical Implementation

See below a block diagram of how the LPVR-AIR system is implemented. While we in principle support any Android based standalone VR headset, we currently focus on the Meta Quest line of HMDs, specifically Meta Quest Pro and Meta Quest 3.

Our solution effectively enables designers to explore a large 3D design space with full high resolution renderings using a lightweight headset. LPVR-AIR even allows for the interaction of several users in a design space. An example of such a use case is shown in the video on the top of this post. Two users in our office in Tokyo, being tracked by LPVR tracking, explore a car design together.

Improved Design Process

This opens new possibilities for automotive, industrial, architecture and many more design applications, leading to increased performance of designers and a higher success rate of their designs. LPVR-AIR is based on the ALVR wireless streaming engine, which we have extended to work with our FusionHub sensor fusion solution.

Long term, the ALVR engine makes it easy for us to support a number of different HMDs, additionally to Meta Quest also the Varjo and eventually the Apple Vision Pro series as shown in the image below. With VRED we have an outstanding rendering solution at the base of LPVR-AIR that allows designers to create photo-realistic content while providing extensive collaboration abilities.

If you would like to move towards immersive and interactive 3D design, don’t hesitate to consult with us and give our LPVR-AIR on-premise collaborative design solution a try!

Immersive Driving Assistance with LPVIZ

How LPVIZ Augments Driving Reality

Going beyond a simple screen replacement, LPVIZ is an augmented reality driving assistance solution for the car. It allows displaying related content to a driver or passenger in 3D, superimposed to reality. Content can be placed anywhere inside the car, such as a virtual speedometer over the dashboard, and anywhere outside of the car, such as point-of-interest markers or navigation guidance.

The video on top of this post shows what a drive around the block in Azabujuban, Tokyo with LPVIZ looks like. A virtual dashboard is projected onto the center console of the vehicle. Arrows on the ground show lane guidance to the driver. Red Google Maps-style markers show points of interest. The virtual dashboard stays fixed to the same location in the car, even when the vehicle turns. The navigation arrows move smoothly and the point-of-interest markers are globally anchored.

Perfectly Tuned Components

LPVIZ consists of several components that all have to interact perfectly to create a compelling and safe augmentation experience. The below illustration shows a block diagram of how the hardware components are connected.

Accurate tracking is required to display useful content to the driver: the HMD pose in the local car coordinate system and the vehicle pose in a globally anchored frame. Precise calibration of all components of the solution is essential to provide the highest visual fidelity and driver safety. Our LPVIZ product makes all parts of the system available in a compact form factor, ready to be integrated with any vehicle.

The Past, Present and the Future

In the current development stage we’re focusing on the most essential aspects of the solution: displaying a virtual dashboard, navigation information and points-of-interest. While this is our proprietary content, we’re opening our software to work with 3rd party developers to create their own content building on our platform.

Currently we’re offering LPVIZ as a B2B solution for prototyping, design and research. However, we’re working on reducing system complexity to make it work as a consumer facing automotive after-market solution to be released later this year.

Towards a Consumer Product

We are very proud of the progress our team has made in the past months. We’re moving closer to making our vision of an augmented reality driving assistance system a reality for everyone. One very important take-away from our recent developments is that it’s indeed possible to provide real utility to the driver using technology that is readily available. It might still be early days, but we’re edging towards a product that could appeal to a wider consumer market. This is just the beginning.

LPVR-DUO in an Airborne Helicopter

In-Flight VR

Imagine soaring through the skies as a pilot, testing the limits of a helicopter’s capabilities while feeling the rush of wind and turbulence. Now imagine that you don’t see the real world outside and the safe landing pad that your helicopter is approaching but a virtual reality (VR) scene where you are homing in on a ship in high seas. The National Research Council Canada (NRC) and Defence Research and Development Canada (DRDC) have brought this experience to life with their groundbreaking Integrated Reality In-Flight Simulation (IRIS).

IRIS is not your ordinary simulator; for one, it’s not sitting on a hexapod, it’s airborne. It’s a variable-stability helicopter based on the Bell 412 that can behave like other aircraft and can simulate varying weather conditions; combine that with a VR environment and you have a tool that allows safely training operations in the most adverse conditions. In particular it is used for Ship Helicopter Operating Limitations (SHOL) testing.

Mission-Critical Application with LPVR-DUO

The LPVR-DUO system is what makes VR possible on this constantly moving platform. This cutting-edge AR/VR tracking system seamlessly merges the inertial measurements taken by the headset with the helicopter’s motion data and a camera system mounted inside the cabin to provide the correct visuals to the pilot. The challenges of using cameras to track the VR headset inside the tight environment of the helicopter while lighting conditions are ever-changing are overcome by using an ART SmartTrack 3 system. This system follows an arrangement of reflective markers attached to the pilot’s helmet. The VR headset is attached to the helmet in such a way that the pilot can wear it as if it were a pair of night vision goggles. Put together, this allows displaying a virtual world to the pilot, even in the most extreme maneuvers.

To ensure an authentic experience, the IRIS system incorporates real-time turbulence models, meticulously crafted from wind tunnel trials. These turbulence effects are seamlessly integrated into the aircraft’s motion and into the VR scene, providing pilots with precise proprioceptive and vestibular cues. It’s a symphony of technology and innovation in the world of aviation testing.

In-Cockpit Implementation

The optical tracking system relies on highly reflective marker targets on the helmet to track movement in three dimensions. Initially, only five markers were installed, strategically placed for optimal tracking. But the pursuit of perfection led NRC to create custom 3D-printed low-reflectivity helmet molds, allowing them to mount a dozen small passive markers. This significantly improved tracking reliability in various lighting conditions and allowed for a wider range of head movement.

Recently, NRC put this remarkable concept to the test with actual flight trials. The response from pilots was nothing short of exhilarating. They found the system required minimal adaptation, exhibited no noticeable lag, and, perhaps most impressively, didn’t induce any motion sickness. Even the turbulence effects felt incredibly realistic. Surprisingly, the typical VR drawbacks, such as resolution and field of view limitations, had minimal impact, especially during close-in shipboard operations. It’s safe to say that IRIS has set a new standard for effective and immersive aviation testing.

Publication of Results

The NRC team presented their results at the Vertical Flight Society’s 79th Annual Form in two papers [1] and [2] and they also have a blog post on their site.

NOTE: Image contents courtesy of Aerospace Research Centre, National Research Council of Canada (NRC) – Ottawa, ON, Canada

LPVR Support for Varjo XR-4

Varjo Releases Mixed Reality Headset XR-4

A leap forward in XR comes from our partners at Varjo, who have been pushing what is possible in VR and AR for the past few years.

Today, they announced their new flagship headset, the XR-4! We have worked with them to ensure that our LPVR software series is ready for it from the start.

The XR-4 features unmatched visual fidelity not only of VR content (expanded to a field-of-view of 120×105 deg), but also of mixed-reality video pass-through, reaching a pixel density of 51ppd in the central viewing area. Light sensors further improve the quality of immersion by adjusting the image display to external lighting conditions.

LPVR Support is Ready

All this is great, but what is a headset without a world to immerse in? What is a VR racetrack without feeling the real motions of the car? Varjo clearly understood and prepared, so they collaborated with us to get LPVR ready for the launch of the XR-4.

We have everything prepared for you!

  • Use your existing camera systems and props to augment the virtual world.
  • Use the industrial-grade precision of ART, OptiTrack, and Vicon tracking systems with the Varjo XR-4 and our custom marker holders.
  • Integrate the HMD with your race-car simulator or fighter jet platform.
  • Do all of this with the XR-4. LPVR-CAD, and LPVR-DUO will make sure that you are tracked perfectly.

 

We offer full solutions of state-of-the-art tracking systems and content using the Varjo XR-4. Contact us for more information.

1 2 3 14