Motion Tracking solutions for VR Arena LBE business
1. Introduction
I have mentioned in my previous article that motion tracking technology is a critical part of the VR Arena LBE business as it provides essential elements of the product experiences. It actually provides one of the key differentiating factors for VR (especially out-of-home VR on large arenas) comparing to alternative entertainment products. VR Arena experience is not about wearing a VR helmet but about being completely transferred from the real world into the game experience including all your movements and vestibular senses. Better you do the motion tracking, better the level of the immersion and the whole experience is. Some people also say that they would give the game visual effects quality (which is not perfect anyway with existing helmets’ tech and rendering engines) to the realism of motion within the experience.
So let us review which motion tracking solutions are available on the market and which of them can be used within the VR Arena environment. We will concentrate only on those which are suitable for large VR Arenas venues (bigger than 100–150 m2) assuming VR Arena business for free roam multiplayer full-body motion tracking experiences.
Trending AR VR Articles:
But before we do that, let us also define major technical and business requirements for that sort of solutions:
- Up to 20 (10 minimum) players within game scenario simultaneously
- Tracked Arena sizes 200–500 m2
- Full-body motion tracking (at least 6 tracked positions for IK model)
- Guns tracking (every person)
- Other in-game objects tracking (e.g. torches, swords, sticks, planks, etc.), e.g. up to 20 objects
- Total number of minimum ~100 simultaneously tracked positions
- Low latency for all tracking telemetry (less than 3-5 ms)
- Good quality tracking telemetry (persistence and consistency of tracked IK models)
- An operationally-friendly solution, self-calibrating setup
- Should allow working in complex geometry of the location
- An affordable solution which costs preferably do NOT depend on m2 of the Arena.
2. Review of technologies and solutions
There are many technical options and types of motion tracking. But there is no generally accepted terminology on that. Therefore sometimes it could sound confusing or too generic to understand how it works and which benefits and pitfalls every technology has (or the particular implementation of it). I will try to describe major types or groups of tracking technologies in this section.
2.1. Optical outside cameras as active telemetry base stations
“Cameras as active telemetry base stations” means here that telemetry data is collected and transferred to data processing server at multiple base stations (cameras) which are located “outside” - on the walls or ceiling of the Arena. The base station is equipped both with flash and camera. IR flash illuminates the area and camera catches the light from markers (which could be also passive/reflective or active/emitting light) located on players or tracked in-game objects.
There are several vendors providing that sort of solutions including OptiTrack, Vicon, Qualisys, ART and others. Normally they use 2 types of cameras within the tracking setup: a) cameras with more Mpixels to allow better angular resolution and b) cameras with better framerates to enhance time-resolution and catch fast moving objects. Also, the recent trend is to use only active trackers on gamers and in-game tracked objects.
Solutions are technologically homogeneous and provide off-the-shelf full-body tracking capabilities (depending on number and location of markers on gamers - one will need at least 6 tracked positions to rebuild good quality VR avatar within the experience) with low latency and good quality telemetry if special care is taken about number and position of cameras for particular Arena geometry.
Most of VR Arena LBE networks use either of them for their venues. They are good enough, proved to work and quite practical to use for local operations.
There are only 2 pitfalls there:
- All these motion tracking solutions require more and more cameras if you build a bigger arena. Setup costs are roughly 1,000 USD / m2 and grow linearly with square meters. You can build it for 100m2, for 400 m2 it gets very expensive and it’s a full showstopper for football-field size arenas (not to speak about technical problems and limitations setting up the sport this big).
- Setup is sensitive to the Arena geometry. Preferably it should be empty square (rectangular) geometry without internal columns/pillars. Ceilings should be at least 3.5–4 m high. Geometric complications will make your solution even more expensive as it will require extra cameras to get rid of dead tracking zones.
2.2. Optical with inside trackers as active telemetry base stations
“Trackers as active telemetry base stations” means here that telemetry data is collected and transferred to data processing computer at every single tracker which is located on gamers or other tracked objects. “Outside” are only illuminating base stations located on the walls or ceiling of the Arena. These base stations normally only emit laser flashes which every tracker picks up to calculate telemetry data.
Most well-known vendor of such technology is VIVE with their VIVE SteamVR Tracking technology. But there are some others including, for example, StepVR and VR Tracker.
Those solutions are also homogeneous and can provide full-body motion tracking providing that enough markers for players (especially in the multiplayer environment) and objects are supported.
Despite these solutions look technologically interesting and there are working examples though they are not so widely used. Major reasons are:
- There is no proved technology which can support large locations (more than ~6x6 m2). VIVE SteamVR Tracking 2.0 technology released early 2018 was supposed to support that requirement allowing more base stations and “roaming” between ~6x6 m2 cells, but it is still not available for commercial use. Worth saying that another vendor, StepVR, claims that their solution supports large setups (more than 200 m2) but I didn’t have a chance personally to make sure it works well (though it might be OK).
- There are some limitations for the number of trackers that can be used within the setup and the way they are connected to the computer (wired or wireless) which limits multiplayer scenarios and in-game objects tracking. Just count that for instance 10 gamers and 12 in-game objects (e.g. 8 guns and 2 torches) will require at least 72 trackers working simultaneously and at least 6 (7 with the gun) of them should be wirelessly connected to the same gamer’s backpack (each has its own wireless dongle).
2.3. Optical inside-out tracking with markers
In this technology tracking devices collecting telemetry are optical cameras located on gamer and in-game object (same as for inside-out markerless tracking) and markers (passive or active) are located outside (on the ceiling and/or walls). There are solutions which support IR cameras and passive IR reflecting markers, optical cameras with passive markers which look like QR-codes, or active markers located on the floor or ceiling/walls. Usage of markers makes the problem of depth sensing and objects/movements recognition more simple comparing to markerless optical tracking and there are working examples available on the market (though not as ready-to-use platforms but as enthusiasts’ activities building solutions for their local use). Which is why it might be first to work within commercial setups.
Major benefits are:
- Solutions are relatively cheap as cameras are located on gamers and objects and overall setup cost does not depend on Arena m2 (“outside” markers are really cheap).
- One can use the location of almost any size and geometry, including rooms, doors, narrow passages, ladders, etc.
It all sounds really good and a lot of teams are trying to move forward with that type of tracking though there is significant pitfall: it is difficult (impossible now with solutions available on the market) to support full-body motion tracking without implementing sensor fusion adding some other types of tracking technology for that. It works fine for the position, head and hands tracking but there are problems with legs and in-game objects.
Once those obstacles are removed and there is reliable and low latency tech sensor fusion tech for that, it could be a game changer for the whole VR Arena full-body motion tracking LBE market same as we mentioned above regarding optical outside-in markerless tracking.
2.4. Optical outside-in and inside-out markerless tracking
One could see many recent examples of optical markerless tracking which is pushed forward by advances in computer vision algorithms (depth sensing, SLAM, pose estimation, etc.), robotics and AR technologies. It requires a lot of math modeling and data processing power compared to “conventional” tracking techniques. Good recent examples pick up e.g. real optical camera shooting or YouTube video and provide real-time full-body motion tracking capabilities from “normal” 2D video. Some pieces of this technology are now very popular as mobile applications allow real-time retargeting (transferring your movements or face mimics shot by smartphone camera into movements of artificial avatar).
Outside-in and Inside-out differ only in a way where the camera is located: outside (walls, ceiling) or inside (on gamer or on a tracked object).
This technology could be the real game changer for full-body motion tracking application providing the following conditions are satisfied:
- It can be reliable (telemetry is continuous and consistent).
- It can work in real-time with minimum latency (less than 3-5 ms).
- It can support multiplayer environments solving problems with occlusion, correct key tracking positions identification and grouping them to right rigid body etc.
- It can recognize and track all required in-game objects with the same quality as people.
- It can cover large locations without costly setup.
At the moment we are not yet there to use this tech commercially as most of the issues mentioned above are still open.
2.5. Inertial tracking
Inertial tracking is precise and low latency with all trackers being compact and located on gamers or objects with the wired or wireless connection to the computer. No external devices required. It is also relatively cheap.
There are many vendors providing that sort of solutions. You can buy ready to use sets like those which e.g. Xsense provides, or purchase individual components from dozens of potential vendors.
There is still one pitfall which makes that sort of solutions practically unusable for VR Arena application in the homogeneous environment. Inertial tracking telemetry has significant drift (which will grow along with your game timing reaching unusable values quite soon) which can not be avoided without some other technology being used to provide “calibration” or “synchronization” for inertial tracking data. It basically means that inertial tracking systems cannot be used without the sensor fusion approach, i.e. tracking technology which assumes that there is no homogeneous environment but one needs to integrate on physical and logical levels telemetry from several different sources.
2.6. Homogeneous vs. sensor fusion technologies for full-body motion tracking
It is not clear whether some cheap homogeneous solutions will be used on VR Arena full-body motion tracking market or some heterogeneous sensor fusion technologies will prevail. It might depend on ongoing progress with different types of the tracking technologies mentioned above and with the market of VR LBE growing big enough to attract the attention of major vendors rather than several engineering enthusiasts.
It should be mentioned that the solution is supposed to be not only technically advanced but also allow practical usage on locations assuming multiple wearing on/off cycles during the day, the reliability of the components, simplicity of repairings etc.
2.7. Other tracking technologies
There are some other tracking technologies, e.g. those based on a radio-frequency band instead of optical (e.g. Fraunhofer IIS solutions). The advantage is a possibility to cover large areas (e.g. football field size) but pitfalls are with precision and full-body tracking support (works for position tracking only).
2.8. Mediation
When you integrate your tracking system into the overall VR motion tracking platform and finally into your VR experience you need to take care of interfaces from the telemetry system to your gaming engines like Unity or Unreal. Most of them do have such interfaces. Nevertheless, your solution might not work well unless you implement additional tools which help to manage tracking data. They are used to define IK (inverse kinematics) models and mediate raw tracking data to get persistent and consistent data-stream. For that purpose, one would normally use either game engine plugins from the corresponding marketplaces or independent solutions (e.g. IKinema software). It improves significantly the way your virtual avatars look and move bringing in related IK math models and sometimes ML tools (motion “patterns”).
3. Conclusions and trends
As of now, the only working full-body motion tracking technology for VR Arena LBE applications is optical tracking with markers and outside cameras as tracking base-stations. It works OK (most of VR Arena LBE networks use one of those solutions) but gets very expensive for large size Arenas which is a serious slowing factor for VR Arena LBR business expansion.
There could be several trends or options for the future, but it’s quite difficult to predict which will be mainstream due to multifactor dependencies — market sizes, technical issues, tech complexity, practical issues related to operations, etc. Let’s mention most important cases:
- Optical tracking with markers and outside cameras as tracking base-stations gets significantly cheaper (which is unlikely to happen due to several reasons).
- HTC SteamVR Tracking (or similar) solutions will work for large Arenas providing modular, simple and cheap setup out-of-the-box. Might happen if VIVE will see the market worth their efforts to make the product work fine or if there will be a similar platform dedicated to VR Arena LBE business.
- Sensor fusion solutions (e.g. a mix of inside-out tracking with markers + inertial) will be provided as ready-to-use platforms or components which will not require complex technical and software integration. There are no any available at the moment (as far as I know).
- Optical markerless tracking solutions will become fast (low latency), precise and reliable within cheap and modular outside-in or inside-out (or combination) setup. That would be the best option from a practical perspective as it removes many operational issues and complications.