Building in AR: Some basics and where to start

--

Have you heard of Augmented Reality or even played around with some AR experiences and thought “hey this is cool,” but felt overwhelmed by all the different platforms, devices, and details that everyone else seems to know?

You are not alone.

AR is an emerging field in terms of popularity, and has experienced monumental shifts the last few years. This makes getting up to speed a bit tough, because you’ll read about a platform that sounds like it has everything you need but only to find out they were acquired or went bankrupt.

Trending AR VR Articles:

1. Augmented Reality Robotics

2. Infographic: The Future of Virtual Reality

3. Mario Kart in a real vehicle with VR!

4. Virtual Reality Headsets: What are the Options? Which is Right For You?

In this post I will give a high level overview of the current AR landscape, helpful frameworks /tools, and what are the emerging use-cases, as a way to get everyone up to speed and thinking about AR.

Augmented Reality encompasses any interface that can be applied to any sensory based Human-Computer Interaction.

Yes these are very broad terms, but AR comes in many forms and I’d rather not limit imaginations. For this post, most of the AR devices and platforms we will discuss operate along similar core principles: use sensor data to enhance the user experience by making it feel more immersive.

During my time at Blippar as a solutions engineer, I had the opportunity to build Augmented Reality experiences for a range of clients (M&M’s, General Mills, Porsche, CocaCola, Kelloggs, L’Oreal) across a wide range of industries . Even though one of the biggest struggles is the concept, once you have the concept the next set of challenges feel like running hurdles in a track meet.

Let’s assume you have your idea, the next set of questions are:

  • What platforms are available to me?
  • What are the trade offs and considerations?
  • Which platform do I choose?
  • Are there any frameworks that I can leverage?

If you follow the AR space and read the headlines it’ll just add to the list of questions. For example, Apple recently released ARKit3 and RealityKit, does that mean SceneKit is done? Facebook and Snapchat filters have been seeing great engagement, will an AR filter suffice? What’s the story with WebAR and should you build a web based AR experience? Are there any drag and drop platforms like Wix or Squarespace but for AR?

Structure of blog:

  • Key terms
  • Platforms overview
  • Tools and Popular frameworks
  • Trends and use cases

I will apologize now for my lack of transitions as I try to keep things concise, given the breadth of topics to be covered.

Key Terms

Augmented reality inherits many concepts and terminology from the existing 3D vernacular. For this reason some terms may seem foreign to a newcomer.

  • Mesh — A collection of points that are connected to form planes to represent the geometry of a 3D model. The number of planes within the mesh is known as the poly count.
  • Model — A single mesh or a collection of 3D meshes and 2D images that dictate how light interacts (reflection, refraction, absorption), color) with the model. Resolution is dictated by the number of points in the mesh, for example a “low poly” model (as needed for AR) has fewer points which in turn form fewer faces resulting in a low poly count. The manner in which the 2D images are connected to the 3D faces is known as UV mapping.
low to high poly models of the same 3D object — source: https://cathyatseneca.gitbooks.io/3d-modelling-for-programmers/content/3ds_max_basics/3d_representation.html
  • Scene — A collection of 3D models and virtual lights that are represented in the form of a scene-graph.
  • Renderer — A framework or application that generates a 2D visual representation of a virtual scene; In augmented reality the renderer uses the camera and motion sensors to capture the users viewpoint and accurately draw in the virtual scene.
  • Tracking (face, marker, world) — In Augmented Reality virtual objects are projected into the real world; There are many ways achieve this effect, but each instance computer vision is used to detect a feature of interest (facial features, an object, a 2D image or the world itself) within the video input, and then that data is passed into the renderer to accurately draw the virtual content into the video, with proper scale and rotation (using the device’s motion sensors). In the instance of world-tracking, the computer vision uses the motion sensor data to understand the device’s movement in relation to specific points within the video input.
ARKit debug mode, displays the feature points the device is tracking — source: https://hackernoon.com/supercharge-your-arkit-interior-design-app-with-a-resume-session-feature-9da31252a266
  • Voxel — In 2D graphics a pixel represents the smallest unit of measure for an image, but when moving to 3D we must account for depth so the pixel takes on a new name “volumetric-pixel” or voxel for short.

Platforms

One of the first decisions you’ll have to make when building an AR experience is which platform is the best for your use-case. There are many options each with their own trade-offs. Lets first take a look at what are all the options and a little about each .

NOTE: I won’t dive too much into detail as there are tons of resources online for each, but I will name the platforms worth knowing, give a general overview and some noteworthy details. In the end it’s up to you to pick the best platform for your project.

Native

In the mobile space native apps tend to have better performance for computationally intensive operations (such as AR), though web based AR has made lots of progress. For very immersive experiences it’s recommended to use a native platform. When talking native there is pure native, and there is cross-platform native. We’ll start with pure native and get into cross-platform next. In the mobile OS world each major player has their own framework for AR, Android has ARCore with SceneForm and Apple has ARKit with SceneKit and RealityKit.

Let's start our platform evaluation with Android. Google was first to market with an AR framework for Android devices, known as Tango. The framework required built-in depth sensors, so only a few devices were released commercially that leveraged the Tango framework. Though Tango never gained much mainstream success, shortly after Apple announced ARKit, Google was quick to respond with ARCore, it was Tango without any reliance on depth sensors, it used the device’s camera and motion sensors. The framework supports plane detection/tracking, SLAM tracking, and FaceTracking. ARCore uses SceneForm as the render engine, though you could bring your own render engine, keep in mind this requires some amount of development resource.

source: https://www.mobileappdaily.com/2017/5/09/googles-latest-gift-to-android-users-brings-augmented-reality-with-arcore

ARCore has one major limitation, (compared to Apple’s Arkit), it’s not as simple as saying only newer Android phones support ARCore, because of the fragmentation issues that face the Android ecosystem. Have no fear, Google provides a list of supported devices.

In the past ARCore didn’t ship natively with the Android OS, meaning that even if a device’s hardware supports ARCore, the user (not you, the user) had to install a service app from the PlayStore. This has changed more recently so all new devices that support ARCore ship with the needed services installed.

Rewind to 2017 (before ARCore) Apple released iOS11, the first iOS to include ARKit. At a high level, ARKit is a framework that leverages the iPhone’s camera and motion sensors. ARKit and ARCore have significant differences, where ARCore is more limited in tracking and rendering than ARKit. With the release of ARKit2, in iOS 12.2, Apple improved upon ARKit 1.5’s plane tracking (adding in vertical plane detection/tracking) and added support for face tracking, dynamic environment maps, shared/multi-user experiences, and persistent experiences, most of which don’t exist as features in ARCore.

With ARkit3, part of iOS13, Apple released upgrades to added support for people occlusion (3d content can appear behind a person), multi-face tracking, using front depth sensor along with rear camera, using both front and rear cameras simultaneously, and RealityKit. When ARKit first launched it used SceneKit, which can be considered a legacy render engine, that was adapted for AR, so it makes sense that Apple would release a render engine written and optimized specifically for AR (RealityKit). Don’t worry you can still continue to use SceneKit with ARKit3, but who knows if that will hold true as ARKit continues to mature.

A few other things to note, RealityKit takes a component entity approach with an effort to simplify some of the verbosity of SceneKit. RealityKit is only half of the suite of tools offered by Apple. As part of MacOS Catalina, Apple includes the RealityComposer studio to preview RealityKit 3D content and make it easier to visual configure the AR content for the scene. To my knowledge RealityComposer does not work with SceneKit, and also has some limitations. For example, when setting up a model for use with a face anchor, within RealityComposer it only supports a single face anchor, to support multiple faces you have manually manage the face anchors.

Though RealityKit has some very powerful features, I feel its still pretty new and lacks some of the maturity of SceneKit. For example RealityKit does not support programmatically creating geometry from a set of vertex points, it only has its preset primitive shapes. This means, within RealityKit you can use one of their primitive shapes (cube, sphere, cylinder) or you have to supply a a Callada or USDZ file for the 3D model. Though this may seem rather niche, there are other small nuances that can factor into considerations when choosing which render engine to build with when using ARKit.

If you are interested in doing a QuickStart on RealityKit check out the “Getting Started with RealityKit” series from Max Cobb

As with all native development, if you have to choose a single platform, then it really depends on your target audience and the devices available to them. This means that even if ARKit has all the features you need it still may not be the best choice if your target audience is based in a country where iPhones are not popular and the ones that are in market are older models that might not even support ARKit. For supporting older devices there are some cross-platform frameworks that use OpenCV, DLib, or some proprietary version of those frameworks, but we’ll get into that in the next section.

Cross-Platform

Now that we have an understanding of the core technologies that drive AR on Android and iOS, we can take a look at cross-platform engines that are built on top of ARCore and ARKit.

We’ll start with Unity as it is the most popular tool and render engine for games and immersive experiences. Unity is used for a wide variety of games and cinematic production. Unity is the development tool of choice for most AR wearables, including HoloLens and Magic Leap. Unity has a strong community with tons of plugins and examples. Unity has their own AR framework known as ARFoundation which leverages ARCore and ARKit but provides developers with a unified set of APIs which Unity refers to as subsystems. Unity uses C# as its coding language. While Unity offers APIs and libraries to do very complex effects, it also offers visual scripting tools and many settings options to minimize the need for coding knowledge. It’s a great tool for digital artists regardless of coding capabilities.

source: https://unitylist.com/p/k59/AR-Foundation-Image-On-Plane

We can’t mention Unity without also discussing Unreal Engine. Unreal is a development suite and render engine similar to Unity. Unreal offers a more mature development tool/render engine, though its community is not nearly as strong or expansive as Unity. Unreal is known for the quality of render detail that the engine is able to provide and has been the goliath within the Gaming and entertainment industries, with Unity working to close the gap. Conversely, within the AR space it’s Unreal that has been playing catchup to Unity. Unreal supports many of the same VR and AR hardware as Unity such as the Oculus devices, Magic Leap, and all ARCore/ARKit mobile phones. Unreal uses C++ as its language of choice, but also offers visual scripting with its Blueprints.

Don’t worry, if your app doesn’t require all the weight of Unity/Unreal there are other options. Viro Media offers a ReactNative library. ReactNative is one of the more popular cross platform frameworks that enables developers to use JavaScript to build native apps. ViroReact enables ReactNative developers to take advantage of ARCore and ARKit in their cross-platform app.

Two words of caution when using Viro Media, the features are limited to those available both ARCore and ARKit. Secondly, Viro Media recently open-sourced their ReactNative library so this could mean lots of libraries and plugins from the developer community but also lots of unsupported code and modifications/hacks that could eventually be floating around.

In mid 2018 Amazon released AWS Sumerain, a 3D development platform akin to Unity/Unreal except the editor is web based. Sumerian also comes with integrations into AWS Lex for NLP and Ai, Poly for text to speech, Amazon Lambda, DynamoDB, and more. Once the Sumerian experiences are built, they are deployable on Oculus Rift, HTC Vive, Android and iOS using ARCore and ARKit.

In my opinion it feels like the entire Sumerian offering is for teams that are already using AWS products and are looking for ways to incorporate AR/VR features into their existing AWS based ecosystem.

There are a few smaller cross-platform AR solutions worth noting, such as Vuforia, Wikitude, Blippar and Zappar.

Vuforia was the earliest pioneer of the AR platform, forging early partnerships with Unity and other giants in the AR space. Vuforia was a Qualcomm startup that was sold to PTC. Vuforia continues to grow and innovate their platform, and after joining the PTC family it seems the company has a clear focus geared towards enterprise and industrial applications.

Wikitude is built atop ARCore/ARKit, with proprietary extensions for each along with modules for both Unity and Unreal. Wikitude has one of the most expansive cross-platform offerings, covering iOs, Android, Windows, Xamarin, Flutter, and Cordova. Wikitude also offers a Javascript API that allows developers the option to build AR with HTML and JS (I’ve never tried it but it’s something the company has advertised for years).

Blippar has its own render engine and offers developers the ability to script their AR experiences using Javascript (BlippJS) but with the performance of native code. Blippar also offers a web-based drag-n-drop editor (Blipp Builder), which offers a limited version of its API to not technical users. Blippar does a great job of democratizing AR for non-technical users with its online editor, but also offers a robust enough API that allows developers to build complex AR experiences.

For deployment Blippar supports using their branded Blippar app, or you can license their SDK and build AR functionality into your own app. Blippar also uses ai for its marker recognition, which allows users to upload images of their product/label without having to add a QR code or manipulate it in any way. This is very useful when you want to activate a product that already exists in the market.

Zappar has a similar offering to Blippar, but also offers a desktop creation tool. The biggest drawback to Zappar is their use of Zapcodes, their own version of a QR code.

ROAR is another cross platform creation suite with a web based editor and self branded app. They have a white label offering as well, but it looks to be a custom app that the ROAR team has to develop.

The last option to consider if the aforementioned platforms don’t meet your needs, you could always deploy a custom render engine using Vulkan or OpenGL (open-source render APIs). This is the approach taken by Blippar, Zappar, Facebook, and Snapchat. For discussion’s sake, let’s assume you want to implement a custom render engine, this is not trivial and has quite a few trade-offs and considerations.

Speaking very generally, if you plan to use OpenGL, Vulkan or any other render engine then consider you’ll have to implement your own scene-graph, shaders, etc. You can use an off the shelf library (OpenSceneGraph, Vulkan Scene Graph), or build your own. Regardless, implementing your own scene-graph and all the details that come with it is what makes this a non-trivial task.

If I think I missed a company working within the Native space worth mentioning, please leave a comment.

Web

With all the latest advances in web technologies, AR is possible within the web-browser without a special app. Web based augmented reality is possible thanks to Web Assembly and WebGL. WebGL is used for rendering the content, and Web Assembly does all the heavy computation needed to do image tracking, face tracking, or even SLAM within the browser. Web based AR is the most accessible with the lowest barrier to entry for the user.

To start from the beginning. One of the earliest projects to gain mainstream attention was ar.js (by Jerome Etienne). The framework uses Aframe (based on Three.js) to handle the scene graph and rendering and then uses JSARToolkit to do the marker tracking.

source: https://www.youtube.com/watch?v=hdXY1tFQ2Hw

AFrame is a library built upon Three.js, with the aim of simplifying the syntax into more of an entity component structure, similar to HTML markup instead of the pure javascript approach that Three.js has taken.

While Three.js is a great framework for WebGL, not only is it’s pure JS approach tough for developers just getting started, the framework also suffers from a lack of maturity as a whole. Three.js has a great community with tons of examples and snippets, but that’s how they all remain; scattered unorganized and dated. When I say the framework lacks maturing, I mean the authors/maintainers have done very little to absorb some of the more common features into the framework, so users are stuck trying to follow outdated examples to implement features that one would expect to be built in.

I won’t jump down the rabbit hole on this one, but for some context when searching for something as simple as a “glass material in Three.js”, check out the questions and responses. There’s too much variation for something as common and standard as glass. For comparison, Babylon.js has a Glass material built-in. Babylon.js was originally designed and released by Microsoft but is now open-source. The framework has a strong community, and the maintainers keep up with bugs as well as absorb features from the community into the framework.

If you aren’t looking to use something simpler, Google offers ModelViewer. A library that allows web developers to quickly and easily embed 3D content into a web page. ModelViewer also supports Augmented Reality by linking to a USDZ model using Apple’s AR QuickLook. USDZ is a zip file containing a .USD file (3D file format from Pixar) and the accompanying texture files.

Wearables

Wearable AR devices come in many forms. While there are some really amazing projects out there (like Bosche’s AR Glasses) and new ones being launching constantly (Lynx pass through AR headset), I’ll focus on the wearables that exist in the market as of today (Feb 2020) and come with an SDK and developer tools.

Let’s start with the whale in the room, Magic Leap.

source: https://www.youtube.com/watch?v=GbpqwUUfMAQ

As an early pioneer in the AR wearable space, Magic Leap has generated a lot of buzz and hype for the industry as a whole. The company has raised about $2.3b, but has only released 1 version of their device. When it was originally released, the device was pitched as the “Creator” edition for developers to get started working, there was an implied promise of a more mainstream version to follow. Two years after its initial release, Magic Leap rebranded the Creator edition as Magic Leap One.

The Magic Leap One is an untethered 2 piece wearable device with add-on hand controllers. One part of the device is disc shaped and contains the battery/cpu/gpu which can be held, strapped to your wrist, or clipped to your pocket/pants. The other component is the headset, which is connected via a cable to the battery/cpu and houses various sensors (depth, motion, etc.), cameras (internal and externally facing), and the with the lens onto which the graphics are projected. Additionally the MLOne ships with a hand held controller for navigating through menus and interacting with the virtual world. The MLOne does support hand tracking and gesture control, but it’s not a complete replacement for the hand-held controller.

source: https://shop.magicleap.com

Magic Leap offers 3 options for developing native app for its platform (Lumen OS). There are plugins for Unity and Unreal that allow you to build and export apps to the Magic Leap device. Magic Leap also offers a C++ SDK for those developers that want to work with the bare metals of the Lumen OS. An interesting (yet less mentioned) product offering is Magic Script, Magic Leap’s JavaScript tag that can be embedded into any html page and when that page is viewed on a device running LumenOS , the user is able to see 3D content come to life.

Magic Leap offers a grant program for AR creators looking to build for Magic Leap. The business recently finished their first round but are teasing the idea that there will be a round 2

Currently Magic Leaps biggest shortcomings are tied to its accessibility for the average user, its price tag ($2,295). The device is great for experiential marketing events, art installations, or any other event where the environment can be controlled to some degree, and the device can be provided to the user at no charge.

If we are talking Magic Leap then it makes sense to mention Nreal, though it’s still in pre-order/beta. Shortly after making its splash debut at CES last year, Nreal and its founder Chi Xu were sued by Magic Leap for IP infringement. Chi Xu worked for Magic Leap and the company contends that he stole trade secrets to launch his own competitor, the case is still making its way through the system. Two notable differences: Nreal leverages the user’s phone for computational power instead of a stand-alone cpu/gp like the Magic Leap One; the Nreal glasses are a little more than half the price of the MLOne ($1199).

While Magic Leap may be the infamous “whale” in the AR space, Microsoft’s HoloLens was the first untethered device to enter the market (early 2016). The HoloLens is an all-in-one headset, with a similar set of sensors and projected display as the MLOne but without the clunk of a separate battery/cpu or the need for the hand-held controller. The HoloLens relies completely on hand gesture recognition. With the launch of the second edition HoloLens, Microsoft also announced shifts to an Enterprise focus for the device’s target audience. Unfortunately even before its official enterprise focus, HoloLens was also priced beyond the reach of the common user ($3500). Microsoft has announced partners from whom you can lease the device, but I imagine this is meant more for business who don’t want to commit to the costs upfront.

source: https://www.windowscentral.com/microsoft-hololens

Microsoft doesn’t currently offer any grant program, though they did run a HoloLens for Research RFP, but that has since ended and candidates announced. Microsoft does offer a HoloLens Partners program.

Vuzix offers a line of “smart” glasses with an SDK/API for developers to build AR apps. Vuzix uses Wikitude as its AR framework. The product is geared towards the enterprise market and documentation is hidden behind a login form.

The aforementioned wearables are all focused on visual augmentation, yet spacial audio has been growing in parallel to its visual focused counterparts.

Bose is a giant within the audio industry who is helping pioneer the spatial audio space. Spatial audio leverages the user’s position (through motion sensors) to adjust the digital audio to synthesize the effect of sound coming from a specific location or direction in the real world. Bose AR allows app developers to build more immersive audio experiences for Bose glasses and headphones, but doesn’t limit the developer to just audio. The Bose AR API gives access to the sensor data so a developer could build an app that is controlled by the user’s head movement.

source: https://www.bose.com/en_us/products/frames/bose-frames-alto-bose-0.html

While there are a number of interesting wearables hitting the market, these are the more popular ones that have been making a buzz in the news and have had booths at developer events throughout the country.

Social

Facebook and Snapchat (through acquisitions) have both built AR platforms within their own respective social networks. Each company offers a desktop app for technical artists to build their AR face-masks and world filters. The creator studios have similar interfaces, though they differ slightly in things such as deploying to device for testing.

source: https://www.youtube.com/watch?v=OHBQYAr79Dc

Each platform offers very robust APIs to make interesting filters. For example, Facebook’s studio supports visual scripting and drag and drop hand tracking. Where Snap’s API provides developers with advanced image manipulation effects. If you are planning to launch a face filter on only one of these two platforms, you should consider making the effort to reuse the assets and launch a filter on the other platform as well.

source: https://lensstudio.snapchat.com/templates/face/trigger/

The walled gardens that both Facebook and Snap have formed around their AR platforms is the only limiting factor. The argument can be made that each platform has such a vast audience that it’s worth the effort to build specifically for them. The other major limitation to both platforms is file size of the AR experience. Since both use cloud delivery to load the assets, they force small file sizes to minimize download times. Unfortunately neither platform allows for making http requests to remote servers to download the assets dynamically. For brands looking to make a simple filter this is not an issue, but for anyone looking to use it as a platform to launch complex AR experiences, then you should look elsewhere.

Tools/Frameworks

Given the infinite expanse that is the internet, it’s safe to assume that there are many tools and frameworks for every platform I‘ve mentioned above but most fall are either ARKit Only or Cross-Platform (Unity/Unreal).

One of the biggest pain points when developing for ARKit is Apple’s lack of support for providing the composited image. Within ARKit, Apple provides all the motion tracking data and the raw image captured by the framework, but for whatever reason Apple doesn’t provide access to the compositied image (real world image with virtual content drawn in) as a buffer, they only give you the snapshot which is not efficient if you want to allow users to record their AR content. It could be argued that Apple provides ReplayKit for recording the screen, but that puts a lot of onus on the user and will record your app’s UI. Thankfully ARVideoKit exists. It’s a framework that implements an offscreen renderer and provides methods that make it easy to take Live Photos, Gifs, and Video of your AR scene.

source: https://github.com/AFathi/ARVideoKit

While on the topic of AR pain points, another struggle is the initial setup for each 3D scene. When a 3D designer exports their content, it needs to be imported and adjusted because every render engine has slight variations. An app that seems to streamline that process is Adobe’s Aero. The mobile app allows designers setup an AR scene and import their own 3D assets or choose from those in Adobe’s starter collection. Once the content is imported, the designer can add behaviors such as taps and animations. The Aero app provides simple animation such as rotations but also allows the user to draw out a complex path for the 3D content to follow.

source: me using the Aero app.

Once the designer is done setting up their scene in Aero, they can export it as a RealityKit file or a USDZ for use with Apple’s RealityKit and QuickLook (respectively). Aero is relatively new to the market (launched with iOS13) but its clear that there is a lot of potential to solve a major pain point for creators in the industry.

Monetization within AR is another point that a few startups are trying to tackle. Arelene.io provides an SDK for ARKit developers to easily add 3D native ads into their AR apps. Admix is another startup with a similar monetization offering but for Unity developers.

A feature that has become increasingly popular within AR is geo-location based experience. Whether its a scavenger hunt or virtual tour, leveraging the users geo-location proven to be a key immersive feature. There are paid frameworks (see above — Vuforia, Wikitude, Blippar, Zappar) that offer geo-location functionality, but there are also projects like ARKit +CoreLocation. This framework merges ARKit sensor data with CoreLocation data to power AR experiences that are location based, such as scavenger hunts, indoor and outdoor navigation, etc.

source: https://github.com/ProjectDent/ARKit-CoreLocation

Continuing along the lines of geolocation based AR, Google and Microsoft offer competitive (cross-platform) products for publishers that want to leverage Cloud Anchors.

As part of its expansion of ARCore, Google has released cross-platform Cloud Anchors, which are available on both Android and iOS (leveraging ARKit). ARCore Cloud Anchors allow users on Android and iOS (who are co-located) to share the same AR experience yet view it from different angles. This is a powerful feature for anyone interested in building collaborative AR experiences. ARCore Cloud Anchors have plugins for Android, iOS, Unity, and Unreal.

In connection with its Hololens offering Microsoft launched Azure Spatial Anchors, similar to ARCore Cloud Anchors, but with the added support for Hololens.

Keeping with the geo-location theme, Niantic is slowly rolling out its Unity based AR geo-location framework, to enable developers to create the next PokemonGo or Harry Potter Wizards Unite.

Shifting focus to the the Web AR space, 8th Wall provides industry leading in-browser SLAM and marker tracking. 8th Wall provides integrations for Three.js, AFrame, Babylon.js, Amazon’s Sumerian and PlayCanvas.

source: https://8thwall.com

TorchAR also offers a web based AR platform. Torch leverages 8thWall for their WebAR.

Lastly, what’s AR without 3D? It’s worth noting that there are a number of 3D resources available to developers. Free3D.com offers an assortment of free models. TurboSquid and GTrader offer an assortment of paid models. Sketchfab offers a hybrid model of free and paid models. Sketchfab also offers a number of tools for 3D designers and developers.

source: https://www.youtube.com/watch?v=uVCRigBuf-M

Trends and use-cases

Thanks for bearing with me, now that we’ve gotten through the landscape and understand the tech available we can take a look at the various trends and use-cases that have emerged in the industry.

When Augmented Reality first started to gain momentum, many of the emerging experiences were driven by marketing dollars or as part of interactive gameplay. Within AR advertising, the majority of implementations are branded experiences. In this use-case, a brand pays an agency to develop a stand-alone AR experience (app or web). The experience can then be tied to the product itself using marker tracking, or it can leverage world tracking to allow the user to place a product in their world and interact with it.

Slowly as devices/platforms matured and more creators entered the space, new use-cases (beyond advertising) have been realized. Take geolocation based AR, one of the pioneering implementations was PokemonGo but as the technology matures we see companies like ARUtility leveraging geolocation and AR to help visualize buried utilities (cables, pipes, etc). It’s not just startups though, Google is also leveraging geolocation and AR to create a more immersive experience within their Maps app. Facebook also recently announced their acquisition of Scape.io, so it will be interesting to what new features or products FB will be releasing using the tech.

source: https://www.arutility.com

After PokemonGo and Google Maps, geolocation seems like a no-brainer, so let’s consider some even more emergent use-cases. As part of the Agora.io team, I’ve seen a number of uses for AR within real-time communications and broadcasting. Real-time communications is video chat and real-time broadcasting can be thought of as 1 way video chat. Think of it as being like a Facebook Live or YouTube Live session, where 1 or more users are broadcasting a video stream and users are watching the stream without sharing their video. In both aforementioned use-cases you have a camera first experience similar to any standalone AR experience, so developers are taking advantage of AR to enhance their live streams with 2D and 3D interactive elements.

Take remote support for example, this is a space where AR and real-time communication can be used to reduce frustration and ambiguity. In the remote support use-case, one user needs some assistance. Traditionally this person would make a voice call and attempt to describe their problem. In the AR evolution, the users join a shared video conference and the users can draw on their respective screen and have it appear augmented into the video feed. Vuforia was an early pioneer with Vuforia Chalk, but it didn’t take long for others to come up with their own solutions like TeamViewer Pilot and CareAR.

Vuforia Chalk — source: https://mobile-ar.reality.news/news/apple-ar-assisting-technically-challenged-is-easier-with-vuforias-new-app-0180353/

Another area of growth for real-time communications and AR is the Virtual Collaboration space. In this use-case, users share a virtual space and can make updates that are visible by all their peers (local and remote). Companies like Spatial are using Unity and the HoloLens to build a completely AR driven experience. While companies like Roomful are also using Unity to build apps for desktop and mobile devices making it within reach of the common user.

source: https://spatial.io/demo/

Social AR experience that can be launched at scale only exist within Facebook and Snap’s walled Gardens. Each platform has invested heavily into AR to increase engagement and dwell time of their platforms. I think as live video (chat or broadcasting) gains steam we will start to see AR proliferate into other social platforms as well.

Speaking of social, Gaming is another space that AR can be used to enhance the user experience. Take for example LaserDraw by LoveShark, the game is similar to Pictionary in AR. Two users join a shared AR session, one user is given an object and they have to draw it in AR and the other person is watching in real-time trying to guess the object.

source: https://www.youtube.com/watch?v=WtB-67asXgU

Beyond just advertising products with AR, companies are using AR to enhance the shopping experience. Ikea, Wayfair, and Amazon all offer AR previews of certain products. Amazon even offers these 3D products to developers through Sumerian. Glasses.com, Ray-Ban, Warby Parker, and Gentle Monster are all using AR for virtual try-ons to enhance their e-commerce experiences. Brands like Nike offer virtual shoe try-on to help users get a better fit when buying online.

Wayfair AR — source: https://medium.com/@goatsandbacon/a-quick-guide-to-designing-for-augmented-reality-on-mobile-part-2-fb76fe87dd41

Conclusion

While AR is an emerging field, the technology exists today to do some really amazing things, and enhance the user experience. I’m looking forward to all the amazing projects that will be released as the AR space matures and more developers are building applications that leverage the technology.

If there are any topics or popular frameworks you think I missed, please leave a comment.

Don’t forget to give us your 👏 !

--

--

Director of DevRel @ Agora.io … former CTO @ webXR.tools & AR Engineer @ Blippar — If you can close your eyes & picture it, I can find a way to build it