Interaction Design Workflow for HoloLens
For my first mixed reality design project I had to adapt my 2D workflow to the third dimension. For three weeks I got the chance to work as a freelance Interaction Designer with the talented guys at afca.ch. This is how I went about it.
The Project
Most of this is covered by a NDA, meaning I can’t talk about the App itself. I’m allowed to tell you it was an educational App used at the ETH Zurich by computer assisted drug design students. The micro-cosmos these guys are working in is incredible. My previous biology knowledge was bad. In a biology test at school I had to draw the bond of two atoms — I drew a comic of them meeting at a bar :)
That sums up how much I knew about this topic going into this project. Having an expert explain to me how molecules fold, atoms interact, drugs work or researcher discover new possible medical components was mind blowing. I wish I had such a teacher at school years ago.
Preparation— reading up on a new topic
I’ve never used a Mixed Reality device before and had very little VR experience. If I’m about to design for a new platform I try to learn as many technical terms and concepts ahead of time. I did this by watching YouTube videos and reading most of the Microsoft design guidelines. This prevents me from asking very basic questions and wasting anybodies time.
Unfortunately, I didn’t find a lot of articles outlining the difference between a 2D and 3D UX process. Nor did I find many articles on a HoloLens specific workflow. So, if you’re in the same situation I hope this is helpful to you.
Trending AR VR Articles:
2. Infographic: The Future of Virtual Reality
First Step — Kickoff and Stakeholder Interviews
This was no different to a traditional 2D workflow. We had the initial meeting with our client. After the general project briefing with everybody we had an hour or two with the lead scientist. We asked questions to understand his goals and motivations.
Second Step — Technical Proof of Concept, User Insights and Goals
We weren’t 100% sure our idea was technically feasible and the data import needed for it was possible. Our dev and 3D artist spent the first two days or so researching and building a proof of concept.
This gave me some time to gather insights and come up with a general plan. I had access to last year’s students so I asked them some questions. Their answers where distilled into a dozen Jobs-to-be-done, three very primitive Personas and three goals:
- Assist the tutor with explaining several fundamental principles in an interactive manner
- Help students retrieve the core concepts when they must apply them later on
- A first-time user should spend 15 to 20 minutes exploring before they remove the HoloLens
Third Step — More Research, App Flows
I felt I needed more hands-on experience with the HoloLens. For the next couple days, I spent the first hour at the office trying out different apps. First, I would casually play with the App to get a “naive” feeling for it. Then I observed what design aspect made me feel that way and took notes what I liked or disliked. And I wrote down those wow-moments when the technology itself got me excited. This “unproductive” time was well worth it. It showed me what worked and I could avoid the biggest mistakes made by others.
Together with the lead dev I then created detailed App Flows for our Jobs-to-be-done. This helped us figure out the exact screens we needed and gave us an idea of the scale of our app.
Fourth Step — 8ninth Framework for Storyboards and WireFlows
If you haven’t already stumbled upon this framework, I fully recommend it. In short it gives you four categories you should design in each frame of your app.
- Physical Space
where the hologram or UI element is shown
(walls, tabletop, ambient space, …) - User Input
how the user interacts with it
(gaze, gesture, voice, …) - Holographic Form
how it is displayed
(3D object, UI element, animation, …) - 3D Sound
what sounds are used
(interaction feedback audio, spatial audio-cue, object-specific sound, …)
The framework forced me make conscious design decisions for each aspect. This process took me longer compared to how I’m used to design in 2D , but it had two very positive effects:
- Me being a Noob to the 3D world, I didn’t know much about conventions within production teams. You know, the ones that lead to the “Oh, I thought this was obvious from the pattern I used”-moments the devs love so much. I took my time to sketch all interaction in great detail and describe those four aspects for every single frame.
- All those details made it easier to communicate my designs to the 3D artist and our devs. While implementing the components they had less questions and thus I got interrupted less often. Even though I spent more time upfront designing each frame I felt more productive.
And, as an added benefit, it prevented me from forgetting to design a certain aspect. If I don’t have sound in a certain frame it is a deliberate choice and not just me forgetting it.
If you’re interested in this framework I’ll have a link for you at the end of the article.
Fifth Step — Menu Design, Voice Over
My first real design challenge was to come up with an intuitive navigation. While testing other apps I felt that most desktop navigation patterns didn’t translate well to mixed reality. For example, a dropdown menu is something very familiar on a screen. But in most implementations on the HoloLens it felt clumsy and error prone. The same goes for Navigation Bars. On a Screen we always have a border where we can anchor the navigation. In Mixed Reality there are no screen edges. Thus, navigation bars felt out of place or, worse, where in my way to interact with the main subject. I suspect this is because on a screen the navigation is on a meta-layer, decoupled from the content. A hologram though feels much more like a physical object then a screen.
But then how do interactions happen with physical objects?
In most cases the simple answer is by touching them.
This led to my first rule: the menu has to appear where the object was “touched” (or where the HoloLens cursor is).
My second rule was to design with short ways, large targets (Do you remember Fitts’s law?) and high error tolerance in mind.
The design pattern I chose was the circular menu. It isn’t very popular in 2D apps but often used in games. We can pop it up around the cursor and the distance to all the click targets is as short as possible. Also, if you overshoot a button you don’t activate the next one. This happened a lot to me with list menus. To further improve error tolerance, I increased the selectable area. It was roughly 30% larger than the visible button to account for unprecise or quick head movement. There are some subtle touches to the menu that further improve usability. For example, the menu starts at the far end of the hologram and animates to its final position in front of the user. By doing this it gently lifts the cursor from its position on the object to the navigation in front of the user. This provides a visual connection from the menu to the location it was spawned.
Another task was to write the text for the voice over actress. We decided on having a very simple “AI” explaining the app to the user. It shouldn’t get a full personality like in a game but have some fun and motivating lines.
Sixth Step — Usabilty Test
By the end of the second week we had a running prototype. We met up with the client in their lab and had some students take part in a usability test. (hehehe, I totally geeked out around those high-tech lab machines)
Our main users will be first time HoloLens users. That’s why we gave our test participants only the most basic instructions on how to AirTap (click). Then we watched them figure out how the menus and interactions work. After five minutes of free exploration the actual usability test started with several tasks.
What made it particularly difficult for us is that you can’t just glance over their shoulders to see what they see. Instead we had to live stream from their HoloLens to a beamer. Due to very bad wireless reception this had a lag of roughly 10 seconds and the pairing of multiple HoloLens’s didn’t work. This lead to some very funny situations where the participant would mention something, I’d “aha mhm” next to them to confirm I understood what they where saying but had no clue what they were talking about until the picture showed up 10 seconds later. :)
Those technical difficulties aside it was a very valuable experience. The scientific principles where easily understood. We achieved the initial goal to get the users attention for at least 20 minutes. And we found some important usability bugs we had to fix for the final product.
Seventh Step — Adjust App, Testing, Finishing touches
On my last week I first redesigned the parts that failed during the usability test. Then our devs rebuilt them and I gave the whole app a thorough test. I found various smaller usability bugs, which I documented and prioritized in OneNote.
Finally, I created an animated intro in Unity (the game engine used to program HoloLens Apps). It took me a whole day to build something totally simple, but it was a fun experience.
Eighth Step — Documentation
Isn’t this everybody’s favourite activity? :) Okay, I know some people enjoy this but I’m not a very good writer. And it wasn’t anything special at all, but I wanted to mention it here to give you the whole process. And a good Documentation is important.
Conclusion
So, this was it. Three weeks of full-on HoloLens code camp goodness. After all it wasn’t as different as I’d expected it to be. Don’t get me wrong, the medium itself is exhilaratingly fun and new and cool and everything! But if you’re about to start with your first mixed reality project I’d encourage you to build upon your current workflow. Most of the fundamental methods I use in my 2D workflows performed pretty much the same in 3D. My biggest take away was the 8ninths-Framework. This, and the meticulously designed wire flows worked so well I’ll incorporate them into my 2D process.
One note on body storming
I found some articles where they tinker with card board, sticky notes and a stick as a cursor to replicate the app. I had reservations towards this. I felt like they only got answers to very, very basic conceptual questions. Testing the finesse of actual interactions seemed difficult. In hindsight I’m glad I didn’t do it in my first project. While working on the project I learned so much about what is possible and how the HoloLens works. I simply wouldn’t have been able to act this out in any reasonable way and get valuable insights from it.
Once I’m more experienced, I might give it a try. If you’ve used such techniques on Mixed Reality projects — please let me know how this worked out for you in the comments! I’m very interested to discuss your experiences and learn how you do rapid prototyping in mixed reality.
If you remember how your first project went, tell me about it.
How did you approach it, what methods did you use
and what worked/didn’t work?
Ressources
Here are some articles, videos and apps I found usefull. If you have any other links, feel free to share them and I’ll add them to this list.
Partners in Crime
Articles
Videos
Apps
Some of the Apps I tried out and liked for various reasons:
On a side note: If you’re new to UX or would like to learn more about the topic, check out my first ever online tutorial:
