Augmented Reality Will Save Your Dinner
Why one of the world’s most secretive startups holds the key to home-cuisine in the digital age
I am, unfortunately, an incompetent cook. Scrambled eggs? I got you. Toast? Sign me up. But when things start getting more complicated, I find myself falling quickly out of my depth. I can follow directions perfectly fine; in fact my roommate makes fun of me for how closely I read recipe cards. What gets me is the sheer amount there is to manage. When do you start the rice so that it will be done when you finish cooking your chicken? How do you remember to check on your vegetables so that they don’t burn while you’re focused intently on seasoning your soup?
These are answerable questions, certainly. I’ve set timers, left myself sticky notes, you name it. A fairly large piece to the puzzle is just practice and general kitchen experience. But as a college student getting some of my first tastes of the real world, I find the whole process quite stressful. Two days into my internship last summer, I almost burned down my Airbnb when I left a burner on for several hours.
Enter Magic Leap. Makers of one of the most advanced augmented reality headsets on the market, they raised over $1.4 billion dollars in seed funding before they even released a beta version of the product. Investors include Google, Andreeson Horowitz, Kleiner Perkins and others.
DALI, a student-driven software development lab at my school, was able to acquire two of these headsets. After demoing the device a few times, I was thrilled by its potential. For my capstone Computer Science project, I worked with four other Dartmouth students to use Magic Leap to make the cooking process easier and more accessible for everybody.
Trending AR VR Articles:
1. Five Augmented Reality Uses That Solve Real-Life Problems
2. Virtual Reality Headsets: What are the Options? Which is Right For You?
A note up front: this project served primarily as a proof of concept and a learning experience. Nobody on our team had done Unity development before and we were entirely unfamiliar with best practices for augmented reality programming. We have no plans to publish the app to Magic Leap’s store at this time. That said, I’m proud of the product that we created and wanted to share it, as well as explain why I think there’s so much potential in this space.
The app is called Cook Along, slogan: “Unlock Your Inner Chef.” Cook Along centralizes all information you might need for a recipe, aiming to replace both the cookbook and the random Googling around and checking and re-checking the recipe on your phone. Users wear the Magic Leap while they cook, the small computer tucked into a pocket.
After navigating through initial welcome screens and a tutorial (should they wish to complete it), users scroll through a list of pre-filled recipes to choose from. They can learn more about a recipe by clicking on it. There’s also a web interface to edit the recipes to their liking or add their own.
Once a recipe has been selected, users are brought to the main cooking window, where they are walked through the recipe, step by step. They can place the window where they’d like by holding a button on the controller and looking at the desired location. Releasing the button locks it in place. From here, most of the navigation occurs via hand gesture. A thumbs up takes you to the next step while an L-shape takes you back.
At each step, users see the instruction, pictures and amounts of the ingredients required, a help menu listing the gesture controls and an interactive table of contents laying out all the steps in the recipe. In the top left corner, users can see and manage a list of all active timers. Particularly difficult steps contain gesture-controllable videos demonstrating the technique.
The proposed value-add is this: save time by not needing to continually consult the cookbook or your phone or YouTube when you don’t understand something. Cook higher quality food by personalizing and editing the recipes in the way that makes the most sense to you. Minimize mistakes by letting Cook Along multi-task for you and alerting you when a timer runs out or you need to go back to a step. We demoed the project at a school Computer Science fair and students made french toast on the spot using only a microwave and a mug.
There is so much potential for more here too. Magic Leap contains voice recognition capabilities which could supplement gestures and make control quicker and more intuitive. Instead of videos, you could have 3D models with looping animation demonstrating the steps. You could use image recognition to suggest recipes based on the ingredients you already have or to overlay guiding lines for cutting meat or dicing an onion. We were partly inspired for our project by the video below (credit: Sameer Bansal) which lays out a vision for some of these more advanced possibilities.
Magic Leap is still evolving as well. Halfway through development for this project, they released The Lab, which makes rapid iteration and testing even easier. The Magic Leap Toolkit, updated frequently, helped us simplify controller input management. As more and more developers build on the platform, they’ll be more and more resources available for solving perpetually tricky problems.
I, for one, am truly fascinated to see where this technology goes. It is entirely possible that the company fails to deliver on its massive investment and consumer AR headsets fade as a passing trend. But it is also possible that Magic Leap and augmented reality rethink not only how we cook but also how we learn anything and how we interact with the world more generally. That’s the possibility I’m rooting for, especially if I’m going to have to somehow feed myself next year.
The code for this project can be found here: https://github.com/dartmouth-cs98/cook_along and here: https://github.com/dartmouth-cs98/cookalong-backend and here: https://github.com/dartmouth-cs98/cookalong-interface. You can view the interface at http://cookalong-interface.surge.sh/. The front-end was developed in Unity using the Magic Leap SDK and Toolkit. The back-end was developed using Java Spring Boot and deployed using Heroku.
Team members: Zack Johnson, Erika Ogino, Brian Tomasco, Danielle Fang, Anders Limstrom. Special thanks to our professor, Tim Tregubov, and all of our friends who tested and gave us feedback. If you have any comments or suggestions for future development, please feel free to let us know in the comment section below. Thanks!
Don’t forget to give us your 👏 !
