Menu

Project page

Eat It, Augmented Reality

About

Eat It, Augmented Reality

This is my first project for the CS491 Virtual and Augmented Reality class at UIC. It consisted in creating an Augmented Reality breakfast table, with some given targets: a placemat, cereal boxes, cans, magazine. The goal was to create a scene of how breakfast could be in the future, by augmenting everything with 3D objects, particles, sounds and implementing interactions between objects. I chose the theme of Alice in Wonderland because it's the first thing that came to my mind when the professor was explaining the project specifications.

Stay with me in Wonderland, and I'll show you how deep the rabbit hole goes.
Download entire project
The custom target can be downloaded here
Please notice that I modified the default method onTrackableStateChanged of the TrackableEventHandler class to signal to my scripts when specific trackable objects are lost

MODELS

This is a list of the models I took from the web and imported into my application without modifying them at all, you can click on the model image to go to the source website and download the model.


MY MODELS

This is a list of the models I created from scratch or got from the web and modified, such modifications include: mesh transformation into low-poly mesh for the purpose of reducing the size and the graphic computation to render the scene, exploding object and taking pieces of it, UV Unwrapping and creation of textures to color models that had no material. If a model comes from somewhere in the web the link will be its source page, and the modified object can be found by downloading my entire project. Models created from scratch by me have a link to download the model in formats of type .FBX, .obj or .dae, complete with textures.

2D Graphics

The 2D graphics on the boxes, can and magazine were created starting from some images found on the web, here are the source links:
Caterpillar
Caterpillar 2
Hat
Mad Hatter
Nutrition Facts
Cookbook
Food
Card
Alice Cards

Sounds

These are the sounds source associated with each of the objects.

Main scenes

Scene I

Mad Hats cereal box


The first cereal box contains yellow cereals in a hat shape, in the Augmented Reality scene you can see them floating around the cereal Box and going from the 2D bowl on the back in the 3D bowl in the front.

The 3D objects in the scene are: the mad hatter, another one of his beautiful hats, a pair of scissors he needs to build hats, a bowl, a spoon and the cereals. I entirely created the bowl and the cereals, while for the mad hatter I took a High Poly model, transformed it in a low Poly, UV-unwrapped the 12 pieces of which it's made and created a texture for each one. The faces of the box are completely replaced by 2D graphics.

This cereal box interacts with all the other boxes and the can, when it's close to one of them the mad hatter starts saying advertising for his cereal creations. It also interacts with the placemat, that displays nutrition facts as soon as it is placed onto the placemat, and as explained in the Placemat interactions section, it could trigger a sad or happy 3D emoji.

The cereals sound plays when the box is put onto the placemat.

Scene II

Smokey Os cereal box


The second cereal box contains grey donut-shaped cereals, in the Augmented Reality scene you can see them been blown out from the mouth of the 2D Caterpillar which is smoking Shisha on the front. The smoking cereals in the animation tend to disappear over time thanks to the diminishing alpha component of the faded material applied to the object.

The 3D objects in the scene are: animated green and blue mushrooms, red and white mushrooms, white smoke (particle object), the animated cheschire cat head, a duck-vinegar object, a caterpillar on the back, the cereals created with a displacement modifier in order to present a rough surface. I entirely created the cereals and the white and red mushrooms, the faces of the box are completely replaced by 2D graphics.

The smokey Os box interacts with the mad hats and when they're close enough a "we're all mad here" 3D text by the cheshire cat starts to appear and fades away. It also interacts with the placemat, that displays nutrition facts as soon as it is placed onto the placemat, and as explained in the Placemat interactions section, it could trigger a sad or happy 3D emoji.

The smoke sound plays when the box is put onto the placemat.

Scene III

Aced Tea


The augmented can became a iced tea beverage called Aced Tea since its main theme is the Aces cards from Alice in wonderland.

The 3D objects in the scene are: the door handle, the animated card that is spat out from the mouth of the door handle, the ace cards, one of which's spear is animated, a mug and a floating "drink me" sign. I entirely created the mug and its texture, the faces of the box are completely replaced by 2D graphics.

The aced tea interacts with the mad hats and when they're close enough the "Drink me" signs starts to have a periodic movement. It also interacts with the placemat, that displays nutrition facts as soon as it is placed onto the placemat, and as explained in the Placemat interactions section, it could trigger a sad or happy 3D emoji.

The tea kettle sound plays when the can is put onto the placemat.

Scene IV

Rabbit Hole


The rabbit hole is a special food container, it contains a random sample of a type of food.

The 3D objects in the scene are: the rabbit hole, a pencil-animal, a coke bottle, a jam jar, the rabbit with the animated "Follow me" text. I created the abstract rabbit hole in the form of a black hole.

This cube interacts with the mad hats and when they're close enough the rabbit hole behaves like a black hole, swallowing the objects around it.

The engine sound of the rabbit hole machine plays when it's swallowing objects.

Scene V

The Alice in wonderland Cookbook


The Alice in wonderland Cookbook is ideally a recipes book inspired by Alice in wonderland, you can use the virtual buttons, displayed in the pages as logos of an open book and closed book, to flip pages back and forth.

The Book also contains instructions on what your healthy breakfast should be like. In particular, it says that it's okay to eat any of the two types of cereals with aced tea, but it's not recommended to eat both cereals as your meal would be hypercaloric. This is also reminded by the placemat, as soon as you place your chosen items on it, a smiley or sad face will appear depending on your choice. Put any of the cereal boxes and the can on the placemat to make the 3D smiley emoji appear, put both the cereals onto the placemat to make the sad emoji appear instead.

Interactions among objects

The mad hatter drives everyone crazy, just put any of the other boxes or can around him and magic animations will start.

Interaction I

Mad Hats and Aced Tea


When the mad hatter is close enough to the Aced Tea, he will start screaming and advertising for his cereal creations. On the other hand a "Drink me" sign starts floating with a periodic movement.

Interaction II

Mad Hats and Smokey Os


When the mad hatter is close enough to the Smokey Os, he will start screaming and advertising for his cereal creations. On the other hand the cheshire cat will start saying "we're all mad here" in a 3D text which eventually fades away.

Interaction III

Mad Hats and Rabbit Hole


When the mad hatter is close enough to the Aced Tea, he will start screaming and advertising for his cereal creations. On the other hand, the rabbit hole will start to behave like a black hole and will eat all objects around it. The sound of an engine will also start playing in this occasion.

Placemat interactions

Interaction I

Cereals and can with placemat


As soon as a cereal box or the can are placed on the placemat, a display with nutritional information relative to the specific object appears, the quick sound of the holographic display is a laser sound. Moreover, the cereal boxes and the can will start playing their sounds: the mad hats' sound is the one of crunchy and yummy cereals, the Smokey Os' is that of the caterpillar smoking, while the Aced tea's one is that of a tea kettle whistling.

Interaction II

Smile emoji by eating any cereal box and the can


As explained in the Alice in Wonderland CookBook/"Magazine", the placemat reacts based on the type of food the user is trying to eat, in particular, a good breakfast suggestion is to have any of the cereal boxes along with a good mug of Aced Tea. In this case a smiling 3D emoji will set as a sun on the background.

Interaction III

Sad emoji by eating both types of cereal


As explained in the Alice in Wonderland CookBook/"Magazine", the placemat reacts based on the type of food the user is trying to eat, in particular, having both the cereal boxes together is a bad meal because it would consist in too many calories. In this case a sad 3D emoji will set as a sun on the background.

During the development

Some screenshots during the development and the final picture.

Screenshot I

UV Unwrapping on Mad Hatter


Screen

Screenshot II

Painting the texture of the smile emoji


Screen

Screenshot III

Painting the texture of the Smokey O cereal


Screen

Screenshot IV

Final picture


Screen

Augmented breakfast in the future

This is a discussion on how an augmented breakfast could be of use in the future 23rd September 2018



It is likely that Augmented Reality will become pervasive in the near future and start appearing in every context. It will certainly be with us on the dining table, and we’ll wake up with an augmented reality breakfast every morning. This could be a very interesting situation because they say that breakfast is the most important meal of the day and it’s very crucial to eat properly in order to start off our day in an optimal way. What if we could have a little help? An augmentation that would show us what is better to eat every day, what to mix together to reach the perfect amount of carbohydrates and vitamins to be ready to face the day with enough energy.

One of the big problems that leads to a hasty and unhealthy breakfast is time, no one in the morning has time to think what to eat, to look at nutrition facts and try to calculate the calories based on the amount of product consumed, the only solution would be to use a memoization technique, pre-compute and store the exact amount of each product to consume each morning. But what if for some reason you wake up and you want to change your ordinary breakfast or integrate it with something else? This would complicate things.

What could be helpful for this problem is an augmented reality system, embedded in our glasses or contact lenses which is able to locate and recognize any type of food and display to us in real time how much of that to eat, suggest us what to eat it with and how to mix it to prepare an impeccable quality dish. As soon as any product is put on our placemat information such as nutrition facts appear in front of us and we cannot ignore it as much as we would like to, this could really help us start eating healthier.

Another interesting feature could be the possibility to keep track of what the user is eating throughout the day, showing total calories and set small milestones to reach during each day, suggesting each time what to eat next to be able to reach the needed nutrients. The application could also be integrated to other types of applications and for instance monitor the user’s vital signs and modify its behavior based on that, or it could create a shopping list suggestion for the user as soon as he opens the fridge and anything the user is used eat is noticed to be missing by the system.

All this could be achieved by being able to recognize and track every type of food. This seems like an impossible task nowadays, but in the future this could be possible, the integration of a quick look-up functionality on the recognized food using deep learning could help retrieve in real time the information needed, and the act of eating itself is recognizable by where the object is going and where it disappears.

A thing that could potentially ruin everything about this application are the companies that sell food, if they’ll be given the possibility to buy augmenting ads and appear in front of the user as soon as something like a competitor’s product triggers them, and start screaming something to try and make the user switch for their product instead. This could become a nightmare for the users and they would try in every possible way to get rid of this application, losing all the benefits it could bring them. But let’s be positive and hope that this doesn’t happen, because honestly, I cannot wait to see this type of technology with us every day.

Video

Video presentation

Contact Us

Reach out for a new project or just say hello

Send Me A Message

Sending...
Something went wrong. Please try again.
Your message was sent, thank you!

Contact Info

Where I live

Bussero, MI
20060 Italy

Email Me At

mrk23 at hotmail dot it

Call Me At

Mobile: