Math is Hard!
Math is Hard!
If you’ve been following along in our series on Augmented Reality (AR) challenges then you know that computer vision is not like human vision (part 1) and that not all print shops know what ‘matte’ means (part 2). You also learned a bit about how we can use software tricks of the light to resolve the balance of high levels of detail and scenes a smartphone can render in real time (part 3).
Why can’t you just put the model on the image? Is what Steve says when I ask him if he’s thought about making that tile matte. It turns out that putting a 3D model on an image is very easy. It’s built into the basic AR frameworks. It can even be done without writing a line of code using tools like Reality Composer. With some careful work in an image editor and in 3D software like Blender, you can ensure that scale and orientation of the model matches the image. So, what’s the problem? The first problem is a numbers game.
That is, recognizing one forest is easy. Recognizing two or more creates a problem. The problem is that the AR frameworks need to not only recognize the image once but keep recognizing it as the camera moves. It can’t really tell the difference between one tile moving a lot and two tiles. With a bit of code writing you can place a model at the same coordinates in space as the image and then tell the framework to start looking for the image again. Uncontrolled you get something like this:
The UEDF Colony tile was repeatedly recognized as the camera moved and multiple copies of it were placed. A little more work and we can ignore any overlaps. This ongoing loop of find, delete, find at 60 frames per second takes a toll on the phone’s processing. That’s why we added the “Map Scan” phase of game setup. We use this method to find all the terrain, place them in the real world and then stop. From that point on we don’t recognize any new terrain, we just remember where it is. Of course, it’s all relative.
Coordinates in 3D AR (x, y, z) are always relative to some virtual origin (0,0,0). This is usually the location of the camera. Every time the camera is activated for AR the origin is set. Then, as the camera moves, it is tracked in the same 3D system of coordinates. As long as the camera is active those terrain models remain in the same place in the real world. What happens, though, if you get a phone call?
The AR system does what it was designed to do. It puts the virtual terrain at the exact coordinates we set. Unfortunately, our origin has moved and now poor Zoe has a game board hovering over her head. This same problem also crops up when we want to share the game board across multiple players. Every player has their own origin. Like all problems, we solve this one by finding a common ground.
Every game of Rift Zone: Contact includes two half-tiles called AR anchors. You only need one but the second can help you avoid undue stretching. We’ll cover the second AR anchor later. We solved the problem of multiple terrain tiles by recognizing them and then remembering the real-world coordinates. This created a new problem. Fortunately, we can solve this problem by going back to image recognition. There can be only one AR anchor (of each type) at a time. With that guarantee we have found our common ground. All information about the game board is relative to the AR anchor and we use the frameworks to continuously track where that anchor is.
The AR anchor gives us a known, shared location that doesn’t move no matter how many players you have and no matter how many selfies you need to take with the Mauler. The AR frameworks give us the real-world coordinates, as well as scale and rotation, of the AR anchor. We use linear algebra to convert to and from real-world coordinates and game board coordinates.
𝐏_board = 𝐓_board⁻¹ × 𝐏_world
Each of these is a 4x4 matrix representing the location, scale, and rotation in three dimensional space. If you want a great explanation of why it’s 4x4 and not 3x3 you can find one here : Math for Game Developers: Why do we use 4x4 Matrices in 3D Graphics?
And the reverse, board coordinates to real world
𝐏_world = 𝐓_board × 𝐏_board
I’ll be honest. After graduating with my computer science degree I was pretty certain I’d never need to remember linear algebra. It turns out that linear algebra is at the foundation of most 3D processing. Usually, it’s handled for you by a framework but sometimes we push the limits and need to dig it back up.
We are working to make Rift Zone: Contact’s AR the best it can be. Remember how we described the phone using a gyroscope and an accelerometer to know when the camera moves so objects can appear to stay in one location? Guess what happened when Steve and I tried to play a game on a train!
Don’t worry! We’ve got some more tricks up our sleeve to solve this problem, too! Stay tuned!
We are thrilled with the results of our hard work in AR. All players share a window into the world of Proxima Centauri B and the game board comes alive. Getting the terrain right is a fundamental part of that experience as is the display showing unit positions, ranges, and movement. When you add music, sound effects, and vibration it is a thrilling multi-sensory journey to a far off world where you fight for the survival of your species.