Apple VisionOS — From Zero to Hero

The Apple Vision Pro (AVP) uses many of the same technologies as your iPhone and iPad. Still, it does that slightly differently now … For those who have already worked with ARKit and before SceneKit, it will probably be straightforward to follow this article. Still, no worries. I will also cover some basics here and there. At that point, it’s my goal that every developer has all the information needed to write his apps or games. We start with straightforward ones. But make it step by step to something worth uploading to the AppStore ;)

Programming a Game

The most important of a game is probably the game logic — the code we need to implement the game mechanics. At first, we do something straightforward. It will be a VisionOS version of the game I wrote for the SceneKit tutorial some years ago. So, the pitch for the game looks like this:

1. You have to fly a little airplane through rings.
2. When you pass a ring, it gives you points.
3. You win after you pass all the rings in a given time.

Of course, we can not port these one by one to a spatial device for many reasons:

1. In VR or classical 3D gaming, we can manipulate the game’s environment. In spatial computing, the environment is given (natural), and we have to create the game in that space (if we do not want to go fully immersive)
2. Every environment and room is different, which is a challenge when we want to create a game-level
3. We have a completely different user input. Yes, we can also connect an Xbox controller, but we want to use the possibilities we have on the AVP.

These things sometimes end even in the fact that a game or its mechanics do not make sense in spatial computing. For our game idea, there is an approach that may still be fun if we do it right, but for this, we slightly change the game’s logic and even its mechanics.

We will create the game so that the user first places the rings in the room and flies after a little airplane through them in a given order. So let’s start implementing…

Place Rings in Space

This will be our first challenge in spatial computing. How can we place the rings in the room, and how can we even select the position in the space where we want to place the ring? We have several possibilities:

1. Place the ring where the user stands (device positions)
2. Look at any position, make the pinch gesture, and place it there.
3. Place it relative to room elements (like windows, walls, etc.)
4. Scan the environment and select a place in the scanned environment.
5. Use hand tracking and gestures.

All of them are more challenging to implement than it first seems, but I will cover them all during the newsletter series.
But at first, we start with a simple one: Solution #1

This is an outtake of an article I wrote in my VisionOS newsletter on Substack. Interested in reading the complete article and downloading the source code to try yourself?

Consider to subscribe on https://visionos.substack.com