27 May Hitting The Brick Wall

As you probably know, we just released our new game Nikko RC Racer. It’s a sweet little off-road RC racing game, and we are proud of it – while it doesn’t pack all the bells and whistles of major racing titles, it has something that most iOS racers don’t have: multiplayer ghosting. We think it’s a blast! 🙂

However, much as we like it, in the past few days users have reported crashes on their devices. Even after intensive testing, we did not catch the crash bug. At Paladin we take quality very seriously, and I speak for the whole team when I say that we are truly sorry (and frankly, embarrassed) about the crashes.

We just released the update to fix this crash bug, so the problem should be solved now. However, I want to let you know what was going on, so this post is to give you a status update on the problem and solution. I will also provide some technical background information for your entertainment. 😉

Launch Stats

First some statistics of the game launch:

1. The game has been downloaded over 330,000 times since the launch – and thousands of players are joining the game every day
2. We got 85 1-star reviews, most of which were related to crash issues
3. By comparison, we got 32 5-star reviews, and 16 in-between reviews
4. The most popular countries are China, Thailand, France, UK, and The Netherlands
5. Players have logged over 1.1M sessions, and the median session length is roughly 3 minutes

Those are some awesome stats, and we hope you guys are enjoying the game as much as we are! 🙂

Crash Stats

Soon after release we got the first crash reports, and right away we started intensive testing on our side. The problem was that we had never seen the game crash during development. It turns out that in order for the crash to happen, you need to do a completely fresh install. For new users, that is always the case, but for us, we need to completely remove every trace of the app before we start the test (I’ll explain why in the next section).

(Our test setup the past days)

When we did the fresh-install-with-background-processes tests, these were the crash results:

1. iPhone 4: 0/32
2. iPhone 5: 0/10
3. iPhone 4S: 1/15 (jailbreak)
4. iPad 2 A: 3/25
5. iPad 2 B: 0/15
6. iPad Mini: 0/29
7. iPad 4: 0/10
8. iPod touch 4th gen: 10/10

Looking at this data, we concluded a couple of things:

1. iPhone 4, iPhone 5, iPad Mini, and iPad 4 worked fine in all cases
2. When it’s a jailbreak device, the game is more likely to crash
3. iPad 2 crashes sometimes (roughly 1 out of 13 times)
4. iPod touch 4G always crashes (which was expected, as the device is unsupported)

Note that this is a *very* small slice of tests, and the problem could have occurred on some of the devices where we found no issues (such as iPhone 4). The good news is that the crashes occurred on a small set of devices, and only some of the time. We know that there have been iPhone 4 users with crash issues, so we suspect that the iPhone 4 might be more prone to crashes than the tests suggest.

The most notable crashable device is the iPad 2 A – a specific subset of the iPad 2. To make it crash we needed to run plenty of background processes, like Spotify and Skype. Otherwise it seemed to run fine most of the time, even with these processes running in the background. The iPad 2 B did not crash at all, which could be a coincidence, but it could also be because it was a different type of iPad 2.

One note on the iPod touch 4G. It is not supported, because it has too little memory for all the 3D models to load. Unfortunately we cannot restrict the iPod touch 4G users from downloading the game, because Apple does not provide a way to block this device without blocking other supported devices too. That said, with version 1.0.9 it crashed instantly, also when it loads the main menu. So we will provide users with a message that their device is not supported – which is less crappy than a hard crash.

What was going on?

After intensive monitoring, we found the cause of the issues.

The problems happen because of the way we initialized the app. For this we use a technique called mipmapping. It is a technique where a high-resolution texture is scaled into multiple resolutions. We use mipmapping to automatically scale the resolution of textures to the appropriate device resolution. This way we only have to create one build for the best possible device, in this case the iPad 4, and let mipmapping gradually scale down the textures. This works wonderfully – and the irony is that we use this technique to prevent memory issues and maintain quality.

This is a mipmapped image

That’s all nice and wonderful. So where did it go wrong?

The application checks the resolution of the screen, and determines which quality setting suits the device best. So for iPad 4, it picks the highest resolution, and for iPhone 4 it picks a lower resolution. However, on the first startup, Unity has to load the first scene before it can determine the type of device. It happily loads the first scene at the highest quality – in our case the iPad retina display.

So those beautiful retina textures are loaded in memory, even on older devices. Most devices can (easily) handle it, but when there are many applications running in the background, the device hits the memory limit and it closes the app. Sometimes. And only on the first startup. But when it happens to you, the simple fact is that you won’t be able to start the game. That’s enough to get pissed off – and rightly so.

The solution

After the painstaking bug hunt, the solution was simple: We added a new startup scene that handles the initialization of the game. Only when that first scene is loaded, we move on to the main menu, and the textures are loaded in the right size. That first scene is as small as possible, and even though it’s loaded in the highest quality, older devices will be able to handle it.

We have submitted this update to Apple, and they were kind enough to provide an expedited review process and approve the app. I sincerely hope the crashes are history now.

Again, I apologize for the screwup. We have learned a lot of hard lessons from this, and it will not happen again. I hope you guys will be happily back to RC racing again. The good news is that we cut the loading times in half too, so go ahead and grab the update from the App Store and have a go! 🙂

Greets!
-Derk