The importance of doing a beta phase in making a successful app

, ,

The process of creating an application does not only involve developing it. As discussed in a previous blog post, the marketing is a crucial aspect. There is another important aspect that we tend to avoid or not taking seriously enough: the testing phase, which I call the beta phase.

No matter which platform you build your app for, you need to test your app at some point. Yes, you worked really hard for many weeks and months on your app. Yes, you know your app by heart. Yes, it is bug free.

I don’t want to disappoint you, but there is always a little something that you didn’t think of, or a special usage scenario you didn’t pay attention to.

I speak from experience. I have submitted apps without doing a beta phase. I have also submitted an app where a crash could be easily reproduced. It is nothing to be proud of, but at least now, I do a beta phase.

True fact: developers are not good testers.

Before we released DualShot, we ran a beta phase.

So, how do you attract testers?

  • Contact your friends
  • Advertise on social networks that you are looking for testers. In my case with DualShot, I tweeted about it at two different times.

Believe me, you’ll get testers pretty easily. People are curious. Depending on how well known you are, you can keep track of the emails manually like I do or you can share an Excel sheet or use MailChimp like one of my friends does.

Unless you pay the testers, don’t expect to get feedback from everyone. This is normal, so don’t take it personally. Sometimes it is just not the tester’s type of app, or perhaps they are just too busy.

Tip: I use BugSense to keep track of crashes and I use Flurry for the analytic events. I highly recommend using these services (or any equivalent services) during the beta phase. As I previously mentioned, you might not get written feedback from all the testers, but at least you’ll receive traces of what they did with your app.

During the beta phase, the tester can provide feedback about the user experience (UX). As a concrete example, in DualShot, Vincent designed the following page:

clip_image001_thumb[1]

I really liked this page and I didn’t see a problem using it. When I submitted the beta to my testers, in the same evening, three users complained that the UX was really bad. It was clear to us that we needed to put more work into that view. We ended up with the following design:

image_thumb[2]

Unless you have the complete Windows Phone collection at home, you are most likely going to have only one or two devices to test with. When you have testers, you increase the chances that the app will be tested with many different Windows Phones. Don’t assume that all Windows Phone 8s from different manufactures behave the same. In DualShot, the image capture with some HTC 8XT (only this model) does not work. We didn’t have a tester with this rare device and Murphy’s Law struck again.

Not only will the testers find bugs/crashes and give you feedback, they will often give you two thumbs up. It will give you the extra energy to polish your app before certification.

Windows Phone

If you are developing a Windows Phone app, you are lucky because the Windows Phone team developed a feature in their portal where it is easy to run a beta phase. Essentially, you publish an app as if you were submitting it for certification. However, the app is kept private for the testers that you have specified and the certification passes automatically in the subsequent two hours.

In the Windows Phone developer portal, you need to select Beta as the Distribution channel, then you enter the list of tester email addresses (using a ‘;’ between addresses).

clip_image002_thumb[1]

After that you can fill out all the info and screenshots.

Tip: avoid flooding your testers with daily emails if you submit a new beta every day. If you submit an update, the tester will be notified via the Store Live Tile. Take advantage of this automatic notification to enter all the new features/bugs into the app description. The first time that you send the deep link of the beta app, you can tell your tester to check the app description when a new update is available.

Windows 8/8.1

Unfortunately, in the current Windows developer portal, it is not possible to easily distribute an app to testers. You need to create a package and send the package to your testers. The testers then need to manually install the app. Hopefully the Windows team will soon mirror the awesome work of the Windows Phone team.

Conclusion

I know the feeling when an app is finally complete and ready to be submitted for certification. It is so tempting to submit it right away in order to see it live in the store ready to be downloaded. Please resist this temptation and do a beta phase. It is better that your friends find the problems than strangers, as strangers will most likely give you a 1-star review if they find a bug or a crash.

Develop, test and release!