In-Country Testing Options

I would classify the options for in-country testing into the following:

  1. Crowd-sourcing
  2. Test Vendor(s)

The first – crowd-sourcing – basically means getting users who are in-market to help with testing the app. Some options for this include using Social Networks such as Twitter or Facebook to request for volunteers to test. Another common option is the Beta test. Both of these require a certain level of rapport with users. Usually the messaging (ie. how the request is put across to users) is key and it would make sense to test out the request message with friends before using it.

Another means of getting volunteers is to give an incentive – like a prize or some form of store credit. Unfortunately checking the validity of the test result may be an issue. Best to get a few results to triangulate.

The second method – getting vendors – requires sufficient budget and a good vendor or group of vendors. This usually means a process of vendor selection. In terms of ease of project management – a single vendor is best. uTest ( is a test vendor that has testers all over the world.

The issue with a single vendor is that they may not be the best in the game for each country. One good group of candidates for such projects are local VAS (or Mobile Content) providers. They are ideal for the following reasons: (a) they usually know the local operator’s nuances very well (b) they are already set up to test their own mobile apps eg. test handsets, sims, test process.

One very key success factor in managing such outsourced testing is the test plan. It should be (a) easy to understand by the tester – test procedure, expected result (b) use screenshots where possible (c) easy for tester to record result and accurately describe issues – multiple choice questions with open ended option are the best.

It is usually not very efficient to test in every country. It may make more sense to do exception based testing ie. only test those countries (or operators) that are exhibiting issues. One way to identify which countries or operators to zoom into is to use application logs with entries classified by carrier/country. Usually the countries/operators to zoom into are the ones that ought to have high traffic but are not. To find out where the IP belongs to, you could use one of many “whois” tools online eg.

Another way is to include a email address on the app for the user to report issues. Even better would be to also include an easy to use troubleshooting tool. If it is easy enough to use, you will be surprised how many users are willing to help.

In the following posts we will cover in-country testing in more detail for SMS, Mobile Web and Native Apps.

Leave a comment

Filed under In country testing, QA, Testing, Useful tips

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s