Testing SMS apps

I recommend reading my previous 6 posts on Messaging starting from A very brief history of messaging if you are new to SMS as important concepts are detailed there.

There are usually 3 main parts to testing SMS apps: functional testing, connectivity testing and end-to-end testing. Functional testing means testing the app without actually sending or receiving an SMS from an actual handset. Connectivity Testing means to test (a) MTs sent from the server hosting the app will reach a handset (b) MOs sent from handset will reach the server hosting the app. End-to-end testing means to test the app with a handset, sending actual SMSs.

It would be a big mistake to jump directly to end-to-end testing. Rather, it would be best to test the frst 2 seperately (and if possible concurrently) before engaging in End-to-end testing. The reason for this is, as mentioned in the preceding sections, there are many separate components that make up the path from your app to the user’s handset. This means many points of failure. Hence it would be best to be pretty sure the app’s functionality is working correctly and messages are able to be sent and received before putting them together for End-to-end.

Another very important reason for being able to test app functionality separate from SMS transport is app regression testing. SMSs experience latency often as mentioned in the preceding posts. As a result, it’s much faster to test directly on the app sans the aggregator and operator network. It is also not easy to automate a user sending/reading SMS on a handset (imagine 1000 test cases done manually!).

Functional Testing

The main objective here is to test the app’s functionality without actually sending real SMSs. Hence there needs to be a way to simulate sending and receiving SMSs from and to the app. This could either be built into the app itself or an external app that mimics the SMS aggregator or operator gateway.

If the API provided by the aggregator or operator gateway is a HTTP based one, then you just need a webserver and HTTP client depending on the API. If however, you need to use one of the SMS protocols (eg. CIMD2, SMPP) there are several options out in the market.

Ideally you should have a means to simulate sending MTs and recieving MOs that is independant of the SMS protocol – this allows you to focus on pure functional testing of the app. This is absolutely critical in eliminating functional bugs when troubleshooting. This is also extremely useful when performing regression testing for app functionality as it is much easier to automate and faster to run.

The simplest way to implement this would be to have the app write to a file to simulate sending MT and read a file to simulate recieving MOs. A ‘|’ separated format is useful for readability purposes and it is also a seldom used char is SMSs. Below is a simple format for MT:

[timestamp (very useful!)] | [shortcode] | [MSISDN] | [Message]

A simple format for MO would be:

SEND | [MSISDN] | [shortcode] | [Message]

Now SMS apps can get more complicated than this of course. Sometimes eg. chat apps, the interactions are such that a response is sent based on the message recieved. In this case, the test setup will need to be able to handle this. The MO file would then have to look something like:

SEND | [MSISDN] | [shortcode] | [message]
WAIT | [shortcode] | [MSISDN] | [expected message or regex pattern]
SEND | [MSISDN] | [shortcode] | [message]

In other words, send a first message, wait for a message string (and fail if no correct message is recieved in a timeout period), then send the next message. This would of course mean that your test logic should be able to read the MT file and perform the shortcode, MSISDN and expected message check. This set up is very easy to create and very useful for automation.

One very useful tool to have when designing functional tests for SMS is the event diagram. This allows you to both design pass cases and fail cases. The figure below shows such a diagram. This easily maps to the MO and MT file.

Event Diagram showing MO and MT messaging for SMS application

Sometimes the app complexity is not in the interaction but rather the encoding. An example of this is a SMS notification service where the notification is being sent in several different languages depending on where the user is from.

One key thing to note is that while a standard ASCII char set SMS is 140 chars, other languages are likely to be less – for example CJKV (Chinese, Japanese, Korean, Vietnamese) require double byte chars (eg. UTF-16). This effectively shortens the char limit by half ie. about 70 chars. It is recommended that you check the limitations for encoding in the aggregator or operator’s API spec since details may vary.

One likely cause of problems for language encoding is the translation between standard ASCII (or UTF-8) representation used by the app and the translation to the encoding adopted by either the aggregator or operator. An example of a complex situation is one where the app is hooked up to operator A using SMPP and operator B using HTTP. In this case, it would be necessary to test the translation of the messages from say UTF-8 to the representation used by SMPP and test the URL encode format for operator B.

One recommendation for testing such translations would be to isolate the code within the app that performs this translation and build test code around it to test it separately. This test should then be automated and run as part of the regression sequence.

Connectivity Testing

The main objective here is to test the SMS connectivity is to test that SMSs sent from the server hosting the app are able to reach the handset and vice-versa.

Now, preconditions to this test are that the SMS connectivity has already been set up, shortcode is ready (for MO initiated apps). Now, sometimes there are network firewalls to cross, or VPNs to set up – these all depend on what sort of arrangement you have with either the aggregator or operator.

To perform this test, you will need a means to send a MT SMS, receive a MO SMS, test handset with sim (if using operator on GSM network). The number of sims you need depends on the number of operators you are tesing for. If the app is hooked up to an aggregator, then for most cases, you’ll just need to test this for 1 operator within the list that are supported by the aggregator since you are testing for the connectivity between the app’s server and the aggregator.

The means to send the MT could be a script or simple Java code (eg. using HTTPClient) to perform perform a HTTP Get or Post to send a simple SMS to the aggregator or operator gateway. The means to receive a MO could be a simple PHP script on Apache or equivalent ASP code on IIS or Servlet on Tomcat. All it needs is to log the call somewhere – parsed or not parsed is up to you – you just need to be able to check it against your assumptions.

Next, you will need a test handset. This could be as simple as getting a sim from the operator store (in some parts of Asia, you can get a sim from a vending machine), or as complex as getting test CDMA devices from the operator. There may also be restrictions on whether the messages get routed to prepaid only/postpaid only, certain number series only (for testing purposes) and designated test cell towers – in such cases, you will need to work with the operator to obtain the necessary sim or mobile phone (for CDMA).

As mentioned earlier, SMS shortcodes seldom roam. So how do you test if the operator is in another country? Now if the app is a pure MT app, you can still test using a roaming sim (and incur roaming charges). However, if there is a need to send MO to a shortcode then you may need to somehow test this “in-market”. One option to do this is to use Device Anywhere (http://www.deviceanywhere.com)  – they have hooked up mobile handsets (with sims) to several operators in US and Europe. Another option is uTest (http://www.utest.com) – they have in-market testers around the world.

The actual test itself is pretty straight forward: (a) send a MT from your client – then check to see if it arrives on the handset; (b) send a MO from the handset and see if it arrives on your server. This is the most basic test and is the first that should be carried out. For a start, send a simple message that consists only of 1 word (in english). If that works, then try more than 1 word, then special chars, then if needed – other language encodings. This will then allow you to isolate possible causes when problems occur (and they will believe me!).

End-to-End Testing

Once you’re sufficiently satisfied with the functional workability of the app and connectivity with the aggregator and operator – it’s time to put things toget in the End-to-End testing phase.

The simplest way to perform this test phase is to use a mobile phone and actually test out the functionality from end to end. Unfortunately, there are times it’s not so simple. Some gotchas include:

  1. operator is in another country – shortcodes do not roam normally
  2. the app is hooked up to several operators (which is usually the case)
  3. frequent changes in the app’s logic that warrants frequent end-to-end testing

If the operator is in another country – this has been covered in the section on Connectivity Testing.

For the case of several operators – it is good practice to perform a full test pre-launch, but for regular regression, it is probably not scalable – this is especially so if the number of operators is more than 5. One good way to check for issues is checking usage over a time period – if there are unaccountable drops in traffic (eg. known operator outages) it could point to a problem and would be a good idea to do a full check on. For such cases, it does make sense to put in logs counters to check for rate of message flow. You should be able to establish over time how much traffic is normal at different days of the week and hour of the day to use them as thresholds for alert triggers.

If there frequent changes to the app’s logic – it may make sense to automate the end-to-end testing. One option to do so is to hook up a GSM modem (see Julian Harty’s Mobile Testing site for info on this). However, this is not very scalable if the number of operators is large and when roaming is an issue. Another option is Device Anywhere’s automation feature – they do have a means to automate the key presses and screen captures.

It would be best to use the same MO file as input data for automated end-to-end tests. Of course the timeouts to expect would have to be longer due to network and operator latencies.


Leave a comment

Filed under Messaging, Mobile Messaging, QA, Testing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s