MobileActive.org

This is a guest post by our colleague Michael Benedict who is currently working in Tanzania and Uganda. It was originally posted on his blog and is published here with permission.

While working with SMS-based applications I’ve noticed an air of mystery around the issue of reliability. I hear colleagues say ambiguous things like “carriers consider SMS to be lower priority than voice”, or “SMS delivery isn’t guaranteed”. My personal experience has been that messages are almost always delivered quickly and correctly, but I’ve heard stories of hours-long delays, corruption of data, and occasionally messages that never arrive. Since I am working on two projects that depend on reliable SMS service — one involves field-based data collection and another employs SMS as a transport layer between computers — I am interested in learning about how factors such as network, location, and time of day impact message transmission. I found myself in Mwanza, Tanzania last week with two GPRS modems and a local partner who was distinctly unenthusiastic about the work I was there to do, so I decided to try an experiment. I bought SIM cards and airtime for three of the major TZ networks, put two at a time in the models, and wrote a simple python script using Adam Mckaig’s excellent pygsm library.

a) Airtel and Zantel

Zantel is one of the smaller networks in TZ, but I was having technical problems with my Tigo SIM so started with Zantel and Airtel. Airtel was known as Zain until about two months ago when Zain was acquired by Baharti Airtel, based in India. The script ran from about 1AM on January 14 to just before 5AM on January 15.

Here are the results:

331 messages were sent and 10 (0.3%) were dropped. Delivery times fell into three bands. Airtel->Airtel (i.e. the Airtel SIM sending to itself) was easily the fastest, with messages typically delivered in 6 or 7 seconds. Messages in either direction between Airtel and Zantel took roughly twice as long, typically between 13 and 16 seconds. One outlier, an Airtel to Zantel message sent between 2 and 3 AM on Jan 15, took 24 seconds to arrive, and the only dropped Airtel->Airtel message occured around the same time. Delivery times for Zantel->Zantel messages were split unevenly: one cluster near 6 seconds, another around 15 seconds, and a third around 32 seconds. 9 messages sent from Zantel->Airtel were not received.

Some aspects of this plot make sense to me or at lease seem reasonable. Airtel is a larger company with wider network coverage, so it’s not too far fetched to imagine that messages sent from an Airtel SIM to itself would be the fastest to arrive. By that reasoning I’d expect messages sent between the two networks to take somewhat longer, and they do. The Zantel->Zantel delivery times are less intutitive. I don’t know what would account for the three distinct time ranges for delivery. It is also unexpected to see that 9 Zantel->Airtel messages were never delivered but no Airtel->Zantel messages experienced this problem.

b) Airtel and Tigo

192 messages were sent and 3 (1.6%) were not readable by the modem. Four distinct delivery time ranges emerged. I eleminated one outlier from the plot to make it easier to view: on January 15 at 11:47, a Tigo->Airtel message took 129 seconds to arrive. I believe that the 3 unreadable Tigo -> Tigo messages reached the modem, but the script raised an exception and the messages were not read. Unfortunately I was not logging pygsm’s debugging output so I am marking the missing messages as “unreadable” instead of dropped. I will have to log output in future runs and see if I can reproduce the problem.

I did not expect Tigo->Airtel messages to arrive slower than Airtel->Tigo messages. The difference is small but it would be interesting to know the technical reason for it.

c) Two Tigo SIMs.

The purpose of having the modems send messages to themselves in the previous two runs was to test delivery times within a network. It occured to me that sending from a SIM to itself may not be equivalent to sending between two different SIMs on the same network, so I bought a second Tigo SIM:

164 messages were sent and 1 (0.6%) was not received. The data shown excludes one outlier: the first Tigo A->Tigo B message took 457 seconds to arrive.

So is SIM to SIM within a network equivalent to sending from a SIM to itself? Sometimes. I don’t know the reason for this difference but it’s worth noting that the number for Tigo A is +255716379091 while Tigo B is +255717435798. In particular, after the 255 country code the numbers have different prefixes, 716 for A and 717 for B. Typically Tanzanian SIMs on the same network have the same first 3 digits, and the discrepancy here may indicate that Tigo’s servers treat the two numbers differently. The Airtel/Tigo data above used the SIM that is labeled as “Tigo B” here.

Thoughts and Next Steps 

Unexpected patterns emerged in the data, including asymmetric delivery times between networks, that I would like to understand better. The few dropped messages do not seem to follow a clear pattern, which is a concern for both human and computer-focused SMS applications where missing messages can cause confusion and frustration. It seems possible that having the modems send all 4 messages with only a 2 second delay between sending could be impacting the results. It would be interesting to revise the script to randomly offset the message send times by a few minutes and see if that increases reliability.

This is not a scientific experiment since there are variables such as time of day, day of the week, network traffic, modem location, weather, and likely many others that are not being controlled for. Still, I think the data gives a representative snapshot of network performance. Overall SMS delivery was quite good. With the exception of a few of outliers most messages were delivered in under a minute, and Airtel consistently delivered on-network SMS in less than 10 seconds. Occasional dropped messages show that safeguards need to be in place if SMS is used for critical applications. Some preliminary timing data I took in Uganda suggests that the networks there are not nearly as reliable. I’m looking forward to looking more carefully at Ugandan networks when I’m back starting next week. And, if you like this sort of thing and also have too many modems and not enough social life, feel free to try out the script yourself.

Editor's Note: For Part 2 of Michael's SMS reliability testing, read more on his blog!

Mobile Use by Micro & Small Enterprises, Wedding Cakes and more
Digital Customization: Mobile Wedding venues in Chicago are offering apps or mobile-responsive websites where couples can customize their venue layout, pick décor themes, and even select lighting moods.
This level of customization ensures the couple can envision and shape their special day in real-time.