(The following case study was written by Kate Otto of The World Bank; it is reprinted here with permission.)
With the proliferation of innovative mHealth tools across the developing world, there comes an inspiring enthusiasm for health system reform. But mHealth raises a critical question as well: how do we know mHealth tools are actually changing health outcomes for the better?
I work with the World Bank and Addis Ababa University in Ethiopia on the evaluation of an mHealth intervention that enables rural community health workers to improve the quality and reach of their maternal and child health care services. We have set out to see if this tool is just a very cool device, or if it actually improves maternal health and decreases maternal and infant mortality.
You may be used to reading about the development and testing of mHealth tools aimed at scaling up to farther-reaching implementation. Our work, however, is a bit different: we built a tool on which we will do no further development, and instead rigorously test the impact of this tool on health outcomes of rural mothers and children through a randomized evaluation.
Basically, we are searching for hard evidence first, and leave a discussion of scale-up for once we have the data.
Since our final results will not be available until early 2013, I’ll share with you today a bit about our research design and the key question we are investigating: Does the use of a mobile phone-based tool for patient registration, appointment reminders, and inventory management in the hands of community health workers result in improved maternal and child health outcomes in a rural Ethiopian setting?
Background: In 2010, I worked for the World Bank to help develop the evidence base on mHealth (how do interventions influence health outcomes?) as part of its eTransform Africa initiative. From a World Bank perspective, it is crucial for Managers to be able to cite rigorous research when providing guidance on health reform via mHealth to partner country government – does it work, and how? Or is it just a fad without impact? Since there is little such research yet available, we invested in the creation of this research, and began in Ethiopia.
What kind of health outcomes did we set out to influence? I met with colleagues at Addis Ababa University who had already been hard at work brainstorming mHealth interventions, and thanks to their history of fieldwork in rural health posts, we identified maternal and child health as a key priority area for an initial intervention. Ethiopia has one of the highest maternal mortality ratios globally, at 673 per 100, 000 live births in the year 2005 which accounted for around 21 percent of all deaths among women aged 15 – 49. The country also has one of the highest under-five mortality rates at 14 of every 100 children (DHS 2005), with more than 90% of child deaths due to five preventable diseases; pneumonia, malaria, diarrhea, measles and malnutrition. (National Strategy for Child Survival in Ethiopia, 2005).
In addition to focusing on maternal and child health, we focused on Health Extension Workers (HEWs) as the end users of the mobile phone tool – these are the community health workers of Ethiopia, a well-developed national program largely standardized across all sub-districts. Unlike the general population, whose mobile phone penetration rate is low around 10% and literacy is low, about 90% of HEWs report using mobile phones regularly and all are literate, high school graduates. HEWs are usually not fluent in English, but reported regularly using their phones to communicate in Amharic with Latin letters. Structurally, two HEWs are assigned to every village, or kebele, of 2,000 – 5,000 people, and each pair is assisted by a set of voluntary community health workers (vCHW) who are normally assigned in pairs per 50 households.
Putting an mHealth tool in the hands of Health Extension Workers for maternal and child health – but what would this tool do? Extensive qualitative interviews with rurally based HEWs and mothers revealed several health system bottlenecks that they felt were most urgent to address with an mHealth tool.
- While pre- and post-natal coverage rates were reported by HEWs as very high, reports from mothers and pregnant women were conflicting. From the mom’s side, the paper card system in place to record and remind about appointments was inconsistently utilized, and if they wanted regular care they needed to take the initiative to seek it. They could not rely on HEWs to remind them or to come to their homes, and oftentimes go without care. From the HEWs were heard that their work in a single day could be unpredictable, and many different maternal health visits were not necessarily planned or prioritized.
- Vaccinations are delivered on a monthly basis to health posts from the health centers, however the health center calculates the amount delivered based on population estimates, as opposed to actual counts of new births. This meant that HEWs were sometimes faced with shortages, and other times excess that went wasted.
- Regarding referrals, HEWs reported frustration that when they sent mothers to higher care in case of obstetric emergencies, there would oftentimes be no doctor there to see the woman, and sometimes resulted in injury or death.
- Regarding deliveries, there remains conflict between HEWs and traditional birth attendants (TBAs). While the MoH is training HEWs to assist with deliveries, most mothers (naturally) still prefer TBAs, who are more skilled and experienced to help deliver. However, the TBAs, unlike HEWs, are not equipped with gloves, towels, misoprostol, and other supplies. HEWs keep these on hand to help ensure a clean and safe delivery, yet they are oftentimes deliberately not informed when births are expected or occurring.
Taking this feedback into account, our team designed a tool that would respond to their concerns.
The use cases of the HEW mobile phone tool include the following:
1. Improve Antenatal Care (ANC) and Delivery Services
In the first Use Case, we address low levels of full ANC that deprive women of the opportunity to detect health problems associated with pregnancy and avoid unsafe delivery.
- HEWs will submit a form to register every mother in their kebele who could need ANC care over the next several months (assumes that women who are pregnant know they are pregnant, and that women who know they are pregnant are forthcoming in sharing this news with the HEWs).
- Submitting this form will create a longitudinal patient record for that mother, and an SMS will be automatically sent back to the HEW containing her patient’s unique numeric ID. The HEW will write this number down on the mother's paper family folder.
- HEWs will then submit the woman’s ID#, Name, Location, and last date of most recent menstrual cycle (or months of amenorrhea), and an Expected Date of Delivery will be automatically calculated.
- As a result of calculating this date, ANC appointments will be scheduled and reminder messages will be sent at weeks 14, 24, 30 and 36.
2. Improve Vaccination Coverage
In the second Use Case, we address mothers who do not always come back to clinics to vaccinate their children on the date written on their child’s health folder (they may lose the card, forget the appointment, or miss the appointment due to other obstacles such as lack of transportation). We aim to help HEWs as well, who have no way to know the exact need in their kebele, as estimates could leave them with too many or too few vaccines on the day of the vaccination campaign.
- HEWs will submit a Child Registration Form to register every child at or under the age of 11 months in their kebele.
- Submitting this form will create a longitudinal record for that child, and an SMS will be automatically sent back to the HEW containing the child’s unique numeric ID. The HEW will submit a form for this child with their unique ID # and their birthdate, and as a result vaccination appointments are scheduled for each child at 6, 10, 14 weeks and 9 months, rounded up to the nearest month to accommodate the ministry's approved vaccine delivery schedule.
- A week before the Woreda’s monthly vaccination campaign, HEWs will be sent an SMS containing the names and IDs of children in their kebele who should be vaccinated in the upcoming vaccination day. One of the HEWs in each health post will submit a vaccine request form to her immediate supervisor indicating how many vaccines she needs for the subsequent month.
- After each vaccination day, the HEW will submit a report indicating the number of children immunized for each vaccine. A number reporting that is higher than the expected amount will indicate that registration of children in the Woreda is incomplete or children are not showing up for vaccination on the recommended schedule, while a number much smaller than expected would indicate incomplete vaccination rates.
3. Facilitate Emergency Referrals
In the third Use Case, we address the need to find transportation for referrals to emergency care, and the need to alert Health Centers, who are oftentimes not prepared with appropriate staff and equipment in order to manage incoming emergency patients from rural Health Posts, thus causing unnecessary maternal and infant mortality.
- Although ambulances are not available at Health Centers to dispatch to Health Posts, there are many local drivers of small three-wheeled vehicles (bajaj) or other type of vehicles (mini-buses, land cruisers, etc.) who may serve the role of an ambulance in emergencies. All HEWs will have their local bajaj driver numbers programmed into their phone from the start, and would thus be immediately able to call for a vehicle (this doesn’t ensure that someone can pay for the ride or that a driver is available, it only ensures that in cases where someone can pay and there is a driver, that the driver is notified and takes the woman).
- HEWs will call the Health Center in emergency referral situations so that Centers are prepared to receive patient.
How do we test the efficacy of this tool?
Our team randomly selected three sub-districts in the SNNP Region into one of the following groups:
- Treatment 1: All HEW received mobile phones equipped to perform the three use cases
- Treatment 2: All HEW and 2 VCHW within each kebele received mobile phones; HEW phones are software-equipped for the three use cases, and vCHW phones are dumbphones intended to make missed calls only.
- Control: No mobile phones were distributed.
In Treatment two, as you can see, we engage the vCHWs and dumbphones, which is a very simple and low-cost and non-technical intervention – no coding required! – because we’re interested in seeing if there will be a difference in health outcomes if this option is available. The vCHWs will be expected to use their dumbphones to:
- Take calls from HEWs to follow up with pregnant women if appointment times need to be changed.
- Missed call HEWs when they meet or hear of mothers in households who are not yet registered.
- Missed call the HEWs to request that they come register a child.
- Take calls from HEWs to follow up with a child's family to notify them of an upcoming vaccination day.
- Missed call HEW if a birth is going on and no HEW is present, so that they can at the least bring their supplies and materials for the TBA to use.
- Missed call to alert HEWs of any other maternal/child health emergency.
Because the risk for spillovers was too high if randomization was done at the health post level (i.e. control health posts seeing what the treatment posts were up to and wanting in on it!), our design randomizes one treatment per sub-district, which normally includes on health center and 5 satellite health posts. We excluded from the possible selection any sub-districts that have no mobile network coverage, any adjacent words, and any words with significantly different access to health services at the village level (as measured by population to health post ratio).
And at the end of the day, what are we trying to see changes in? A few things. In terms of ANC and Delivery, we will analyze changes in antenatal care (ANC) attendance – timing and # of visits, in numbers of clean and safe deliveries (of births delivered in kebele) – meaning HEW is present with her supplies, regardless of who delivers, and in number of deliveries assisted by a skilled attendant (of births delivered in kebele) – whether the TBA and/or HEW delivers. In the case of immunizations, we’ll be monitoring changes in immunization coverage (TT2, Penta3, Measles). And in regards to referrals, there is not a whole lot of change we can attribute to the phone, but will measure Instances of referral to HC in which women is seen upon arrival.
As a research team, we are still in discussion over appropriate outcome measures and would be very happy to hear about others’ experiences and suggestions here. We think it is important to disseminate our ‘results’ – the lessons learned along the research design road – throughout the study, not just at the very end. And we invite you to be in touch with questions and suggestions!
Many thanks for reading, and many thanks to the amazing team of researchers driving this study: Drs. Asfaw Atnafu and Solomon Shiferaw, of the AAU Center for eHealth, Dr. Mesganaw Fantahun, Head of the Dept of Reproductive and Family Health and Nutrition at AAU, and Dr. Rahel Bekele, Dean of the School of Information Sciences at AAU. Also, we developed the initial software in collaboration between Medic Mobile and the AAU School of Information Sciences, under the leadership of Henock Lulseged, and I would like to acknowledge the brilliant leadership of Dr. Mieraf Taddesse, my colleague at the Bank in Addis, who keeps our project moving forward always!
Do m-Health Tools Really Work? Testing The Impact of Mobile Technology on Maternal and Child Health Care Locations
(The following case study was written by Kate Otto of The World Bank; it is reprinted here with permission.)
With the proliferation of innovative mHealth tools across the developing world, there comes an inspiring enthusiasm for health system reform. But mHealth raises a critical question as well: how do we know mHealth tools are actually changing health outcomes for the better?
I work with the World Bank and Addis Ababa University in Ethiopia on the evaluation of an mHealth intervention that enables rural community health workers to improve the quality and reach of their maternal and child health care services. We have set out to see if this tool is just a very cool device, or if it actually improves maternal health and decreases maternal and infant mortality.
You may be used to reading about the development and testing of mHealth tools aimed at scaling up to farther-reaching implementation. Our work, however, is a bit different: we built a tool on which we will do no further development, and instead rigorously test the impact of this tool on health outcomes of rural mothers and children through a randomized evaluation.
Basically, we are searching for hard evidence first, and leave a discussion of scale-up for once we have the data.
Since our final results will not be available until early 2013, I’ll share with you today a bit about our research design and the key question we are investigating: Does the use of a mobile phone-based tool for patient registration, appointment reminders, and inventory management in the hands of community health workers result in improved maternal and child health outcomes in a rural Ethiopian setting?
Background: In 2010, I worked for the World Bank to help develop the evidence base on mHealth (how do interventions influence health outcomes?) as part of its eTransform Africa initiative. From a World Bank perspective, it is crucial for Managers to be able to cite rigorous research when providing guidance on health reform via mHealth to partner country government – does it work, and how? Or is it just a fad without impact? Since there is little such research yet available, we invested in the creation of this research, and began in Ethiopia.
What kind of health outcomes did we set out to influence? I met with colleagues at Addis Ababa University who had already been hard at work brainstorming mHealth interventions, and thanks to their history of fieldwork in rural health posts, we identified maternal and child health as a key priority area for an initial intervention. Ethiopia has one of the highest maternal mortality ratios globally, at 673 per 100, 000 live births in the year 2005 which accounted for around 21 percent of all deaths among women aged 15 – 49. The country also has one of the highest under-five mortality rates at 14 of every 100 children (DHS 2005), with more than 90% of child deaths due to five preventable diseases; pneumonia, malaria, diarrhea, measles and malnutrition. (National Strategy for Child Survival in Ethiopia, 2005).
In addition to focusing on maternal and child health, we focused on Health Extension Workers (HEWs) as the end users of the mobile phone tool – these are the community health workers of Ethiopia, a well-developed national program largely standardized across all sub-districts. Unlike the general population, whose mobile phone penetration rate is low around 10% and literacy is low, about 90% of HEWs report using mobile phones regularly and all are literate, high school graduates. HEWs are usually not fluent in English, but reported regularly using their phones to communicate in Amharic with Latin letters. Structurally, two HEWs are assigned to every village, or kebele, of 2,000 – 5,000 people, and each pair is assisted by a set of voluntary community health workers (vCHW) who are normally assigned in pairs per 50 households.
Putting an mHealth tool in the hands of Health Extension Workers for maternal and child health – but what would this tool do? Extensive qualitative interviews with rurally based HEWs and mothers revealed several health system bottlenecks that they felt were most urgent to address with an mHealth tool.
- While pre- and post-natal coverage rates were reported by HEWs as very high, reports from mothers and pregnant women were conflicting. From the mom’s side, the paper card system in place to record and remind about appointments was inconsistently utilized, and if they wanted regular care they needed to take the initiative to seek it. They could not rely on HEWs to remind them or to come to their homes, and oftentimes go without care. From the HEWs were heard that their work in a single day could be unpredictable, and many different maternal health visits were not necessarily planned or prioritized.
- Vaccinations are delivered on a monthly basis to health posts from the health centers, however the health center calculates the amount delivered based on population estimates, as opposed to actual counts of new births. This meant that HEWs were sometimes faced with shortages, and other times excess that went wasted.
- Regarding referrals, HEWs reported frustration that when they sent mothers to higher care in case of obstetric emergencies, there would oftentimes be no doctor there to see the woman, and sometimes resulted in injury or death.
- Regarding deliveries, there remains conflict between HEWs and traditional birth attendants (TBAs). While the MoH is training HEWs to assist with deliveries, most mothers (naturally) still prefer TBAs, who are more skilled and experienced to help deliver. However, the TBAs, unlike HEWs, are not equipped with gloves, towels, misoprostol, and other supplies. HEWs keep these on hand to help ensure a clean and safe delivery, yet they are oftentimes deliberately not informed when births are expected or occurring.
Taking this feedback into account, our team designed a tool that would respond to their concerns.
The use cases of the HEW mobile phone tool include the following:
1. Improve Antenatal Care (ANC) and Delivery Services
In the first Use Case, we address low levels of full ANC that deprive women of the opportunity to detect health problems associated with pregnancy and avoid unsafe delivery.
- HEWs will submit a form to register every mother in their kebele who could need ANC care over the next several months (assumes that women who are pregnant know they are pregnant, and that women who know they are pregnant are forthcoming in sharing this news with the HEWs).
- Submitting this form will create a longitudinal patient record for that mother, and an SMS will be automatically sent back to the HEW containing her patient’s unique numeric ID. The HEW will write this number down on the mother's paper family folder.
- HEWs will then submit the woman’s ID#, Name, Location, and last date of most recent menstrual cycle (or months of amenorrhea), and an Expected Date of Delivery will be automatically calculated.
- As a result of calculating this date, ANC appointments will be scheduled and reminder messages will be sent at weeks 14, 24, 30 and 36.
2. Improve Vaccination Coverage
In the second Use Case, we address mothers who do not always come back to clinics to vaccinate their children on the date written on their child’s health folder (they may lose the card, forget the appointment, or miss the appointment due to other obstacles such as lack of transportation). We aim to help HEWs as well, who have no way to know the exact need in their kebele, as estimates could leave them with too many or too few vaccines on the day of the vaccination campaign.
- HEWs will submit a Child Registration Form to register every child at or under the age of 11 months in their kebele.
- Submitting this form will create a longitudinal record for that child, and an SMS will be automatically sent back to the HEW containing the child’s unique numeric ID. The HEW will submit a form for this child with their unique ID # and their birthdate, and as a result vaccination appointments are scheduled for each child at 6, 10, 14 weeks and 9 months, rounded up to the nearest month to accommodate the ministry's approved vaccine delivery schedule.
- A week before the Woreda’s monthly vaccination campaign, HEWs will be sent an SMS containing the names and IDs of children in their kebele who should be vaccinated in the upcoming vaccination day. One of the HEWs in each health post will submit a vaccine request form to her immediate supervisor indicating how many vaccines she needs for the subsequent month.
- After each vaccination day, the HEW will submit a report indicating the number of children immunized for each vaccine. A number reporting that is higher than the expected amount will indicate that registration of children in the Woreda is incomplete or children are not showing up for vaccination on the recommended schedule, while a number much smaller than expected would indicate incomplete vaccination rates.
3. Facilitate Emergency Referrals
In the third Use Case, we address the need to find transportation for referrals to emergency care, and the need to alert Health Centers, who are oftentimes not prepared with appropriate staff and equipment in order to manage incoming emergency patients from rural Health Posts, thus causing unnecessary maternal and infant mortality.
- Although ambulances are not available at Health Centers to dispatch to Health Posts, there are many local drivers of small three-wheeled vehicles (bajaj) or other type of vehicles (mini-buses, land cruisers, etc.) who may serve the role of an ambulance in emergencies. All HEWs will have their local bajaj driver numbers programmed into their phone from the start, and would thus be immediately able to call for a vehicle (this doesn’t ensure that someone can pay for the ride or that a driver is available, it only ensures that in cases where someone can pay and there is a driver, that the driver is notified and takes the woman).
- HEWs will call the Health Center in emergency referral situations so that Centers are prepared to receive patient.
How do we test the efficacy of this tool?
Our team randomly selected three sub-districts in the SNNP Region into one of the following groups:
- Treatment 1: All HEW received mobile phones equipped to perform the three use cases
- Treatment 2: All HEW and 2 VCHW within each kebele received mobile phones; HEW phones are software-equipped for the three use cases, and vCHW phones are dumbphones intended to make missed calls only.
- Control: No mobile phones were distributed.
In Treatment two, as you can see, we engage the vCHWs and dumbphones, which is a very simple and low-cost and non-technical intervention – no coding required! – because we’re interested in seeing if there will be a difference in health outcomes if this option is available. The vCHWs will be expected to use their dumbphones to:
- Take calls from HEWs to follow up with pregnant women if appointment times need to be changed.
- Missed call HEWs when they meet or hear of mothers in households who are not yet registered.
- Missed call the HEWs to request that they come register a child.
- Take calls from HEWs to follow up with a child's family to notify them of an upcoming vaccination day.
- Missed call HEW if a birth is going on and no HEW is present, so that they can at the least bring their supplies and materials for the TBA to use.
- Missed call to alert HEWs of any other maternal/child health emergency.
Because the risk for spillovers was too high if randomization was done at the health post level (i.e. control health posts seeing what the treatment posts were up to and wanting in on it!), our design randomizes one treatment per sub-district, which normally includes on health center and 5 satellite health posts. We excluded from the possible selection any sub-districts that have no mobile network coverage, any adjacent words, and any words with significantly different access to health services at the village level (as measured by population to health post ratio).
And at the end of the day, what are we trying to see changes in? A few things. In terms of ANC and Delivery, we will analyze changes in antenatal care (ANC) attendance – timing and # of visits, in numbers of clean and safe deliveries (of births delivered in kebele) – meaning HEW is present with her supplies, regardless of who delivers, and in number of deliveries assisted by a skilled attendant (of births delivered in kebele) – whether the TBA and/or HEW delivers. In the case of immunizations, we’ll be monitoring changes in immunization coverage (TT2, Penta3, Measles). And in regards to referrals, there is not a whole lot of change we can attribute to the phone, but will measure Instances of referral to HC in which women is seen upon arrival.
As a research team, we are still in discussion over appropriate outcome measures and would be very happy to hear about others’ experiences and suggestions here. We think it is important to disseminate our ‘results’ – the lessons learned along the research design road – throughout the study, not just at the very end. And we invite you to be in touch with questions and suggestions!
Many thanks for reading, and many thanks to the amazing team of researchers driving this study: Drs. Asfaw Atnafu and Solomon Shiferaw, of the AAU Center for eHealth, Dr. Mesganaw Fantahun, Head of the Dept of Reproductive and Family Health and Nutrition at AAU, and Dr. Rahel Bekele, Dean of the School of Information Sciences at AAU. Also, we developed the initial software in collaboration between Medic Mobile and the AAU School of Information Sciences, under the leadership of Henock Lulseged, and I would like to acknowledge the brilliant leadership of Dr. Mieraf Taddesse, my colleague at the Bank in Addis, who keeps our project moving forward always!
Post new comment