MIT evaluation and report of water test kits

But household technologies that ensure water is safe to drink are few and far between.

MIT researchers have just launched a new report evaluating water test kits in Ahmedabad, India, where water-testing technologies are widely used by local governments and nonprofits, but are not yet available at the household level. The report, “Streamlining a Methodology for Product Evaluation: Water Test Kits in India,” details the study design and findings of the latest experimental evaluation implemented by the Comprehensive Initiative on Technology Evaluation (CITE), a programme supported by the US Agency for International Development (USAID) and led by a multidisciplinary team of faculty, staff, and students at MIT. The report also describes some of the lean and innovative techniques used in this evaluation to streamline CITE’s research approach.

Launched at MIT in 2012, CITE is a pioneering programme dedicated to developing methods for product evaluation in global development. CITE researchers evaluate products from three perspectives, including suitability — how well a product performs its purpose, scalability — how well the product’s supply chain effectively reaches consumers, and sustainability — how well the product is used correctly, consistently, and continuously by users over time.

Test kits affordable, but not available for DIY water quality testing
In India, 91 million people lack access to an improved drinking water source, putting them at risk for water-borne diseases like diarrhea, typhoid, and cholera. And in monsoon season, which generally runs from June through September, water quality can be especially unpredictable.

Without access to reliable, low-cost water testing labs, many development organisations and governments have turned to portable water test kits as a stopgap solution for testing community water quality, said CITE Water Test Kit Evaluation lead Jennifer Green.

“Low-income consumers don’t have the luxury of taking a water sample to a local lab to see if it’s safe to drink,” Green explains. “The labs are located in larger cities which take both time and money to get to, and even if they travel to the lab, the tests are expensive and not always accurate. It can take weeks to get results back, and in the meantime, those people may be continuing to drink contaminated water.”

“It’s critical to make these technologies as accessible as possible to low-income households,” says CITE Director Bishwapriya Sanyal. “We really have to work to ensure that not only are these technologies affordable from a price point perspective, but also from the perspective of the cost of time required to make use of the technology. For any person, and particularly a resource-constrained person, time is money.”

CITE studied two primary types of water test kits available on the local market: single-parameter, which measure just one aspect of a water sample, and multi-parameter, which measure multiple aspects.

Researchers found two single-parameter water test kits to be widely used by development programmes in India, both are which are manufactured in-country. The tests are simple: If hydrogen sulfide-producing bacteria associated with fecal contamination are detected in a water sample, the water turns black. If not, the sample turns yellow. CITE found that the technical performance of the two water test kits was nearly identical: Both kits were 94% accurate when used correctly.

To explore the viability of introducing personal-use test kits at the household level, CITE interviewed local consumers. Interviewees from low-income households were willing to pay for the product — each test costs just $0.50 — and 95% of those interviewed said they were likely to recommend the product to a neighbour or friend. However, the products were not available in local retail stores, and interviewees said that they would not know where to buy the product.

In addition, CITE ran a small experiment to assess how easy the tests were to use. Researchers found that consumers who saw a demonstration of how to use the test were nearly 70% more likely to use the test correctly than those who had just been read the instructions.

This suggests that if water test kits were to be effective at the household level, their sale may have to be accompanied with training to have a positive impact, Green says.

“Having English instructions in fine print on the side of a vial is not sufficient to teach consumers in villages how to do the test correctly,” Green explains. “A simple demonstration by a shop owner would help a lot, but some may be unwilling to invest the time with such a low profit margin on the tests. Graphical posters or community radio spots could also promote correct usage.”

Streamlining an evaluation approach
An evaluation of multi-parameter water test kits gave CITE the opportunity to test out new, streamlined evaluation approaches.

“Since the CITE programme began, we have explored the trade-offs between depth, breadth, and time required to perform our technology evaluations,” explains CITE Associate Director Joanne Mathias. “If we aim for exceptional rigor and address all possible research pathways, we might only be able to complete a few evaluations a year. If we just evaluate one aspect of a product, we could conduct many more evaluations, but they wouldn’t be all that valuable. We wanted to see if we could find a happy medium and produce actionable research.”

Research for this portion of the evaluation depended on remote research techniques, including a literature review, telephone interviews with stakeholders, and online surveys. This approach to an evaluation takes significantly less time and is less resource-intensive, but there are trade-offs.

“One of the key trade-offs is that we were not able to independently verify the performance data provided by the manufacturers of these tests, or to truly understand how the users would interact with the product,” Green says. “Both of these factors are critical to our evaluation. That said, learning as much as possible about the product and stakeholders before traveling to the field helps us focus our research and make our time spent there more efficient.”

In addition to experimenting with remote research techniques, CITE researchers used this evaluation to explore new kinds of multi-criteria decision analysis, an approach that allows CITE to compare alternative options and products against one another. CITE interviewed stakeholders from eight local organisations who use multi-parameter water test kits in their work to understand which aspects of the technology were most important to them: technical performance, ease of use, availability, affordability, demand generation, and environmental and health impact. CITE researchers then ran this input through a Group Decision Support System to inform weightings of those six criteria for the evaluation.

“This support system asks a select group of experts to think about how important each criteria is to them, but also dives deeper into decision theory to understand how groups using decision making tools reach consensus and how they influence one another,” Green says. “This is especially important when dealing with stakeholders from diverse socioeconomic backgrounds since there may be a wide range of opinions and some stakeholders may have much more power than others.”

In this way, Group Decision Support Systems and other such approaches can be valuable for stakeholders who would like to customise the results of a product evaluation to better fit their context and preferences. The full findings of the multi-parameter water test kit evaluation are now available on CITE’s website.

Looking Forward
This year, the CITE programme looks forward to publishing evaluation reports on malaria rapid diagnostic tests, post-harvest storage solutions, and educational technologies as well as completing its evaluations of solar water pumps, food aid packaging, and wheelchairs.

CITE’s research is funded by the USAID U.S. Global Development Lab. CITE is led by Principal Investigator Bishwapriya Sanyal of MIT’s Department of Urban Studies and Planning, and supported by MIT faculty and staff from D-Lab, Priscilla King Gray Public Service Center, Sociotechnical Systems Research Center, the Center for Transportation and Logistics, School of Engineering, and Sloan School of Management.

In addition to Green, co-authors on the report include Innocent Tumwebaze, Jonars Spielberg, Linda Annala, Madhi Zarghami, Sara Lynn Pesek, Sydney Beasley, and Vihar Parikh. CITE conducted its research in partnership with local Indian universities including the Indian Institute of Management, Ahmedabad and TERI University, New Delhi.