More on Using Crowdsourced Data to Find Big Picture Patterns (Take 3)
Thanks to commenter Differance bringing up in response to to our last post on this topic that made me want to take a new tack. You're absolutely right that information quality comes from people and that data's fitness for a particular purpose is very contextual. To continue in this direction, let’s look at how people use this information.
The people who are in most need of information about humanitarian disasters are the organized responders. [Commenter Iraqi Bootleg might have some very helpful ideas/examples here.] They are especially in need of big picture information that will help guide their response to do the most good with the resources employed. Civil authorities, humanitarian organizations, military units with a humanitarian mission, all hopefully have well-trained and experienced professionals in positions to make these critically important decisions. Let’s call our example professional Captain Lopez.
Successful approach to crowdsourcing data: Captain Lopez’ team gathers information from every source imaginable. Maps, helicopter overflights, satellite imagery, field reports from first responders, as well as phone calls and SMS messages from the general public. Captain Lopez helps her team assess the quality, quantity and usefulness of these different streams, each of which plays a part of the picture of the situation. SMS messages asking for help are bona fide requests, and help fill in information not otherwise available. Captain Lopez would probably rather have information from 911 calls, with trained operators following a carefully crafted protocol to extract the most crucial information, but the 911 system is swamped. So, SMS messages provide less valuable information, but their value is providing information that may not otherwise be available. And of course, in a less developed country, there unlikely to be a well-functioning 911 system: SMS may be the best way of signaling a specific need. Captain Lopez is a sophisticated user of information, and can direct her team appropriately. Success!
Unsuccessful approach to crowdsourcing data (on a map): Captain Lopez’ political boss turns down her urgent request to use a helicopter to make a survey of the building damage patterns. Why? Because the crisis-map of SMS messages is the “most comprehensive map” of the disaster we have (a quote from a senior agency head used in a recent Ushahidi presentation) and “SMS mapping has been shown to be predictive of building damage” and “it’s so much darn cheaper.” These last two are manufactured quotes, but based on the claims in the original post to which we’re responding. So, Captain Lopez ends up using the crisis-map in real-time to guide her team, and happens to miss the area of worst damage because of any number of real world reasons (inoperative cell towers comes to mind). It takes an extra six hours for the real picture to come through, and that delay has real impact. The helicopter survey is a more expensive, but more effective tool for getting the big picture. Failure!
I hope that this underscores the seriousness of this issue and makes it more tangible. We’re not having a purely academic/technical debate: rapid humanitarian response in a disaster saves lives. Delay costs lives.
As a thought experiment, imagine this approach being used in the Japan tsunami. How plausible would be to put SMS messages on a map and point to it, and say, there’s where the most severe damage is? There would be giant spots without little or no SMS traffic: the towns that were most severely affected. Using SMS for its purported predictive capabilities would likely to have been a second disaster. But, using it for what it is: real specific instances of needs would have been fine as part of a comprehensive assessment using all channels of information.
When claims of the “comprehensive” and “predictive” nature of a new tool is made, it’s naïve to expect that some people won’t leap to the conclusion that the tool should be expected to be comprehensive and predictive. And when it’s not, or when it’s less good than today’s standard of practice, people who make decisions believing that it is predictive or comprehensive will be in real danger of failing to meet their obligations to the people they serve. We need people to see SMS crowdsourcing through the successful application scenario above (and others like it), while being cautious to avoid the mistakes that stem from the second scenario.
As technologists working in highly important areas, it’s crucial we get this right.
The people who are in most need of information about humanitarian disasters are the organized responders. [Commenter Iraqi Bootleg might have some very helpful ideas/examples here.] They are especially in need of big picture information that will help guide their response to do the most good with the resources employed. Civil authorities, humanitarian organizations, military units with a humanitarian mission, all hopefully have well-trained and experienced professionals in positions to make these critically important decisions. Let’s call our example professional Captain Lopez.
Successful approach to crowdsourcing data: Captain Lopez’ team gathers information from every source imaginable. Maps, helicopter overflights, satellite imagery, field reports from first responders, as well as phone calls and SMS messages from the general public. Captain Lopez helps her team assess the quality, quantity and usefulness of these different streams, each of which plays a part of the picture of the situation. SMS messages asking for help are bona fide requests, and help fill in information not otherwise available. Captain Lopez would probably rather have information from 911 calls, with trained operators following a carefully crafted protocol to extract the most crucial information, but the 911 system is swamped. So, SMS messages provide less valuable information, but their value is providing information that may not otherwise be available. And of course, in a less developed country, there unlikely to be a well-functioning 911 system: SMS may be the best way of signaling a specific need. Captain Lopez is a sophisticated user of information, and can direct her team appropriately. Success!
Unsuccessful approach to crowdsourcing data (on a map): Captain Lopez’ political boss turns down her urgent request to use a helicopter to make a survey of the building damage patterns. Why? Because the crisis-map of SMS messages is the “most comprehensive map” of the disaster we have (a quote from a senior agency head used in a recent Ushahidi presentation) and “SMS mapping has been shown to be predictive of building damage” and “it’s so much darn cheaper.” These last two are manufactured quotes, but based on the claims in the original post to which we’re responding. So, Captain Lopez ends up using the crisis-map in real-time to guide her team, and happens to miss the area of worst damage because of any number of real world reasons (inoperative cell towers comes to mind). It takes an extra six hours for the real picture to come through, and that delay has real impact. The helicopter survey is a more expensive, but more effective tool for getting the big picture. Failure!
I hope that this underscores the seriousness of this issue and makes it more tangible. We’re not having a purely academic/technical debate: rapid humanitarian response in a disaster saves lives. Delay costs lives.
As a thought experiment, imagine this approach being used in the Japan tsunami. How plausible would be to put SMS messages on a map and point to it, and say, there’s where the most severe damage is? There would be giant spots without little or no SMS traffic: the towns that were most severely affected. Using SMS for its purported predictive capabilities would likely to have been a second disaster. But, using it for what it is: real specific instances of needs would have been fine as part of a comprehensive assessment using all channels of information.
When claims of the “comprehensive” and “predictive” nature of a new tool is made, it’s naïve to expect that some people won’t leap to the conclusion that the tool should be expected to be comprehensive and predictive. And when it’s not, or when it’s less good than today’s standard of practice, people who make decisions believing that it is predictive or comprehensive will be in real danger of failing to meet their obligations to the people they serve. We need people to see SMS crowdsourcing through the successful application scenario above (and others like it), while being cautious to avoid the mistakes that stem from the second scenario.
As technologists working in highly important areas, it’s crucial we get this right.
Comments
Cultural familiarity is an important aspect.
Just in case you wanted to take a look at this brief review of the potential uses (and limitations) of Web-based data for public health and beyond that mentions your concept of "sophisticated users":
http://www.unglobalpulse.org/blog/mining-web-digital-signals-lessons-public-health-research
Best
Emmanuel
UN Global Pulse