Sensor networks have already changed the way your mobile device locates itself, and they are set to change the way you commute, connect to the Internet, pick your network, and get your big datasets.
Automated, distributed processing of big data has a long history. Even before MapReduce frameworks such as Hadoop were developed, SETI@home was using a net of distributed personal computers to crunch data. Since then, cloud technologies have brought a sea change in processing ability, giving small enterprises possibilities previously available only to research institutions. Perhaps key to the future of big data is an analogous way of collecting information: sensor networks that collect and distribute data from devices automatically. These big sources will increase the resolution of datasets while reducing their cost.
There are more than 1 billion smartphones in the world, each of which is a potential node in sensor networks. Not only is the number of possible nodes growing, but the capabilities of each are increasing as manufacturers improve and expand the sensors on mobile phones and tablets.
No Going Back: The Addition of Sensors to the Galaxy S Series of Phones
Barometer
NFC
Gyroscope
Accelerometer
Bluetooth radio
WiFi radio
FM radio
Cell radio
Front camera
Rear camera
GPS
Magnetic field
Light flux
Battery temp.
Microphone
Touch
S
S II
S III
This massive pool of hardware is relatively untapped; only a few large-scale projects are drawing on it. Probably the largest sensor network in the world, with around 1 billion active nodes, is the Google Maps mobile app. Unless the user opts out, the app collects data on the location of cell towers and WiFi routers. Its tremendous success owes something to the way that data is fed back into Google's services, helping them locate your phone. It was discovered in 2011 that, unbeknownst to iPhone users, Apple was collecting a similar dataset, likely for location-as-a-service.
Building a sensor network
What is preventing sensor networks from scaling up is not the availability of hardware or the ease of distribution (iTunes and Google Play reach billions worldwide), but incentives for users to install and download apps. SETI solved this for data processing by offering a $1 million prize to anyone whose computer detected signs of alien life. Google Maps solves this by providing a fantastic service, and this is key to any app's success. Waze is using its app to map traffic flows in real-time. It feeds this information back into its navigation app, allowing you to optimize your commute. Waze has achieved more than 5 million downloads on Android.
A complementary approach to incentivizing users is to make the data freely available. WiFi crowdsourcing (a.k.a. war-driving) projects such as Wigle have pioneered this. RootMetrics, Sensorly, and OpenSignal (which I co-founded) all collect cellular metrics and use them to create browseable coverage maps as an independent alternative to those provided by carriers.
Crowdsourcing on mobile devices is not the only approach to building a sensor network. Other devices populating the Internet of Things -- including smart home energy monitors such as those from Bidgely, wireless scales such as Withings, and activity trackers such as Fitbit -- could provide massive near real-time datasets.
These new big sources will allow midsized enterprises to access data that was previously unavailable or unaffordable. As an example, cellphone carriers have collected data on their own performance for many years. A top-tier US carrier would have a fleet of up to 100 vans driving around and testing the network. When it takes hundreds of employees to collect a dataset in just one country, it's going to be expensive. At OpenSignal, we're a team of five collecting a similar (but bigger) dataset from more than 400,000 active users.
Every industry is connected in some way to what people do and the physical environment they inhabit. Sensor networks are unrivalled in their ability to measure both at a large scale. It's now up to app developers and others building the Internet of Things to make good on this potential.
Saul Sherry,
User Rank: Blogger 12/28/2012 | 5:42:57 AM
Re: Engaging the User - A challenge The payment idea isn't without conundrums though. Facebook, which doesn't pay for a user's input, is largely thought of as returning more accurate 'truthful' data. As soon as people feel more watched (which is something a paying situation brings up) people are more guarded.
The engagement James gets is transparent; offer us your data, you get to see the compiled data. It's clean, transparent, and useful.
MDMConsult,
User Rank: Exabyte Executive 12/20/2012 | 7:43:54 PM
Re: Engaging the User - A challenge Yes, Engagement measured by a user interacting can be by viewing, sharing, voting, commenting, reviewing, polls, etc al. Cost Per Engagement has the ability to leverage performance to online advertising. What can be brought out is quantity vs quality.
James Robinson,
User Rank: Blogger 12/19/2012 | 4:41:36 AM
Re: Engaging the User - A challenge Our approach has always been to make the app as useful as possible in itself. One way has been by feeding back crowsourced cell locations to show users the direction their signal is coming from. We're just about to roll out a big update that will also feed back our best estimates of accessible Wi-Fi networks by looking at networks used by several different app users.
Whenever a user contacts us from somewhere the app is not showing cell tower locations or Wi-Fi we ask them to just persist a little with using it as they will seed that area - getting this initial data is the hardest and for that we rely on people who are enthusiastic about this type of project.
Saul Sherry,
User Rank: Blogger 12/19/2012 | 3:08:54 AM
Re: Engaging the User - A challenge @James what process did you go through to decide what would act as a suitable reward for the users of OpenSignal? I guess the idea has the reward inherent in the information it produces?
James Robinson,
User Rank: Blogger 12/18/2012 | 5:59:01 AM
Re: Engaging the User - A challenge Nice idea, it could definitely supplement the income from banner ads, a lot will depend on the economics - new metrics will need to be developed instead of CPM there'll need to be some measure of the value of each type of data point.
Saul Sherry,
User Rank: Blogger 12/18/2012 | 1:56:05 AM
Re: Engaging the User - A challenge Could that be the kind of innovation that saves newspapers/magazines? As the ROI on straight banner advertising continues to drop, would sensors pose a possible answer?
Saul Sherry,
User Rank: Blogger 12/18/2012 | 1:54:55 AM
Re: A treasure trove of data. @legalcio, that brings to mind those mannequins watching us, which still makes me shudder. The key is always going to be incentivising the users to get involved with the sensors... but that's probably becoming less of an issue as the generations coming up have been opting into Facebook, Foursquare and all sorts since their early teens - and for them content in exchange for privacy/become an anonymized stat seems like a fair swap.
James Robinson,
User Rank: Blogger 12/17/2012 | 4:14:39 PM
Re: Engaging the User - A challenge User engagement is certainly the big challenge here.
That said there are other approaches than the ones above that could be explored. Here's one idea: instead of monetising via adverts, application developers include code (in the form of an SDK) that collects anonymized data, the developers are then paid per data they contribute.
As the value of data collected in these ways is discovered, we'll see more money invested ways of developing these sort of sensor networks.
legalcio,
User Rank: Exabyte Executive 12/17/2012 | 3:44:41 PM
A treasure trove of data. Proliferation of smart devices will provide a treasure trove of data. A small to medium sized business could measure customer population in stores, possibly using the information to adjust inventory. Traffic flow is another obvious data mine. How many customers are checking product web sites in a retail outlet would be a good thing to know too.
Saul Sherry,
User Rank: Blogger 12/17/2012 | 3:40:01 PM
Engaging the User - A challenge Great insight James, each app you mentioned there has a great method of encouraging engagement, but it seems to be quite a difficult thing to achieve. For each of these success stories there are dozens of crowd sourcing ideas which fall away because that dynamic of engaging the end user doesn't have enough of a hook.
Seems the tech/programming side and the emotional/user awareness side come to gether perfectly in your examples.
To save this item to your list of favorite Big Data Republic content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
