Robert Mullins, Technology Journalist, 5/14/2013 Comment now
A newly-released survey shows that the US government is gradually gearing up to develop systems to analyze big data stores to make government more efficient, but that state and local governments in the US are way behind.
Not even a conversation
Forty-four percent of state and local governments aren't even discussing a big data analytics project, ...
Mithun Sridharan, Business Development Manager, Corporate Quality Consulting, 5/10/2013 Comment now
Over the course of this decade, we could expect big data applications to enable truly personalized healthcare solutions, based on an individual's and his or her family's medical history, characteristics, and progression over time.
John Edwards, Technology Journalist & Author, 5/7/2013 Comment now
The largest biotech companies are amassing multiple petabytes of data containing an array of useful drug development resources, such as patient outcomes, gene expression profiles, DNA sequencing data, and biomarker reliability. Putting all of this big data to productive use in the quest to find new treatments and streamline research and development is ...
Sam Zindel, Data Strategist at iCrossing Digital Marketing, filled us in on how supermarket delivery giant Ocado uses big data to identify and serve specific content to vegetarians once they spot them on their website.
This was part of Sam's talk at The Big Data Show about putting the customer at the heart of digital marketing, as well as making the most of the data you already have.
On the opening day of the Big Data Show, Mike Cornwell, CEO of The IDM, was generous enough to give us some time to discuss his afternoon panel session. He also offered a word of caution on the state of marketing data. We're all getting excited about big data, but it seems most people still can't deal with their small data in the best possible manner.
"Just knowing enough to find some insight from information and using it intelligently for marketing still seems to be beyond a lot of organizations," he said.
Does this resonate with your business? Have you got the small data figured out before you invest time in de-siloing and bringing more information together?
Big data is awash with acronyms at the moment, none more widely used than HDFS. Let's cut to the chase... it stands for Hadoop Distributed File System.
This is the system of distributing files that allows Hadoop to work on huge data sets at speed. It spreads blocks of data across different servers, as well as duplicating those blocks of data, and storing them distinctly.
Let's see why with an example.
Sarianne works in the financial markets, and runs a lot of predictive models to make sure her investments are minimum risk.
Utilising HDFS, her queries through Hadoop can run quickly because the data blocks are stored separately -- meaning all the computation can happen in one go, rather than queuing up behind each other.
As an added benefit, if one server fails (as one is bound to, given the amount of servers and disk drives needed to run big data projects) it won't stop Sarianne's models from pulling the data they need, because HDFS duplicated those blocks -- meaning Hadoop can return Sarianne's results in double quick time.
Pig basically simplifies the processes needed to get analytics done through Hadoop on your big data sets.
Like the animal, Pig is not a fussy eater, getting its name from its ability to crunch through data, no matter what form it takes. It acts as a scripting interface to
Hadoop, meaning a lack of
MapReduce programming experience won't hold you back.
Example: Harvey works in a government office, looking to formulate new solutions for his city's parking problems. He knows how to use data, but writing his own mapper and reduce functions is a little beyond him.
Luckily, he's been set up with access to the databases through Pig, meaning he can draw on sources like parking ticket records and population density maps. Taking advantage of Pig's eat-anything attitude, he can also mine topics from a call for email suggestions his department sent to local residents, as well as sensor information about the amount of traffic on the roads. In spite of his limited programming capabilities, Pig allows Harvey to query these data sets and sketch out some draft suggestions he can use to alleviate the local parking problems.
In the first of a series of interviews with business leaders who leverage big data, we talk to James Robinson, CTO and co-founder of OpenSignal.
OpenSignal combines big data technologies and sensor data from mobile phones to give insight to both mobile consumers and telecommunications giants. Robinson is also a contributing writer on Big Data Republic.