Although big data is on the minds of just about everyone, IT departments are still struggling to find ways to capture, manage, make sense of, and ultimately get cash value from all this data and information we are accumulating.
Gently does it
Here are my top common-sense recommendations about big data and how to generate value from it by working within your means -- and with the tools, systems and skills you already have. The key: Everything needs to be evolutionary, not revolutionary.
First, beware of technology in search of problems. Identify the pressing problems first and then how to apply the new technology to address them.
Second, don't be so focused on the future you forget the past. Semantic analysis, for example, is very attractive, but what about identifying what is in front of us and figuring out how to integrate data sources like data warehouses or operational systems first?
Third, pick useful, relevant problems to attack. Think about gathering CRM data and combining it with your web traffic history to see if there is a correlation between clicks and revenue, for example. That can be what makes your big data project concrete.
Fourth, given the volume of big data, it is important to identify what should be kept and what should be thrown out. Take machine status logs. Even though the frequency readings in the log are in milliseconds, collecting a thousand "status OK" records is useless.
Don't be afraid of mistakes
Learn to match the big data problems you have with the skills you already have in your toolbox. Don't waste time lamenting that you don't have the right skills or worry about how long it will take to get up to speed with new skills like MapReduce. Above all, don't be afraid of making a mistake. Instead, look at any mistake as a fruitful learning exercise. The goal should be, not to run away in fear from any problems, but to run toward them in order to address them.
The biggest bit of big data common sense I can suggest for you, however, has nothing to do with product, technology, or strategy. It's to engage in big data discussions with the business whenever possible in order to move forward productively. Why? If you stand still, you'll be passed by your competitors, who are chasing the same customers in the same tough market you are.
Re: The big disconnect in Big Data @Netcrawl. I can't advocate for shadow IT except in cases where central IT is completely non-responsive to reasonable requests for new projects/initiatives. Even in that case, the department head who wants to establish a shadow IT team had better get clearance from the CEO (who has the power to act on the non-responsive IT group).
There certainly are times when central IT gets stubborn. However, from what I've seen, they have come a long way in the past 10 years, opening up to new tech concepts. In most cases where they say "no" to an initiative today it's for one of two reasons: security or resources. The advocates for a departmental initiative that, lets say involves use of a third-party cloud service, need to go to IT with an open mind when it comes to security. IT will have concerns and questions, and for the department to disregard those concerns would be unreasonable.
If IT signs off on that type of initiative without vetting the security issues and a breach results, the CIO's neck is on the line. If staff resources are an issue for IT, then maybe the business unit will step up and fund a body or two. In any case, working with IT and fitting a departmental initiative into the IT security and data management structures ends up being a much better solution than sneaking services and servers in through the back door as shadow IT.
User Rank: Petabyte Pathfinder 7/8/2013 | 7:25:26 PM
Re: The big disconnect in Big Data @Saul shadow IT could be a quick and easy way for end users to improve their agility and competitiveness, if done corrrectly it has the potential to spur innovation, traditional IT companies are taking longer to achieve businees goal because they spent a vast majority of time on fixing day to day issues, they missed the big picture, they focused so heavily on one area that they can no longer respond to another works, and the worst thing- they can lose sight of real business requirements.
Sometimes employees became frustrated with their IT system's inability to respond to their needs, and decided to take matters on they own hands- the result was more agility for employees.
Re: The big disconnect in Big Data @Saul and Dcawrey. When we consider whether big data is being discussed within IT, I think one roadblock we run into is defining what IT is these days. If we are talking about the traditional "manage the infrastructure" IT organization, big data probably isn't on the table. They have enough on their plate. However, a lot of companies now have more of an expanded IT organization where some groups are very focused on data management and analytics. In some cases they report directly into IT, and in others they are sort of shared (dotted line reporting) between IT and some business unit.Then there are companies where business units have their own IT resources who are focused on big data. They are still IT, just on a different org chart. If those companies are smart they ensure that the departmental IT folks have a good relationship with the backroom folks, because the backroom is where the network and servers sit.
Re: The big disconnect in Big Data so @netcrawl - does that make you in favour of shadow IT being used to prove what can be done when you tear down beurocracy and just immerse yourself in goal driven data projects?
Re: The big disconnect in Big Data I imagine there's a flipside to that arguement @dcawrey, that is that IT have been discussing big data for a long time - but just not using the language popularised in this iteration of the hype machine.
A lot of businesses will find they aiming for the same goals, but IT and the business minded decision makers are using different languages so far... as everything matures, projects should be quicker to launch and with a keen eye on their final use.
User Rank: Petabyte Pathfinder 7/8/2013 | 8:15:07 AM
Re: The big disconnect in Big Data yeah, it is time consuming, it takes a lot of effort and resources-human and capital. but the pay off is equally good, i guess. The progress and analysis you do reap better results and if we are very clear as to how to use bigData then being patient with the long training sessions will not be a problem. we just need to have a better focus.
User Rank: Petabyte Pathfinder 7/8/2013 | 6:32:22 AM
Re: The big disconnect in Big Data @Kiran I agree with you, I believe companies believe in big data's potential but are frustrated by lengthy IT projects, complex tools and lack of special traning in big data analysis, the reality is the complexities of big data play a "critical role" in the dissatisfaction among those companies, there something to do with inability to effectively analyze data. And there's the last one- its a very lengthy project to take and finish.
User Rank: Petabyte Pathfinder 7/8/2013 | 2:29:00 AM
Re: The big disconnect in Big Data there has been enough talk and discussion about BigData, its advantages and why should companies use it and opt for it. however, we should now put it to action and see for its results and output for real.