BYOBD: Bring Your Own Big Data

The hack/reduce model of experimenting with data should be embraced within the organization, because it could lead to a combination of four rewarding results.

Many organizations would like to experiment with big data, but they are put off by the setup costs, a lack of internal expertise, and an unclear understanding of the outcomes they may expect to receive. Clearly, this is a poor business case to present to the CFO signing off on the project and its budget.

One alternative is to look for external entities that can provide the necessary components on an outsourcing arrangement. Unfortunately, not too many environments support this type of computing need. As such, perhaps now is a perfect time for you and your organization to promote the idea of developing a collaborative big data venture -- in essence, establishing a facility with partners to leverage the technology, people, and resources needed to succeed in big data.

The hack/reduce model
One model that could be used as a framework is the nonprofit hack/reduce organization based in Cambridge, Mass., whose tagline is "Code big or go home." Hack/reduce is a partnership involving the state of Massachusetts, local firms, MIT, Harvard, and several other universities. It links participants with a host of industry sponsors (Google, IBM, Microsoft, Dell, etc.) and specialist firms such as GoGrid, a provider of cloud hosting and automated provisioning architectures, and Greenplum, a division of EMC that offers analytics solutions. There are also venture capital partners such as Bessemer and Bain Capital, making that potential for a big data venture even more accessible.

Though hack/reduce's membership is based on a meritocratic system (Fellows are gurus, and Resident Hackers are programmers working on "something incredibly cool"), there is room for more mortal types (as Contributors) who want to learn and collaborate on big data projects.

By following the hack/reduce model, big data development mashups could be created anywhere, but the best target locations are within technology centers. In the UK, tech clusters such as London, Reading, Manchester, Oxford, and Cambridge are obvious targets. However, they could be established anywhere with the potential for building a cluster of industry players, academic computing, and data-driven business. It would also be ideal if these ventures were supported by private equity or venture capital investments.

The payoff
What would the potential payoff be? Four outcomes would result.

1. In the BYOD model, the partners would share the facility costs. Many organizations already have high-performance computing clusters and systems. The costs of establishing and running them can be reduced by pooling resources into a common datacenter. Software licenses and operational costs would also be shared, and vendors supporting the project could be persuaded to act as collaborators, rather than purely as vendors.
2. The learning curve would be reduced for all participants. One of the biggest barriers to entry in the big data space is access to talent. The BYOBD approach would allow easier access to talent and training from vendors and partners. University resources could be leveraged in a centralized location, with students engaged in projects eventually becoming highly talented potential employees.
3. The environment could spin off new entities and businesses, creating a virtuous cycle of innovation that further builds the talent pool for big data. One way to achieve additional levels of ROI is to encourage innovation. By allowing startups to use their facility, large enterprises could buy in to these startups, develop them, and spin them off for a profit.
4. You can use these facilities to literally bring your own big data and exploit its capabilities.

In light of these advantages, executives need to act fast, move out of their silos, and collaborate. It has been done before in other industries: Covisint in the automotive supply chain, GS1 in B2B data, healthcare exchanges in the insurance industry, and Orbitz in the airline industry. The first movers in these collaborative spaces got the best ROI, an early-mover advantage, and leverage. The laggards lost out, and many are fighting to recoup lost marketshare.

Success in big data requires thinking outside the traditional datacenter box, and the hack/reduce model may well be the place to house those thought processes. At the end of the day, big data is not traditional data processing; it's more akin to technology innovation, and that requires an entrepreneur's mindset. Where better to begin than by building a big data startup? Who knows? It may grow to be bigger than your current business.

Related posts:

• Free Big Data Education: A Management Perspective
• How to Hire a Qualified Data Scientist
• Hadoop Won't Turn Your Dump Into a Data Warehouse

— Robert Plant, Associate Professor, School of Business Administration, University of Miami