Given the fact that investor success (or failure) hinges on the availability of large amounts of useful information, it should come as no surprise that a growing number of traders are looking to big data as a powerful, perhaps even revolutionary, trading tool.
"Buy low and sell high" is the fundamental principle followed by every successful stock investor. Maximizing investment returns, however, requires informed trading decisions supported by timely, insightful information. Market tickers, news reports, weather sensors, social media gossip, analyst reports, government research, and an endless number of other data feeds all help create accurate buy-sell decisions.
An evolving technology
Big data technology hasn't yet reached the stage where analytics vendors can supply individual traders with systems that automatically select and collect a wide range of structured and unstructured datasets, and then provide instant insights. Individual investors, even relatively wealthy ones, simply don't have the skills or resources required to dabble in big data in any sort of meaningful way.
Yet, capital markets firms, including major stock exchanges, investment banks, brokerages, and financial advisory services, are typically well equipped to work with big data, and many of these organizations are already getting involved with the technology.
That should come as no shock. Financial industry firms have already spent billions of dollars over the past several years in attempts to develop full and accurate datasets. Although data management has improved significantly over this period, particularly within certain asset classes, seamless data sharing remains a challenge, since each business unit typically prefers to refer to its own set of data for its calculations, making enterprise-wide data analysis extremely difficult.
Some stock exchanges, such as NASDAQ and NYSE Euronext, have already deployed data warehouse applications such as EMC Greenplum and IBM Netezza in an effort to aggregate big data. These tools are designed to store and manage unstructured data collected from various sources, and then process the information in a way that makes it actionable.
Vendors, including EMC, IBM, Infosys, Oracle, Sybase, Tableau Software, and Teradata are already zeroing-in on capital markets organizations as prime big data customers. These vendors can offer customers an array of trading analytics tools for tasks like high frequency trading, predictive analytics, and pre-trade decision-support analytics, such as sentiment measurement and temporal/bi-temporal analytics.
How can financial markets benefit
from big data use?
Bottom line
Capital market tactics have evolved greatly over the past few decades, evolving from relatively simple strategies, such as 1980s-paired models, to today's complex gaming techniques. In recent years, new trading strategies, such as such as Twitter-based trading systems, have begun incorporating unstructured data. It appears almost inevitable that more types of unstructured data will be used in future trading strategies, and that big data analytics will eventually become a mainstay for all types of traders.
Edwin Willems,
User Rank: Bit Player 2/28/2013 | 1:09:15 PM
Re: Big Data & the Stock Market @john - aren't brokers using big data already for a long while? Looking at the information they put at the disposal of their customers on their trading platform, there doesn't seem to be much much difference now compared to 10 years ago. Unless they're not sharing it with their clients. Any view on this?
alvb1227,
User Rank: Petabyte Pathfinder 2/27/2013 | 7:59:47 PM
Re: Big Data & the Stock Market That's exaclty what I was thinking @Saul. It makes sense to me for big data to play an important role in how analysts make decisions, however, I wonder at the same time how much big data could be used to "manipulate" the market. Obviously traders, etc. will do whatever possible to gain an edge on their competition.
Re: Big Data & the Stock Market @Saul, these trading algorithms were well kept secrets from what I could see, and everyone gave the impression they were successful. Of course intellectual pride would dictate that if the algorithms failed, they'd never talked about it. It isn't something anyone wanted to share and open source certainly wasn't entertained. I don't think anyone wanted their algorithms to fall into the hands of anyone on Wall Street.
MDMConsult,
User Rank: Exabyte Executive 2/26/2013 | 1:33:12 PM
Re: Big Data & the Stock Market @Saul SEC still has its uncertainity challenges when it comes to its technology ambition and budgets passing through Congress as funding is decided by the law. However there is great potential being done here where the new Market Information Data Analytics, or MIDAS initiative was recently chosen. This is a real time software that is in place to help the SEC "monitor and understand mini-flash crashes, or pick up on possibly troublesome or illegal behavior" benefiting the long term investors protection. Even more greater opportunities for data scientists. The full speech on SEC's new initiatives can be found here:
Saul Sherry,
User Rank: Blogger 2/26/2013 | 12:07:30 PM
Re: Big Data & the Stock Market Is there a reason to stop there @MDMconsult? Could the laws be governed by data driven rules? Coudl these laws use big data to respond in real time and see unnecessary risk and unfair movement?
Saul Sherry,
User Rank: Blogger 2/26/2013 | 12:05:32 PM
Re: Big Data & the Stock Market @Daniel... that sounds like it could be a bit of a revelation!Were they generally successful or was it a bit hit and miss?
MDMConsult,
User Rank: Exabyte Executive 2/25/2013 | 7:56:23 PM
Re: Big Data & the Stock Market Yes, the SEC is based on strict rules and regulations one that should change to a more data driven culture. SEC should be able to recruit these data scientists and retain employees who understand how to use the analytics tools and big data effectively.
Re: Big Data & the Stock Market @Saul, when I was in grad school for a data science program, it was common knowledge that most of the professors and grad students had their own trading algorithims used for individual portfolio accounts; its like the first thing you do when you get a graduate level perspective in machine learning - build technology to make $ in the market. From what I saw, especially from the professors, these algorithms could be sold to any number of Wall Street firms, but they generally prefer to keep the technology private.
AlphaEdge,
User Rank: Exabyte Executive 2/25/2013 | 2:36:28 PM
Re: Big Data & the Stock Market Good point. It might be difficult for SEC to understand the Big Data analytics results. :) However, there are many places Big Data can make a big difference in financial industry anyway.
AlphaEdge,
User Rank: Exabyte Executive 2/25/2013 | 2:34:32 PM
Re: Big Data & the Stock Market @Saul, It might be difficult to get around regulation by using big data. For example, a machine learning algorithm might have less explanatory capability comparing to a simple regression or simple statistics analysis. On the other hand, it is true that with more inpformation available, it might be difficult to break the rules.
Join this radio show to truly understand what a CIO needs to do to build a successful private cloud and what skills and values the IT team will need to embody.
To save this item to your list of favorite Big Data Republic content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
