The advent of technology and its high rate of evolution has impacted on all aspect of life from Artificial intelligence to 3D printing of human organs. This impact has greatly impacted on the financial industry creating such phenomenon like fintech, revolutionalised trading systems and platforms advanced payment system among others. The change had been at dizzying speed such that it is very difficult for very discerning organizations to cope with the speed. This is despite the fact that being left behind by technology will make such technology laggards to become  part of the ever expanding corporate graveyard. For the global financial services sector, one of the most powerful tools of coping with the challenge is Big Data Analytics.

Environmental Dynamics

The business environment globally has been going through rapid changes since the market crash of 2008. These changes are being accelerated and exacerbated by unfolding market events both political and cyclic, technological advancement and evolving dynamics in risks and their management regulatory reforms. The effect on the capital market had been very great with unprecedented reaction from regulators since the financial catastrophe arose over recent years. Also, market events, technological changes, and regulatory reforms continuously throw up new challenges for the capital markets industry. Moreover, problems like dwindling revenues and increased cost pressures are snowballing continuously.

Consequently, the capital markets industry has been forced to continuously think of solution to improve revenues and reduce costs at the same time. Solutions to many issues that the  industry faces today lies in house, and leveraging available data is one of them. The capital market industry captures data in each and every operation, which results in huge volume data. The maximum potentials of the massive data troves can be earned through Big Data and can act as a game changer. Big data analytics can influence the business by finding the solutions to current business problems as well as identifying new business opportunities.

Nevertheless, big data analytics as virile tool can sift through massive data warehouse that are present in modern business environments. These tools can analyze both structured and unstructured data and create logical patterns to help business decisions.

Big Data Analytics as Virile Tool

While the different aspect of bid data analytics has been applied to a whole host of industries, the capital markets have been relatively slow in adopting these strategies, especially in developing countries. Within the financial services sector, big data has gained far more traction within retail banking due to the increasing desire of these financial institutions to profile their customers in a similar manner to early adopters of big data such as Amazon or Google. On the institutional side of the capital markets, there has traditionally been far more customer stickiness; hence there has been less incentive to apply big data in this manner. Big data strategies have, however, begun to make some impact in a select few areas of the capital markets over recent years, including area of sentiment analysis for trading, risk analytics, and market surveillance.

Defining Big Data

Though big data analytics has its unique characteristics, the term is often used synonymously with related concept such as Business Intelligence (BI) and data mining. While there is no doubt that that all the three terms relate to data crunching and in many cases advanced analytics. This uniqueness of big data concept is not only in terms of data volumes but more of data complexity and variability; dynamics of transactions and the dimension of data sources are so dissimilar and complex that they require special methods and technologies in order to draw insight out of data (for instance, traditional data warehouse solutions may fall short when dealing with big data). This also forms the basis for the most used definition of big data, the four V’s of big data: Volume, Velocity and Variety:

  • Volume: Large amounts of data, from datasets with sizes of terabytes to zettabyte.
  • Velocity: Large amounts of data from transactions with high refresh rate resulting in data streams coming at great speed and the time to act on the basis of these data streams will often be very short. There is a shift from batch processing to real time streaming.
  • Variety: Data come from different data sources. For the first, data can come from both internal and external data source. More importantly, data can come in various formats such as transaction and log data from various applications, structured data as database table, semi-structured data such as XML data, unstructured data such as text, images, video streams, audio statement, and more.
  • Veracity: This relates to biases, uncertainty, noise and volatility of the data.

This leads us to one of the most widely used definition in the industry. Gartner (2012)[1] defines Big Data as “is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation”.

The Domain of Big Data Analytics

It is unquestionable that the four Vs alone cannot be considered to represent a big data problem in the capital markets. Strategies that focus only on coping with a high volume of data have existed for some time and have not previously been labeled as big data strategies. The gradual increase in trading volume across financial markets over recent years, for example, has not posed a significant threat to current front, middle or back office technology (aside from areas where manual intervention or workarounds are rife). Tick data has always been a large data set and firms have been dealing with that in a structured manner for years. They may not be perfect, but trading and securities processing technologies have generally been able to scale up to meet increased electronic flows of data resulting from market structure change and increased electronic activity.

Related to this, in the front office, high frequency trading (HFT) technology has adequately coped with much higher velocity of data moving the industry from dealing in seconds to milliseconds to microseconds, but this is not traditionally considered solely the realm of big data (there are plenty of other buzzwords for that space). Analytical manipulation of complex and varied data sets related to instruments such as over-the-counter derivatives has also been in big data in capital markets existence for much longer than big data has existed as a concept.

Complexity or variety of data alone is therefore not sufficient to warrant being tagged a big data problem. So what exactly constitutes a big data challenge in the capital markets context? There appears to be the most take-up of the term in areas related to projects that involve multiple variables, such as high volumes of complex data that must be cross-referenced in a specific timeframe. These tasks do not necessarily need to be performed in real time, though they can be in the context of sentiment analysis for trading, but there is a focus on reducing the latency of data aggregation tasks for ad hoc regulatory reporting or risk analysis, for example. As large firms focus on trying to eliminate silos, especially in response to regulatory mandates, this aggregation has presented significant challenges.

Currently, there are no viable public examples of enterprise-wide big data projects in place within financial institutions active in the capital markets; hence it is very early days overall for the development of these strategies and techniques within the sector. There are plenty of projects being conducted at the business unit level but no public case studies at a group-wide level.

Benefit of Big Data Analytics Strategy

The definition and characteristics of big data summarized succinctly the main benefits of big data analytics. These include:

  • To draw insight from data,
  • To make better decision based on the insight, and
  • To automate the decision and bake it into a business process, hence process automation.

In a more detailed level, each big data solution may address particular business problems the organizations face and the business value of the solution is further connected to the original business problems. When building a business case for big data analytics project, it is important to start with a business problem, not data or technology. Collecting data or purchasing technology without a clear business target is a losing strategy. A business case for analytics must solve real business problems that an organization faces.

Other general benefits of big data strategies include the following:

  1. Costs reduction

Appropriate application of big data is expected to improve efficiency of the system with demand-driven production and optimum utilization of input. Cost saving can also arise from automation and use of AI to reduce manual processes in production and operations.

  1. Higher workforce productivity

Prompt and accurate decision permitted by big data make decision makers to be more pro-active and productive by boosting confidence among the employees.

iii. Setting up competitive pricing

Organisation can leverage on big data to create differentiated pricing strategies for develop competitive pricing and earning the associated revenue benefits. This may be achieved through complex market segmentation. demographics based sales strategies

  1. Driving brand loyalty

Behavioural science suggests that customers are likely to respond more to relationship-driven marketing. Through detailed analysis of prior buying activities, organization can simulate and predicts the specific reaction of current and potential customers and offer products and services accordingly. This leads to higher customer value to the organization through long-term relationship.

  1. Hiring smarter people for smarter jobs

HR analytics is one of the most common application of data analytics in many organizations as HR managers can now select candidates by accessing profiled data from social media, business databases and job search engines.

Big Data Analytic in Capital Market Development

Big data analytics plays a significant role in organizational efficiency. The benefits that come with big data strategies has helped many companies to gain a competitive advantage over their rivals, generally by virtue of increased awareness which an organisation and its workforce gains by using analytics as the basis for decision making. Below are immediate areas for profitable applications of Big Data for Capital Markets institutions.

1)   Revenue generation

  • Behavioral analysis for trading strategies (scanner algorithms), and understanding of customer interactions, i.e. use the data provided by Session Description Protocol (SDP). e-platforms represent a tremendous source of new information about clients, such as the types of transactions clients do or queries they run, or research they look at.
  • Trading analytics: includes analytics for HFT, predictive analytics, pre-trade decision support analytics. Insights from market indicators, economic indicators, and sentiment analysis for stocks and events may be used to enrich the information set used by traders and investors alike for making investment decisions.

2)   Regulatory compliance

  • Market surveillance and fraud detection: the ability to consume different channels and types of data – including instant messages, phone recordings, emails and internet content and consolidate all this into usable database allows advanced pattern matching analytics to spot anomalous behavior.
  • Regulatory reporting: big data enables the cross-referencing of key sets of internal data related to derivatives instruments in order to facilitate trade reconstruction and reporting (as illustrated by Dodd-Frank requirements). Compliance to a growing set of regulations (Dodd Frank, Solvency II, EMIR, audits…) adds more pressure on banks to develop sustainable long-term data management strategies.

3)   Risk management

Big data introduces a revolution in risk management, as it allows the production and monitoring of real-time, on-demand performance metrics and risk measures across product lines. To be more precise, big data allows more consolidated views of risk, gives prediction capability for expected risk, more flexible tools to better match banks’ changing business environment, better reporting format (interactive, dynamic, leveraging data visualisation technologies), better allocation monitoring of Scarce Resources across region and business lines. Stress test becomes a powerful monitoring and business steering tool, all the more so as the real-time dimension allows optimal hedging and reduces associated costs. Also, big data is a key component of cyber security as it allows enhancing detection in unstructured networks.

Scaled Risk solves the growing needs for smart data processing in the capital market industry by providing a Big Data and in-memory analytics platform that assures real-time historical and live trade data analytics to help investment firms accomplish real-time enterprise-wide risk management and comply with current and future regulatory demands.

4)   Cost reduction and operational efficiency

  • The data aggregation process for ad-hoc reporting, to feed both internal and external reporting functions, is often painful and costly, and big data addresses this cost. For instance, matching/reconciliation of trades across various systems can result in operational risk of invalid, duplicated or failed trades. Data tagging allows to easily identifying trades and events like corporate actions.
  • Similarly, maintenance (storage, handling and processing) and consolidation of data for various asset classes, product lines, service layers (FO, BO) and coming from various vendors, is extremely challenging, and common dictionaries to handle all this information are hard to find. This is solved by big data.
  • Versioning and audit of trade transactions: structured databases require keeping all versions in the same table and flagging latest version, with the associated difficulty to identify modified fields. With big data, cells are versioned (timestamp), and each revision can be retrieved easily by API.

The Future of Big Data Analytics in Capital Market

Big data analytic in capital markets is a long way off being considered mature; it has yet to reach a tipping point in usage within top-tier sell-side firms, let alone across the industry as a whole. Though there are viable reasons for investment, these tend to be focused on specific use cases rather than an over-arching big data strategy. Those that are currently aware of the benefits of big data and have successfully realized some of these internally (in spite of the hurdles), are certainly keen to extend their usage of these strategies and technologies. These implementations will, however, likely remain piecemeal for the near future.


Big data analytics has will continue to evolve and become the new normal in many aspect of capital market operations. As such, the knowledge of bid data analytics must be central to the continuous/mandatory training of the Institute of Capital market Analyst. Members must improve their fluency in data manipulation and application of analytics tools. This must at least include bid data management technology, bid data capturing methodology (from point of sale systems, web forms, sensors), big data storage facilities (like cloud computing), data visuaalisation softwares, descriptive analytics tools, diagnostic analytics strategies, predictive analytics methodologies, prescriptive analytics and automation.