Lesson 1 of 7 · All lessons in this module
Free lesson: the first lesson of every module is open to everyone.
Lesson 1 of 7 · 12 min
Fintech and the four Vs of Big Data
Fintech gives analysts two things, far more data and smarter tools to read it, and Big Data is defined by its volume, velocity and variety, plus veracity once it is used to predict.
In short
- Fintech here means technological innovation in the design and delivery of financial products and services; it has moved from automating routine tasks, to rule-based execution, to machine-learning decision making.
- Two fintech areas matter most for quant analysis: the analysis of large datasets (traditional plus alternative data) and new analytical tools based on artificial intelligence.
- Volume: millions to billions of data points, now measured in terabytes and petabytes.
- Velocity: data are recorded and sent in real time or near real time, which is now the norm.
- Variety: many sources and formats: structured, semi-structured and unstructured.
- Veracity: when data are used for inference or prediction, the credibility and reliability of each source becomes critical.
What fintech means for an analyst
In its broadest sense, fintech is any technology-driven innovation in financial services. This reading uses a narrower meaning: technological innovation in the design and delivery of financial products and services. In everyday use, the word also covers the companies that build these technologies and the business sector they form.
Fintech has grown up in stages. The first wave was data processing and the automation of routine tasks. Next came systems that carried out decisions by following set rules and instructions. Today's frontier is decision making driven by machine learning, where programs get better at a task over time and can sometimes perform far beyond human ability.
Along the way fintech has created new ways to deliver investment advice, financial planning, business lending and payments. Many of these innovations challenge the business models of established financial firms.
For a quantitative analyst, two areas matter most. The first is the analysis of large datasets: on top of growing traditional data (prices, financial statements, economic indicators), managers can now feed alternative data from social media, sensors and other non-traditional sources into decisions, aiming to add alpha and limit losses. The second is new analytical tools: artificial intelligence (AI) techniques may find complex, non-linear relationships in huge datasets better than traditional statistical methods can.
| Stage | What the technology does |
|---|---|
| Early | Processes data and automates routine tasks |
| Rule-based | Executes decisions according to specified rules and instructions |
| Current | Learns from data over time and makes complex decisions, sometimes better than humans |
This is not only a story about 'quant shops'. Fundamental managers also use AI tools, for example to sift filings and earnings calls, and combine them with their own judgment in a hybrid decision process.
The three Vs: volume, velocity and variety
Big Data is the enormous flow of information produced by industry, governments, individuals and electronic devices. It mixes traditional data (from exchanges, companies and governments) with alternative data (from devices, social media, sensor networks and the by-products of running a business).
Three characteristics mark a dataset as Big Data. Volume is sheer size: many millions or even billions of data points held in files, records and tables. Sizes have moved from megabytes and gigabytes to terabytes and petabytes, where each unit is 1,000 times the one before.
Velocity is the speed and frequency at which data are recorded and transmitted. Real-time or near-real-time delivery is now normal in many areas, not an exception. Variety is the range of sources and formats: one analysis might combine database tables, web pages, photos, audio and video.
| V | Question it answers | Example |
|---|---|---|
| Volume | How much data? | Billions of card transactions a year |
| Velocity | How fast does it arrive? | Prices streamed tick by tick in real time |
| Variety | How many sources and formats? | Filings, posts, satellite photos and call audio combined |
| Veracity | Can we trust it? | Is a product review genuine, and is the source reliable? |
Veracity: the fourth V
Volume, velocity and variety describe the data themselves. A fourth V, veracity, comes into play when the data are used for inference or prediction. It asks how credible and reliable each data source is.
Judging source quality is part of any empirical study, but it becomes critical with Big Data because the data come from so many different, often unvetted places. Big Data makes the old problem of separating quality from quantity much harder.
In practice: a billion noisy or fake observations can mislead a model more confidently than a thousand clean ones. The bigger and more varied the dataset, the more effort must go into checking where each piece came from.
More data is not the same as better data. Whenever an exam question pairs Big Data with 'credibility' or 'reliability', the answer is veracity.
Structured, semi-structured and unstructured data
Structured data fit neatly into tables, usually in a database where each field (column) holds the same type of information, such as an SQL table of daily closing prices. They are ready to analyse.
Unstructured data are disparate and disorganised and cannot be shown in rows and columns: social media posts, emails, text messages, voice recordings, pictures, blogs, and raw scanner or sensor output. They usually need specialised applications or custom code before an investment professional can use them. A program must first parse a stack of emails, for instance, before anything in them can be counted.
Semi-structured data have features of both. HTML code for a web page is the standard example: its tags impose some order, but the content inside them is free-form.
- Structured: SQL table of prices or financial ratios.
- Semi-structured: HTML page.
- Unstructured: video message, photo, voice recording, social media post.
Formula
- Data volume units
Megabyte = one million bytes, gigabyte = one billion, terabyte = one trillion, petabyte = one quadrillion. Each step up multiplies by 1,000.
Worked examples
Example 1 Converting data sizes
A Dublin hedge fund stores 2.4 PB of tick data. Its new cloud plan is priced per terabyte. How many terabytes, and how many gigabytes, does it hold?
Example 2 Sorting data by structure
An analyst in Oslo collects (i) a database table of daily NOK government bond yields, (ii) saved HTML pages from a financial news site and (iii) audio recordings of earnings calls. How should each be classified?
Example 3 Which V is the real risk?
A fund in Singapore streams millions of geotagged social media posts per hour, in text, photo and video form, to forecast weekly sales at a restaurant chain. Which characteristics of Big Data apply, and which most threatens the forecast?
On the exam
Expect short concept items: which V a description illustrates, the fact that real-time data are now common, or whether a format is structured, semi-structured or unstructured.
- Veracity is about credibility and reliability, not size. Do not confuse it with volume.
- Real-time or near-real-time data are now the norm, not rare: that is velocity.
- HTML is semi-structured, not structured: it has tags, but the content is free-form.
- Big Data includes traditional data too; it is not only alternative data.
Check yourself
Q1Which of the following is most likely a characteristic of Big Data?
Q2Which characteristic of Big Data becomes especially important when the data are used for inference or prediction?
Q3A vendor delivers 750 GB of satellite images per day. Over 400 days, the total volume delivered is closest to:
Q4An HTML page downloaded from a company's website is best described as:
Q5Which of the following is least likely one of the fintech areas most directly relevant to quantitative investment analysis?
Q6The most recent stage in the development of fintech is best described as:
Answer the 6 questions: +10 XP for each one you get right first time.
Liked this lesson? Get the rest free for 7 days
Create a free account to open the other 6 lessons in this module, with videos, flashcards that come back before you forget them, the question bank and full mocks. No card needed.