Key Learning Points for High-Frequency Data Analytics
- Data analytics and AI depends on historical data to make predictions of the future.
- During COVID-19, the change in behaviour and increased frequency of global events was enough to overwhelm most models, even sophisticated AI models.
- Data analysis using data from months ago is no longer relevant, especially in a financial context.
- The way to overcome this is by using high-frequency data.
How has traditional data analysis evolved during the pandemic to arrive at high-frequency data analytics?
Traditional data analytics looks at taking historical data and either performing descriptive analysis (what were our sales last quarter?) or predictive analysis (what will sales be next quarter?). Of course, prescriptive analytics (what should we do to increase sales?) is another major component of the analytics discipline, that we will leave out of scope for this article.
Descriptive and predictive analytics is largely dependent on the present and the future somewhat resembling the past. For example, a business may traditionally do well in the summer months and their forecasts would include this. Changes in the macroeconomic and microeconomic environments are modelled and expected to affect the results, however, this is usually a gradual change. For example, an increase in interest rates may decrease sales as customers have higher mortgage repayments or decide to save more to take advantage of the rates, however, these changes are unlikely to be made and change within the day. These changes are not immediate but come into account in a 12-month period. Largely, analytics is dependant that behaviour will repeat itself over time[1].
In March 2020, the world was thrown upside down. At the beginning of March much of the world was carrying on as normal, and by the end of March, most countries around the world were in some form of hard lockdown. On the 11th of March, the WHO declared COVID-19 can be characterized as a pandemic[2]. By the 13th of March, Europe had become the epicentre of the pandemic, and by April 2020 about half of the world’s population was in lockdown[3]. On the 1st of April 2020, Wimbledon was cancelled for the first time since World War Two[4]. During this timeframe, there was a sudden increase in panic buying[5] and people stocking up for an uncertain time period[6], together with a massive decrease in travel, both international and local.
To say that both financial and other analytics models did not predict this is an understatement, often even downside scenarios did not take into account the full impact of the pandemic. Financial models, often running one to three months behind, were showing a world that no longer existed. Most descriptive analytics either had to ramp up to show real-time data or risk showing data that was no longer relevant or helpful. In March, April and May of 2020 the world was changing radically on a daily basis, and data that was even a week old was completely irrelevant. In January of 2020, not one toilet paper manufacturer was expecting to see a surge in demand, certainly not enough to create a shortage! What transpired was that “nearly half of all grocery stores in the United States were out of stock of toilet paper for some part of the day on April 19”[7]!
Ecommerce was one of the early fields hit by the sudden changes in behaviour. During April 2020, the top Amazon searches and purchases included toilet paper, face masks, and hand sanitiser[8].
