Pattern Theory and Effective Analytics Programs
This week, on The Tuesday Standard, I will attempt to pull in some information from the world of applied mathematics in a way that a) makes me seem more intelligent than I am, b) doesn’t invoke any questions that I will be unable to answer, and thus embarrass myself, and c) provides some understanding of the complexity that can be introduced to the world we work in.
Patterns – we see them every day in the analysis we do as analytics professionals and they are part of the stories we tell those in our organizations that are looking to the data to make decisions. The key is to be able to recognize these patterns in a timely manner so that the information is able to fuel constant improvement efforts of your organization.
Patterns are also extremely timely right now as the field of artificial intelligence and analytics collide both at the theoretical level, as well as, increasingly, at the tool-set level (i.e. Google Analytics, Saleforce.com, IBM Watson, etc.).
It all started with a Swedish Statistician and Professor of Applied Mathematics at Brown by the name of Ulf Grenander. He introduced the term "Pattern Theory" in the 70's as a name for a field of applied mathematics which gave a theoretical setting for a large number of related ideas, techniques and results from fields such as computer vision, speech recognition, image and acoustic signal processing, pattern recognition and its statistical side, neural nets and parts of artificial intelligence. The problem that "Pattern Theory" aims to solve, can be described as – “the analysis of the patterns generated by the world in any modularity, with all their naturally occurring complexity and ambiguity, with the goal of reconstructing the processes, objects and events that produced them and of predicting these patterns when they reoccur.” (David Mumford – American Mathemetician – from “Pattern Theory: The Stochastic Analysis of Real-World Signals” in case you are interested.)
For the purposes of this blog post, we will focus perhaps on a more mundane approach to patterns than our new friend Ulf was looking at. I wanted to look at two areas that I believe we need to be aware of, and understand as we structure measurement and analytics in our organizations to support decision making.
In the work that I have been doing, where our analytics does have correlation to the health of the economy, unemployment rates, immigration and housing supply, to name a few variables, spotting patterns in how this plays out with our analytics is both exhilarating, and provides a stepping stone to the world of predictive analytics. Something we will explore in the future.
One - at a basic level Pattern Theory helps us establish business processes to support activities in support of analytics programs that understand Patterns, and then develop a framework to react when there is deviation from the patters for further exploration.
Effective analytics programs are not just about measuring what happened and storing that away as nice to know but not actioned information. Our role in the organization is to make sure leadership has the information they need to take the steps necessary to ensure objectives are being met. The powerful role that pattern theory plays is that it enables the notion of learning – and in some cases machine learning - to help organizations arrive at an understanding of what is happening and make corrective decisions far before a look at historical data will ever do. In some cases, the look back can be so late that decisions are constantly happening too slow to do anything more than harm the organization.
Here is an example that should help demonstrate this: Activity on a government website focused on income tax follows a pattern based on a number of constants: ever evolving tax code with yearly changes following Government Budgets, tax filing periods and deadlines, exploration of processed tax returns. In parallel, there are changes to interest rates and the stock market performance. There are several inputs that can be used to understand any patterns that may appear, including the number of website visitors and their behavior on a government website, tax filings statistics, RRSP contribution data, etc. These patterns – if they exist – come from a variety of different data sources – ultimately provide the information needed to, if not predict future activity, then surface what is happening and explain why things are unfolding as they are.
Two – The second area of patterns that we should understand at this point brings the theory one step further to help us predict, based on both patterns in intrinsic data, what could come next. The applications for this are already in play, and include predictive analytics for suggestion engines in e-commerce websites, helper apps for health and weight loss, and financial applications. Predictive analytics is patterns at work – collecting data – recognizing patterns – and then instead of raising alerts, presenting options in a user experience guided by this information. As in all intelligent systems, machine learning will then learn your reactions to the presented information and measure its success based on what was presented.
As an example, Google uses patterns to help it make suggestions of content you are looking for, and then based on how long + how much you interact + how much sharing you do of this content, scores its own suggestions against the patterns it used in the first place. This tension between pattern recognition and pattern testing rolled into the machine learning of prediction systems helps them constantly improve on their own: if built correctly.
Pattern Theory may have been a lofty topic to tackle on The Tuesday Standard, but I do believe at least a cursory examination was warranted given the important role it plays in the work we are doing as analytics professionals. I also believe it is a skill that should be developed as artificial intelligence is integrated more and more into the applications that our customers, citizens, etc. use on a regular basis.
If you have any questions or comments about this post, or are able to lend greater expertise to the topics of Patterns or Machine Learning, please do not hesitate to reach out. Thank you.