Audrey Messer, founder and chief analytics officer of Centricity Analytics, explains how bias can be tackled. She also explains how organisations can implement analytics.
Bias exists and needs to be addressed. Many practitioners in the field do not necessarily acknowledge that. They think that since they are working with data and models, the output is then ‘pure’ or free from bias. These are the observations of Audrey Messer. “In the past few years more and more instances of bias, from the benign to the possibly illegal, have been revealed. The model’s biased output generally starts at the beginning with the training dataset and control group selection.”
Black box
For many companies a key challenge with AI and machine learning is that it is opaque and ‘black-boxy’. As these complex models are used to drive and impact the business it is crucial that internal stakeholders, like executives, sales teams, and even lawyers, need to be involved. “If they don’t understand it, they will have the power to veto its use. Processes will need to be put in place to test for bias, to review whether data is being used properly, according to GRPD, CCPA and HIPAA guidelines.” In order to tackle bias, Messer recommends the following:
1. The new world really needs to accelerate and address the fact that there is bias – intentional or not.
2. Selecting sample sources and selecting representative training sets are equally important. “Select incorrectly and the results can drive incorrect, business-impacting, and in limited cases illegal outcomes.”
3. In market research a study can be derailed by other factors than just poor sample selection. “The way the questions are asked, when/how the survey is deployed and the types of bias of the human researcher all influence the outcomes. Similarly, be aware that machine learning and AI are vulnerable to bias in other areas of the model’s design, development and deployment.”
Fighting data with data
Having worked for Metlife, one of the largest global insurance providers, Messer has experienced up-close the insurance industry’s transformation from a product focused strategy to one of customer centric focus. She describes insurance executives as data driven decision makers, with many of them coming up through the actuary ranks or with a sales agent background; professionals who always had clear KPI’s. “The source of misconceptions was that either they had seen data from five to twenty years ago, or they just hadn’t any data to suggest an alternative. So we found success in fighting data with data; setting up experiments that had clear goals, hypothesis and action standards; and being patient in validating models, having conversations about concerns or beliefs.”
As for CLTV (customer lifetime value) models, a crucial tool for insurers to understand their clients, Messer identifies the following pitfalls:
1. Small historical datasets: “For a new company, when is it a good time to leverage the customer data to calculate and implement the model?”
2. Predictions are not set in stone: “This is more about the implementation of the model. When used incorrectly, CLTV is used to justify a self-perpetuating model of only caring about high-value customers.”
3. Later acquired customers generally do not look the same as initial customers (early adopters): “As customers come in from a range of sources, CLTV can change in unexpected ways, particularly if it is being calculated as just one number. Generally, to help uncover the nuances of CLTV, building out segments and cohorts can assist in overcoming this.”
Keeping it simple
Regardless of the segment, it’s Messer’s strong conviction that analytics should always drive business decisions. “Yes absolutely, unequivocally and indisputably. Many companies and executives think that analytics is just a fancy buzzword. They either A. steer away from it because it is seen as too expensive, difficult or complex, or B. tools and processes have been adopted without a strategic alignment and clear problem definition.” She offers solutions for both scenarios:
A. “There are some foundational analytics that can be incorporated using a simple tool like Excel. You don’t have to start with AI/machine learning. For instance: start with some simple descriptive analytics that describe what happened, and then create simple charts that show yearly trends. Good visualizations will identify areas for further investigation. Gaining trust and incorporating data into decisions will help advance a stronger analytics team in the long run.”
B. “There needs to be a strong relationship – trust, communication – between the analytics and executive teams. Do the executives truly understand what they will get from an analytics model; what types of questions it can and cannot answer? Is everyone even aligned on what AI/machine learning is? Is there alignment on what decisions the ‘computer’ will make versus the decisions a person will make? If there are risks in the model, has there been a conversation about how to mitigate those risks?”
Three challenges
When it comes to the biggest barriers for using analytics within organisations, Messer identifies three key areas:
1. Lack of alignment: “The business goals need to drive the analytics and research problems. Many times, I have seen solutions go in search of problems. This can be dangerous; it can create distrust between teams, and for executives it devalues what analytics can offer. I also talk a lot about alignment of KPI and metrics definitions. I have seen it create a lot of confusion; do the executives really understand what a key performance indicator is versus, say, a vanity metric? Are all team members aligned on key definitions, like retention? Is the formula consistent across the business? Analytics leaders need to be communicating, building relationships within the organisation, and be clear about the problem the analytics model is looking to solve.”
2. Access to data: “This can range from not doing an analysis because the data isn’t perfect, to having the data but it isn’t quite in useable form. Issues with data quality can also be in this bucket. Furthermore, is the data comprehensive enough to provide a fair result? A solution is to start small; good analytics work creates more analytics work when done correctly.”
3. Access to talent: “We need analysts that have both strong technical and interpersonal skills, which is difficult to find. One solution is to partner with local colleges and universities. Once you find a great analyst, keep him or her in your network.”
Finally, as a woman in a male-dominated industry, one area of bias that Messer talks about frequently is the human bias insights professionals bring to their work. “Our community can enrich itself through diversity of thought, background and experience, so that sources of bias can be identified at the get-go.”
About Audrey Messer
Until recently, Audrey headed up global marketing analytics for MetLife. Now she runs her own analytics consulting company, Centricity Analytics, LLC. “For me running my own firm provides a level of freedom to focus on the type of analytics I am particularly passionate about – customer analytics – as well as be able to work with fast growing companies and firms that need help in building out and unlocking the power of analytics.” While there are no typical engagements, Centricity Analytics focusses on solutions such as customer lifetime value, segmentation modelling, customer journey analytics amongst others in both B2B and B2C business models.
