How AI and deep learning enhances market research

It is an undisputed fact that consumer behaviour is changing. There are many contributing factors, but one of the major influencers is the impact of technology in everyday life which is speeding up and altering the customer journey.

For instance 46% of purchases in the UK are now made via a smartphone (desktop and tablet purchases are declining) and 88% of shoppers now carry out research on their phone before making a purchase, according to a study by online retailer, mobile.co.uk. .As a result, the role of analysis, insight and market research has never been more important for organisations to keep up with consumers. With the advent of big data which enables businesses to determine what customers are doing, what they are thinking and what motivates them, combined with greater and more cost-effective computational power, market researchers have the ability to revolutionise what has traditionally been a slow and expensive process. Ultimately leading to business decisions being made with greater agility, accuracy and speed.

There are, however, a number of challenges that need to be addressed. The first is breaking down the silos that exist in many organisations. The technology department, research team, data analysts and marketing are all trying to answer the same questions but are using different techniques, different thinking, different data and as a result often get to different answers. Compounding the issue is the fact that each of these business units have different systems which means trying to bring together disparate sets of data has traditionally been a major cause for the silos remaining in place.

It is also impossible to discuss data without referencing GDPR. Under the new legislative landscape data management has become increasingly complex. And where once it was possible to fill out a table with multiple rows of data, today these rows have huge gaps as consumers have the power to say what businesses can and can’t do with their personal information. Added to this is the fact that data comes with a sell-by-date and it cannot sit nicely in a table forever. Increasingly there has been a reliance on digital data to plug these holes but it is looking ever-more likely that ePrivacy legislation is going to change this too.

As a result data will become even more fragmented as consumers will have greater control over their digital data and organisations will probably be prohibited from tracking online customer behaviour without explicit consent.

So the question is, what do we do now? We go back to the future.

The evolution of segmentation through deep learning

Many businesses are hampered by legacy thinking, legacy technology and systems. These create elastic bands to our past which keep pulling us back rather than letting us explore future opportunities. A case in point is the continued use of out dated customer classification tools that operate on the premise that ‘birds of a feather flock together.’ Whilst this was the case 50 years ago when these segmentations were built, things have moved on. How many of us still use technology from the 1970s in our daily lives – so why are businesses? The solution lies in the evolution of these processes, what we refer to as ‘deep learning segmentation’ – a segmentation system that has been enhanced using artificial intelligence to identify patterns that are too complex for human comprehension. This is a way to break through the barriers and realise the opportunities that big data affords.

Deep learning can find patterns that are too complex for humans to identify

Traditional data analytics are based on linear patterns and testing hypotheses such as is this true or not? But in this more complex world there are far more predictive and insightful patterns to be found that lie outside of the linear. Moreover, deep learning finds patterns that do not have human bias or preconceptions.

An example of this is in the prediction of a home move. The linear approach says that when a household applies for planning permission they will be staying put. Instead of moving they are concentrating on home improvement. However, if you look at older residents that live in coastal communities the application for planning denotes the total opposite. AI has identified a pattern that says when people in this specific group of householders apply for planning permission they will move house. This is because they will be selling their prime location property to a developer who will be building top spec homes that are currently in demand in areas such as Sandbanks and Harbour Heights in Dorset.

A modern, science-led approach to delivering actionable insight at speed

Traditional segmentations can solve many problems from understanding the effect of seasonality to reducing customer churn. But one of the major issues is that the more generic the segmentation the less predictive it is of specific behaviours.

…the more generic the segmentation the less predictive it is of specific behaviours.

This is why deep learning segmentation builds up layers of insight to provide a more accurate view and understanding of consumers. For example, it incorporates persistent and stable data such as property, locality and demographics along with transactional data which enables better targeting. It also allows an understanding of spending behaviour and lifetime value providing guidance on acquisition and retention costs. In addition, transient data such as clicks, likes and emotions which need to be acted on quickly can be analysed through real-time segmentations and automated responses can ensure opportunities to engage with customers are not missed. Plus time and trends data can be added in to give context to segment movement.

This approach brings together multiple types of data and allows organisations to deal with complex and missing data. Not only that, but because it unifies disparate data it breaks down the silos that exist in organisations and provides a common language to describe the customers which can be used across all business units and levels of seniority.

Having a deeper understanding of what customers are doing and their motivations means it is possible for researchers to measure the value and behaviours of each segment and predict where each is heading and then determine exactly what this means for the business. The added benefit is that these types of custom segmentations can be built very quickly.

The insight process once described what the customer is doing, but through the application of AI and deep learning it is now possible for researchers to drive better business outcomes by anticipating exactly what the organisation needs to do next to better serve its customers.