Challenges & Issues

The Value of Partial Data

Felix Rios

As I’m typing this blog post, CES 2015 is taking place in Las Vegas, Nevada. If you don’t know it, the Consumer Electronics Show is the biggest and most important consumer-centric technology event in the world. This event is well known for showcasing everything from the most absurd gadgets that will never make it to production, like the iPotty, to truly revolutionary gadgets that shaped the way we consume media like the CD player in 1981.

The CES is the Monaco Grand Prix of the gadget world. This is where products and trends are made or come to die. In the 2015 edition there is an overwhelming number of manufacturers showcasing IoT technology, smartwatches and wearables. This could only mean one thing: these are not just a trend; these devices will stick around for a while. They are the logical progression of the gadgets we own. As we become more dependent on being online, it only makes sense that all the devices we interact with on a daily basis are also connected for our comfort and ease of use.

In a previous blog post (Data: The new gold fever), I explained the endless possibilities that will come for researchers with so many devices collecting our data. The implications for the market research industry are huge. To get the information we need, until now, our main tool for data collection has been asking participants questions directly. But what if asking questions is no longer the only way to collect measurable and actionable data? What if asking questions and waiting for participants to answer them becomes the slowest and least cost-effective source of data?

As IoT devices and wearables become more affordable they will be widely adopted. Most of our devices will inevitably be connected. They are putting at our fingertips the possibility to gather the type of data that cannot be collected by asking questions. These new technologies could show us that the data we get from passive monitoring is going to be richer and more contextual than we have ever seen in the past – maybe much more than any answer from any participant. In no way are we saying that this will kill the other existing data collection techniques (e.g., computer-assisted telephone, online, face-to-face, etc.). Each will have its own space and usability.

The amount of data generated is already larger than ever before, and it comes in a different form than we are used to. It is mostly unstructured, but highly contextual if seen through the right glasses. The advertising industry has found that using large amounts of unstructured data, in the right way, helps create very accurate user profiles that allow them to target online advertising to very specific segments of the population. There are a lot of learnings that we can take from such experiences:

Facebook: A Data Giant!
A few weeks ago, Facebook announced that it started to index more than 3 trillion posts. This will allow users to search their posts and their friends’ posts. Many times I have found myself trying to dig out something that posted a long time ago and I painfully have to scroll through pages and pages of posts until I get there. Now, by simply remembering a few key words, I am going to be able to find it.

What this also means for Facebook is that they are going to be able to build links with all the data that I have been feeding onto their platform. They are also going to be able to compare it and blend it with all of the data from my contacts to extend these links and create patterns. This will give them ways to understand how I behave, think and even feel – all in the spirit of selling more accurate advertising and products.

It gives me shivers down my spine when I think of how powerful this is, not only will they be able to know what I like, think etc., they will know the same information for every single one of the 1.35 billion users that are currently active on their social network.

You may think, “Well, I don’t share much on Facebook, so I’m safe,” but what little you are sharing right now is enough to match to someone else’s similar profile and decode pretty well what your interests are. 1.35 billion users provide a lot of data. I like to think that people are quite unique, but the sad truth is that we are very predictable. This is the foundation of the market research industry; we try to understand how a group of the population behaves by studying a statistically representative subset of that population. Think of the largest panel that you have ever worked with and it is merely a fly in the wall compared to everyone on Facebook.

Facebook is one of the most important human data repositories to ever exist. This is not something that will happen in the future — it is already happening. Until recently, Facebook knew a lot about us, but now they understand it.

Some Important Lessons

First, the online advertising world is moving fast. Google, Baidu, and the most important online advertising delivery organizations all have similar initiatives that involve artificial intelligence and machine learning. They are embracing the changes and riding the wave.

Second, partial data is valuable: Facebook doesn’t need to know everything about you to be able to accurately target advertising that is relevant to you. They just need to know enough information so they can find your similar self. There will be someone out there sharing similar information to you in bigger quantities that will match your profile. If you think of this same approach, how many of you are using partial data from dropouts, terminates, etc. from your surveys? Are you discarding this data?

John Puleston from GMI (@jonpuleston) is well known for his “bonsai surveys” approach and his campaign to “eradicate boring surveys.” This method allows you to set question level quotas, so you can stop asking a section of your survey if you have enough data. Or maybe use that partial data in your results if the entire survey is not completed. In consequence you end up with shorter surveys that are more engaging and quicker to complete.

Have you thought of how much money you can save if you could use partial data from dropouts in your surveys? Have you thought of using all the information from your completed surveys to create profiles that help you validate that partial data? This radical approach changes the conversation in our industry, where the “complete” is not the most important unit, but data itself.

We need to have the right mindset and embrace the changes coming our way. There are many companies out there already thinking like this. Companies like Ugam are willing to work with you to rethink the future of research. To look at your data differently and find new ways to understand it and create insights. Without any doubts, technology is disrupting every aspect of our lives, and it will continue to rock the boat of the market research industry. The most important thing is to have a disruptive mindset. We shouldn’t be afraid to try new things. Our clients will thank us for it!

Felix Rios, Market Research Technology Manager at Ugam

Leave a Comment

* By using this form you agree with the storage and handling of your data by this website.
Please note that your e-mail address will not be publicly displayed.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles