Often experts pit Big Data and market research (MR) against each other, as if Big Data will make MR increasingly irrelevant. It’s a misguided match-up. Recently, At AbsolutData, in my capacity as SVP, Analytics, I had a discussion with a CPG senior executive in charge of insights, market research, for a large international CPG company.  He is vitally interested, as with many leaders in the industry, in how to best combine Big Data with market research insights: Finding the best ways of bringing the two together.

I’ve taken the liberty of organizing that discussion into a Q&A format to bring out the most useful points. The discussion addresses common concerns about types of Big Data and illustrates high impact opportunities for it when combined with MR.

Question 1: How representative is Big Data, particularly for categories like kids yogurt where the core target is mothers who may or may not be the people expressing opinion/views on social media, chats, blogs etc?

Response:

Big Data can quite well represent a population. On a high level, some countries, like the US, have very high internet penetration and social media use by moms, suggesting that geographic internet penetration is a factor in determining degree of representation for given populations. Having said that, extent of representation is not a one-size-fits-all concept but is applied differently based on the context. So to more fully address the issue, we need to step back and put this question into the context of the intended use of the Big Data –

• Will we be trying to forecast demand?
• Assess overall brand or product satisfaction?
• Decide to make a product or a product line change?
• Understand the market-wide impact of product or advertising changes?
• Trigger more in depth research activity?
• Understand trends?
• Clarify consumer language and perception, etc?

There are many potential uses of Big Data and depending on the kind of Big Data we are referring to, it can have an impact on each of these areas.

The extent of representation depends on the kind of Big Data and the brand category. Furthermore the appropriateness of a given level of representativeness depends on the application of the Big Data.

Illustration –

If we define Big Data as Social Media – which includes tweets, FB comments and likes, Pintrest, Instagram, blogs, reviews, chats etc. these kinds of data will be representative of a given kind of population. For a category like kids yogurt, customers who mention the category or specific brands will be those who are involved in the category and are likely heavy users. These people are also most likely influencers in their circles of friends.

In fact, these people would well represent a valid target for the purposes of such issues as

• Assessing brand image
• Success or problems with product changes
• Tracking or anticipating extremes of satisfaction (which really is the zone of interest when examining satisfaction over the longer term)
• Exploring usage occasions, and
• Identifying most compelling features and benefits for communications

As another example, innovative companies like BlueFin Labs (recently acquired by Twitter) have shown a direct correlation between twitter comments and advertising effectiveness when correlated in real time with video streams in TV.

The application has a bearing on the appropriateness of representation – but the kind of data also bears upon the issue of representativeness. As detailed below in response to the second question, Big Data means more than Social Media and the data has different properties of representativeness, especially when integrated with disparate data sources.

Question 2: Therefore where and how can Big Data be used? And where can it not be used? What, if any, are its limitations?

Response:

In addition to the Social Media aspect of Big Data, and its associated specific uses, broader use cases become relevant when considering aspects of Big Data. We often think only in terms of social or web activity but we can include other data streams such as

• Daily or hourly sales by location
• Stock out information and distribution information
• Temperature and weather
• Socio-psycho-demographic information based on location
• Daily information on promotions, coupons, mailers, etc.

Because of the velocity, resulting volume, and variety of these kinds of data they can be used in solutions which are particular to what we consider “Big Data” these days. Not only can we map, in real time, these different elements together but we can model them as well to create early warning systems, recommendation systems, and communications and operations real time interventions. Social Media monitoring with text analytics can be overlaid geographically against sales, promotion usage and distribution information to react in real time with retail partners on pricing decisions or display options. We can then include demographic and even psychographic zip+4 information, and traffic information to maximize sales or communications effectiveness.

In short, when taken in its entirety and integrated together, the different aspects of Big Data that have varying degrees of representativeness when individually applied can be blended for general representativeness and applied innovatively to very important decisions. And, innovation is the fertile ground of competitive advantage.

Big Data can further be applied when approached more assertively. Big Data is a flow of information. It is somewhat reactive (but there is more to be said about this aspect) and, if we think in terms of social media, not particularly specific in all the ways we would prefer.

It must be harvested, filtered and interpreted. Or it can be blended with more traditional data to imbue more meaning.

If you are looking to test specific ideas, new product concepts, advertising copy, new product line configuration or pricing, then Big Data generally, used passively, will not provide the answers.

If you want to dive deeply into your target market to connect lifestyles, behaviors, attitudes and needs for segmentation purposes on a strategic level, you will need more than what is commonly available in Big Data.

This does bring up an interesting point, that of using Big Data passively or more actively creating data flows to more directly address specific issues.

On a very high level this is what we do with A/B testing of offers, promotions, web site enhancements, etc. We actually put something out there that customers react to in their normal activity and we measure those reactions and mine them for insights.

Likewise in the Social Media sphere much can be done to elicit more targeted and in depth information. Contests, viral announcements, online events, and such can be used to find out what people think or how they feel about very specific kinds of issues, products, features or benefits.

Question 3: How can one integrate Big Data with primary data and what would be its applicability?

Response:

Many feel very excited about the combination of Big Data with primary data.

On one level the excitement stems from the promise of enriching Big Data flows with the intimate and more detailed information that comes from primary data.

And vice-versa, the ability to further enrich our primary data insights by appending or blending in associated Big Data information.

There are many levels of integration that are possible now and yet will be possible in the near future. Here are a few examples:

• Augmenting a large portion of brand and advertising effectiveness tracking with social media – the best solution is a blended solution now
• Sampling for primary research from people who are part of a multi-device panel so that we know their primary data as well as their web and social interactions across a period of time
• Sampling for primary research directly from social media platforms, and therefore being able to tie into that users past data on that site and maybe even be able to glean information from their social-graph
• It is more typical to use social media listening and text analytics to identify trends, problems, and opportunities that can then be further explored using the more precise tools of primary data collection.

More specifically, AbsolutData has created several tools that specifically link Big Data and primary data. We can:

• Mine the social sphere to come up with an initial set of brands, attributes and levels to offer the most realistic testing using conjoint
• Track brands allowing the issues and image items we are tracking to take form from what people are talking about
• Develop a more realistic set of brand driver models based on social media information
• Administer surveys on consumer’s mobile devices allowing us to tie into social and local data both parts of the Big Data information flow.

In short the main areas that currently can blend Big and primary data would include

• Tracking
• Product development
• Advertising and promotional testing and effectiveness
• Segmentation and
• U&A type landscape work.

We can further combine primary data with other kinds of geo-demographic Big Data which could also include data on sales, inventory, pricing, promotion and distribution to more fully explain patterns observed in the primary data. In this area we have specific experience in

• Combining transactional data with behavioral data from survey for a segmentation exercise undertaken for a leading US based sporting goods retailer
• Combining internal CRM data with survey data to predict travel wallet share for a leading global hospitality major
• And other studies for our global clients

Conclusion

These ideas represent some of the increasing ways that Big Data and MR work hand in hand to generate more valuable insights. The two types of data do not contend for the insights but complement each other in deepening those insights. Each brings its own critical elements of value.