Out of the darkness: A call for change in neuromarketing

The biggest threat to neuromarketing is the secrecy of its methods – the black box within which the “magic” occurs.

Over the past decade the field of psychology has been jolted by a reproducibility crisis – the inability to replicate some of the most popular and well-known findings in the literature. Recent articles in journals such as Nature Neuroscience and Science have detailed numerous failed attempts to reproduce published results.

Traditionally, academia has rewarded quantity of publications, not necessarily the quality of the science. These pressures sometimes have led researchers to twist statistics, consciously or unconsciously, to make a result appear significant. Indeed, a recent study comparing Ph.D. dissertations to the papers eventually published from those dissertations found that nearly 30 percent of results originally reported as non-significant miraculously became significant in the published studies – although no additional research was conducted.

Bolstered by the creation of the Open Science Framework, leaders in the field of psychology have worked tirelessly to shift the reward and peer-review systems toward more responsible research practices. Journals are now encouraged to accept papers before data are collected, based simply on the merits of the research question and the methodological design. In addition, top tier journals are requiring scientists to make data from their experiments, along with the coding used to analyze them, freely available in raw form so other researchers can verify the reproducibility of the results. 

Since these practices have been implemented the field has witnessed a notable increase in published studies reporting null results (results that fail to support the investigators’ hypothesis).  Although these studies often contain valuable insights, researchers and journal editors previously would have labelled them failures and relegated them to the proverbial dustbin.

While these essential reforms have been transformative for psychology, we should be alarmed by the nearly complete absence of similar safeguards in the practice of neuromarketing research.

It is common for neuromarketing suppliers to cloak their methods and analytic strategies, often presenting them as their proprietary “secret sauce.”  In other areas of business where confidentiality is necessary to maintain a competitive advantage (pharmaceutical research, for example), there are organizations (such as the U.S. Food and Drug Administration) charged with imposing independent oversight of the quality of research design and methods and data analysis before the output of that research becomes available for public consumption. There is no such protection for clients in the field of neuromarketing.   

Clients can be their own worst enemy, too. Understandably, they don’t like messy results, so neuromarketing vendors perceive an incentive to frame their results as a clean, simple story. Furthermore, when a new vendor is hired there is pressure – sometimes implied, sometimes explicit – to deliver insights that are different from what the client has seen before. In effect, this discourages replication and could lead to questionable research practices.

In neuromarketing, researchers’ processes are even more opaque than they were in the bad old days in academia. Therefore, it is reasonable to infer that the quality of research may be substantially worse.

We cannot expect all client-side research buyers to be neuroscience experts. However, clients could benefit from creating a neuromarketing ombudsman position. This individual should be someone from outside the organization who is formally trained in cognitive neuroscience methods, who is current with the literature, and who therefore is positioned to assess best practices. The ombudsman would vet the design of a neuromarketing study in advance, confidentially evaluate the raw data a vendor has collected, and ensure that the analysis plan is appropriate. Vendors could defend any challenges to their approach, but when appropriate they should modify the design as requested to bring it in line with currently accepted best practices. Ultimately, clients should not use research that fails this scrutiny.

An alternative is to create an industry-wide panel of experts to verify that vendors’ research practices are conducted with sufficient rigor. These methodologies could then receive a seal of approval – akin to a Good Housekeeping Seal or the UL stamp on electronic equipment. This would assure clients that suppliers’ methodologies have been appropriately vetted, and vendors could charge a premium for methodologies that have passed this rigorous third-party examination.

Transparency and adherence to standards need not result in homogeneity. Ask two artists to paint a landscape using the same brushes and the same palette and the results will vary dramatically depending on the artists’ talent and imagination. Vendors can differentiate based upon the creativity of their research design, their ability to integrate methods, and their skill in formulating insights. Concocting a “secret sauce” is not the only path to distinctiveness and is antithetical to the practice of science.

In the absence of transparency, research is best judged with a healthy dose of skepticism. It is unwise to make important strategic decisions based upon what could be junk science. Clients must demand better and vendors must do better.

References
Kaplan RM, Irvin VL (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLoS ONE 10(8): e0132382
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017). The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles. Journal of Management, 43(2), 376–399.