“There are three kinds of lies: lies, damned lies, and statistics.” is a phrase from Mark Twain in his autobiography. This comment can be seen as a perspective on the use, or misuse, of statistics to support weak findings. Statistical analysis is crucial at a time of big data and data-based decision-making. However, there’s also the problem when statistics are not being taught at universities correctly or used in an appropriate way.
Over the years I’ve have sat through several presentations of highly statistical research findings in seminars, conferences, and doctoral defenses, where it appears the postgraduate student or early career researcher presenter doesn’t fully understand the results, but is presenting the material due to pressure from a supervisor or senior colleague, or just that is what is expected that they do.
How can I tell? There are a few clear signs, such as:
- The ‘deer in the headlights’ look when the presenter comes to the first results Powe rPoint slide
- The commencement of the presentation with the words “We discovered that there was a data set with 100,000 responses so we thought that we would look to see what we could find….”
- (Also with a ‘royal we’) something like “we wanted to run modelling of decisions based on Hertzenburg’s three quadrant S-test discrimination constant based on dummy variables and non-dummy variations taking into account R-cubed, sin-cos-tan equilibrium scores, and so we found this data set with 100,000 responses and…”
- An interesting line is “I will be determining attitudes by analyzing the demographic responses …”
- Some wild assumptions such as “using census data we have found that those who took public transport to work were obviously left-wing, tree-hugging loonies who hate cars, technology, and our country”
- That classic response to a question “because my supervisor said so” or “you will have to ask my supervisor”.
Of course, statistical analysis is vital for marketing research, but there is a problem when it is being misused and poorly taught by marketing educators at a postgraduate level. This is a concern, especially if the graduates are then using these under-developed, over-confident skills in industry.
Also, I believe that this is evolving into an actual problem at some conferences. It’d appear that some presentations are so statistic- or data-driven that the student presenter may have lost sight of the actual purpose of the study. To some the final answer is irrelevant to the statistical road trip to get there, even though most people (including the presenter) got lost along the way.
While there are many extremely interesting and important findings resulting from highly statistical studies, sometimes these are better presented by people who actually know what they are talking about.
Finally, I would like encourage supervisors to make sure that the statistical analysis used in their student’s work is suitable for the task intended, and that the student understands it. The last thing we would want is a new generation of under-skilled researchers who then go on to teach others or make industry recommendations as consultants.
